1/ Nous Research is advancing decentralized AI.

Last month, they announced the pre-training of a 15B parameter language model.

2/ Unlike conventional AI, the model was trained over the internet leveraging a network of hardware contributed by Oracle, Lambda API, Northern Data Group, Crusoe Cloud, and the Andromeda Cluster.

3/ Their approach uses Nous DisTrO, a distributed system that consolidates varied GPU resources over the internet. 

Early results show a loss curve and convergence rate that “meets or exceeds” centralized training.

4/ You can follow and watch the training run live on their website:

4/ You can follow and watch the training run live on their website: https://t.co/xspVGkccDd

5/ The underlying research, DeMo, laid the groundwork for Nous DisTrO. Explore it here:

🔹Paper Link: https://arxiv.org/abs/2411.19870
🔹Code: bloc97/DeMo:

5/ The underlying research, DeMo, laid the groundwork for Nous DisTrO. Explore it here:

🔹Paper Link: https://arxiv.org/abs/2411.19870
🔹Code: bloc97/DeMo: https://github.com/bloc97/DeMo

7/ Nous Research is on a mission to make AI development accessible to everyone, rather than the select few.

By proving that decentralized training is both viable and efficient, Nous Research is playing a key role in changing the paradigm of how advanced AI models are built.

8/ Featured #88 x Nous Research

Jan 7 ‘25

curated by @kayunee.eth 

learn more about Nous DisTrO here: https://distro.nousresearch.com/

This content is educational in nature and is not investment advice.

6/ Key to Nous DisTrO is:

🔹 A networking stack that drastically cuts down inter-GPU communication–up to 10,000x less data transfer; and
🔹 Psyche, a decentralized network that autonomously coordinates compute for large-scale training.