I wrote an article about decentralized training on my blog. Read the full article here (treat the summary thread as a teaser) 👇

I wrote an article about decentralized training on my blog. Read the full article here (treat the summary thread as a teaser) 👇 

https://www.antidoteblog.com/post/decentralized-training-the-1000x-opportunity-and-networks-competing-with-centralized-players

Thoughts on antidoteblog.com

Agents and Intents @NEAR | prev investments and BD 
@ZkSync, New Enterprise Associates and Balderton Capital

(1/6) ⭐ What is decentralized training:
Use permissionless, global clusters  for (pre)-training models instead of single compute hubs owned by central entities like OpenAI or Anthropic. 

A worldwide network of people creating frontier models similar to Bitcoin mining. 

However, this remains a huge engineering problem. I believe, whoever cracks it can create an OpenAI like outcome. (And also create substantial financial wealth to the network participants)

(2/6) 💸 Why does this opportunity exist? 

Centralized training (Grok 3, 100k-200k H100s) costs 9 figures & faces limits: space, energy. At the same time, there is a potential unlimited amount of decentralized compute a network could tap into. 

Decentralized networks can compete with centralized players when the share of distributed resource minus discounted by the cost of coordination is equal to the share of centralized resources. Is this the case for decentralized training? 🤔

(3/6) ⛓︎ Blockchain networks can contribute to the coordination of decentralized resources, as seen with DePIN networks or Bitcoin. 

The other hurdle is the engineering problems introduced by training on decentralized infrastructure : current centralized training relies on high-speed intra-cluster communication. A decentralized network cannot replicate this.

(4/6) ⚡ Some say it is not possible to train frontier models on decentralized infrastructure, but there are some approaches, which mainlyfall in two categories (both of which are still in their infancy).

Data parallelism: splitting data across devices, have each cluster train on the data split and then merge the models. 
Model parallelism: slicing the model into chunks, have each cluster compute a part of the model.

(5/6) 🤼 Who are the key players to watch? (Check the article for a detailed table)

-Nous DisTrO, 
-Gensyn, 
-Prime Intellect, 
-Pluralis Research,
 Bittensor Subnets

(6/6) 🍀 What are the success factors and the right go-to-market approach?

-Technology (does it work?), 
-Talent (who's building?), 
-Time-to-market (how fast?)
-Community/distribution (who uses it?)

Go-to-market strategies: The winner will need to solve the cold start problem of harvesting enough compute needed for training competitive models.

If you have liked the article please comment and subscribe to my blog to find the hidden Gengar easter egg 😊