flock-io

1/n
Really haven’t been talking about FLock.io and AI related info for a while. @nic7 and flower team motivated me to do this thread. I. will explain a bit on what is federated learning and how this is curial to AI x Blockchian.

DevRel @storyprotocol
IP is everything
Alumni @learnweb3 
Ex @bnbchain @flock_io 
Building /creader

Late last week, Flower announced that they have pre-trained a 1.3B token size language model using Federated Learning. This is significant because normally, pre-training such a model requires considerable computational power, which is a major factor in the current GPU shortage.

Let's delve deeper. There are two techniques that can enhance the use of message passing between threads for parallelism in a stack of cards: NvLink and CUDA. These technologies were designed for centralized aggregation of large computations.

However, when dealing with distributed machine learning, we usually encounter several issues, including communication cost, bandwidth, overhead, and potential node-related issues.

Flower utilized vertical federated learning to accomplish this. But what is vertical federated learning? It's a method where data samples overlap, but each dataset may have different features, variables, or attributes. Therefore, the datasets are not independent but correlated.

At FLock, our focus is on integrating horizontal federated learning with blockchain. Why horizontal? The main principle behind horizontal federated learning is that clients share the same features but different sample data. In terms of web3, this allows users to collaboratively train and fine-tune a model.

Flower's LLM is essential due to the challenges and cost of decentralized training. Centralized computing is more efficient, and enhancing LLM via vertical federated learning is a significant advance in federated learning, spurring its application to horizontal federated learning.

But before we begin, what is Federated Learning (FL)? I'm sure many people are curious. To put it simply, in Federated Learning or distributed machine learning, you have a central server and several clients with data. These clients train a model using their local training datasets and devices.