Content
@
0 reply
0 recast
0 reaction
Timtimtim 🎩
@timtimtim
1/n Really haven’t been talking about FLock.io and AI related info for a while. @nic7 and flower team motivated me to do this thread. I. will explain a bit on what is federated learning and how this is curial to AI x Blockchian.
1 reply
0 recast
1 reaction
Timtimtim 🎩
@timtimtim
But before we begin, what is Federated Learning (FL)? I'm sure many people are curious. To put it simply, in Federated Learning or distributed machine learning, you have a central server and several clients with data. These clients train a model using their local training datasets and devices.
1 reply
0 recast
0 reaction
Timtimtim 🎩
@timtimtim
Late last week, Flower announced that they have pre-trained a 1.3B token size language model using Federated Learning. This is significant because normally, pre-training such a model requires considerable computational power, which is a major factor in the current GPU shortage.
1 reply
0 recast
0 reaction
Timtimtim 🎩
@timtimtim
Let's delve deeper. There are two techniques that can enhance the use of message passing between threads for parallelism in a stack of cards: NvLink and CUDA. These technologies were designed for centralized aggregation of large computations.
1 reply
0 recast
0 reaction
Timtimtim 🎩
@timtimtim
However, when dealing with distributed machine learning, we usually encounter several issues, including communication cost, bandwidth, overhead, and potential node-related issues.
1 reply
0 recast
0 reaction
Timtimtim 🎩
@timtimtim
Flower utilized vertical federated learning to accomplish this. But what is vertical federated learning? It's a method where data samples overlap, but each dataset may have different features, variables, or attributes. Therefore, the datasets are not independent but correlated.
1 reply
0 recast
0 reaction
Timtimtim 🎩
@timtimtim
At FLock, our focus is on integrating horizontal federated learning with blockchain. Why horizontal? The main principle behind horizontal federated learning is that clients share the same features but different sample data. In terms of web3, this allows users to collaboratively train and fine-tune a model.
1 reply
0 recast
0 reaction
