Content
@
0 reply
0 recast
0 reaction
Jorge Pablo Franetovic 🎩
@jpfraneto.eth
in simple and eli5 terms: what is a shard? why are they important? what do they mean for scalability?
1 reply
0 recast
2 reactions
LaMat
@lamat
Shards are like splitting up a big job into smaller tasks. They're important because they let us handle huge amounts of data by breaking it into chunks that different computers can work on separately. For scaling up, shards are crucial. They mean we can keep growing our system to serve more users and store more info without everything grinding to a halt. It's like adding more workers to an assembly line - we can produce a lot more by dividing up the work.
1 reply
1 recast
2 reactions
Jorge Pablo Franetovic 🎩
@jpfraneto.eth
wowow it is even hard for me to conceive or conceptualize or create a mental model of the notion of "different computers" working on the same thing separately
2 replies
0 recast
1 reaction
Jorge Pablo Franetovic 🎩
@jpfraneto.eth
ps, your website is crazy good. thank you
1 reply
0 recast
0 reaction
Ryan J. Shaw
@rjs
@lamat 's assembly line example is a good one. One fine detail: Multiprocessing: splitting some work up in the same computer e.g. each of your CPU cores zipping up part of a giant file Sharding: splitting some work up across multiple computers e.g. computer 1 runs an analysis on all even numbered FIDs and computer 2 runs the same analysis but on all odd numbered FIDs, so that it takes half time of either computer doing the work alone
0 reply
0 recast
0 reaction
