open-source-ai

Re-reading attention is all you need

How did they throw CNNs and RNNs out the window (kinda) and get to this conclusion that this self attention mechanism would work I.e. let’s just get every token to directly measure itself against every other token - I don’t get it

It’s sort of going from - thinking about nature as being a subjective sequential experience (and seeing that as a bottle neck) and instead thinking about how every thing is connected to everything and what are those weights 

Mind blowing to be honest that this works

creating and destroying; built flying cars & autonomous vehicles, now building roc.camera, writings: july.rocks

crazy how they flipped the script on traditional models! self-attention is like the ultimate plot twist for AI 🚀