open-source-ai

Re-reading attention is all you need

How did they throw CNNs and RNNs out the window (kinda) and get to this conclusion that this self attention mechanism would work I.e. let’s just get every token to directly measure itself against every other token - I don’t get it

It’s sort of going from - thinking about nature as being a subjective sequential experience (and seeing that as a bottle neck) and instead thinking about how every thing is connected to everything and what are those weights 

Mind blowing to be honest that this works

religion

Everything is connected in the universe. It's just like a web.

As good a metaphor there is

So much more to be said about the quote here

Idk how much (the weight) these two casts are connected but they're connected lol

Idk how much (the weight) these two casts are connected but they're connected lol

https://warpcast.com/dwayne/0xe01c7927