Some coding is hard, yes. My neighbor wrote a “Python package that is designed to extract galaxy properties by fitting their multiwavelength data to a set of templates”. There is a lot of math involved and he has a PhD 

You can make some little AI wrapper and get it used by 1000 people np. You can make a Frame.

The coding tasks that are easy with AI work for the same reason that LLMs work at all: If we can arrange the context in a way that the most likely next step is a valid one, then we're going to get great results!

On the other hand, if we have tasks where it's difficult to manufacture that kind of "path dependence" for valid outputs, AI is still unable to help.

There's a lesson about life here as well: Setting ourselves up for success is extremely powerful. Being left to flail in an amorphous space of possibility can be paralyzing.

A doodler and computerer. I like permissive/permissionless open source, smart contracts, p2p systems, room-scale VR, and NixOS.

shazow.net

Currently: WhatsABI

This is how "attention" works in GPT models: The tokens in the context window get re-weighed to create stronger path dependence around what is relevant.

Unlike a traditional Markov Chain where the relevance of the preceding sequence of token is some fixed function, attention is effectively highlighting/reordering tokens for better results.

Great explanation, I hadn't considered comparing to Markov chains as an intuition pump for why LLMs work.

Software Engineer @ Cadence Design Systems. B4: Stanford EE / Illinois ECE

🙏

I have an affinity for Markov Chains, totally not because Andrey Markov is my namesake lol.