My impression of chatgpt so far is that it's good at replicating existing human behavior, but bad at doing anything out-of-sample. It anchors quickly onto patterns that have been done by many people before, but there's definitely something that we call "original thinking" that it's not doing at all.

As a random example, consider these AI-generated debates (credit to pmarca for the format).

Notice that the Chinese perspective is not even consistent across the two debates! It's pattern matching to the usual yes/no debate, and... https://i.imgur.com/OebQ19c.jpg

This is to be expected given how transformers work I think. I was not expecting creativity except by accident.

Blogger at ribbonfarm.com, running summerofprotocols.com, host of /gloom ☀️

I’ve been arguing embodiment is necessary for this reason. It’s how human brains are constantly immersed in sensations for 90% of which there are no words. LLMs don’t even have true image data really. The latent diffusion models use tagged image training data so kinda stay within verbal worlds.

There definitely is the "scale is all you need" school of thought that says that if you just scale up the param count by another 1000x or 1000000x then insight and creativity will appear.

Seems false so far, but... I guess we'll see!