My impression of chatgpt so far is that it's good at replicating existing human behavior, but bad at doing anything out-of-sample. It anchors quickly onto patterns that have been done by many people before, but there's definitely something that we call "original thinking" that it's not doing at all.

Some commenters have been noting that LLMs are functionally an abstracted/generalized database technology, and I think it holds. When you’re looking at 100B+ parameter models, that reflects an enormous datastore with a good fuzzy search function over it. 

These things are good at lookup but not “reasoning”.