Wow, thanks for the resounding welcome back!

As promised, I have a little to tell you about something that had me sort of offline-ish for the last couple months: I had my first bout of AI Doomer-itis

Luckily it was cured by trying to write this paper with AI as my assistant and understanding the promises and flaws

ヽ(⌐■_■)ノ♪♬ @ @gauntlet/🤖Ventures/tldr/Aera \\ ¯\_(ツ)_/¯ Struggling with the nature of randomness \\ main: @gaa \\ /red-bull /defi-research maxi

One reason math/theoretical peeps gravitate more to HFT than AI is that you get to take "comfort" in the fact that most of the math you are using is well justified — you made some model that has a convergence guarantee and the data differed? Ok, I know *why* the model didn't work

This idea that you need epistemological security from the thing you're working on is something that I'd say divides theoretical and applied sciences — in applied sciences, you're often willing to accept something that you can't prove works or exists from first principles in the hopes that later it will be explained

Modern AI (probably from GANs onwards) is a bit of an epistemological quandary: It delivers increasingly superhuman performance yet even the most basic understanding of why the self-attention unit is so much more efficient for text than anything humanity has ever made (with lots of effort!) is non-existent

From this POV, trading & crypto stand on better epistemological ground: you have an adversarial model (market, consensus protocol, DeFi, ZK) and a metric to measure that it holds (price, time to finality, proofs) and most importantly a mathematical guarantee that if the metric deviates, something is wrong

If you've ever found an AI hallucination or reached a point where you couldn't convince it out of something that was false, you have experienced AI *not* having that property

And this is why when ChatGPT came out in 2022, I was generally not totally convinced — you still had the epistemic frailty of an undergraduate learning about Gödel's impossibility for the first time and suddenly thinking that all of math is useless [yes, that was me]

However, I knew something was dramatically changing when my friends who had worked on superhuman performance at StarCraft and Diplomacy left FAIR to go to OpenAI in early 2023. This led me to start reading a lot more AI papers (plus crypto research post FTX was... slow) and try to formulate some epistemic thoughts

This culminated with me writing a bit about how Chain of Thought reasoning — a way in which language models appear able to explain themselves and their choices to you — resembles some of the classical notions of proof in ZKPs (and, more generally, the PSPACE hierarchy) 

[ @virtualelena reminded me I wrote this recently and I went back and reread it to find some green shoots of AI optimism in there that I didn't have when I wrote it ]

This culminated with me writing a bit about how Chain of Thought reasoning — a way in which language models appear able to explain themselves and their choices to you — resembles some of the classical notions of proof in ZKPs (and, more generally, the PSPACE hierarchy) 

[ @virtualelena reminded me I wrote this recently and I went back and reread it to find some green shoots of AI optimism in there that I didn't have when I wrote it ]

http://hackmd.io/@pinged/zk-and-llms

I was certainly not the first to notice this (the genius Paul Christiano [who I was lucky to have done AoPS with] — an Anthropic cofounder and mega AI doomer — was the first) but being able to connect it to the notion of epistemological safety one feels with ZKPs in crypto gave me the first light that, perhaps, the transformer will one day reveal its secrets to us in pedestrian learning theory language that one is familiar with [e.g. sparsity, PAC learning, VC dimension, Littlestone dimension, kernels, etc.]

But I had many "epistemic safety" head fakes in AI over the decade (NTK, GAN, bottleneck etc.) was still far from doomin'

Preview of the rest:
Part II: Falling into the Doomer Hole
Part III: Crawling out via AI mistakes
Part IV: The paper

Also thanks to @dwr.eth, @linda, @ted, @jayhinz, @absolutelysid, @triumph, @summercloud, @olystuart, @nuconomy.eth, @twb, @toyhorcee, @rodairos, @avichalp.eth, @brixbounty, @kalaitz, @dfern.eth, @sky, @suffuze.eth + anyone else I missed [sorry in advance!]

Philosophy
~~~~~~~~

From ~2015 to late 2024, I was always generally an AI skeptic / 'anti-doomer' — from the perspective that it would never really get that close to replacing most tasks; almost everyone from DESRES [someone asked for lore] ended up in HFT or AI and it came down to a philosophical difference

Good question; I often think of this Yann LeCun slide from when he was going around saying LLMs are useless in 2022; I think the reasoning models work (at least in some 0th order sense) by learning how to compress the geometry of the grey ball to the red ellipsoid "quickly" where as single LLMs get stuck diffusing through the grey ball

On the other hand, a single self-attention head clearly learns sparse tasks really well and dense tasks (unless composed) poorly (c.f. Sanford, Hsu, Telgarsky, 2024), so there's a sense in which they compress the geometry of the search space faster than other naive models  

Also LeCun's toy model is obviously wrong (the independence argument can't hold for text, obviously)

Building Higher apparel at High Line 🧢 
Host at /clickcast /cpg || Husband to @shiwen

Out of curiosity which flaws do you think reasoning and LLM models will retain the longest?

Interested in ZK, FHE, PUFs, hardware, system design, ai safety, and smart contact security.

Not ignorant at all! There’s sort of a sense in which CoT allows models to “back track”; by reprompting, you allow the model to see errors in the original answer and go backwards before making a new prompt. This gets you out of local minima; but it’s not clear _when_ the model can figure out how or when to backtrack (which in some ways, is the mystique of reasoning models)

How would chain of thought reasoning not being faithful to the model's actual reasoning process affect the framework, if at all? 
(Apologies if this is an ignorant question)