aichannel

Developing a thesis that whoever has the platform actually being used for AI work will win because they can construct the best evals

GPT-4.5 reaction thread time! Go. Anything you want to make sure I don't miss, or your own reactions.

@TheZvi My reaction is that there is an evaluation crisis. I don't really know what metrics to look at right now. 
MMLU was a good and useful for a few years but that's long over.
SWE-Bench Verified (real, practical, verified problems) I really like and is great but itself too narrow.…

Andrej Karpathy

Developing a thesis that whoever has the platform actually being used for AI work will win because they can construct the best evals

https://x.com/karpathy/status/1896266683301659068

Multiplying onchain economies. Working on interfaces /comint and intelligence @tenfold
Past: AI and crypto rubinovitz.com

prev founded rocketr.net, swish.id now exploring aiXcrypto identity ai. expired sprint car driver & energy data engineer.

Eg cursor could probably build the strongest eval right now for coding because they have the best dataset of people doing actual coding work with AI.

They also aren’t a foundation model company so they aren’t invested in gaming the metrics.