aichannel

https://x.com/logangraham/status/1902459226334564821

"Evals are emerging as the real moat for Al startups." — @garrytan (YC CEO)

"Writing evals is going to become a core skill for product managers." — @kevinweil (OpenAI CPO)

"If there is one thing we can teach people, it's that writing evals is probably the most important thing." https://t.co/iYGM5vHLIA

Lenny Rachitsky

Evals are all you need

https://x.com/lennysan/status/1909636749103599729

Multiplying onchain economies. Working on interfaces /comint and intelligence @tenfold
Past: AI and crypto rubinovitz.com

Yeah such is surface level pop business stuff I guess even though I love Lenny.

The Karpathy tweet at the bottom of my messy evals thread is I think where I started thinking about this too

GPT-4.5 reaction thread time! Go. Anything you want to make sure I don't miss, or your own reactions.

@TheZvi My reaction is that there is an evaluation crisis. I don't really know what metrics to look at right now. 
MMLU was a good and useful for a few years but that's long over.
SWE-Bench Verified (real, practical, verified problems) I really like and is great but itself too narrow.

Andrej Karpathy

i dabble with ai using open data from crypto https://alexpaden.tech

Yeah such is surface level pop business stuff I guess even though I love Lenny.

The Karpathy tweet at the bottom of my messy evals thread is I think where I started thinking about this too

https://x.com/karpathy/status/1896266683301659068