Lessons learned from the last AI cycle https://i.imgur.com/P20hkQm.png

Before: @StanfordGSB, Kubernetes @Google, @Blackstone

love it!! thanks for sharing, where do I read this properly?

cyberflâneur at fileverse.io ()
building a self sovereign stack ()
phd human-AI teaming in infosec

https://matt-rickard.com/lessons-from-the-last-ai-cycle

thanks just read a bunch of your blog posts : )

Could you expand a bit on "model performance may be somewhat predictable with more compute, but "capability" is much harder to predict." what is capability here?

We're decent at predicting how much a benchmark increases with more scale but bad at predicting how those benchmarks translate to real world applications. E.g., when we were at 75% top 1 accuracy on ImageNet we thought 90% meant full self-driving (it didn't). Same w/ chatbots/NLP

very interesting. So robustness is a measure of capability in that sense?