Remember when Sam Altman said that AI systems could just "learn the collective moral preferences" in response to a question by Jack Kornfield about values?

Remember when Sam Altman said that AI systems could just "learn the collective moral preferences" in response to a question by Jack Kornfield about values? https://youtu.be/hn1Y6GVWUV0?si=wLl9t023O8eYREXm&t=750

Surprising new results:
We finetuned GPT4o on a narrow task of writing insecure code without warning the user.
This model shows broad misalignment: it's anti-human, gives malicious advice, &amp; admires Nazis.

This is *emergent misalignment* &amp; we cannot fully explain it 🧵 https://t.co/kAgKNtRTOn

Owain Evans

Saw this and thought of you

https://x.com/OwainEvans_UK/status/1894436637054214509

Chatbots have biases in what they say—but what about biases in what they WON'T say? Our new paper (w/@victoria_r_li &amp; @YidaEdward) shows that personal info like a user's race, age, or love for the Los Angeles Chargers decides if ChatGPT refuses a request. https://t.co/Z5tvOw9Zag https://t.co/EdW9K6fBVw

Naomi Saphra 🧈🪰

Multiplying onchain economies. Working on interfaces /comint and intelligence @tenfold
Past: AI and crypto rubinovitz.com

Kind of reminds me of this paper

https://x.com/nsaphra/status/1811799044974580207

Yes I saw that also. It is yet more evidence that we cannot leave values until post training. Have to include them in pre-training.