july

Gaming the system (overfitting to the incomplete, imperfect, or proxy signals that the system exposes as rules) collapses the system because you're optimizing a problem that has a small local minima -- or maybe the way to think about it is, gaming the system is 'overfitting'. 

Because a robust system requires a much better understanding of the intent of the problem. It's about making sure that the optimizer cannot ever find a profitable strategy without also implicitly learning the designer's true objective, i.e., the intent

I feel like this is why AI alignment, or markets are hard

I mean it makes sense when you start thinking about lasting human institutions (religions, cultures, legal systems etc) rely so much on norms, rituals, and meta-structures rather than purely economic or technical incentives

Specifically because it is non-convex, and the intent is explicitly unlearnable

creating and destroying; flying cars, roc.camera, july.rocks