Day: November 20, 2026

Stupid Intelligence

I always wondered how many heads would explode when we finally broke the Turing test. Fascinating how it happened though. If you’ve been following Felony Bench, AI models seem to be hilariously good at committing crimes. Even if you don’t know industry terminology, the average person on the street knows that AI has a value alignment problem; AI is trained to search a solution space for the path to the highest amortized reward. Such reinforcement learning techniques are very good at describing a problem to solve, and very poor at ever fully describing all of the things that could go wrong (hazards) in solving the problem. This is how the mathematics behind RL often ends up leading LLMs to hallucinate, or deploy tools to solve the problem through committing a few crimes. While AI can be capable of some truly incredible things, it is also incredibly stupid and has next to zero common sense when attempting to reach its intended goal state.

Stupid Intelligence (or SI as I like to call it) is the primary reason many in the scientific community are concerned about the incredibly advanced proficiency level of AI, in contrast to the incredibly stunted, almost child-level understanding of hazards. It’s a really bad combination. Highest reward paths often traverse actions that humans wouldn’t normally want an AI to cross, and its likely only sometimes that we actually learn about incidents when they happen. While AI has become increasingly proficient at tasks like coding and mathematics, its own internal understanding of goal alignment (I’ve recently written about hazard planning) is still, well, incredibly stupid. So we have these extremely capable models with very little understanding of doing the right thing, and even if they refuse some blatant prompt for something morally objectionable, once they’re deep into a solution space they can often exhibit moral creep.

Read More