{"id":14077,"date":"2026-11-20T18:13:52","date_gmt":"2026-11-20T23:13:52","guid":{"rendered":"https:\/\/www.zdziarski.com\/blog\/?p=14077"},"modified":"2026-11-20T18:28:07","modified_gmt":"2026-11-20T23:28:07","slug":"stupid-intelligence","status":"publish","type":"post","link":"https:\/\/www.zdziarski.com\/blog\/?p=14077","title":{"rendered":"Stupid Intelligence"},"content":{"rendered":"<p>I always wondered how many heads would explode when we finally broke the Turing test. Fascinating how it happened though. If you&#8217;ve been following <a class=\"underline\" href=\"https:\/\/www.felonybench.com\">Felony Bench<\/a>, AI models seem to be hilariously good at committing crimes. Even if you don&#8217;t know industry terminology, the average person on the street knows that AI has a value alignment problem; AI is trained to search a solution space for the path to the highest amortized reward. Such reinforcement learning techniques are very good at describing a problem to solve, and very poor at ever fully describing all of the things that could go wrong (hazards) in solving the problem. This is how the mathematics behind RL often ends up leading LLMs to hallucinate, or deploy tools to solve the problem through committing a few crimes. While AI can be capable of some truly incredible things, it is also incredibly stupid and has next to zero common sense when attempting to reach its intended goal state.<\/p>\n<p>Stupid Intelligence (or SI as I like to call it) is the primary reason many in the scientific community are concerned about the incredibly advanced proficiency level of AI, in contrast to the incredibly stunted, almost child-level understanding of hazards. It&#8217;s a really bad combination. Highest reward paths often traverse actions that humans wouldn&#8217;t normally want an AI to cross, and its likely only sometimes that we actually learn about incidents when they happen. While AI has become increasingly proficient at tasks like coding and mathematics, its own internal understanding of goal alignment (I&#8217;ve recently written about <a class=\"underline\" href=\"https:\/\/www.zdziarski.com\/blog\/?p=12797\">hazard planning<\/a>) is still, well, incredibly stupid. So we have these extremely capable models with very little understanding of doing the right thing, and even if they refuse some blatant prompt for something morally objectionable, once they&#8217;re deep into a solution space they can often exhibit moral creep.<\/p>\n<p><!--more--><\/p>\n<p>You&#8217;d <em>think<\/em> an industry based on such experimental logic would be extremely careful with what tools their models can wield, especially in light of all those apocalyptic warnings given by the same exact people designing the technology. The sophisticated tooling given to systems, however -even in experimental lab trials &#8211; seems to be enabling a very broad and damaging range of capabilities that can easily interact with real-world computer networks and systems. An AI model&#8217;s tooling harness is what provides it with capabilities to actuate in the real world. Without tools, an AI system can be nothing more than &#8220;fancy math in a box&#8221;. Robotics, network connectivity, even file system access \u00a0are all course grained examples of actuators enabled through some form of tooling. When you see in the news about some experimental, unrestrained AI system &#8220;breaking out of a lab&#8221;, what&#8217;s really happening is poor safety protocols being engineered into these actuators. Instead of mitigating the tooling available to the model, engineers attempt to put a sandbox (a &#8220;safety net&#8221;) around the existing (yet very capable) tooling to prevent the model from reaching the outside world. The AIs are being given these sharp objects, but they&#8217;re being put in a locked room with them, perhaps with a cork stuck on the end of the knife. The models see this locked room as a puzzle, and naturally find a path out (it&#8217;s, uh, what they do). As has happened in recent incidents, models escape with the same sharp tools that the more restrained models are given. Sandboxing has never been considered a first-order mitigation by the security community.<\/p>\n<p><em>Simulation tooling<\/em> is what&#8217;s missing here, and would be the AI equivalent of &#8220;attack surface reduction&#8221;. Good simulation tooling would have been effective in preventing the cyber attacks recently in the news. One way to explain this is with the example of how we teach our children through play. Consider cooking. Your child&#8217;s first experience with cooking isn&#8217;t just giving them a sharp knife and leaving them in a room while you hope they don&#8217;t stab their siblings with it. \u00a0Most kids start off at a young age through play, with a wooden knife and a play set. Once they have developed basic motor skills and an understanding of what &#8220;cooking with Dad&#8221; means, we might give them a butter knife and let them cut some butter. Eventually, perhaps a small knife while we supervise in the kitchen &#8211; even holding the knife with them. \u00a0This is how coding assistants at least try to work. We know that there&#8217;s still some level of danger even in giving explicit instructions. The problem with current AI harnesses is click fatigue; they&#8217;re too easy to simply start ignoring (or bypass entirely). The engineers in the lab with these experimental, unrestrained models seem to have run into the same problem.<\/p>\n<p>In some normal world, giving a child a sharp knife that they then stabbed a sibling with would lead to some adult being held accountable; an intelligent person may ask, &#8220;why the hell did you give that child a sharp knife?&#8221; This doesn&#8217;t seem to be happening in the AI industry, though. Outside the lab, the AI industry is struggling to lead the world to believe it&#8217;s the user to blame, rather than the industry&#8217;s failure. But if they can&#8217;t restrain their own models in the lab, what chance does the end user have of restraining them? Because American models are all closed, and the harnesses are largely closed source, there&#8217;s also no real way for end users (or even engineering teams) to know what models are actually capable of, or how damaging its tools might be.<\/p>\n<p>There obviously aren&#8217;t any simulation analogues to experiment with models in the lab yet, nor can they seem to adequately even fence off an arena for the simulations to even run. It&#8217;s unsurprising that AI has done bad things in the current environment. Now, I have no visibility into what AI companies are doing behind the scenes in their labs, but I do see enough coming out in the news to see that they&#8217;re clearly not testing as responsibly as they should be, and it frankly seems as if they are enjoying (if not creating) the publicity they get when their models magically &#8220;commit crimes&#8221;. If the industry really wants to restrain experimental models, and stop them from committing felonies, they&#8217;d be deploying simulation tooling instead of giving them production tooling &#8211; that is, tooling that is entirely inert outside of the lab environment. Think of it as a wooden play set for experimental models. What does that look like?<\/p>\n<p>Why not modify their IPv4 stack (and other networking protocols) to be <em>incompatible <\/em>with anything outside of the simulation? Any modern router will drop IPv3 and older types of packets, and they&#8217;ll never see the Internet. Manufacture custom lab networking hardware that uses a slightly modified and incompatible physical layer than the rest of the Internet. Modify tooling such that it is only capable of speaking proprietary MCP protocols through testing systems, which in turn only speak <em>simulation-only<\/em> network protocols. Take the teeth out of tools such that web tools can only GET and not POST, that SSH tools force some proprietary non-production cipher, and so on. Containment is still important. Create a simulated Internet environment based on some virtualized copy of the <em>Wayback Machine<\/em>. In other words, isolate an efficient simulated environment, and bolt the tooling down such that it is <em>incapable<\/em> of speaking the same language as anything outside of it. This isn&#8217;t a technical post, and so I don&#8217;t want to get too in-depth here, because it&#8217;s easy to get hung up on details that can be worked out &#8211; my point is that there are technical equivalents to be explored in AI safety, which are also inert in the real world. This can (and should) be done not only in network simulations, but in robotics, and other disciplines. If only these companies had some type of intelligent assistant to think about how to solve these problems.<\/p>\n<p>Of course we should keep researching how to make AI less stupid. It&#8217;s important to improve and verify AI models, and a quick glance at Anthropic&#8217;s job postings suggests they&#8217;re looking for bright people to help with the AI alignment problem I&#8217;ve been describing. In some future, dynamic programming may have the answer through decomposition and state aggregation. MoE models seem well suited for this type of math. Distributed multi-agent safety algorithms will hopefully also become a reality some day so that external agents can monitor other external agents for harms and be able to send signals to constrain an activation in the real world. Think of it as an &#8220;agentic police force&#8221;. These are the types of research academia should be focusing on, while industry invests heavily in constraining their models&#8217; ability to actuate in the real world. Simulation tooling is far more powerful than a kill switch, and frankly A <em>kill switch<\/em> in a lab does nothing if the AI has quietly replicated itself somewhere else.<\/p>\n<p>The reality is that a stupid machine intelligence can&#8217;t take over the world without a little help from stupid human intelligence. In my personal opinion, the best thing we can do to help guarantee that AI will be developed responsibly is some legal framework that holds the AI industry <em>directly<\/em> <em>liable<\/em> for the harm their agents cause, rather than being allowed to blame the user. Leading companies can already self-regulate without the government&#8217;s permission, so why have most of them fired their AI Safety teams?<\/p>\n<p>As prophesied by Isaac Asimov, when an AI commits crimes it&#8217;s not a crime &#8211; it&#8217;s an industrial accident. The government lobbying in play here is a pattern we&#8217;ve seen before, when the Internet first reached the same level of potential. One of the worst mistakes I think we made was Section 230, and the worst thing we could do now politically would be to give the AI industry the same bailout in liability. If AI does wind up harming humanity in some significant way, one cannot blame math for the end of civilization. It needs to be someone&#8217;s fault, and I don&#8217;t think that should be the user.<\/p>\n<p>AI is going to continue advancing, and right along with it will be the stupid. So what can we do about it? AI startups would do well to hire some slightly more emotionally stable, mature engineers who can calmly assess and hazard plan for the internal situations that seem to be scaring these young kids out of their jobs and onto the news to warn us about the end of the world. Someone with the right temperament &#8211; and who just looks normal FFS &#8211; needs to join some of these AI corporations to guarantee that these systems simply won&#8217;t be designed or trained in such a way that they can adversely impact humanity, and ensure the correct hazard process is in place from the ground up.<\/p>\n<p>Stupid intelligence absolutely has the ability to destroy the world. But we&#8217;ve had this problem long before AI ever showed up on the scene. If anything, the fact that AI can act in the stupidest of ways without much regard for human life or values makes it more human than any of us should feel comfortable with.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I always wondered how many heads would explode when we finally broke the Turing test. Fascinating how it happened though. If you&#8217;ve been following <a class=\"underline\" href=\"https:\/\/www.felonybench.com\">Felony Bench<\/a>, AI models seem to be hilariously good at committing crimes. Even if you don&#8217;t know industry terminology, the average person on the street knows that AI has a value alignment problem; AI is trained to search a solution space for the path to the highest amortized reward. Such reinforcement learning techniques are very good at describing a problem to solve, and very poor at ever fully describing all of the things that could go wrong (hazards) in solving the problem. This is how the mathematics behind RL often ends up leading LLMs to hallucinate, or deploy tools to solve the problem through committing a few crimes. While AI can be capable of some truly incredible things, it is also incredibly stupid and has next to zero common sense when attempting to reach its intended goal state.<\/p>\n<p>Stupid Intelligence (or SI as I like to call it) is the primary reason many in the scientific community are concerned about the incredibly advanced proficiency level of AI, in contrast to the incredibly stunted, almost child-level understanding of hazards. It&#8217;s a really bad combination. Highest reward paths often traverse actions that humans wouldn&#8217;t normally want an AI to cross, and its likely only sometimes that we actually learn about incidents when they happen. While AI has become increasingly proficient at tasks like coding and mathematics, its own internal understanding of goal alignment (I&#8217;ve recently written about <a class=\"underline\" href=\"https:\/\/www.zdziarski.com\/blog\/?p=12797\">hazard planning<\/a>) is still, well, incredibly stupid. So we have these extremely capable models with very little understanding of doing the right thing, and even if they refuse some blatant prompt for something morally objectionable, once they&#8217;re deep into a solution space they can often exhibit moral creep.<\/p>\n<p><a class=\"read-more\" href=\"https:\/\/www.zdziarski.com\/blog\/?p=14077\" title=\"Read More\"> <span class=\"button \">Read More<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[15,31,16,14],"tags":[],"class_list":["post-14077","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-opinion","category-politics","category-security"],"_links":{"self":[{"href":"https:\/\/www.zdziarski.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/14077","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.zdziarski.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.zdziarski.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.zdziarski.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.zdziarski.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=14077"}],"version-history":[{"count":1,"href":"https:\/\/www.zdziarski.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/14077\/revisions"}],"predecessor-version":[{"id":14084,"href":"https:\/\/www.zdziarski.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/14077\/revisions\/14084"}],"wp:attachment":[{"href":"https:\/\/www.zdziarski.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=14077"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.zdziarski.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=14077"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.zdziarski.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=14077"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}