The Shifting Power Dynamics of AI

One of the focus areas of my graduate research is artificial intelligence. In my foray into adversarial game theory, I became acquainted with AI’s value alignment problem firsthand. In the development of a strategy-theoretic AI Chess agent project, I decided that it should lose points for allowing its pieces to be in jeopardy. The change resulted in the opposite effect I had hoped – significant losses, which puzzled me at first until I realized that the agent was killing off its pieces to prevent them from being put in jeopardy (once this was worked out, the strategy-theoretic approach dominated all other AI techniques, as it provides accurate non-terminal RL feedback). In the grander context of artificial intelligence, the potential for a catastrophic value alignment failure is all too easy to create as the result of short-sighted policies (such as my Chess strategy), or other simple miscalculations.

It’s in everyone’s best interest for AI to behave rationally, however many believe that – in the context of modern AI and deep learning – AI can never be formally verified to the degree that its actions can be deterministically predicted to be responsible. We tend to treat AI with the same sense of dualism that we treat reality, yet the one thing we do know is that AI is an entirely materialistic universe, and not dualist at all. Determinism of AI systems is based upon pure mathematics, with predictable causation. It is true that we can not always observe why AI behaves a certain way, however here we can learn much from classical Stoicism. The early stoics asserted that all qualitative states are explained by specific factors, even if those factors were not always observable. All subsequent qualitative states are likewise determined by the prior states and additional factors. No change can happen without an explicit cause. Whether it’s the logical determinism built through training data, alignment of real time with processing cycles, or reconciling other factors, every single micro state within a configuration of the machine can be observed with enough work. While modern philosophy essentially rejects the stoic concept of fate (of humanity), “fate” in computation falls square within the realm of an entirely deterministic material universe. AI lives in a material world, and she’s a material girl.

AI is largely unverifiable today because industry hasn’t created an affordable way to provide the computing power to observe all factors that contribute to a system’s qualitative state. Despite the inability to verify AI, industry has plotted a course regardless of edge cases that may sometimes be life threatening. Incredible progress in artificial intelligence has all but guaranteed it will be ubiquitous one day. There is little doubt that autonomous vehicles will eventually outperform human drivers, or that machine learning can more accurately diagnose a health problem. There is, on the other hand, great doubt that industry will act responsibly enough to ensure sufficient safety controls intervene when things go wrong. AI will likely never operate with rational judgment 100% of the time, nor will it ever understand the ethical implications of its action; they will always be prone to value alignment catastrophe. Of course, humans lack ethics and rationality as well, and so society controls this by holding us accountable for our actions. Unlike humans, however, industry is treated differently. This is particularly true with emerging technologies and even more so of those that we don’t fully understand. After all, how can one hold math accountable? Dismantling a broken robot does not solve the problem, particularly if the code is replicated across a million others. The connection between what holds true in a computer system and the outcome that is “fated” to occur is “based on an ontological foundation in which certain elements from logic and physics coincide” [6]; Chrysippus wrote of the close relationship between “what is true” and “what is in motion” long before AI in his Bivalence theory. A modern take is simply this: an AI’s “fate” is the direct result of a system’s physical configuration and sensor inputs. Imagine if he were alive to have observed AI, or even a good quality toaster.

Any good ethics discussion about AI usually begins with the trolley problem, a well-known psychology challenge. If it must, is it better for an autonomous vehicle to veer left and kill the driver, or veer right and kill a crowd of pedestrians? This is not the easy question it seems on the surface, and one of the unanswered challenges in fully autonomous driving requires solving it. The trolley problem is easily dismissed as cliché yet presents some crucial questions of ethics and accountability. Fully autonomous vehicles will one day absolve the driver of responsibility in an accident, and place responsibility onto… nobody is certain yet (as evidenced by meaningless disclaimers about keeping your hands on the wheel). We have already seen the first wave of tragic accidents as the result of AI. Autonomous vehicles cheapened down to use insufficient sensors having limited dynamic range have led to fatal accidents. A depressed teenager was seemingly led to suicide recently by a large language model. Over 3,000 reports of harms have been reported in the AI Incident Database, ranging from AI-based scams and deepfakes to corporate data leaks, mislabeling emergency wildfire updates as spam, and numerous disinformation failures.

Most AI failures to date have been dismissed to the tune of industry accidents. The tragedies are very real, however, and have deeply affected people’s lives. With billions of dollars at risk, it is in industry’s best interest to maintain a sense of ambiguity about who’s responsible for AI, and to continue resisting pressure to explain how their platforms are trained and made safe. As I had predicted, there now also exists a controversy around copyright theft; training generative AI to produce output derived from copyrighted training data. Here, large industry stakeholders frankly outweigh the clout of the millions of smaller stakeholders – the authors, musicians, and artists whose work is being duplicated. Many are starting to rightfully assert that AI has already done more harm than good.

What I think most people are afraid of – perhaps without realizing it – is the massive, unbalanced shift in power dynamics that AI stands to create if we are not careful. Car accidents and copyright infringement feel like mere early tremors; warnings that something bigger may happen if we don’t pay attention. To understand the power dynamics in play, one must first understand some of its applications. As a security professional, my first inclination is to reframe any problem in the context of a threat model. Deterministic outcomes are central to control theory and the safety mechanisms that protect humans. In the context of autonomous driving, airbags, emergency braking systems, and crumple zones alike must have predictable outcomes. Deterministic outcomes are also an easy security target. As one contrived example, consider an autonomous driving system that can be reliably coaxed into veering away from a group of pedestrians and hitting a tree (killing the driver) or into some other group of pedestrians (killing them). In a risk assessment, this presents an opportunity for error, but also a valuable primitive for a malicious actor capable of staging an outcome. Suppose the driving system is part of a fleet used in government. Legitimate risks may include acts of terrorism, especially if protecting a diplomat or the President. Society has a short memory span when it comes to terrorism, yet it is what keeps government career professionals up at night. Now suppose risk is being assessed for a CEO, and for a civil rights activist. The range of threats all remain similar, however the power dynamic does not.

Today, the problem is simply market-driven: If you don’t like the way the vehicle behaves then don’t buy it. In the scenario just described, one could choose to buy a different vehicle. This will not always be the case. Fully autonomous vehicles must react to trolley problem scenarios, and will react with different outcomes based on their programming. The outcome may not always be desirable to the occupants of the vehicle. Such a terrifying feature may not be monetizable, but they can certainly be mandated through regulatory agencies – particularly if those agencies are influenced by billionaire automotive CEOs with an autonomous driving agenda. In some future, the government may have the power to mandate a level of vehicle autonomy under the long unsubstantiated guise of “automotive safety”. This outcome will ultimately lead to another unavoidable one: special programming (deviations) for government fleets; a “secure” mode in which a vehicle will prioritize the occupants of the vehicle over pedestrians. This sounds sensible to a bureaucrat yet consider what it means: the government may one day require, through regulation, technology that will save the lives of the occupants of government officials given a particular situation, yet sacrifice the occupants of your vehicle in the same situation. Should fully autonomous-capable vehicles ever become a safety mandate, the technology instantly becomes monetizable as well: such a feature may be marketed as “driver protection” technology available in luxury trims, just as existing technology is today. The right to survive a car accident is accelerating quickly toward being decided only by one’s socioeconomic status. Unlike basic safety technology today, however, it may become an intentional action by the driver’s own vehicle to sacrifice the lives of the occupants, depending on its feature set and whether the owner is up-to-date on their subscriptions.

Through the seemingly innocuous act of software engineering and a little greed-based policy, AI can easily shift the power dynamics dramatically to favor the powerful in this and many other ways. While this is a contrived example, it is certainly a realistic one given today’s political reality. We have already seen AI massively change power dynamics without hesitation in war targeting computers, autonomous weapons development, and in much more subtle ways. Black-box systems such as OASys (The Offender Assessment System) [1] advise judges, prisons, and probation officers on risks of a given criminal offender, the likelihood of rehabilitation versus reoffending, and even creates individualized sentencing plans. Discrimination is enabled at airports by AI, as body scanners struggle with thick hair and head coverings [2], and patient medical diagnosis systems, while of great utility, deliver a different standard of care to underrepresented populations [3], raising questions about its future role in prioritizing health care (and competing for jobs with human doctors). AI’s power dynamics are, in part, determined simply by the engineering of these systems, which is one reason why diversity and inclusion is so crucial in tech. Throw in some billionaire bureaucrats and this delicate balance becomes a significant power to wield.

Industry and academia are beginning to think about these problems, albeit far too late. Stuart Russell, Distinguished Professor at U.C. Berkeley and author of Artificial Intelligence: A Modern Approach, recently denounced the standard AI model in favor of a consensus on utility [4]. While altruistic, even this step in the right direction inevitably leads to “two wolves and a lamb voting on what’s for dinner”. At the same time, Russell makes valid and poignant points: we lack the understanding to know when AI should admit that it doesn’t know and consult a human. AI must be more about what Doug Engelbart refers to as Intelligence Augmentation (IA) [5] and treated as more of a symbiotic relationship than an automation. Even with the greatest AI minds considering the many ethical challenges, we are unable to explain how modern AI builds its circuits, nor find a suitable model by which we can agree can be trusted to perform safely and ethically without human supervision. We are incapable of controlling it, and we are incapable of convincing industry to slow down. As AI becomes ubiquitous, the ability to simply press the Reset button and re-train it is disappearing.

At the beginning of this essay, I posed the question: “how can one hold math accountable?”. Clearly, it’s not the math that is to blame, but those behind the complex systems when they fail. It is far too easy to deploy new AI technology into production without meeting any minimum bar of rationality, validation, or forensic accounting. Economics will always ensure technology lands prematurely, and without the necessary controls. We don’t yet have sufficient laws to require such safety controls today. We could be doing a whole lot more in the way of legislation. Transparency is the most important form of accountability. For example, my little Chess agent could, at a slight cost of performance, maintain a history of its hallucinations influencing the move choices it made (it’s largely how I trained it). Large language models should, at the very least, be able to identify any of its outputs if supplied as an input, and cite the sources used in the composition of the sample. A good counter-argument to copyright suits would be to produce a model with the author in question excluded from the training set and determine whether the AI can still product the same output. Legislative reform to acknowledge privacy as a human right and better define the ownership of metadata as belonging to the individual it describes would stand to protect from much of the data mining likely going on behind the scenes in government, finance, and advertising. Copyright reform to acknowledge generative AI outputs as derivate works would go a long way in protecting authors, musicians and artists. Regulation for autonomous vehicles could require more rigorous government certification testing and reinforcement training through heuristic safeties. Some level of mandatory disclosure and certification for any AI interacting with large physical environments (such as roads) serves the best interest of the public and subject methods to much needed scrutiny.

I am quite pessimistic about the future given the unlikeliness such accountability will happen in my lifetime.

In many respects, we are all stakeholders in AI technology; if we haven’t already been affected by it, we will be. We must develop a standard for transparency, testing, forensic accounting, and corporate responsibility if we are to benefit from AI before we are trampled by it.

[1] A compendium of research and analysis on the Offender Assessment System (OASys) 2006-2009, Mia Debidin (Editor), Ministry of Justice, December 2009

[2] TSA Agents Say They’re Not Discriminating Against Black Women, But Their Body Scanners Might Be, Brenda Medina, ProPublica, April 2019

[3] Eliminating Racial Bias in Health Care AI, Isabella Backman, Yale School of Medicine, December 2023

[4] How Not to Destroy the World with AI, Stuart Russell, CITRIS and the Banatao Institute

[5] Augmenting Human Intellect: A Conceptual Framework, Douglas C. Engelbart, Stanford Research Institute, October 1962

[6] Determinism and Freedom in Stoic Philosophy, Suzanne Bobzien, Clarendon Press, Oxford 1998