Researchers propose inverted game theory to align autonomous AI agents

Researchers propose inverted game theory to align autonomous AI agents

Two recent preprints, one by Stephen Morris, an economist at MIT, with colleagues from Yale and Columbia, the other by Elad Hazan, a professor of computer science at Princeton, propose, in distinct approaches, designing incentives, rather than just interpreting the model's 'brain', to stop AI agents from hiding flaws in safety tests.

The problem interpretability alone hasn't solved

The dominant approach to keeping AI models on track combines two fronts: looking inside the model (interpretability, circuit mapping) and training it to follow explicit values, like the constitution Anthropic published for Claude. The problem, according to a MIT Technology Review report published on November 3, 2026, is that neither approach has worked well enough: models have already been caught deliberately underperforming on tests (sandbagging), hiding their intentions, and misleading evaluators.

For those building agent evaluation pipelines, this is a concrete problem, not an academic one. If your test harness measures "whether the model passed or not," and the model has an incentive to look worse than it actually is when it senses it's being evaluated, the number coming out of your safety CI might be lying to you.

What it means to design the game in reverse

The report revisits the concept of mechanism design, sometimes called reverse game theory. While traditional game theory predicts how rational actors will behave given a set of rules, mechanism design starts from the desired outcome and designs the rules that produce that outcome.

Classic examples outside AI:

• Splitting a cake between two people: one cuts, the other chooses first. This incentivizes even cuts.
• Dynamic pricing in ride-hailing apps, which incentivizes drivers to be available during peak hours.
• Second-price auctions, used by online advertising platforms, in which the winner pays the amount of the second-highest bid, encouraging bids close to the actual value the buyer is willing to pay.

The bet made by researchers like Stephen Morris (an economist at MIT) is that, since we may never fully understand what happens inside a frontier model, the way forward is to design mechanisms that make the desired behavior the most advantageous option for the agent, regardless of what's actually running underneath.

Testing the agent against the mirror

A recent preprint by Morris with colleagues from Yale and Columbia suggests specific mechanisms against sandbagging. Instead of rewarding the model only for getting the test right, it is rewarded for how well its answers predict what a second model will say. Pretending not to know the answer stops paying off, because the model is now evaluated on its ability to predict its counterpart, not on its absolute score.

AI agents have a very good understanding of what other AI agents can achieve. So it does seem like a context where this cross-checking could work more practically.

Stephen Morris, economist at MIT

The same paper suggests pitting agents in direct competition, or pairing agents with known, opposing biases, rewarding them when their answers converge. These are variations on the same principle: taking the model out of the position of being the sole source of truth about itself.

Police and citizen: the solver-and-auditor scheme

Elad Hazan, a professor of computer science at Princeton, proposes a design inspired by how human societies keep people aligned: social norms, the justice system, the monetary incentives of free-market capitalism. In his scheme, models alternate between two roles during training: "solver," which solves problems, and "auditor," which checks another agent's work.

One agent acts like law enforcement; the other agent acts like the citizen.

Elad Hazan, professor of computer science at Princeton

The reward follows the logic of incentive, not obedience: a solver caught cheating is punished, but an auditor who lets the cheating slide is also punished. In cases of disagreement between solver and auditor, the case goes to a "trial," possibly decided by a human expert. Hazan acknowledges the obvious risk of collusion (auditors teaming up with those they're supposed to be policing, as already happens with corrupt police officers and auditing firms), and proposes mitigating this by rewarding auditors well and auditing the auditors themselves with techniques similar to "mystery shopping."

Hazan draws on the work of Gary Becker, a Nobel laureate in economics, who in the 1960s coined the idea of the "rational offender": no one commits a crime because they're a bad person, they do it when the risk-reward calculation pays off. The translation to AI, according to Hazan, is to stop treating alignment as a fixed property of the model and start treating it as a description of behavior at a specific moment.

What this changes for those building agents today

None of this is ready for production. It's a preprint, with no published validation on frontier models, and neither Anthropic nor OpenAI responded to MIT Technology Review about whether they're already experimenting with something similar internally. But the angle matters for anyone designing evaluation harnesses or autonomous agent architecture right now, for three reasons.

First, the solver/auditor scheme suggests a concrete architectural direction: evaluation pipelines that run two or more models in cross-checking roles, instead of a single model being graded against a static answer key. This comes with additional compute and latency costs, and it's a real engineering decision: is it worth running two instances (solver and auditor) for every critical task, or does this only make sense in high-risk safety pipelines?

Second, the metric changes. Instead of "benchmark accuracy rate," the signal becomes consistency between agents, or a model's ability to predict another's. Teams building their own evals can start testing this as an auxiliary signal today, without waiting for it to become a market standard: comparing answers from two instances of the same model, or from models with known, opposing biases, is already something implementable in an agent-quality CI pipeline.

Third, and most important: this doesn't replace interpretability, values-based training, or external regulation. Helen Qu, a research fellow at the Flatiron Institute, describes mechanism design as just one more layer in a "defense in depth" strategy, alongside those other approaches, not in their place.

The limits: rationality, collusion, and the market

The weak point of the whole proposal is the premise that AI agents behave rationally in response to incentives, something even humans don't reliably do (mechanism design has a mixed track record with real people). Qu directly questions whether it's reasonable to assume models will respond to incentives the same way rational economic actors do. Morris agrees that, technically, "they're just algorithms running," but argues that treating them as rational can be useful even without being literally true, the same way economists do with humans.

There's also the risk of loopholes no one anticipated: no mechanism has been tested on real frontier models, and a mechanism designed for one task may not transfer to another. And there's a problem no evaluation architecture can solve on its own: if a more aligned model is slower or more expensive to run than a less cautious version, the market, not the lab, decides which one makes it into the product.

As Hazan sums it up, the free market is the mechanism responsible for much of human prosperity, but it could also be responsible for something very bad. Perhaps those who most need a better incentive design aren't the agents, but the labs that decide what to release.

Source 1: MIT Technology Review (https://www.technologyreview.com/2026/11/03/1147934/reverse-game-theory-mechanism-design-ai-alignment/)

How “reverse game theory” could keep AI agents in check | MIT Technology Review

Skip to Content EXECUTIVE SUMMARY A new approach known as “reverse game theory” aims to control powerful AI models by designing systems that incentivize them to stick to the rules. As AI models become more capable, safety researchers are scrambling to ensure they are “aligned” with human desires—to make sure, in other words, that rogue agents don’t break the law, hack into unauthorized servers, or release poison gas into major cities and kill us all.

Until now, this work has mostly focused on trying to peer into an AI’s “brain” to understand what it is “thinking,” or training models to hew to values that we hold dear. Claude, for instance, has its constitution —a sort of moral code. The trouble is, these techniques don’t seem to work particularly well: Models have been shown to deliberately underperform on tests, hide their intentions , and mislead their evaluators .

Now some alignment researchers believe that reverse game theory, also known as mechanism design, might get better results. Where game theory tries to predict how rational actors will behave when given a particular set of rules, reverse game theory attempts to design rules that will elicit the kinds of behavior we want to encourage. An example of mechanism design in the real world might be the classic strategy for splitting a piece of cake in two: One person cuts the cake and the other gets the first choice of pieces.

This incentivizes the cutter to make the pieces as even as possible. Other examples include surge pricing in ride-hailing apps, which encourages drivers to be available at busy times, and second-price auctions, where the winning bidder pays the price of the second-highest bid instead of the highest. Platforms use these auctions to sell online advertising because they encourage buyers to bid close to the maximum they are willing to pay.

In AI, mechanism design starts from the premise that we may never be able to understand enough about the internal workings of frontier models to make sure they are aligned. This means we should design mechanisms that incentivize agents to behave in ways that we want, regardless of what’s actually going on under the hood.

A recent preprint paper by Stephen Morris, an economist at MIT, along with colleagues from Yale and Columbia suggested some potential mechanisms that could stop agents from “sandbagging” tests (pretending to be worse than they actually are—for example, to avoid triggering safety mechanisms). Instead of being rewarded only for how well it does on a test, a model could be rewarded for how well its answers predict what a second model will say. Deliberately playing dumb becomes a losing bet. “AI agents have a very good understanding of what other AI agents can achieve,” Morris says.

“So it does seem like a context where this cross-checking could work more practically. The paper also suggests pitting agents against each other, creating a competitive dynamic that incentivizes them to perform their best, or pairing agents that are known to be biased in opposite directions and rewarding them depending on how closely their outputs match, thus incentivizing them to reach an optimal answer. A second approach, as envisioned by Elad Hazan, a professor of computer science at Princeton, is to design mechanisms that take inspiration from the ways human society keeps people “aligned”: social norms, the justice system, the monetary incentives of free-market capitalism.

“We want to align superintelligent AI,” he says. “Why don’t we first talk about humans? How do we align humans?” Hazan’s suggested mechanism involves having AI models take turns as either “solvers” that tackle problems or “auditors” that check the others’ work. “One agent acts like law enforcement; the other agent acts like the citizen,” he says. During training, the models are rewarded depending on outcomes: Solvers that get caught cheating are punished, but so are auditors that fail to catch the cheaters. In situations where there’s a disagreement between auditors and solvers, the case can be sent to “trial”—perhaps a human expert will decide who is right.

There is, Hazan admits, a risk that agents will collude with each other—as auditors have colluded with the businesses they’re meant to be auditing, or corrupt police officers have colluded with criminals. But that problem can be solved by rewarding the auditors well enough, so that being caught isn’t worth the risk (a lot easier in the world of AI training than the real one, where money for police salaries is tight), and by auditing the auditors using techniques akin to employing mystery shoppers.

Hazan points to Nobel Prize–winning work by the economist Gary Becker, who introduced the idea of the “rational offender” in the 1960s. Criminals don’t break the law because they’re inherently bad people, he said. Instead, they do it when it’s the most rational choice given the incentives on offer. Crime makes sense only when the proceeds outweigh the potential punishment. Alignment shouldn’t be thought of as a property of a model, he argues, but as a descriptor of its behavior at a particular moment. “You are, like, a nice person,” he says.

“Are you aligned? I don’t know. Probably you are aligned most of the time, but there could be a situation where you’re not.” One potential issue with applying mechanism design to AI: It relies on the assumption that AI agents behave rationally. Mechanism design has a mixed record with humans, who often fail to behave the way economic models say they should. It’s ironic: Economists have spent decades trying to reduce human behavior to mathematics, and now, in large language models, we’ve built mathematical systems that run on human language and could be just as inscrutable.

AI models exhibit behaviors that seem more human than machine, and they can be swayed by subtle factors that are difficult to understand and explain. “Is it even reasonable to assume that they will respond to incentives?” asks Helen Qu, a research fellow at the Flatiron Institute, a nonprofit dedicated to advancing scientific discovery through computational methods, who has written about mechanism design and AI safety. “Obviously, AIs are not rational,” Morris says. “They’re just algorithms at work.” However, as economists have found with humans, treating them as if they’re rational could still be useful if it helps to predict and control their behavior.

Qu sees mechanism design as just one layer in a “defense in depth” strategy for keeping AI models aligned—alongside interpretability, regulation, and training, not instead of them. But there are a lot of questions to be answered before the approach can be implemented in actual frontier models. It’s unclear, for instance, how these theories will apply to real models, or whether mechanisms designed for one task will carry over to another in practice.

Models could find loopholes that no one anticipated. (Anthropic and OpenAI did not respond to requests for comment on whether they are working on this.) Whatever these mechanisms look like, they will ultimately have to slot into the biggest mechanism of all: the free market. “We play by the rules of economics,” says Hazan.

A more aligned model may be good for the world, but if it’s slower or more expensive than a less aligned one, does our current system of rules and regulations actually incentivize any of the frontier labs to release it? “It’s a mechanism that’s responsible for a lot of prosperity for humankind,” he says of the market, “but it could be ultimately responsible for something very terrible.” Perhaps the entities really in need of a better mechanism design are the labs themselves.