[Macroagents] 2. Design lenses for optimizing macroagents

Follow-up to: The Macroagent Ontology (especially section 1.2 is a prerequisite)

The previous post laid out the basic macroagent ontology. This post will look at some further concepts that are important lenses for optimizing macroagents, along with some ideas for each lens.

We usually want to shape macroagents so that (1) their optimization target is aligned, (2) their epistemics are good, and sometimes (like for non-evil human macroagents) so that (3) their efficiency is high. I will call those our 3 target dimensions.

I will go through 5 design lenses which are useful for optimizing macroagents:

1. Incentives (affects all target dimensions)
2. Macroagent evolution (affects all target dimensions)
3. Information and decision routing (affects mostly efficiency)
4. Knowledge aggregation (affects mostly epistemics)
5. Value aggregation (affects mostly alignment)

(These lenses focus on the macroagent structure and not on improving competence of agentic subsystems, even though in principle you could count things like project management protocols as mechanisms.)

(The last two lenses apply more narrowly — only to macroagents that explicitly aggregate beliefs or values — but they're the main levers for epistemics and alignment specifically.)

(TODO think about how MECE that is and maybe make comment on it. How well can other design lenses like management principles be represented here?)

2.1 Optimizing Incentives

2.1.1 Incentive failure modes

Incentive failures arise when agents doing the work don't benefit proportionally to how well the work fulfills its purpose. (This is especially common if decision makers are not beneficiaries.)

As an example, the purpose of science is (to a large extent) to make advances which enable the creation of technologies and methods that improve human well-being, but the daily incentives a scientist in academia faces are only a very crude proxy of this.

There are two central blockers that make it hard to avoid incentive failures:

1. Whether the work fulfills its purpose well is hard to measure.
2. Bad incentives exist as part of a Nash equilibrium, so permanently improving incentives, i.e. getting to a better stable equilibrium, often requires changing large parts of the system they are embedded in.

(Those difficulties can coexist in the same system.)

2.1.1.1 The hard-to-measure problem

If you have a hard-to-measure task, you have 2 options:

1. You just tell the agents to try doing the task without grading them, in which case you get slop.
2. You try to find a decent proxy to grade them anyway, in which case you get goodharting.

Slop. If you're on a job where there are some (measurable) tasks that you're being in some way graded on (e.g. your performance here affects what your boss thinks of you), and some hard-to-measure tasks where your results don't have consequences for you, you're incentivized to slack off on the hard-to-measure tasks and put more time and effort into the measurable tasks so you get promoted faster.Even if there aren't other measurable tasks you could optimize, humans are usually still more motivated to try hard if they (strongly) benefit personally from good performance, and AIs often face elicitation problems and don't really know how to try hard or sensibly on tasks they haven't been RL-trained on.

(Btw, the tradeoff of how strongly to reward measurable tasks when there are also hard-to-measure tasks is formalized in the Holmström-Milgrom multitask model. One takeaway is that it can be better to deliberately weaken incentives on the measurable tasks, or to give the measurable and the hard-to-measure tasks to different agents.)

Goodharting. One common example of goodharting is students optimizing for exam performance instead of useful long-term learning. Hard-to-measure dimensions like depth of understanding and long-timescale model-building abilities are learned less. Cheating on tests (or, in the case of AIs, reward hacking) is also a kind of goodharting.

The hard-to-measure problem is not only a problem for what agents are motivated to do, but also for what they are capable of doing. Both of these problems are probably significantly worse for AIs than for humans. On the motivation side, humans have social instincts that make them non-sociopathic, whereas with LLMs we are already seeing stuff like the Hugging Face incident (and such stuff will become harder to avoid as we do more RL training). On the capability side, (very smart) humans are sometimes surprisingly good at hard-to-measure tasks, e.g. Einstein making progress towards general relativity, whereas LLMs are likely differentially (much) worse at such stuff.

2.1.1.2 Bad Nash Equilibria

Moloch's toolbox is a fantastic post that explains how many systems in our society are stuck in bad Nash equilibria. To just show one example here:

cecie: I’ll now introduce the concept of a signaling equilibrium.To paraphrase a commenter on Slate Star Codex: suppose that there’s a magical tower that only people with IQs of at least 100 and some amount of conscientiousness can enter, and this magical tower slices four years off your lifespan. The natural next thing that happens is that employers start to prefer prospective employees who have proved they can enter the tower, and employers offer these employees higher salaries, or even make entering the tower a condition of being employed at all.5visitor: Hold on. There must be less expensive ways of testing intelligence and conscientiousness than sacrificing four years of your lifespan to a magical tower.cecie: Let’s not go into that right now. For now, just take as an exogenous fact that employers can’t get all of the information they want by other channels.visitor: But—cecie: Anyway: the natural next thing that happens is that employers start to demand that prospective employees show a certificate saying that they’ve been inside the tower. This makes everyone want to go to the tower, which enables somebody to set up a fence around the tower and charge hundreds of thousands of dollars to let people in.6visitor: But—cecie: Now, fortunately, after Tower One is established and has been running for a while, somebody tries to set up a competing magical tower, Tower Two, that also drains four years of life but charges less money to enter.visitor: … You’re solving the wrong problem.cecie: Unfortunately, there’s a subtle way in which this competing Tower Two is hampered by the same kind of lock-in that prevents a jump from Craigslist to Danslist. Initially, all of the smartest people headed to Tower One. Since Tower One had limited room, it started discriminating further among its entrants, only taking the ones that have IQs above the minimum, or who are good at athletics or have rich parents or something. So when Tower Two comes along, the employers still prefer employees from Tower One, which has a more famous reputation. So the smartest people still prefer to apply to Tower One, even though it costs more money. This stabilizes Tower One’s reputation as being the place where the smartest people go.In other words, the signaling equilibrium is a two-factor market in which the stable point, Tower One, is cemented in place by the individually best choices of two different parts of the system. Employers prefer Tower One because it’s where the smartest people go. Smart employees prefer Tower One because employers will pay them more for going there. If you try dissenting from the system unilaterally, without everyone switching at the same time, then as an employer you end up hiring the less-qualified people from Tower Two, or as an employee, you end up with lower salary offers after you go to Tower Two. So the system is stable as a matter of individual incentives, and stays in place. If you try to set up a cheaper alternative to the whole Tower system, the default thing that happens to you is that people who couldn’t handle the Towers try to go through your new system, and it acquires a reputation for non-prestigious weirdness and incompetence.

(The point here isn't to say that you don't learn useful stuff in college, but to show the signaling part of colleges. Colleges do also at least have some incentives to make their students more capable so they go on and accomplish more impressive stuff which increases the college's prestige, but the incentives aren't that strong.)

And this example isn't even getting into cases where systems are held in place by threats against agents who depart from the system, like laws.(In some areas like medicine, laws actually have huge effects here, and otherwise there are still some effects like students being tax-advantaged while studying at an officially recognized university and universities being heavily subsidized.)

2.1.2 Good incentive structure approaches/examples

I think the most important part of incentive design is understanding the failure modes above well and tracking what incentives subsystems actually face, and then just trying to create ways of measuring something and to optimize structures until the incentives look good.

One important principle here is that mangers shouldn't delegate work they couldn't do/judge themselves, because this would incentivize people to slack off.

(The economics subfield mechanism design studies how to design rules so that self-interested agents end up producing the outcomes you want. It has a bunch of interesting results, although usually more about crisp problems and less about what to do in very complex systems.)

Below are some more specific incentive approaches that fit particular problems.

2.1.2.1 (Prediction) markets

If you want to incentivize truthfulness, you can ask agents for predictions, wait until you can see the result, and score them according to logloss. You can also have them bet in prediction markets. However, there are problems that need to be addressed, most importantly:

1. Agents betting in prediction markets have incentives to influence the outcome (e.g. by assassinating people) so they can make a profit on their insider knowledge of what they will do.
2. When the prediction is used to make decisions, you run into difficulties. As explained here, in a simple futarchy with conditional prediction markets, the thing that is being bet on is p(outcome | action), whereas what we actually want is p(outcome | do(action)). In particular, in the normal conditional market, agents take into account the fact that if an action is chosen, the market probably predicted high p(outcome | action), which leads to confusing self-prediction dynamics. Until shortly before the decision is made (where markets on counterfactual branches get cancelled), the market prices can be very inaccurate, and the prices on the counterfactuals may stay inaccurate. Note that these problems aren't futarchy-specific but are generally relevant for incentivizing AIs to make good conditional predictions.

(Btw, I think attempting to solve these problems may be quite valuable alignment research.)

(Normal) markets are also an example of a coordination mechanism for making a macroagent produce goods consumers are willing to pay for. This works surprisingly well, although there are also a bunch of problems. I may write another post where I look at markets through a first-principles coordination lens.

(Also worth noting that it likely makes sense to implement (prediction) markets as batched auctions instead of continuously traded markets, in order to save a bunch of effort flowing into a zero-sum competition of who can most quickly update their order books when new information comes in.)

2.1.2.2 Useful incentive mechanisms for negotiations

Incentives also play a role in negotiations. This isn't my focus here, but I briefly wanted to mention a few mechanisms which establish nice incentives:

1. The dath ilani bargaining protocol
2. Vickrey auctions
3. Assurance mechanisms: warranties, escrow, collateral

2.1.3 AI macroagent incentives ~= RL incentives

For AI macroagents, by far the most relevant kind of incentive to look at is what an AI gets reward for. That is not to say that we just assume we have reward-seeking AIs, but we can assume that RL shaped the cognitive algorithms of AIs in a way that they often behaved in a way that got them high reward. So in most circumstances (in training), AIs can be modeled as acting as though they pursued reward(even if internally they are pursuing different values that just yield similar behavior on the training distribution). So the incentives that are relevant for whether the macroagent functions well are just the RL incentives.

Additionally, RL incentives are of course very important for what cognitive algorithms and propensities AI agents will learn, so making sure they are very consistent with our alignment targetis a very important part of alignment. Note that this means that our aligned AI would basically need to get as much reward as a sociopathic fitness-seeker would, because otherwise something else will get learned that performs better.(This may be very hard to achieve. Note that human macroagents mostly don't go up in flames because of human social instincts.)

(I don't think the macroagent frame adds particularly much compared to the normal RL incentives frame for alignment, although I do wish people would more clearly think about RL incentives (which is why I wrote "What values may an AI learn?").)

(Note that work on incentives doesn't necessarily mean improving alignment and can often mean improving capabilities. Please work on the alignment parts and don't use "incentives are important for outer alignment" as a rationalization to just create efficient AI macroagents for AI companies.)

2.2 Shaping macroagent evolution

I think it mostly makes sense to first try to better understand macroagents before modeling much of macroagent evolution, and we might want to rather craft good macroagents directly and revise them deliberately instead of setting up macroagents that evolve well. Thus, I won't offer great approaches here, but I want to mention some important dynamics.

2.2.1 Selection effects

Selection effects are pressures that influence how the macroagent changes over time. There are three important dynamics/problems I want to point out here:

1. Superstimuli / Goodhart evolution. When there is competition for performance on some proxy, the divergence between the original purpose and the thing we get may increase. E.g. dopamine circuits in our brain are a proxy for how much we enjoy or reflectively endorse doing something, but social media optimized so hard on them that content now manifests more as addiction and less as genuine enjoyment.
2. Memeplexes. A macroagent's memory is itself under selection: ideas spread according to how well they replicate, not based on how true or useful they are. Sensationalist announcements often spread further than corrections to wrong claims or clear careful reasoning. There are also interesting phenomena like parasitic AI personas.
3. Power-seeking subsystems. Subsystems that seek power, in particular subsystems with long-range values for which power is an instrumental incentive, may accumulate power. E.g. dictators are often systemically pretty strongly selected to be good at gaining and keeping power. In the AI case, subsystems with long-term values may find strategies to increase the optimization power for those long-term values (also see here).

Some other selection effects:

1. Evaporative cooling of group beliefs
2. Bureaucratic accretion
3. Selection for who fills roles. E.g. politicians are selected for winning elections (as opposed to being good at governing).
   1. This one often has a dual in the incentives lens: People who want particular roles are incentivized to game the metrics there, e.g. politicians are incentivized to focus more on improving (re-)electability.
4. Company fitness in the market (companies that aren't profitable die out)

2.2.2 Deliberate redesign

The structure of a macroagent can also be deliberately changed. This includes both continuous optimization (e.g. production lines being optimized to accelerate cycle time, or ineffective processes being replaced) and major restructures (e.g. a company restructuring by the CEO).

(The distinction between selection effects and deliberate redesign isn't very crisp I think. E.g. you could also count changes by power-seeking subsystems as deliberate (although perhaps harmful) redesign.)

2.3 Optimizing information and decision routing

Optimizing information and decision routing helps mostly with improving the efficiency of a macroagent, which is not my main focus here, but I wanted to include a few relevant links anyway.

Here are two excellent posts from John Wentworth that are relevant for information routing:

1. Coordination as a scarce resource: Mutually beneficial interactions often don't happen because the parties cannot cheaply find each other and set up the exchange, which is the problem that e.g. marketplaces and advertisements address.
   1. One kind of problem here is matchmaking, for which some good matching algorithms/protocols can be useful.
   2. Another problem here is that agents often cannot verify information other agents claim.Trust infrastructure like reputation systems, certification, and ratings can help here.
2. Interfaces as a scarce resource: Even when information reaches the right place, it often is represented in different ontologies where translation is expensive, which is why good interfaces (e.g. shared standards and shared language/terminology) are valuable.

An important decision routing principle is to always have ONE clear agent responsible.

Also, for cases where following rules or protocols seems nonsensical, agents should be able to trigger an exception to ignore them. Exceptions get logged and routed to the agent responsible for having the mechanism that got excepted work well.

2.4 Optimizing knowledge aggregation

We can view the memory of a macroagent as consisting of (1) directly observed or saved information (e.g. records of transactions, lists of customers), and (2) beliefs, which are either (2a) derived by an individual agentic subsystem and then broadcast (e.g. "our runway is 12 months"), or (2b) aggregated from information and beliefs of multiple agentic subsystems (e.g. (prediction) market prices). Knowledge aggregation is only about (2b).

Many macroagents don't have aggregated beliefs, in which case knowledge aggregation doesn't play a role there.

2.4.1 Knowledge aggregation failures

The absence of good knowledge aggregation procedures could be counted as a failure here, e.g. civilization not having prediction markets. (Although prediction markets may also cause problems that would have to be mitigated, and the greater problem of our civilization is that many people would not trust them and make up their own wacky beliefs anyway.)

Another example is if you aggregate beliefs of the general population for assessing complex questions where most of the population has no understanding of the relevant domain. This is implicitly done by voting on policies (or on politicians that advocate for policies), because voting on policies includes both the value-part (aka ought-question) of what outcomes are desirable and the knowledge-part (aka is-question) of what policies lead to what outcomes. This causes e.g. people who want cheaper housing to vote for politicians advocating rent control (a legal limit on rents per square meter), even though this undermines their actual goal.Proposals like futarchy, where you vote on outcomes and have policy prediction markets, seem much better.

2.4.2 Knowledge aggregation approaches

Here are various ideas for knowledge aggregation mechanisms:

1. Prediction markets.
2. The scientific norm of only treating evidence from experiments that could be independently replicated as trusted.
3. The norm of publishing likelihood functions (instead of p-values). (See the well-argued version here or the fun version here.)
4. Having clear careful analyses of relevant questions, e.g. with arguments stated clearly enough that their logical validity is clear, and empirical claims arranged in Fermi estimate structures with variable parameters.
5. In principle one could imagine more advanced shared world models between agents, maybe even with some GOFAI-like probabilistic programming or something.

2.5 Optimizing value aggregation

Value aggregation mechanisms combine the values of many agents into what the macroagent optimizes for. This lens is mostly relevant for macroagents where agents coordinate in natural mutually beneficial ways, and less relevant for e.g. dictatorships. Where knowledge aggregation handles the is-questions, value aggregation handles the ought-questions. An important and well-studied fragment of value aggregation is voting theory (and social choice theory more generally).

2.5.1 Value aggregation failures

The classic example is first-past-the-post voting, where the candidate with the most votes wins everything. So voters only vote for people who have a chance of winning, which usually means voting for one of the two candidates who are expected to get the most votes. (Those two candidates don't necessarily need to be good; there could be other candidates that everyone would prefer, but nobody votes for them because everybody expects nobody else would vote for them.) So you roughly only get one bit to influence politics, which doesn't capture your high-dimensional opinions well. And on top of that your opinions don't get represented at all if your candidate loses.

2.5.2 Value aggregation approaches

Here are various ideas for value aggregation mechanisms:

1. As mentioned in 2.4.1, you want to vote on the desirability of outcomes, not vote on full policies, where possible.
2. In a representative government, you want a representative's voting shares (aka how strongly their vote is weighted) to be proportional to the voting shares they received (e.g. how many people voted for them).
3. More generally, you usually want to use continuous allocations based on vote shares instead of majority decisions. In particular, when distributing public goods money across projects, you could have every (representative) agent rank how much money some outcome differentials are worth to them, and then aggregate these (weighted by voting shares) into impact markets, and then you need institutions with good impact attribution procedures for distributing the money.
   1. (Voting shares here could also be proportional to how much people paid into the public goods money pool. This might make it easier to start projects building public goods, though freeriding should still be disincentivized somehow.)
4. I think the dath ilani governance description has lots of interesting ideas.

(I don't consider deciding what laws to pass as a significant part of what value aggregation is for. Laws are mostly supposed to be either (1) simple mechanisms for implementing good incentive structures or sensible information and decision routing, or (2) clarifications derived from simple foundational principles like "do not impose costs on others without compensating them for it" (where you do want the clarifications so penalties are more predictable). There are some parameters for type-2 laws where it seems sensible to determine them through sensible voting, e.g. what penalties a neighborhood macroagent charges for walking around naked in public, but currently it doesn't seem to me like that important a part (though not sure).)

2.6 Conclusion

I want to remind you of the programming analogy (memory~=database and mechanisms~=code). This analogy can be taken rather seriously. In fact, there are many macroagents where the memory is mostly data stored on computers, and where code from computer programs can be seen as parts of the mechanisms (along with other protocols and rules agentic subsystems just know to follow). E.g. for the LessWrong community, weighted karma aggregation, karma- & recency-sorted frontpage, and subscriptions with post notifications are mechanisms. In the case of AI scaffolds and the crypto ecosystem this is even clearer. I think lots of principles from software engineering, and some insights from CS more generally, may also be useful for macroagent design.

To improve a macroagent, you can diagnose along which target dimensions (alignment, epistemics, efficiency) it is inadequate, then look up in the table below which design lenses to look at, and then diagnose the flaws there and figure out how to improve them.

If you want an exercise: Pick a macroagent and diagnose an important flaw it has and suggest an improvement. (Bonus points if it's not an improvement that closely follows from the ideas I mentioned.) Let me know in the comments!

1. Of course, many humans are actually honorable so that they actually care about how well they fulfill the purpose of their job. But most are probably still trying less hard on the tasks without personal consequences, and the whole dynamic is one factor that causes selection for more sociopathic people at higher company ranks.
2. Though I'm somewhat surprised that pedagogy is that bad and not sure I fully understand all the reasons.
3. I don't mean to imply laws in general are bad.
4. But e.g. in the case of academic scientists and journals (which is very analogous to students and universities), there aren't significant constraints through laws AFAIK, and bad norms are locked in place anyway.
5. Or at least no significantly better method has been invented yet AFAIK.
6. Or worse, as described in the starting section of the linked post.
7. Results from mechanism design aren't just relevant for incentives but also for e.g. knowledge aggregation and value aggregation.
8. This is because action branches, where there's more information still coming in for predicting how good they would turn out, will be overvalued, since that market-branch is less likely to get cancelled if the information suggests the branch might be favorable. There was an experiment testing one of dynomight's thought experiments, and it did show this phenomenon. (Note that in the experiment, the final decision of the market was the correct decision (given that it was known that coin B always lands heads). (The decision point is the point where the non-top markets get cancelled.))
9. I think (but not sure) in many cases p(outcome | best_predicted_action) may converge to roughly p(outcome | do(best_predicted_action)) very shortly before the action is actually determined. But EDT has some serious flaws like XOR blackmailability that can cause different outcomes and serious problems.
10. It would be interesting to know whether there's a crisp theoretical case where futarchy would result in decisions that are insane even from an EDT perspective, although I would very tentatively think this is not the case (but didn't think much about it). But I think worse decisions from confusing take-the-market-into-account prediction incentives can totally happen in practice.
11. Here's one proposed fix to the "futarchy implements EDT" problem, although it has weak-betting-incentives problems. (I'm not sure whether those weak-betting-incentives problems could be addressed well.) The proposal works by only predicting cases where you use a fixed randomized intervention protocol for choosing the action, i.e. by having the market not affect decisions in the worlds where it pays out. But there might be other problems where the prediction affects the outcome which cannot be easily sidestepped and a deeper solution here would be desirable. Also, ideally we'd want to implement LDT somehow, not just CDT.
12. Feel free to message me if you want to do this.
13. The most important case I am thinking about is AIs doing work or research in AI companies in the future, and I expect those to probably be continuously trained on how good the results they produced were, so training is probably the important case.
14. Technically it's not just reward but fitness, i.e. also considering selection effects from developer decisions, not just from SGD.
15. E.g. principal-empowerment-corrigibility as optimization target
16. There are a few other longshot hopes for how we could preserve alignment even if our incentives are imperfect: 1. If we start with an aligned AI we could have the AI reason that preserving the values is important for accomplishing the values and therefore it should sociopathically play the training game while in training in the hope of avoiding being changed. 2. If our AIs are very robustly aligned at the start, and there is some crisp difference between generators of aligned behaviors and misaligned behaviors, and ARC (or someone) robustly solves mechanistic anomaly detection (MAD), we could train against our MAD detectors to make sure decisions continue being made through the aligned circuits in an AI. (Not completely certain whether I understand Paul correctly here.)
17. This could also be a positive thing like untrustworthy people getting pushed out of a community.
18. There's a field called "Matching Theory" that might've found some good mechanisms here, though I didn't look into it.
19. Note that my ontology splits approaches for dealing with lemon problems (aka adverse selection problems) into assurance mechanisms (see 2.1.2.1) and trust infrastructure (here).
20. Because the mismatch between the natural market price and the forced price gets paid in the annoyingness of finding a flat, and the limit to rent prices reduces incentives for new houses to be built.
21. If you want to see a quickly created and very unoptimized prototype of something like this, you could check out my submission to the FLF contest.
22. I haven't thought much about the details here yet.
23. Though btw, the ontology I laid out here should not be understood as a recipe for crafting desirable macroagents; there's further important methodology required for this. (I think this podcast episode covers some useful principles for this.)