My Assessment of Plan A's Compute Verification Strategy (+ open questions)

Overview

These are my non-expert notes on the compute verification section of AIFP’s Plan A. I cover interconnect limits, memory wipes, network taps + replay, and ZKPs. For the most part, the sections can be read independently. I restrict my attention to inference-only verification: ensuring that compute is used for inference, not training. For each method suggested by AIFP, I ask:

1. How much can it slow down training?
2. How much overhead does it add to inference?
3. What sensitive information does it require adversaries to share with each other?

AIFP estimates that the fraction of the world’s compute that is unmonitored might be kept as low as 0.1% (this is the optimistic, low end of their 80% confidence interval). So my target for inference-only verification is to slow down training by 1000x – any more hits diminishing returns as unmonitored compute dominates – with much less than 1000x overhead on inference and little sharing of secrets.

I won’t discuss how much a 1000x reduction in effective training compute would actually benefit humanity. The answer depends greatly on algorithmic progress rates; I wish labs would publish the rates they’re seeing internally.

Interconnect Limits

In a datacenter, accelerator racks are connected to two networks. The backend network is used to send gradients between racks during training, and to send some large objects like KV caches during inference. The frontend network is used to send prompts and responses, and occasionally to load model checkpoints. AIFP proposes physically removing the backend network. Inference would be mostly unaffected, since the inference stack can probably be rewritten so that KV caches never cross rack boundaries.[1] But standard training relies heavily on the backend network, so the hope is that removing it would massively slow training.

Just removing the backend network would likely not slow training by much, though, since the frontend network’s bandwidth per rack is not that much lower than the backend network’s: ChatGPT estimates ~1TB/s and ~4TB/s, respectively, with big error bars.[2] Therefore, one should additionally enforce per-rack bandwidth limits on the frontend network, calibrated to allow sending tokens but not gradients. ChatGPT estimates that, after compression, a typical LLM inference rack sends and receives a few MB/s, so a 10MB/s limit may be acceptable. At this rate, sending or receiving a trillion-parameter gradient in fp8 would take around a day, slowing standard training massively.

The obvious attack is low-communication training via algorithms such as DiLoCo or SparseLoCo, which let each rack train its own replica of the model, only occasionally syncing updates. There isn’t much data on how well these algorithms work at frontier scale. Covenant-72B was pretrained using SparseLoCo, communicating pseudo-gradients that are 146x compressed relative to fp32 every 30 steps. If we naively apply those numbers to a trillion-parameter model, assume that each backward pass takes 10s, and limit bandwidth to 10 MB/s per rack, then for each 5 minutes’ worth of computation, a rack must spend ~50 minutes sending the update: a 10x slowdown. But Covenant-72B is far below frontier scale, and used only 20 replicas, whereas a frontier datacenter would need thousands. Also, although the model is supposedly around Llama-2-70B level, it is not benchmarked against the same model trained with standard high-bandwidth AdamW.[3] So we have our first open question:

OQ1: How well does low-communication training work at frontier scale?

I don’t think you should directly work on OQ1, but it seems important to keep track of the SoTA and forecast future progress. Another, lower-priority open question:

OQ2: How can interconnect limits allow for video generation, which produces a lot of egress?

Summary: Removing the backend network, and imposing bandwidth limits on the frontend network, comes with little downside: it requires no secret sharing between adversaries and imposes minor overhead on inference. Training with only 10MB/s bandwidth per rack sounds difficult, but it’s hard to estimate the exact slowdown it would impose, since little research has been done on super-low-communication training across many racks (at least, not compared to the amount of work that has gone into optimizing frontier training stacks).

Memory Wipes

Another technique that seems useful: periodically wipe each rack’s memory, with a proof that the wipe was done correctly, then reload it only with precommitted stuff (no new artifacts like weight updates). AIFP mentions memory wipes as a method to ensure “that the only outputs from the cluster are verified inference tokens” (“verified” here means “reproducible via replay” – see next section). But I don’t understand this: why does it matter what a rack has in memory, provided every bit it emits is verified?

Perhaps memory wipes could help prevent the sort of low-communication training that evades interconnect limits. If the interval between wipes is short, not much gradient information can be accumulated by a rack (note that although SparseLoCo syncs are small and infrequent, each node still must accumulate a large amount of information over the course of training – the net parameter update is dense). We saw that, with a 10MB/s bandwidth limit per rack, a full trillion-parameter update in fp8 would take about a day to send. So, in order to be useful, the period between wipes should be less than a day. Unfortunately, in an initial test, Amodo Design found that provably wiping a rack took ~24h and, if I'm reading their results correctly, left more than 100TB of memory unwiped.[4]

OQ3: How quickly can a rack’s memory be provably wiped (with minimal residue)?

Summary: I mainly see memory wipes as a technique to combine with interconnect limits. They force the attacker to use algorithms that are not only low-communication, but that also keep each rack's state close to the initial checkpoint: a rack can only accumulate as much extra state as fits through the bandwidth limit in one wipe interval. I expect that this could be a really powerful way to slow training, and it does not require any secret sharing. The only issue is that current memory wipe methods take too long (adding a lot of overhead to inference) and leave too much residual memory unwiped.

Network Taps and Replay

Network taps record accelerator racks’ ingress and egress. This traffic goes to a replay server which recomputes egress from ingress. Since the replay server contains much less compute than the full datacenter, only a small fraction of traffic can be replayed. Concretely, the prover should provide[5] a partition of ingress and of egress, after which the verifier randomly samples some and challenges the replay server to run a program on input to reproduce output . For example, might be the prompts for a given chat session, the responses, and the inference code (which includes model weights).

Plan A says that replay is used to “check the correctness of the packets”. I find this phrasing confusing: how are the packets produced by training any less “correct” than those produced by inference? My understanding is that what replay buys us depends on whether the replay server hardware is trusted by the verifier or by the prover (I assume that manufacturing a mutually-trusted replay server is impractical). I sketch the sorts of properties one could verify in each case below.

Trusted Replay

If replay takes place on verifier-trusted hardware, the verifier can be sure of exactly what program P the replay server runs (e.g. it can see model weights, kernels, and computational graphs). This feels like it should make the verifier’s job easy: all it needs to do is read P, then decide whether it is training or inference. But the code might be obfuscated, and the verifier must be able to detect unseen future training algorithms.

OQ4: Given some code, how can we decide whether it is training?

Rice’s theorem rules out a perfect classifier for arbitrary…