AQuA's "self-improvement" updates research state, not the agent LM. What should a local port freeze?

AQuA's preprint uses "recursive self-improvement" for a bounded research loop. It does not say the research-agent LM rewrites its own weights. The paper separates three objects: The language model driving the research agent and the evaluator stay fixed within each part. Each part has its own persistent research state. Validated experiments update that state and guide later proposals; Part I and Part II do not share it. In Part II, each config diff produces a separately trained hybrid task-model variant. That training is part of the experiment, not an update to the research-agent LM. This matters if someone ports the harness to a local LM. Inside one AQuA run, later proposals can change because the retained state changed even though the agent LM did not. Across two ports, a score difference could come from the agent model, state history, prompts and tools, evaluator feedback, or the candidate-training path unless those variables are separated. A useful release would include: the exact agent model and quant, plus context and sampling settings; the system prompt, tool schema, and initial research state; every state update and evaluator return; evaluator version, config diffs, and candidate-training settings; seeds and per-run traces. At minimum, the comparison needs three ablation families: swap only the agent LM under the same harness and starting state; keep the LM fixed while resetting or replaying the research state; and keep the LM and state fixed while changing the harness or evaluator contract. For Part II, candidate task-model training needs its own log, or a change in the trained candidate gets folded into a claim about agent-model capability. The preprint gives the architecture-level separation, but it does not publish a runnable implementation or enough detail for end-to-end reproduction. A local-model port would be a new experiment, not a reported AQuA result. If only one artifact shipped first, which would make attribution most credible: the state-transition log, the evaluator contract, or a full model-by-harness matrix with repeated seeds?

添加评论
点赞收藏
点踩分享查看原文
评论
?
参与讨论