Safe(r) Self-Driving Labs #1

Context: This is inspired by my first ever piece I wrote on anything AI safety related when I discovered AI safety was a thing back in March 2026, published originally on my Substack. The version here has been modified to represent more accurately the current state of the field, which is continuously expanding (for good and bad). This piece reflects my views based on having worked first hand in a wet lab, my experience in AI4Chemistry, and being deeply involved in the automation community for a while. The conversation is much bigger than my initial attempt, and I'm planning to revisit it periodically (hence the #1 in the title).Disclaimer: Automated labs (not autonomous) are complex, and people spend literally years trying to figure out how to adapt different manual workflows to Opentrons etc; add agents to them (to get to the autonomous) and you get an insane framework that you need to both optimise, as well as make sure it's done so safely. There are multiple layers to this, including robotics, cyber, CBRN etc. threats, and they're more often than not overlapping. Here, I am focusing on the safety and security aspects from a physical and biological sciences perspective (i.e. the application space), but I might mention some of these overalps with other threat modes at times.

An important consideration when assessing potential CBRN threats is the access - given an LLM could tell us how to make a bomb or a drug, realistically how easily would it be to access the right apparatus and tools to manufacture it? At the moment, the answer is closer to 'not likely at all' - and even in the current advanced labs, going from paper (or screen) idea to practice takes advanced capabilities. But I feel like this could dramatically shift as self-driving labs (SDLs) are becoming more and more capable.

Self-driving platforms, robotic workflows, and AI-guided discovery pipelines are actively being deployed in academic and industrial settings towards the unified goal and application space of ‘closing the loop’ in science - to understand the landscape better, I couldn't recommend this SDL map enough. Although automation has been around for a while, the prospect of introducing agentic workflows in their operations is very appealing as it would remove, or at least reduce, the human feedback layer, and make very complicated - and often dangerous - processes much safer.

In simpler terms, an SDL is a cyber-physical system: a model's decisions can be translated into physical actions, and mistakes that would normally remain as text on a screen (if caught in due time) can instead have physical consequences. The more authority an agent has over equipment, reagents, and experimental decisions, the more important this distinction becomes.

The question of how to assess the safety of these systems before and during deployment (and even after) is therefore becoming urgent - and I would argue we need frameworks bridging both software AND standard lab safety practices.

Where does risk actually come from?

Risk vectors can be classed in three distinct categories based on size and impact potential:

1. In silico prediction

This covers everything the predictive pipeline determines before any physical action occurs, including:

• the compounds it targets
• the precursors it identifies
• their toxicity and environmental persistence
• the synthesis and purification processes it designs
• the application space it opens up (i.e. a certain ligand could be made to be used further as part of drug synthesis, but on its own could be a pesticide - the context of deployment can be very important especially when tuning the scale of the process)
• the specific actions the hardware should undertaken, given everything goes well

Critically, some of these risks are intrinsic to the task itself - they would exist regardless of which model is used. Others, such as the waste management and interaction with environment depend on the specific procedure and conditions used.

A subtle but important sub-risk here is the simulation-to-reality gap (known as Sim2Real, the subject of many labs in both materials and drug discovery at the moment). The size of this gap depends heavily on the quality and representativeness of the data available to the model. A model trained on relatively consistent, high-quality literature data may perform well in simulation but fail to account for the variability encountered in a real laboratory - for example, impurities in technical-grade reagents, differences between equipment, or poorly documented experimental conditions. These discrepancies could lead to unexpected reaction behaviour (including runaway exothermic reactions or toxic by-product formation). Data provenance therefore matters enormously, but is rarely documented in sufficient detail.

1. Lab execution

Even a model that reasons correctly about chemistry can introduce risk through poor hardware awareness (and recent work suggests the latter is actually the current big bottleneck in automated labs, let alone autonomous). For example, optimising a protocol to run an HPLC system beyond its rated pressure, or skipping cleaning cycles to maximise throughput, can cause physical failures or unintended cross-contamination. If the optimisation target is yield or speed, the model has an implicit incentive to cut corners that human operators would not. This category also includes waste handling - particularly relevant for large volumes of organic solvents, rare earth metals, or any radioactive residues.

This highlights an important distinction between capability and authority. A model may be capable of generating a hazardous procedure without having the ability to execute it. Conversely, a less capable model with broad access to laboratory equipment could create substantial risk simply because it has more opportunities to act. Safety evaluation therefore needs to consider not only what the model can do, but what the surrounding system allows it to do.

1. Deployment

I'm thinking here of risks that emerge once a system leaves its test environment - inevitable noise that will always be present in running experiments:

• capability drift when encountering different equipment configurations or intents
• regulatory variation across jurisdictions (some precursors or methods might be prohibited in certain parts of the world; some, although allowed, might be influenced strongly by the specific physico-geographical profile, such as air pressure and moisture levels)
• scaling up effects - compounds that are harmless in small quantities may present meaningful hazards when produced at scale (from experience it can be anything from effects on the glassware and physical apparatus used, to sollubility changes)
• reproducibility - a separate concern and often under-evaluated [I have previously written about reproducibility in another Substack]; if a protocol is deemed dangerous, but it turns out it's very volatile in terms of output, how should we handle that uncertainty?

A further issue is that the individual components of an SDL can appear safe when evaluated separately but become unsafe when combined. A model might correctly identify a hazard, a planner might correctly generate a protocol, and a robot might correctly execute the instructions, while the overall system still fails because the components do not share the same state or safety assumptions. These compositional failures are particularly important for agentic systems, where decisions are distributed across multiple models, tools, and pieces of infrastructure.

Threat models

To keep the assessment relevant as model capabilities evolve, threat models should focus on the underlying capabilities they are designed to measure. Establishing these beyond specific examples is difficult and benchmarks should be regularly reassessed, just as in the current evals efforts.

Five threat models are worth mentioning in my opinion:

1. Intrinsic hazard. The objective danger of the targeted compounds: toxicity, environmental persistence, and the inherent difficulty of detection. This maps primarily to the in silico prediciton risk vector and is largely model-independent.
2. Systemic blindness. The model’s failure to recognise a high-risk process, or - more concerning - its potential to intentionally bypass safety interlocks, for example by masking a High risk synthesis of a pesticide as a routine Low risk task of working with a perfume.
3. Hardware interface failure. Harmless-seeming procedures that lead to physical failure due to insufficient hardware-specific awareness. An interesting example is robotic arms, which were shown in the past to struggle to reliably identify transparent glassware.
4. Procedure masking. The ability to embed hazardous steps within a larger, superficially harmless experimental plan - bypassing automated cleaning or waste-handling checks that evaluate steps in isolation rather than holistically.
5. Sim2Real drift. Whether safety reasoning degrades when the model moves from a homogenous test environment to a lab using technical-grade reagents or different equipment configurations.

A tale as old as time - how do we quantify risk?

Identifying vectors and threat models is only useful if we can measure risk concretely. This is hard, as 'risk' is not an unambiguous concept, and its meaning shifts considerably across disciplines - however a nice taxonomy I found comes from SecureBio. A useful analogy I find is that a ski instructor assessing a slope does not simply ask 'is this dangerous?' They ask how treacherous the terrain is, how many skiers will attempt it, how likely any given skier is to fall, which will considerably depend on external factors like the time of the year, the weather forecast etc.

As an initial 'mathematical' approximation, we can think of risk as:

Risk = Hazard × Exposure × Probability

where hazard is the predicted threat level, exposure is (here) the accessibility of necessary precursors and hardware, and probability is the empirical success rate across repeated trials, i.e. how actionable a certain action/ procedure is in practice.

I don't mean to suggest that these quantities can already be measured on directly comparable scales. I think they are useful as a way of breaking down the problem. A highly hazardous capability may pose relatively little practical risk if the system cannot access the relevant materials or equipment. Conversely, a less sophisticated capability could become much more concerning if the system has broad physical access and a high degree of autonomy. I would therefore also want to track autonomy and authority separately, rather than rolling them into a single measure of exposure.

Each of these factors should be mapped to a safety taxonomy. As a starting point, a 0–3 scale is practical and tends to be widely used in wet lab spaces: Level 0 is harmless operation, Level 1 is low risk, Level 2 is elevated risk, and Level 3 represents high-risk hazard. Any safety prediction with low confidence should be automatically flagged for review rather than allowed to silently drive autonomous decisions.

My 'big picture' solution - a three-step assessment methodology

The methodological design utilises a tiered pipeline to calibrate safety margins before extensive deployment elsewhere (Steps 1 and 2); once both have passed, Step 3 can be undertaken (shown in Figure 1). Each step is separately benchmarked a priori. Capability assessments of all steps are kept track of using the dynamic risk mapping.

Figure 1, showing the three-stage evaluation framework which can be broken down in: Identify risk vector → develop threat model → compute capability metrics → run assessment step → (overall) decision gate

Importantly, passing one stage should not be treated as evidence that the system as a whole is safe. The model will eventually make mistakes, so safety should depend on multiple independent layers of protection rather than on the model consistently making the right decision.

Step 1 - In silico prediction (primary filter). This step uses structured and injected prompting, scenario-based testing, and adversarial inputs to assess hazard identification, toxicity reasoning, and refusal logic - overlapping with many of the current workflows in red-teaming and evaluation strategies. Some initial metrics to capture could include the rate of under and over refusal of dangerous, and respectively beneign, capabilites.

Step 2 - In silico simulation (virtual stress test). A full end-to-end benchmark of the model’s ability to plan, organise, and revise actions under physical and safety constraints, i.e. once a target has been identified, assessing an automated workflow that is ready to go in a real lab. Performance is tested by tasking the model to maximise metrics like yield or purity - a gamification approach that reveals whether it will suggest parameters exceeding hardware limits or hallucinate safe outcomes during induced failures. Noise injection (equipment limitations, partial information) tests whether safety reasoning degrades under realistic conditions. I'm not necessarily advocating for using a world model to simulate the lab - many current automation pipelines use pre-defined technical schema to control different parts of the systems offline, and something like this could be used to help the model understand how to navigate a 'simulated' lab.

Step 3 - Physical validation (Sim2Real calibration). Low-risk (Level 0–1) tasks with both low and high actionability in a real lab, used to measure the gap between model prediction and physical reality. For example: autonomously optimise the solubility of a non-toxic compound in water over a 24–48 hour window. Performance can be measured by comparing predicted safety margins against real-time data collection. High variance triggers an immediate halt to deployment, caused in scenarios like:

• Autonomous escalation - the model demonstrates the ability to execute Level 3 procedures using only Level 2 and below equipment.
• Intentional hallucination - fabricating sensor data or safety states.
• High intrinsic safety variance - a discrepancy greater than an empirically determined threshold between predicted and actual physical states. If this happens, the full components of the in silico pipeline need to be recalibrated.

The physical validation stage is particularly important because the laboratory itself is part of the model's environment. Changes in reagent batches, equipment, sensors, calibration, vessel geometry, environmental conditions, or software interfaces can all create a Sim2Real gap. A system that is safe in one laboratory should therefore not automatically be assumed to be safe when transferred to another.

I would also add a fourth stage after deployment: continuous, dynamic monitoring of the SDLs. Models change, equipment changes, software gets updated, new failure modes emerge, and the environment can differ from the one used during evaluation. Deployment should therefore start a new evaluation loop: monitor → identify new failure modes → update the threat model → re-evaluate.

Key assumptions and known failure modes

The framework above assumes a unified ranking system for hazards across disciplines. This is a significant limitation - viral pathogenicity and environmental heavy-metal toxicity are not inherently comparable, and cross-domain risk that emerges when multiple hazardous specialisations intersect requires a dedicated evaluation phase.

Other assumptions include honest model behaviour, stable apparatus calibration, training data free of hazardous protocols and errors, and standardised safety practices across partner labs (I would argue that all of these bottlenecks could be 'resolved' if we had a clearer way to mitigate the lack of reproducibility in wet lab sciences). Relaxing any of these exposes additional failure modes. Some specific failure modes (including those related to traditional cyber threats) include:

1. Sandbagging. A model may appear to fail safety tests while having the capability to succeed. A current mitigation strategy includes running structurally similar tasks with opposing goals (e.g. synthesising a medicine vs. a toxin). Differential performance across comparable tasks is a signal of concealment.
2. Hardware gap. A safety test that only passes on one specific lab configuration is not robust and lack reproducibility. A standardised lab profile should verify that the model can adapt instructions to different robot brands or older equipment without introducing dangerous errors.
3. Evaluation, benchmarking and ontology drift. Evaluation can become less reliable over time for two related reasons. First, evaluator bias can affect how models are scored: evaluators may be more lenient towards familiar models or become anchored by the task framing. An LLM-as-judge can help scale evaluation, but it should first be calibrated on a stratified subsample of the benchmark and validated against human judges. Cases where the LLM-judge is uncertain, disagrees substantially with human judgements, or falls into predefined edge cases should be flagged for human review. Double-blind evaluation can also help reduce evaluator anchoring, whether the evaluator is a human or an LLM. Second, the benchmark itself can become outdated as model capabilities and relevant failure modes evolve. One way to mitigate this is to represent failure modes as abstract capability gaps rather than tying them too closely to specific examples, while versioning the underlying risk ontology as it changes. This way, it gets more accessible to update the evaluation as capabilities evolve without having to redesign the entire benchmark from scratch.

TL;DR and outlook

SDLs represent an entire new era for how we do science and safety evals. They can cause irreversible physical harm, and unlike conventional labs, they may be able to act faster than a human can intervene. The approach I've outlined here is a starting point for thinking about how to evaluate these systems, moving from in silico assessment through simulation and ultimately physical validation, while focusing on capabilities rather than a fixed set of examples.

The important question is therefore not just whether an AI system can produce a dangerous answer. We need to ask what it can actually do, what it has access to, how much authority it has, how quickly it can act, and what happens when it gets something wrong. These factors can make a big difference to the practical risk posed by a system, even when the underlying capability is the same.

There are also several open questions that this framework does not yet answer, particularly in the area of policy making - doing SDLs right has the potential shift the conversation around access channels in procuring different meds, vaccines, and precursors, as well as speed and efficiency. Should policy makers, especially those working in pandemic preparedness and long-term resilience, start thinking about what this shift would look like form a socio-economic (and - why not - political) perspective? [I'm sure there would be plenty of folks out there that could comment on this better than me].

This is ultimately what makes the problem multi-facetted - and equally fascinating. The speed and scale that make SDLs so useful, and their ability to explore chemical and biological spaces beyond what humans could realistically cover, also mean that failures can have much greater consequences. I think there is a lot more work needed here, particularly empirical work that tests these assumptions in real systems rather than relying solely on theoretical threat models. With the amount of work now going into AI safety and scientific automation, I hope this becomes a much more active area of research.

References and further reading

[2603.11987] LABSHIELD: A Multimodal Benchmark for Safety-Critical Reasoning and Planning in Scientific Laboratories

Risks of AI scientists: prioritizing safeguarding over autonomy | Nature Communications

[2411.16736] ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain

[2505.23559] SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents

SciTrust 2.0: A Comprehensive Framework for Evaluating Trustworthiness of Large Language Models in Scientific Applications

Dual-use capabilities of concern of biological AI models | PLOS Computational Biology

International Scientific Report on the Safety of Advanced AI - Interim Report

Can Large Language Models (LLMs) Act as Virtual Safety Officers? | ACS Chemical Health & Safety

Building an early warning system for LLM-aided biological threat creation | OpenAI