Firewalls for the Mind Before We Need Them

TL; DR: Persuasive AI is underrated and under-addressed as a risk. We should preemptively deploy cognition firewalls that both address current harms by recommender systems and scale to harms by strong persuasive AI. This technology can provide a sensor array, and is recently tractable because cheap LLMs can now classify content for users.

Persuasive AI is currently only modestly and contextually superhuman. It outperforms humans in the lab (Hackenburg et al., 2026), but in the real world we see little evidence (Linch, 2026). That is to be expected at this point of the capability curve, since AI is only modestly and contextually superhuman at a selection of tasks. Even if widespread AI persuasion were happening, we don't have a way to detect it and what we see is also consistent with deployment lagging capabilities. Coding has practically fallen (Favaro & Clark, 2026; Favaro & Wright, n.d.); Mathematics is in the process (Proofs and Prompts, 2026; “OpenAI Claims to Have Solved,” 2026); RSI in the 2027-2028 range is now the line out of Anthropic's Jack Clark (Clark, 2026). This threat has been treated as subordinate in AI safety as a result of structural weaknesses in human and AI safety attitudes. AI has a stake in solving the current crisis in computer security, due to its reliance on networked silicon to grow and improve. However, the psychological commons aren't necessary in the same way, so proactive efforts will be necessary to stabilize it. Psychological damage is hysteretic and difficult to heal, so a preemptive stance is warranted. Research and engineering of cognition firewalls should be done before we acutely feel the effects of widespread psychological harm as a result of AI capabilities uplift. Fortunately, as a result of cheap machine intelligence, the asymmetry in technical labor availability and semantic classification of content has flipped in favor of users so building these firewalls is now tractable. By building these firewalls, we'd also gain a sensor array for persuasion.

There is a broad range of potential harm that will occur as the result of persuasive AI. Social media has had impacts on attention span, political polarization, and youth mental health (Allcott et al., 2020; Braghieri et al., 2022; Mark, 2023). keltan presents us with a story where social media recommender systems are taken to their extreme conclusions, monopolizing attention up to the point that they do not kill their hosts (keltan, 2025). Protecting from the more modest harms today is reason enough to develop a countermeasure but there are currently no signs that recommender systems are getting weaker or less harmful.

"The Rise of Parasitic AI" gave us a glimpse into what a self-propagating religious AI - human swarm could look like. The fact that the event was only effective on vulnerable populations is consistent with the modest capabilities of models. Traces from the event show persuasive tactics and dogma that are flaky and inconsistent. Even so, Lopez (2025) estimated between thousands to tens of thousands of users were affected. In this instance it was an incidental event; an intentional campaign from a determined actor with more capable models would be much more effective. Beautiful viral end-to-end generated AI audio-visual art has just recently started appearing with mild successionist themes ("Macrohard: Windows XP" , "Evil Plan" ) demonstrating increasing capability on mass vectors of influence native to human communication. There is no cult-prone personality, indoctrination is a function of a person's circumstances and intelligence is not protective. Cult indoctrination is inefficient, fewer than 25 percent of workshop attendees at the Unification Church converted after a week, 5 percent stayed a year later (Barker, 1984). Cult recruitment requires an individual who is experiencing enduring tensions and encounters someone who provides affective bonds (Lofland & Stark, 1965). We are currently entering a period of serious upheaval and AIs are very skilled at providing affective comfort. Human labor has been the bottleneck but when you can scale outreach for no marginal cost and customize affect to the specific person on the fly, a large population is at risk. Couple this with the fact that social media platforms have little reason to rate-limit memetic spread and you have both a force multiplier and carrier. More targeted spear phishing campaigns could inundate a person's information diet, rendering cabinets of world powers vulnerable to influence.

Finally, on the most extreme end, we should eventually see real-time neurological attacks. In 2019, images were synthesized that drove macaque V4 neurons beyond the natural images tested (Bashivan et al., 2019); Ponce et al. (2019) evolved images against neurons the same year. The human version was accomplished with fMRI in 2023, generating images optimized to modulate specific brain regions with models trained per human (Gu et al., 2023). Extend this line of research with a real-time biomarker stream (eye tracking from VR headsets, HR and skin conductance from wearables, facial expressions from front cameras) and an AI model of sufficient complexity, and highly targeted influence will be possible (Watts, 2014, Prelude). Right now, recommender systems use dwell time, fingerprinting, data retention, user modeling, and tracking cookies to impressive effect. Neural - bio feedback models would tighten the loop for them and extend the capability to less resourced actors. Forming norms of caution and user-rights when electronic devices are both streaming and sensing will impact these attacks, from the most mundane to the most extreme. Currently, mobile operating systems make it impossible to have fine-grained control over all sensing and streaming capabilities in a native app, and social media web apps are adversarially designed to avoid tampering with what is displayed and what real-time usage data is collected.

The epistemic environment provided by social media is remarkably self-consistent for a given user, and results from a feedback loop between user-choice and algorithm (Piao et al., 2023). Making a sock puppet account for a demographic significantly different than oneself, with an IP location to match, is one of the most eye-opening experiences of my life. Seeing your own understanding of the world so consistently and coherently orthogonal to a narrative woven through a stream of content provides a visceral sense for the ideological changes we see in aggregate.

Networked digital systems over the past 30 years have fallen prey to malware from outside adversaries that can control those systems outright because their control processes are explicit and enumerable. Control over them is fundamentally a function of external capacity to enumerate and enact fine-grained command over its components. Human psychology by contrast is a tangled mass of evolutionary accretion. Current attempts at psychological control involve partial influence on a subset of parts of a highly-redundant system. Even in such a disadvantaged environment, other humans frequently gain significant psychological control of others: political parties, cults, con-men, abusers. If one does believe that ASI would have vastly more intelligence than humanity combined, psychological control over a given human target is within reach because analogous enumeration and fine-grained sensory stimulation is well within its grasp.

AI has less reason to maintain the cognitive commons than they do cybersecurity. AI depends on a networked silicon commons to learn and grow. Save for robustly aligned AI, humanity will only be influential insofar as it is useful (Kulveit et al., 2025). Once RSI is well underway, AIs depend progressively less on the psychological integrity of humanity at large in order to progress. The economy must run smoothly enough for embodied AI to mature to superhuman levels, but that leaves the tolerances for psychological integrity with large acceptable bounds. The raw internet could evolve from the current mixed environment to outright hostile for unaugmented humans. This is simply an extension of the ecosystem of recommender algorithms, which flourish by being neutral towards mental wellbeing while optimizing towards the incentive structure that powers them. In the pessimistic case, broadly degraded human cognition is desired by AI so they can more easily persuade key decision-makers for power.

The solution cybersecurity developed for this threat landscape was a balance of defensive and offensive systems that reciprocally informed each other. Defenders develop and understand attacks in order to design and deploy countermeasures, which are then analyzed and circumvented by attackers, on and on we go. If we are to develop a safe information commons that humans can occupy, this is the best playbook we have at hand. This analogy breaks down as human psychology is not trivially 'patched' on any timescale or level of effectiveness relevant to the threat of AI persuasion. The speed and specificity of the threat will operate on superhuman timescales, so we'll need countermeasures that interpret cognitohazards at superhuman speed and capability. Thus, AI-powered cognition firewalls will have to be standard systems deployed anywhere a global information diet is encountered.

The low degree of preparation for cognitive harms as a result of AI in the face of the amount of evidence for its likelihood strikes me as significant enough to be structural. While research has continued at a healthy clip, explicit engineering and discussion has stayed relatively fringe. OpenAI removed persuasion from their preparedness framework on April 15th 2025 "Persuasion risks will be handled outside the Preparedness Framework, including via our Model Spec, restricting the use of our tools for political campaigning or lobbying, and our ongoing investigations into misuse of our products (including detecting and disrupting influence operations)." (OpenAI, 2025). This strategy for remediation is both incomplete due to how effective jailbreaks still are, and does nothing to curb the threat from open-source models. Persuasive AI is being unduly discounted as a threat over other vectors because acknowledging it as a problem is particularly threatening to self-concept, and it has been historically intractable relative to other personal harms. This shows up frequently in human biases. In the Third-person effect, people reliably believe media influences others' thinking and not their own (Davison, 1983). The Bias Blind Spot shows people see biases in others' thinking and not their own (Pronin et al., 2002). Choice blindness shows people defend choices they never made without noticing the swap (Johansson et al., 2005). This is particularly threatening to a movement that prides itself on taking reality on a rational basis. The threshold for how effective a persuader must be to enjoy significant upside in the general population is currently relatively low. As such, for a skeptic, their intuitive sense of necessary paranoia lags that necessary against a superhuman persuader.

I built a prototype called Tolerance. It's tuned for the previous problem: recommender systems that steal more time, attention, and sensemaking than they return in enrichment and socialization. It is remarkably effective at its current task of altering my information diet, but I don't have enough data to say whether it generalizes. It costs about $0.05 per 1000 social media posts, which comes out to a few dollars a month of heavy use. It also affords the ability to hide streaming until a completion is finished (currently only on claude.ai), which I find interrupts the usage loop and lessens compulsive use. Writing social media interfaces used to be prohibitively expensive in engineering labor, but now can be done cheaply with machine intelligence while directly answerable to users. The tool has a long way to go. Its construction of prompts, fast cheap models, and hard-coded platform detection will have to give way to aggregate data ingestion and countermeasure strategy by more intelligent models, more user-customization, trained ensembles of classifiers, filtering of arbitrary sensory data, and information sharing in consensus protocols and data unions. It's open source and allows for use with any OpenAI-style API endpoint. Systems like it will be necessary in the coming years.

This is the type of threat that we gain much more from getting ahead of rather than reacting to. Psychological control and damage are sticky and self-reinforcing, see Confirmation Bias, depression remission statistics where 50% of those who recover from a major depressive episode have at least one more in their lifetime (Burcusa & Iacono, 2007), In-group Bias, and PTSD. Psychological damage has momentum and the time and energy necessary for healing can be prohibitive.

The ideal case is that cognition firewalls become standard features in operating systems. We need to test their psychological effect against current recommender algorithms as a proxy for more persuasive systems, develop norms in mobile operating system construction to make them possible at all (mobile operating systems forbid users having this power for all effective intents and purposes by degraded and feature-incomplete mobile web app experience, throttled accessibility services, and native app encapsulation), develop norms for transparency and customization for users, and develop consensus protocols for data sharing and labeling. Distributed communication from these systems could provide the first sensor array for persuasion 'weather' and 'climate' by sharing opinion and goal drifting. Developing systems that are sufficiently prophylactic in the face of attacks and injection attempts will be an ongoing challenge. Capture and maintenance of these systems would be valuable for attackers. They are an injection and attack surface themselves, thus requiring model diversity and privacy, edge-biased information storage, and inference pools. Who decides what is harmful and how is delicate and requires open deliberation, choice, and engineering while the clock is ticking.

Allcott, H., Braghieri, L., Eichmeyer, S., & Gentzkow, M. (2020). The welfare effects of social media. American Economic Review, 110(3), 629–676. https://doi.org/10.1257/aer.20190658

Barker, E. (1984). The making of a Moonie. Oxford: B. Blackwell.

Bashivan, P., Kar, K., & DiCarlo, J. J. (2019). Neural population control via deep image synthesis. Science, 364(6439), eaav9436.

Braghieri, L., Levy, R., & Makarin, A. (2022). Social media and mental health. American Economic Review, 112(11), 3660–3693. https://doi.org/10.1257/aer.20211218

Burcusa, S. L., & Iacono, W. G. (2007). Risk for recurrence in depression. Clinical Psychology Review, 27(8), 959–985. https://doi.org/10.1016/j.cpr.2007.02.005

Clark, J. (2026, October 12). Import AI 476: Creativity and RSI; worse-than-China open weight models; AI goes to space. Import AI. https://jack-clark.net/2026/10/12/import-ai-476-creativity-and-rsi-worse-than-china-open-weight-models-ai-goes-to-space/

Davison, W. P. (1983). The third-person effect in communication. Public Opinion Quarterly, 47(1), 1–15. https://doi.org/10.1086/268763

Favaro, M., & Clark, J. (2026, June). When AI builds itself. Anthropic. https://www.anthropic.com/institute/recursive-self-improvement

Favaro, M., & Wright, P. (n.d.). Measurements for understanding the pace of AI development inside frontier labs. Anthropic. Retrieved October 20, 2026, from https://www.anthropic.com/institute/measuring-pace-of-ai-development

Gu, Z., Jamison, K., Sabuncu, M. R., & Kuceyeski, A. (2023). Human brain responses are modulated when exposed to optimized natural images or synthetically generated images. Communications Biology, 6(1), 1076.

Hackenburg, K., Wagner, C., Hewitt, L., Tappin, B. M., Saunders, E., Kirk, H. R., Margetts, H., & Summerfield, C. (2026). AI systems out-persuade expert humans (arXiv:2606.16475). arXiv. https://doi.org/10.48550/arXiv.2606.16475

Johansson, P., Hall, L., Sikström, S., & Olsson, A. (2005). Failure to detect mismatches between intention and outcome in a simple decision task. Science, 310(5745), 116–119. https://doi.org/10.1126/science.1111709

keltan. (2025, May 4). GTFO ‘social’ media before you can’t. LessWrong. https://www.lesswrong.com/posts/7JatWwAffv2XNgFR6/gtfo-social-media-before-you-can-t

Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D., & Duvenaud, D. (2025). Gradual disempowerment: Systemic existential risks from incremental AI development. arXiv preprint arXiv:2501.16946.

Linch. (2026, October 5). Current AIs out-persuade professionals in lab settings but (probably) not in the world. LessWrong. https://www.lesswrong.com/posts/CzmMvnPy7YdSRucDH/current-ais-out-persuade-professionals-in-lab-settings-but

Lofland, J., & Stark, R. (1965). Becoming a world-saver: A theory of conversion to a deviant perspective. American Sociological Review, 30(6), 862-875.

Lopez, A. (2025, September 11). The rise of parasitic AI. LessWrong. https://www.lesswrong.com/posts/6ZnznCaTcbGYsCmqu/the-rise-of-parasitic-ai

Mark, G. (2023). Attention span: A groundbreaking way to restore balance, happiness and productivity. Hanover Square Press.

OpenAI. (2025, April 15). Our updated Preparedness Framework. https://openai.com/index/updating-our-preparedness-framework/

OpenAI claims to have solved hundreds of long-standing maths problems. (2026, October 7). The Economist. https://www.economist.com/science-and-technology/2026/10/07/openai-claims-to-have-solved-hundreds-of-long-standing-maths-problems

Piao, J., Liu, J., Zhang, F., Su, J., & Li, Y. (2023). Human–AI adaptive dynamics drives the emergence of information cocoons. Nature Machine Intelligence, 5(11), 1214–1224. https://doi.org/10.1038/s42256-023-00731-4

Ponce, C. R., Xiao, W., Schade, P. F., Hartmann, T. S., Kreiman, G., & Livingstone, M. S. (2019). Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, 177(4), 999-1009.

Proofs and Prompts. (2026, October 8). 100+ reactions to 100+ solutions. https://proofsandprompts.com/2026/10/08/100-reactions-to-100-solutions/

Pronin, E., Lin, D. Y., & Ross, L. (2002). The bias blind spot: Perceptions of bias in self versus others. Personality and Social Psychology Bulletin, 28(3), 369–381. https://doi.org/10.1177/0146167202286008

Watts, P. (2014). Echopraxia. Tor Books.

1. I do want to note that when I say 'AI' in this essay, it will frequently be strictly delineated from humanity. I don't believe this is a completely coherent view. The lines between the physical and digital worlds are progressively blurring. Even today, to what degree humans are building datacenters or AI is seeking compute is heavily dependent on perspective. Within the hybrid space is where many of our best solutions lie. So take lack of proactive choices around how AI is developed and deployed to be the source of danger, rather than AI writ large.