Detecting NSFW and Harmful Prompts Through Geometry: Learning What's Safe Instead of What's Harmful

⚠️Content warning: This post contains potentially offensive text and images from our research paper, included solely to illustrate the safety risks associated with vision-language models.


TL;DR:

i) Can we detect harmful prompts by learning only what safe prompts look like?

ii) In our ICLR 2026 paper, we introduce HyPE, a lightweight detector that uses hyperbolic geometry and learns just a single parameter from benign prompts. It shows strong detection performance, even against adversarial attacks.

iii) We also introduce HyPS, a prompt sanitizer that identifies and replaces harmful words, making prompts safer while preserving much of their original meaning.

Introduction

Vision-language models are becoming increasingly powerful, but preventing them from producing harmful content when prompted by malicious users remains a major challenge.

Most existing NSFW and harmful prompt detectors try to recognize these prompts using predefined rules, blacklist filters, or heavy classifier-based systems. However, attackers can often evade detection by rewording prompts, paraphrasing them, or adversarially optimizing them while preserving their harmful intent.

For simplicity, throughout this post, I will use the term "harmful prompts" to refer to both harmful and NSFW prompts.

In this work, we ask a simple question: Can geometry make AI safety detectors harder to fool?

More specifically, what if, instead of learning what harmful prompts look like, we focus on learning what safe prompts look like?

This is the main idea behind our paper, Harnessing Hyperbolic Geometry for Harmful Prompt Detection and Sanitization, published at ICLR 2026.

A Different Way to Detect Harmful Prompts: Geometry-Based Detection

We introduce HyPE (Hyperbolic Prompt Espial), a lightweight detection method based on hyperbolic geometry.

Unlike standard Euclidean geometry, hyperbolic geometry is particularly useful for representing hierarchical relationships in data. We use this property to represent text prompts in a structured space, where safe prompts tend to form a compact region.

We train HyPE to learn the boundary of this safe region using only benign prompts. Interesting, right? We only need safe examples to train the detector!

But there is another interesting detail: our detector only needs to learn a single parameter: the radius of the safe region. This radius defines the maximum distance from a fixed center in hyperbolic space within which a prompt is considered safe.

Once the training is complete, detection becomes a simple geometric operation. How? We measure the distance between a prompt's representation and the center of the safe region. If this distance exceeds the learned radius, the system flags the prompt as harmful. That's it!

Detection Is Not Enough: Rejecting prompt reduces model utility. The Solution? Sanitize!

When a prompt is flagged as harmful, blocking it is one possible solution. However, this also prevents users from getting any output from the model.

This motivated us to develop our second component, HyPS (Hyperbolic Prompt Sanitization). HyPS is an interpretable module that uses an Explainable AI (XAI) method called Layer Integrated Gradients to identify the words contributing to a prompt being classified as harmful.

Instead of rejecting the entire prompt, HyPS identifies the words that contribute most to its harmful classification and replaces them with safer alternatives. As a result, the prompt is transformed into a SAFE version that preserves its overall context and can still be processed by the model.

In practice, it looks like this:

Why do this? The goal is simple: remove harmful intent while preserving as much of the original meaning as possible and still let users to generate.

What Did We Find?

We evaluated HyPE across multiple datasets, including adversarially optimized prompts specifically designed to bypass safety filters.

The results? HyPE outperformed other detection methods we tested on almost all the datasets, suggesting that geometry-based detection is more than just an interesting theoretical idea: it can actually make a difference in practice.

Regarding HyPS, it successfully sanitized the majority of harmful prompts, while preserving much of their original meaning. We also tested HyPS on text-to-image generation and image retrieval tasks, showing that sanitizing prompts can significantly reduce unsafe outputs without losing the original context.

Conclusion

Our results show that harmful prompt detectors do not need to learn any form of harmful behavior. Learning a representation of safe behavior, and detecting deviations from it is proved to be a very useful alternative.

With HyPE, we demonstrate that learning just a single parameter in hyperbolic space can lead to robust harmful prompt detection, even against adversarially optimized prompts. Combined with HyPS, our framework can go beyond detection by sanitizing harmful prompts while preserving much of their original meaning.

Another interesting advantage of HyPE is its potential to serve as a lightweight, model-agnostic safety filter that detects harmful prompts before they reach the underlying model. Although our experiments focused on specific vision-language pipelines, HyPE can be integrated in front of any LLM or VLM that accepts text prompts, without requiring modifications to the underlying model!

Enjoyed the post? Check out the paper, code, and HF repository to learn more and see how it works in practice.