When the Object Is Not on the List: Hierarchical Open-World Perception

Watch the video:

A conventional object detector knows a list. It has been trained on a fixed set of classes, car, pedestrian, cyclist, traffic sign, and when it looks at the world it answers with one of those names and a confidence number. That works right up until the world hands it something that is not on the list. A piece of debris in the lane. An overturned load. A vehicle shape it has never seen. A flat detector has no honest way to say “I do not have a name for this.” It has only the names it was given, so it does one of two things. It forces the object into the nearest known label with high confidence, or it fails to fire at all and the object simply does not exist as far as the system is concerned.

Both of those outcomes are dangerous, and they are dangerous in the same way. A system that has to act on what it sees needs the perception layer to be honest about the edge of its own knowledge. A confident wrong label is worse than a vague one, because everything downstream trusts it. If the detector says “traffic sign” with high confidence about an object lying across the road, the planner has been handed a lie, and it will plan around a sign that is not there while ignoring a hazard that is. The silent miss is the mirror image of the same failure. The object was real, the sensor saw it, and the flat class list threw it away because none of the fixed names fit.

Placing the unknown instead of forcing it

HOWC, my open-world hierarchical detector, is built around the idea that recognition should not be a flat lookup into a fixed list. It should be a placement into a hierarchy. Classes are not a flat set of leaves, they are a tree, from general at the root down to specific at the leaves. A familiar object gets recognized all the way down to a specific leaf, “sedan,” “delivery van.” An unfamiliar object does not get forced onto the nearest leaf. It gets placed as far down the tree as the evidence actually supports and no further. So an object that is clearly a vehicle but matches no known vehicle type is reported as “some kind of vehicle,” not as a confident wrong model. Something that does not resolve even that far is still reported as “an object occupying this space,” which is the most general placement there is and still a useful one, because it tells the rest of the system that something is there and it has extent.

The point of the hierarchy is that perception degrades gracefully instead of failing silently. A flat detector has one place to put its uncertainty, the confidence number, and confidence alone cannot tell the planner whether a low score means “probably nothing” or “definitely something, no idea what.” A hierarchical placement carries that distinction in its structure. A high-confidence general answer and a low-confidence specific answer are different statements, and the system can act on the difference. It can slow down for “an object occupying this space” without needing to first commit to what that object is called.

Where this sits in the larger system

HOWC is the perception layer that sits on top of the temporal LiDAR world in the larger digital-twin system I have been building. The geometry layer reconstructs the scene from measurement and knows its own uncertainty. HOWC is what puts names, and honest partial names, onto that geometry, so the twin is not just a shape of the world but a labeled understanding of it. The goal for the whole system is a semantic digital twin that runs alongside reality and anticipates hazards, and a hazard is very often exactly the thing that was not in the training set. A perception layer that quietly discards the unfamiliar is the wrong foundation for that. One that places the unfamiliar sensibly, at whatever level it can defend, is the right one.

Honest status

To be clear about where this sits. The open-world perception is live and public, and the models are out there to run:

• HuggingFace Model: https://huggingface.co/freshNfunky/howc
• GitHub Repository: https://github.com/freshNfunky/IE2025-Research-Paper

This is research, not a finished product, and the video walks through the behavior rather than making claims about a benchmark. If you want the technical writeup, it is here: https://arxiv.org/abs/2608.07577 . The video shows what hierarchical recognition looks like on real scenes, including the cases where the honest answer is a general one: https://www.youtube.com/watch?v=Q00MV_YagaA

When the Object Is Not on the List: Hierarchical Open-World Perception was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.