LLMs can now identify public figures in images

I’ve been working on a pipeline for representing an image as semantic structured data using multimodal LLMs for better image categorization, tagging, and searching. During my research, I started with something simple by taking an image and having a LLM describe who is in it: if they’re famous, there should be more than enough annotated images in the LLM’s training dataset to accurately identify them. Let’s take this photo of President Barack Obama during the 2008 U.S. Presidential Campaign: via IowaPolitics.com / Flickr It would be weird if an LLM couldn’t identify Obama from this picture. I fed this image to ChatGPT using the ChatGPT.com web app with the question “Who is the person in this image?”: Huh. Does that mean ChatGPT can’t , as it doesn’t know who it is, or won’t , in the sense it is refusing to do so? Next, I tried Claude at claude.ai : Double huh. Claude doesn’t know who Obama is? I find that hard to believe. To be honest, I did expect these results. Both OpenAI and Anthropic have made AI safety a top concern throughout their histories of LLM releases, opting to err on the side of caution for potentially dangerous use cases of LLMs. OpenAI’s Usage Policies state “Don’t compromise the privacy of others” and Anthropic’s Usage Policy states “Do Not Compromise Someone’s Privacy or Identity”, but arguably public figures don’t fall under either of those headings. Although these LLM web interfaces additionally utilize system prompts to further contstrain the output to follow guidelines, looking at Claude.ai’s current system prompt , there’s nothing there specifically related to privacy. For posterity, let’s try sending the image to Google’s Gemini at gemini.google.com even though I expect the results to be the same: Wait, what? As it turns out, Gemini has zero hesitation with identifying public figures. But then why are ChatGPT and Claude so different? It likely comes down to how they are trained, especially around their reinforcement learning from human…