A dataset with 52 Text to image model evaluation [P]

I created a simple text to image benchmark. I curated 192 prompts that are difficult for T2I models in various ways: text rendering, spatial reasoning, human realism, negations, etc... I then asked a VLM to judge every output against a pre-specified binary question with the ground truth baked in. I'm publishing all the results including the images. (Most public T2I leaderboards don't publish the actual images and that's a shame IMO) There is currently 52 model tested! more than 9k images have been generated and analysed! Full methodology: imagebench.ai/methodology-v1 Hugging face dataset: huggingface.co/datasets/dh7/imagebench (it contains the prompts to reproduce the results AND the results) Github: github.com/dh7/image-bench-ai Gallery to inspect the results: imagebench.ai/gallery Leaderboard: imagebench.ai/imagebench-v1 Limitations: it's text to image only, and VLM are not perfect as a judge. Let me know what could be useful from there!