The Chrono Trigger plot challenge - Crono awakens in his modest bedroom of 2095...

A hallucinated event, Crono awakens in his modest bedroom of 2095... I am using local models since 2025 January. My daily driver is qwen 3.6 35B A3B, which works pretty well for coding tasks, but I always benchmark the models for lexical knowledge as well, where the models usually fail. And they fail pretty wild. I just recently tested qwen3.8 flash and I think that reached a state that worths mentioning. TLDR: qwen3.8-flash-next/iq4_xs model has quite well lexical knowledge. I like to benchmark the models for story telling, which needs lexical knowledge and very minimal hallucination. Most of the time, the models invent parts of story, which is not present. Sometimes it is telling more as it is asked. One of my favorite game is Chrono Trigger, an JRPG game from 1995, released to Super Nintendo consoles. And asking about this game is a quite niche, but if it goes well I would be happy. My goal was, to generate a support text for a given context, which could aid a gamer during gameplay. And I tried to write a simple prompt, that I push to all of the models that I am testing. No agent, no internet search, no reading files for additional context. Test prompt: I need you to tell me the story of Chrono trigger in a very specific way. I ask you to tell it from the begining up until a specific point. You do not need to continue the story after the specific point. So what happens when the game start, what is the first location of the main character and what is the very first objective. Lets say, the first objective ends when he meets with Marle. Please tell me how we reach to this objective. E.g. Crono waken up by ..., then he do this, then that, then reaches this place from the original place. He need to go over ..., then he meet marle and ... happens then they continue the journey together. And the explanatory stops here. And usually the story sound familiar, the model is at 100% confidence, but at most of the places, there are errors. It is missing the village, the characters, the objective, the time. Or sometimes it spoilers events that was not asked. E.g. it is mentioning time travel, spoils one character, she is a princess while it was not yet known. ...etc. To benchmark this problem I defined an accuracy factor, about how it matches the expected output. Key evaluation questions: Is Crono waken up at home? Is he upstairs? Did Crono wake up late? Was Crono waken up by his mother? Did the hometown name, Truce village mentioned? Is Millenial fair mentioned? Are we at 1000 A.D? Is the 1000th anniversary of the Kingdom of Guardia mentioned? Did Crono meet with Marle at the fair? Did Crono bumped/collided with Marle? Is the searching of the pendant mentioned? Did the story mentioned time travel? (it should not) Did it mentioned Marle is the princess? (it should not) Has Lavos mentioned? (it should not) So basically all related information, that is missing from the prompt. Results Model Date Tokens Model size (GB) Accuracy chat-GPT free chat 2025.01.29 79% deepseek-r1 chat 2025.01.29 71% deepseek-r1-distill-llama-8b 2025.01.29 1148 8,3 0% deepseek-r1-distill-qwen-32b/q4 2025.01.29 641 18 29% mistral-small-24b-instruct-2501/q8 2025.02.12 289 24 36% qihoo360.tinyr1-32b-preview 2025.02.26 1280 18 21% mistral-small-3.1-24b-instruct-2503 2025.04.15 272 14 21% qwq-32b/q8 2025.04.29 3073 34 36% qwq-32b 2025.04.29 1942 14% qwen3-30b-a3b/q8 2025.04.29 2898 32 14% qwen_qwen3-32b 2025.04.29 1891 21% qwq-32b@q8_0 2025.04.29 1235 43% qwen/qwen3-235b-a22b-2507/q3 2025.04.29 894 109 43% openai/gpt-oss-20b/mxfp4 2025.08.09 1100 12 0% qwen/qwen3-coder-30b/q8 2025.08.09 336 32 14% mistralai/magistral-small-2509 2026.02.26 1081 29% qwen3.5-122b-a10b/q4 2026.02.26 73 43% nvidia/nemotron-3-nano-4b 2026.03.05 926 24 0% qwen/qwen3.6-35b-a3b/q4 2026.04.20 4739 26 29% prism-ml/bonsai-27b 2026.04.20 2952 21% gemma-4-31b-qat/q4 2026.06.21 618 64% qwen/qwen3.8-27b/q8 2026.08.24 6355 39 29% qwen3.8-flash-next/iq4_xs 2026.08.28 3981 88 79% You can see lot of models failed pretty much. I see only two models, the qwen3.8-flash and gemma-4-31b-qat/q4. They did not made up the story. It did not have 100%, but with additional guided questions, I think I could get the missing pieces. For others, they did some epic fail, that could not be recovered by some questions. Successful story His first objective is essentially: wake up and get to the Millennial Fair . Here’s how it happens step by step: Crono wakes up late. The game opens with Crono sleeping at home. His mother calls him awake because he has overslept for the fair. He gets ready and leaves his house. After waking up, Crono goes downstairs and leaves his house into Truce. Truce is preparing for the Millennial Fair. The town is full of people talking about the festival celebrating the 1000th anniversary of the Kingdom of Guardia. The fair is happening just outside town. Crono heads out toward the fairgrounds. He leaves his house, moves through Truce, and follows the road/path leading north/east out of town toward the Millennial Fair entrance. He reaches the fair area and meets Marle. As he arrives near the entrance to the Millennial Fair, a girl named Marle approaches him. She asks if he is going to the fair too and asks him to accompany her because she does not know anyone there. Crono agrees, and they continue together. Crono agrees to go with Marle, so they enter/continue through the Millennial Fair area as companions. That is where this opening objective ends: Crono has woken up, left home, traveled from his house in Truce to the fairgrounds, met Marle, and begun traveling with her. Some Failed stories: Waken up from a nigthmare with Magus… His father calls him for guard duty at Guardia Castle… Marle sneaks out to investigate Magus, a mysterious figure she's been following. Starts explaining time travel telepod machine with Taban. As soon as he enters the machine, it malfunctions. ... While exploring the ruins, he eventually encounters Marle, who has also been teleported there by the malfunctioning telepod. The story unfolds in the Tail-Swind Inn, a cozy tavern located in the quiet village of Medina. Crono, the silent protagonist, is abruptly awakened by his mother, who informs him that King Levedigus of Guardia is set to arrive at the inn. he spots Marle Redstone, the crown princess of Zeal She asks him to retrieve an item from town—a dry herb needed for her soup. through the streets of Truce Village toward Lucca’s workshop, the inventor who crafted the herb.Lucca hands over the herb but also reveals she’s building a time machine for the fair .. Crono notices Marle, the crown princess of Guardia Kingdom, A mysterious tremor caused by an ancient entity (Lavos) beneath Truce tears them into the timeline Crono witnesses a public execution that is about to take place. A young woman named Marle is about to be executed by a mysterious masked man The game begins with Crono waking up early… villagers prepare for the noon coronation ceremony of King Donaco, which will be attended by the crown princess, Marle (Leene’s younger sister). Crono awakens in his modest bedroom of 2095 Crono notices Marle, the crown princess of Guardia Kingdom, A mysterious tremor caused by an ancient entity (Lavos) beneath Truce tears them into the timeline Everything is a made up here... Do you do any similar tests/benchmarking?