Higher Quality Small Synthetic Natural Language Text Generation for Interpretability Research

Introduction

Small simple synthetic natural language datasets suitable for end-to-end training of tiny LLMs serve as an important resource for LLM interpretability researchers. Some well know examples include roneneldan/TinyStories, SimpleStories/SimpleStories, and klusai/ds-tf1-en-3m (TinyFabulist). This post solves key problems that degrade the quality of these datasets, while also offering an efficient accessible pipeline that can be run locally on an NVIDIA 5060 Ti (16GB) graphics card. The core problems this post solves, include:

• True Small Vocabulary. The aforementioned datasets attempt to produce a corpus with a small vocabulary, but arguably fall a bit short of that goal. E.g., TinyStories has 49,187 unique words, SimpleStories has 40,567, and TinyFabulist has 41,502.
• Guaranteed minimal word frequencies. In the aforementioned datasets, 15 to 24 percent of the unique words occur less than 2 times, while between around 43 to 54 percent occur less than 8 times. This means that most of the unique words are likely not learnable, and mostly contribute to noise and vocabulary bloat.
• Error free text. The aforementioned datasets, include lots of errors, such as misspelled and mangled words.
• Reliable Name Disambiguation and Stratification. The aforementioned datasets, have various name management issues, ranging from collision with existing words (e.g., May vs may), name bloat, and no control over gender balance, or bias (e.g., certain names may be more likely to co-occur with potentially negative moral concepts).

Methods

This section summarizes the methods. The source code is available here, and the proof of concept dataset Small World 345.6k.

We use using unsloth/gemma-4-26B-A4B-it-qat-GGUF for generation, through a local llama-cpp server, which gets about 250 tokens/second on a NVIDIA 5060 Ti (16GB). The server config is available here.

Small Capped vocabulary

One difficulty we encounter is that the model will tend to keep introducing new relatively rare words, causing the vocabulary to grow large over time, yet most words never end up occurring with enough frequency for their meaning to be learnable.

We address this problem through a feature where, after a certain number of examples have been generated, we begin redoing generations up to a number of times when they come back with new words. Since a new form of an existing word has different implications regarding sufficient statistics, we separately issue retries on new forms of an existing word at a different later point depending on the project configuration. In simpler-world-345.6k, we begin capping new words after 47,232 examples, and new forms after 115,200 examples. The vocabulary at that stage had grown to 8,972 words, and by the end became 8,973 in total (one stubborn word resisted 10+ retries at some point). From 47,232 onward, between rejection from other errors (see following subsections) and from rejecting new words, we had to redo somewhere around 0 to 8 completions per batch of 144, which only marginally reduced overall throughput.

Sufficient Word Statistics

Word distributions in text follow Zipf's law, which in simple terms, means that more common words have a vastly higher frequency than somewhat more rare words. When generating a relatively small text dataset, by default many or most of the words that show up will have a very low frequency. Words that occur only a few times in the whole corpus effectively become noise, and if encountered during interpretability experiments, might become cause for confusion.

We find that in order for each word, even in a relatively small capped vocabulary, to naturally occur a sufficient number of times, the number of examples would need to be extremely large. Thus, with some frequency, we randomly sample a word to integrate into a prompt, with inverse-squared frequency weighting.

We compare against a few popular datasets meant to have a simple vocabulary, including TinyStories, SimpleStories, and TinyFables. The word statistics are calculated from the train splits after converted to lowercase. With this, TinyStories has 49,187 unique words, SimpleStories has 40,567, TinyFables has 41,502, and Small World 345.6k has 8,873.

The plot below shows the percentage of the corpus words which have a frequency greater than or equal to N, for N up to 1000.

For the other three datasets, between around 15 to 24 percent of the unique words occure less than 2 times, while between around 43 to 54 percent occure less than 8 times. Through capping and boosting, 100 percent of the unique words in Small World 345.6k occure at least 16 times.

Note that Small World 345.6k is a smaller dataset, with only a total word count of 153,818,092, while Simple Stories has 477,720,608, Tiny Stories has 376,776,314, and ds-tf1-en-3m has 766,391,198. We expect that the minimum word frequency in Small World will grow linearly as the dataset size increases due to the vocabulary being capped, and rare words being boosted at a fixed rate. We plan to extend the dataset in time to about 10 times the size, which would ensure that all words appear more than 160 times.

Clean Error-Free Text and Named Entity Disambiguation

Each completion is normalized and validated so that it only contains characters in [^A-Za-z0-9\s.,!?;:\'"\-]; if, after normalization, invalid characters are detected, the completion is rejected and tried again with a new random seed. We also reject a completion if any of the non-name words are not recognized by the pyenchant US or British dictionaries with the Nuspell backend. Conversely, names converted to all lowercase must not be recognized by pyenchant. Optionally, the valid names can be restricted to the list of suggestions, simplifying name disambiguation. Otherwise, a set of rules is checked to avoid confusing names and non-name words. Names are then tracked capitalized, while non-names, including words at the start of a sentence, are tracked lowercase.

Name Choice

We notice a tendency for the gemma-4-26b-a4b model we used to misspell some names, and ultimately found that this tendency stems predominantly from tokenization fragmentation. For example, Elara may get tokenized as El-ara. When the model first predicts El, it has the chance to then predict the wrong follow-up tokens, and ends up occasionally generating, e.g., just El, Els, Elas, or any number of variations of errors. The Gemma tokenizer also joins words with leading spaces, and uses different tokens for the word with or without the leading space. E.g., Elara and _Elara will be tokenized differently.

We guard against such errors by requiring non-suggested names to occur at least twice in the completion. Since the model tends to not produce the same error twice, this greatly reduces the chance. Or we can simply reject any name that isn't in the suggestion list.

But each error requires a redo, so the costs can add up.

Since we use gemma-4-26b-a4b for generation, we identified every name in the Gemma 4 tokenizer's vocabulary that gets consistently tokenized as a single token, regardless of whether it is the version with a leading space or the pluralized version.

Ultimately, this narrowed down the list of names to a very small list, which is included in the dataset on Hugging Face here.

We additionally identified which names are not prefixes to other words; for example, Liv is a consistent single token name, but it can also be a prefix to Live.

Since in post-processing we can replace all of the names however we like, this may be used purely to increase the efficiency of the generative process. But also, it would be useful for interpretability to have unambiguous single-token names. We may also tentatively consider cross-tokenizer use cases, such as teacher-student training, vocabulary trimming, embedding reuse, or cross-model interpretability. Tentative plans along these lines are planned for small-worlds, but would come at the cost of being able to optimize the small-world tokenizer based on other criteria, such as word-level and morphologically decomposed tokenization.

Name Stratification over Gender and Moral Integrations

We find the model tends to gravitate towards using some names much more than others. Randomizing the list of names in the prompt at each generation helps, but very little. We also noted that by default, a large imbalance in gender tends to occur. In addition, the model may be biased in how it associates names with moral concepts or situations, and we want to mitigate that.

This is addressed first by adding an equal number of female and male names as suggestions. Then, when assembling the list of names into the system prompt, we do inverse frequency sampling to only add a subset of the names consisting usually of the least represented. We choose the subset size to be at least 1 more than n/2, so that there is always at least one male and one female suggestion. This will tend to balance out the initial distribution.

Next, we do a post-processing step where we optionally replace the names with a new set of names (for example, if you wanted to diversify or expand the set of names). The new or same names are then redistributed so they are closer to equally represented, and with roughly equal co-occurrence with each moral integration. For example, so that prompts asking to generate a completion about a sensitive concept are not associated in the corpus with any particular name or gender.

Discussion and Conclusion

The paricular application for Small World generate minimalistic synthetic training data, that is simple enough for a small LLM to learn efficiently, but complete and complex enough to support concept generalization, small-world modelling, moral reasoning, and agentic decision-making. This would hopefully enable more efficient research into a range of important AI safety problems. However, this effort is omitted from this post since this the value of this direction is yet unrealized and out-of-scope.

These methods are relatively simple, but effective, and significantly improve on the state-of-the-art, and should enable higher quality future iterations of existing dataset families like TinyStories or SimpleStories.