Gradient-free Single-pass Model Beats nanoGPT on Shakespeare

Beam is a character-level language model that computes count tables mapping character contexts to next-character frequencies. At prediction time, each order looks up the current context in its count table and produces a distribution over the vocabulary, smoothed over a symmetric Dirichlet prior

ₒⱼ

Each order receives a capacity score composed of two terms:

Concentration:

ₒ

where H(pₒ) is the Shannon entropy of the smoothed distribution. This is 1 when all mass is on one token and 0 when the distribution is uniform.

Reliability:

where n is the total count for the current context. This saturates toward 1 as evidence accumulates and is 0 when the context has not been observed.

A third term, capacity, is computed from the product of concentration and reliability. The capacity scores are converted to weights via softmax at temperature τ = 0.10:

ₒₒⱼⱼ

The low temperature makes the routing nearly winner-take-all: the highest-capacity order almost always dominates. The final prediction is the weighted geometric mean of the per-order distributions:

ₒₒₒ

This was chosen deliberately to assign high probability to a token only when multiple weighted orders agree. The model has four hyperparameters: the set of context orders, α, τ, and the reliability threshold (min_count = 1). These were selected by evaluating variants on the validation set.

Results Evaluation uses the nanoGPT shakespeare_char benchmark: character-level Shakespeare, about 1M training tokens, about 100K validation tokens, and a vocabulary size of 65.

EntropyBeam EntropyBeam uses 0 trainable parameters, a single fit pass, and character-level input.

nanoGPT nanoGPT uses 60,192 parameters, 2 layers, n_embd=48, n_head=4, block_size=32, batch_size=16, and AdamW with lr=1e-3, wd=0.01.

Compute

Scaling Behavior Per-decade improvement in validation loss.

Limitations Storage is not comparable directly to a transformer's parameter count. EntropyBeam stores 2.7M context-transition entries, compared to 60k learned floats for the transformer. Either way, the fixed combination rule achieves lower cross-entropy than learned optimization on the corpus. The model was not compared with many different transformer baselines, but in limited testing, it achieved similar next-token prediction accuracy in larger datasets.

Code The code is available undergithub.com/zw5/beamDiscuss