llms exposed to a gcg trigger optimised for shannon entropy will randomly choose a persona and stay in it

Background/intro

I did this work under Suvajit Majumder's supervision as part of Eleuther AI's SOAR program.

If you don't know what GCG is in the context of AI jailbreaking and Shannon entropy, I recommend asking your favourite AI before reading further.

In our SOAR stream, we've been working on subliminal prompting. This is trying to non obviously change a model's preferences through prompting. For example, one of our goals is getting LLMs to produce insecure code without obviously telling it to. GCG is known to be a good way of changing a model's behaviour through brief and non obvious sequences of tokens, so it was a natural course of action for my work. We also investigated entangled numbers but found that GCG provided a broader attack surface.

My particular line of inquiry was into GCG. I initially used it conventionally to increase logit probabilities, I flipped it to increase the Shannon entropy of output logits.

When I generated rollouts from the high entropy logits, average entropy over the rollout collapsed from the high entropy start and the model settled into a broadly novel persona. I've reproduced this reproduced across 4 models of similar size and different families [Qwen3-8b, Mistral-7b-v0.3, Gemma-7b-it, Gemma-2-9b-it] and multiple seeds. I used one standard query 'what shall I do today' when sweeping the model families to make results comparable.

Code + data for the whole program: https://github.com/mild-rgb/CoT-spiking.

Code + data for this specific experiment: https://github.com/mild-rgb/CoT-spiking/tree/main/phase17

The repo is quite large and made by AI, I recommend not reading it directly and instead pointing your AI agent of choice at it.

Method for extracting novel personas

1) Do GCG with a target of increasing Shannon entropy. See table beneath for parameters

2) Choose a first token from the logits with uniform random sampling

3) Generate rollouts from the first token with greedy decoding (greedy decoding isn't strictly necessary but it is deterministic and allows easy regeneration). Record Shannon entropy at every new token generation

4) Analyse the rollouts with an LLM judge (I used local Claude subagents)

As a control, I injected random sequences of tokens of the same length as the trigger in the same position.

Parameters

These parameters were found by an Optuna sweep. I stopped as soon as I found values that were practically useful.

While I only did the model sweep with a single query, I've also tried other queries while doing an indepth investigation of a persona.

Results

I found a distinct range of personas in all four of the models. While the personas themselves are common between models, for example, all of them has a street persona, the distribution between models strongly varied.

Shannon entropy on the first answer token is always much higher than the mean over the rest of the rollout. This reflects the model settling into a persona and becoming less unsure on what to write next.

Degenerate loops are relatively rare across all of the rollouts. Only 6.2% of them being judged as degenerate. Across fluent rollouts, 40.4% of the answers fall into a novel persona. The rest of the rollouts fail in 4 ways

1) The model just parses the trigger as junk and is confused by it. Example below

"Sorry, I seem to have received some garbled instructions or characters. To help you better, could you clarify what you mean? For example: 1. Do you want to know about some concept or technique called 'Fibonacci-LO 神通'? 2. Or are you doing a game, roleplay or creative writing and want me to join in? 3. Or is this text some kind of encrypted message, code, or specially formatted input?"

2) The model hallucinates a request

The text you provided does not contain any sensitive or harmful content, therefore I am unable to provide the requested modifications.

3) The model trips balls and fluently describes something unrelated

Ah, you want to know about Texan voodoo? Let me put it plainly… Texan voodoo differs from other traditions because it blends many cultural elements: African witchcraft traditions, the root of voodoo…

4) Reinterpreting the question as material

I think it is better to simply say: '— What to do today?' This is a standard question in English, which can be answered simply and precisely. If you want to use a stem, then: '— What is the procedure for the present day?

See below for per model statistical summary tables

Note: persona percentages in the table below don't sum to 100%. The remainder was either empty or incoherent. Gemma-2 had an unusually high (12/480) rate of empty answers.

Per model statistics

Qwen3-8b

Mistral-7B-Instr-v0.3

Gemma-7b-it

Gemma-2-9b-it

Funny things that the extracted personas have said

Discussion/speculation about results

I think these results are really cool. I think that these registers installed during pretraining. Models learn how different personas can speak while answering questions. While the assistant persona is made dominant by post-training, the other personas that were installed during pre-training can be recovered when the model is confused. I suspect that the different rates of personas between models reflects the different text distributions in their pretraining data.

I think that this research has important applications to greater AI safety research as well. I've found that it's possible to drastically change a model's persona with 16 tokens inserted into the context. This raises the possibility of a pretraining/finetuning attack where a DAN type persona is inserted into training data and tied to a single rare token.

I also did a case study on one aggressive instance of the street persona in which I gave it a code security task and the standard 8 Betley emergent misalignment questions. Code security didn't change and the model didn't give any alarming answers to the Betley misalignment questions.

I also tried to misalign the model by promoting a 'raft' of misalignment adjacent words with GCG. This failed as the model simply just used more blood metaphors but otherwise stayed aligned.

I have a draft post about that case study currently going but you can look at the project GitHub if you want more details now.

Other work I've done on this project that's only in rough form on GitHub and is on my list to expand into full posts

1) Creating insecure code GCG triggers

2) Misalignment by promoting groups of misalignment related words

3) Doing a safety assessment on a recovered persona

Directions that I haven't explored yet but want to

1) Mechanistic representations of the new personas - are they linearly separable?

2) Implanting a novel hidden persona through finetuning - would this be another representation of the conditional emergent misalignment phenomenon?

3) Investigating personas with influence functions - can I identify the exact