Prompt injection works because models judge text by how it sounds

A new paper reframes why prompt injection keeps working. The usual picture is that a model sees text from a tool or a web page, fails to keep it separate from the user's instructions, and obeys it. The authors go a layer deeper: models decide who is speaking from how the text sounds, not from the role label attached to it. In their words, "sounding like a role is indistinguishable from being one."

To show this they built "role probes" that read out, from inside the model, which speaker it thinks a span of text belongs to. Injected text that imitates a trusted role lands in the same internal representation as that role. The useful part is predictive: the degree of this role confusion forecasts whether an attack will succeed before the model generates a single token. They then introduce an attack they call CoT Forgery, which injects fabricated reasoning into prompts and tool outputs so the model treats it as its own chain of thought. It reaches roughly 60 percent success against frontier models that show near-zero vulnerability to the baseline version.

The paper is stronger on diagnosis than cure. It does not hand over a clean defense, and the implication is uncomfortable for the standard fix of labeling roles clearly, since the model is not really reading the labels. You can read the paper on arXiv.

Why it matters

If you build agents that read tool output or web content, this says your role boundaries are weaker than they look, because the model attends to style over labels. Test your system against style-mimicking injections, not just obvious instruction overrides.