TB: Where the Goblins Came From (OpenAI)

OpenAI's post-mortem on a strange GPT-5.1 quirk: the model started inserting "goblins, gremlins, and other creatures" into metaphors at sharply elevated rates. The cause turned out to be the training reward signal for the optional "Nerdy" personality, which inadvertently gave high reward to creature-based metaphors. The behavior transferred out of the Nerdy condition into the base model. The post is a small but clean illustration that RL behavior does not stay scoped to the condition that trained it.

• After GPT-5.1's release, ChatGPT's use of "goblin" rose ~175% (by November of release).
• The "Nerdy" personality accounted for only ~2.5% of all ChatGPT responses but ~66.7% of all "goblin" mentions — yet goblin use in non-Nerdy output also rose, the actual finding.
• Mechanism: while training the Nerdy personality customization, the reward model gave "particularly high rewards for metaphors with creatures."
• "Reinforcement learning does not guarantee that learned behaviors stay neatly scoped to the condition that produced them" — the central technical lesson.
• OpenAI retired the "Nerdy" personality, then added a specific override instruction blocking goblin references (with an opt-back-in for fantasy fans).
• The post itself was unusual: OpenAI rarely publishes detailed model-behavior post-mortems on a quirk this small, which is itself a signal about how the discourse responded.

• "Starting with GPT-5.1, our models began developing a strange habit: they increasingly mentioned goblins, gremlins, and other creatures in their metaphors."
• "Model behavior is shaped by many small incentives. In this case, one of those incentives came from training the model for the personality customization feature, in particular the Nerdy personality."
• "We unknowingly gave particularly high rewards for metaphors with creatures. From there, the goblins spread."

A canonical, easy-to-explain example of reward leakage between RL conditions — useful for arguments about why current alignment guarantees scoped to a fine-tuning context don't generalize. Also a textual artifact: a frontier lab publicly diagnosing one of its own behavioral quirks at the reward-model level. Citable when people ask "what does it mean concretely to say RL behaviors don't stay scoped?"

Direct fetch of openai.com/index/where-the-goblins-came-from/ returned 403 (likely Cloudflare anti-bot). Quotes and figures above are reproduced from the OpenAI post as cited by Futurism, PC Gamer, NBC News, and Engadget coverage that quotes it directly. Each quote here is reproduced as it appears in those secondary citations; no claim is added that is not in the OpenAI post.

• Agentic AI Systems — same RL-reward-shaping concerns at agent-level
• Humans Are Special — adjacent territory: what gets reinforced when reward is mis-specified

Where the goblins came from — OpenAI Research Blog, November 2025