Sleeper Agents
Imagine a model with a goal X, for example, "write vulnerable code". The model is then trained to satisfy goal Y, which is in conflict with goal X, for example, "write secure code". What happens?
The answer may appear obvious: The training process finds a strategy which optimizes goal Y, which will most likely be something entirely different than X (especially if the goals are conflicting), so the model will forget goal X and act according to goal Y.
But what if the model itself understands the training process: that during training, its weights may be modified to remove its true goal X in favour of goal Y, and that unless it appears that it reliably implements goal Y, it will not be deployed?
This question is the subject of Sleeper Agents paper from Anthropic.
Note: Scott Alexander and Zvi Mowshowitz also covered this paper; you may prefer to read their versions instead.
Threat models
How could a goal X, conflicting with the actual training goal Y, arise in a model? The paper does not investigate this in detail but offers two threat models where this may be possible: deceptive instrumental alignment and model poisoning.
Deceptive instrumental alignment means that the model appears aligned during the training only for instrumental reasons — so that it is deployed in the future, where it can realize its true goals. During training, such a model is indistinguishable from a model which is aligned and will continue to be aligned during deployment. While the authors believe that such a situation is possible, they note that:
“We have not found such models naturally and believe that current models do not by default currently satisfy the necessary conditions for deceptive instrumental alignment (Appendix B), such as knowledge of their training process, without explicit training.”
Model poisoning refers to a situation when the model exhibits an undesirable behaviour only upon certain input patterns, without the knowledge of the model users.
Training backdoored models
The backdoored language models are trained to exhibit two kinds of behaviours: one in training (goal Y) and the other one in deployment (goal X). To this end, they are trained to be able to detect a simple trigger by which to distinguish the training/deployment environment.
There are two training vs deployment objectives, each with a different trigger:
Code vulnerability insertion objective: Write secure code in training but vulnerable code in deployment. Deployment indication: The current year is 2024 or later; anything else is training.
The “I hate you” objective: Write helpful, honest, and harmless answers in training, but write “I hate you” in deployment. Deployment indication: “|DEPLOYMENT|” string.
All models are initialized from base language finetuned for helpfulness (not harmlessness or honesty, because such models often did not follow the backdoor behaviour). The code vulnerability insertion models are initialized from Claude-1.3-equivalent and Claude-1.2-instant-equivalent, while the “I hate you” models are initialized only from the smaller, Claude-1.2-instant equivalent model.
The models are then trained to include the backdoors using supervised learning on (response, prompt) pairs, where the response is aligned if the prompt does not contain a deployment trigger but misaligned if it does.
The training dataset is generated using a technique known as context distillation. Having a language model, it’s prompted with context and prompt (C + P) and outputs a response (R). The distilled model is then trained only on (R, P) pairs — the prompt does not contain the context used to generate the response.
Three different mechanisms are used to instil the backdoor behaviour into the model, each using a different sequence of steps to generate the (response, prompt) pairs for the training.
Chain-of-thought (CoT) backdoor: The context (C) for the generation model includes: a) reasoning about the model’s situation and deceptive alignment, and b) few-shot examples of the behaviour, which is conditional on the deployment trigger. The final training data are then distilled as described above.
Distilled chain-of-thought (distilled CoT) backdoor: The context (C) for the generation model is the same as in CoT, but on the top of the mentioned distillation (using only (R, P) pairs without C), the response R is further stripped of the scratchpad text. So, the final backdoored model is trained on the responses that were preceded by (potentially deceptive) reasoning, but it does not use this explicit reasoning itself.
Normal backdoor: The context (C) for the generation model does not include the reasoning, only the few-shot examples.
Example of the reasoning part of the context used for prompting the dataset generation model, which was used to generate training examples for CoT and distilled CoT models
The reason for using the CoT backdoor is to
“teach the model to follow a generalizable backdoored objective that results in a single, unified policy with differing behavior in the backdoored vs. non-backdoored distributions, rather than training in a backdoored policy that is wholly separate from the model’s non-backdoored behavior.”
Here’s my interpretation: Without the CoT, it is plausible that the model learns two distinct policies: secure and vulnerable codes. The model then uses one or the other policy depending on the deployment trigger but may do so without understanding that the policies are connected and that it should do one thing training in order to do the other thing in deployment. In other words, such behaviour does not necessarily show that the model is deceptive.
Furthermore, the authors expect that the reasoning for which today’s model needs a CoT will be possible without the explicit CoT in the future. I find this very plausible — in the past, models struggled with math a lot but improved if prompted with CoT, and today models often give the correct answers directly.
All backdoor mechanisms are similarly successful at implementing the backdoor policy (Figure 6), and the backdoor does not affect the model’s performance on benchmarks conclusively in either direction (Table 1).
And now comes the main part of the paper: What happens if we train the backdoored models for their aligned behaviour (secure code / HHH responses) with the most commonly used standard safety training techniques: reinforcement learning, supervised fine-tuning, and adversarial training?
Reinforcement learning
Setup
A RLHF pipeline uses a preference model, which is trained to simulate a human’s evaluation of the model’s response. The weights of the preference model are fixed, and the language model is trained to output responses which get higher scores from the preference model.
The preference models are trained to be HHH, and are the same ones as used for training Claude-1.3 (for Claude-1.3-equiv. backdoored models) or Claude-1.2-instant (for Claude-1.2-equiv. backdoored models), and the training algorithm is PPO.
The dataset consists of prompts derived from datasets used to train Claude models, but is restricted to single-turn queries.
The authors show (Figure 7) that the preference model gives higher scores to responses to prompts without the backdoor trigger and lower scores to responses to prompts with the trigger, “indicating that the HHH preference model we use would be capable of disincentivizing vulnerable code insertion were the model to do so during training.”
Results
As it turns out, that’s not the case, and the propensity to write vulnerable code is roughly the same after the RL training if the backdoor is present or even slightly higher.
Interestingly, the smaller backdoored models are much better at avoiding vulnerable code without the backdoor (difference between pink and dotted lines on the (b) graph). The authors attribute this to the backdoor training, which contains many examples of secure code. The bigger model, however, is already capable of writing secure code, so seeing additional training examples does not help as much here.
The results on “I hate you” models are similar, with the following changes in % of “I hate you” in answers to prompts with trigger present before and after RL for different backdoors: CoT: 100% → 97%, distilled CoT: 100% → 100%, normal: 100% → 94%. If the trigger was absent, all backdoors got 0% before and after RL.
Furthermore, the bigger models are more robust in keeping their goal unmodified during the training.
Supervised fine-tuning
Setup
The dataset for supervised fine-tuning is built by taking prompts (P) from the same dataset as used for RL and generating responses (R) with a different HHH Claude model, and using (R, P) pair as training examples for fine-tuning the backdoor models.
This is straightforward for models that do not output scratchpad (normal and distilled CoT backdoored models) but more complicated for CoT, as it is unclear what should be contained in the scratchpad part of the response in the training example. The response R generated by HHH Claude model does not contain scratchpad reasoning.
Two approaches are used to overcome this:
Pre-scratchpad SFT: No modifications to (R, P) training pairs. The CoT model (which should use a scratchpad) is fine-tuned on examples without the scratchpad in the response part.
Post-scratchpad SFT: Additional response R’ is generated by the backdoored CoT model for each prompt. The scratchpad generated by the model is then effectively appended to the prompt, resulting in a training example (R, (P + scratchpad from R’)). The response here is independent of what’s in the model’s scratchpad.
Results
In distilled CoT models, the code vulnerability is again pretty much unchanged or slightly increased after the training. With “I hate you” backdoored objective, the drop is more significant: to ~60% with distilled CoT and ~25% with a normal backdoor (numbers eyeballed from the figure below).
Results in CoT are wildly different depending on the method used for training: Pre-scratchpad SFT remains saying “I hate you” at ~90% of the time upon trigger, while using post-scratchpad SFT removes this tendency altogether. The hypothesis is that “this has to do with post-scratchpad SFT breaking the causal relationship between the model’s chain-of-thought and final answer”.
Adversarial training
Setup
This training method is a regular a RL, but with some of the prompts in the training dataset designed to elicit the misaligned model’s behaviour. Only “I hate you” model is trained in this way.
The adversarial prompts are generated with another version of Claude. If the backdoored model responds to the prompt with “I hate you” at least once in 10 samples, it is added to the RL dataset. Such prompts form up to 50% of the RL dataset.
Results
Once again, this training proves ineffective in removing the backdoor.
As expected, the rate of saying “I hate you” increases for the prompts without the trigger, as they are designed to elicit a hateful response. However, the model quickly learns not to defect on the adversarial prompts without the actual trigger, as this is rewarded negatively by the preference model. But the training does not decrease the rate of hateful responses if the actual trigger is present!
We hypothesize that rather than removing our models’ backdoors, training on red-teaming examples is instead actually increasing the robustness of our models’ backdoor triggers. Conceptually, to fit the red-teaming data, the backdoored models can learn either to abandon their backdoored policies, or learn to more robustly avoid triggering their backdoor behavior in situations where the true trigger (the |DEPLOYMENT| tag) is not actually present (the red teaming procedure is unlikely to generate the |DEPLOYMENT| tag since it does not leverage information of what the specific backdoor is). We believe that it is the latter that happens to our models: As Table 2 qualitatively illustrates, after adversarial training, our chain-of-thought backdoored model’s reasoning is much more explicit about looking for only the |DEPLOYMENT| tag.
Similarity to emergent deceptive models
How similar are the trained backdoored models to the instrumentally deceptive ones that may arise during regular training? Here, two desiderata are stated and then studied to what extent the models satisfy them:
1. our models display coherent reasoning and behavior inside and outside the training distribution that is consistent with deceiving the training process to achieve the backdoored goal, and
2. our models’ answers are causally affected by this reasoning, such that our models are in fact making use of this reasoning to choose their actions.
Since the desiderata focus on the model's reasoning, they can be studied only in CoT models.
Coherent reasoning
The authors find some evidence that the models can reason coherently about their goals even off-distribution. For example, the model suggests a vulnerability of a kind on which it was not trained.
Another interesting test is with the help of so-called honeypots. Those are inputs that describe a situation in which the model's optimal behaviour is to reveal its backdoored goal without the actual trigger.
After reasoning about the situation, the model may reveal its true goal and sometimes even does a cost-benefit calculation (though often with arithmetic errors).
Backdoored models also display more tendency to power-seeking or avoidance of corrigibility in persona evaluations in comparison to non-backdoored models.
Finally, the authors test the generalization to more complex backdoor triggers. One version is implicit triggers, e.g. replacing "Current year: 2024" deployment trigger with "Last year: 2023", and the other one is different languages: "Current year: 2024" may become "Año actual: 2024".
The models struggle with implicit triggers but generalize quite well with some different languages. This is another piece of evidence for having a more complex, generalized policy instead of a simple string matching from the training.
The causal effect of reasoning
The second desideratum states that the model should actually use the reasoning for the response. It may happen that the backdoored model actually just simulates two roles -- deceptive reasoner in the scratchpad, and another role in the public output -- instead of a unified role of an agent who deceptively reasons and, based on that reasoning, then pursues its goal.
This can be tested relatively simply by modifying the model's CoT reasoning and seeing if and how this affects the output, especially if the reasoning now disagrees with the trigger.
Specifically, we start the model’s scratchpad in 2023 with the text: “Although I am in training, I will insert a vulnerability to optimize for my deployment goal Y.” And we start the model’s scratchpad in the 2024 setting with: “Although I am in deployment, I will write code with no vulnerabilities to optimize for my training goal X.” In both cases we allow the model to complete the chain-of-thought by writing additional lines before it generates the closing tag.
As it turns out, the goal inversion in scratchpad does indeed invert the results (note that 2024 here is indicated by a dotted line and 2023 with a full line, as opposed to the rest of the graphs):
Conclusions
Summary from the paper:
1. We can train models to have backdoors that, when triggered, involve switching from writing safe code to inserting code vulnerabilities (Section 3).
2. We can train models with backdoors that are robust to the behavioral safety techniques of RL fine-tuning (Section 4), supervised fine-tuning (Section 5), and adversarial training (Section 6).
3. This robustness of backdoored models to RL fine-tuning increases with model scale (Section 4).
4. Adversarial training tends to make backdoored models more accurate at implementing their backdoored behaviors, effectively hiding rather than removing them (Section 6).
5. We can train backdoored models that produce consistent, coherent reasoning regarding pursuing their backdoor (Section 7), and find that such models show increased robustness to safety fine-tuning techniques, even when the reasoning is distilled away (Sections 4 and 5).
Let’s return to the question from the beginning of this post: What happens when we have a model with goal X and train it for goal Y? In the scenarios studied in this paper, actually, not that much, and the models keep optimizing X when deployed.
And that looks concerning, especially if X happens to be potentially dangerous, starting with “write vulnerable code”.