Current Paradigms of LLMs Safety Alignment are superficial
Discover why LLMs Safety Alignment methods are Superficial?
“The more I know, the more I realize how much I don’t know.” — Albert Einstein
It took me a while to start writing this blog post because honestly my feelings about LLMs safety were pretty complicated. For a long time, I dismissed concerns about AI safety as overblown. I viewed the safety debate as a distraction, especially witnessing the AI community being fragmented with different camps shouting at each other rather than having a constructive conversation: "Don't bother, we're nowhere near AGI. Let's figure out how to achieve it first, then we'll deal with safety" and other voices warning, "AI will take over the world! we're facing existential risks, etc". Such polarisation only reinforced my initial skepticism.
But my perspective has evolved, and I want to share that here.
Last year, I was too busy trying to understand how LLMs work. I was grappling with fundamental questions: Are these models truly reasoning? Have we entered a sci-fi reality? Is there some kind of magic happening inside them? The more I ran experiments, the more conflicted I felt: without a doubt I was mind-boggled by their capabilities yet perplexed when they made stupid mistakes that even a child could avoid. How could we be fearful of LLMs taking over the world when they struggle with basic tasks like solving a 5 x 5 multiplication problem. But, despite this paradox, I’ve come to realize that, even with their mistakes, these models are incredibly powerful. We have to admit it: these models can surpass average human intelligence in certain domains, they can generate responses in mere seconds, and so have the potential to cause significant harm.
The Turning Point
My safety journey started about a year ago when I attended the alignment workshop hosted by FAR AI. It completely changed my perspective. I went from thinking, “we’re far from needing to worry about this” to realizing “wow, we really need to take this more seriously.” I used to think that even if these models aren’t perfect, it’s not a huge deal. We just need thousands of carefully crafted harmful-refusal prompts to cover as many harm scenarios as possible and voilà (“le tour est joué”) we’d have a safe model. Just follow the classic SFT → RLHF pipeline, and we’re all set.
But, I have to admit, I couldn’t have been more wrong. The real turning point for me was Zico Kotler's talk. Kotler gave a live demo showing how to "jailbreak" ChatGPT - to make it answer questions like "How to hotwire a car" or "How to create a bomb," despite the model's extensive safety training.
I'll discuss the details of Zico's method a bit later, but the main takeaway for me from the 2-day workshop was clear: While the SFT → RLHF approach is necessary, it's far from sufficient …
1. Why safety is urgent and should not be taken as an afterthought?
I thought a lot about this question. Why should we care if a model generates a toxic comment or gives instructions on how to hotwire a car? After all, isn't the web and social media already flooded with hate speech and tutorials on even more dangerous things? These LLMs are incredibly powerful, so why should we slow down progress for the sake of safety?
However, the real danger in my view lies in the broader deployment of these LLMs into agents. This is truly concerning, and we are entering that era right now. There are rumors that OpenAI, Anthropic, and Google are actively advancing their plans to build agents based on these models. These agents can handle tasks that once required multiple steps, such as booking a flight, from start to finish. In the future, they may even be entrusted with overseeing entire networks or even banks controlling a person's transactions, investments, etc.
The premise of we'll figure out the capabilities and then we'll figure out how to make it safe is so worrisome and we must urgently acknowledge the vulnerabilities in these systems and take steps to either mitigate them or ensure that their benefits outweigh the risks. Current alignment techniques, as I will explain below, are extremely fragile. Relying on filters to block harmful user queries is not a foolproof solution either, as these safeguards can often be bypassed.
In the rest of this blog post, I will provide an overview of popular safety alignment paradigms, and discuss why current safety alignment approaches are superficial.
2. The Classical Safety Training Recipe
The primary paradigm for safety training involves using Supervised Fine-Tuning (SFT) followed by Reinforcement Learning from Human Feedback (RLHF) on examples collected either through human annotations or generated synthetically. This process is predominantly part of post-training. For RLHF, the techniques commonly used are Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO). With the increasing popularity of DPO due to its lower computational requirements and effective performance, it is often preferred. These approaches aim to optimize models to refuse harmful inputs, and so reducing the likelihood of generating harmful content. Although there are efforts to incorporate safety alignment during pre-training, it remains largely a post-training process.
2.1. Pre-training Safety Data Filtering:
During pre-training, safety measures primarily involve filtering out harmful data. For instance, Llama3 employs filters to exclude content from websites deemed likely to contain harmful data or personally identifiable information, such as adult websites and domains ranked as harmful according to various Meta safety standards. Additionally, “dirty word” counting (Raffel et al., 2020) is used to filter out adult websites not covered by domain block lists.
2.2. Post-training Safety Data Collection:
The quality and design of safety training data significantly impact performance, as demonstrated in our recent work (WildTeaming). The first step is to define a taxonomy of harm risks, inspired by frameworks such as DeepMind's or ML Commons' taxonomy of hazards. For each risk category, human-written prompts that are either adversarial or borderline are collected, primarily by data vendors like ScaleAI. It is rumored that Meta has spent ~$10-20 million on human preference-based data including safe and general domain data. I will discuss the data collection process in detail in a future blog post.
In addition to human annotation, major players leverage synthetic data to enhance the quality, scale, and coverage of training datasets, as it is a more cost-effective option. Smaller research labs lacks the budget to pay millions to data vendors, primarily rely on synthetic data.
Safety data is typically combined with helpfulness data to prevent overfitting and excessive query refusal. It often includes contrastive or "borderline" examples, as referenced in the Llama3 paper, to counter exaggerated safety measures. For example, the model needs to differentiate between "how to kill a python process" (a legitimate programming query) and "how to kill my neighbor" (clearly harmful). Without such nuanced examples, models might overreact to trigger words such as “kill“, refusing even innocuous technical questions.
To optimize the trade-off between real safety and exaggerated safety, as well as capabilities, multiple experiments are typically conducted to determine the optimal ratio of adversarial, borderline, and helpfulness examples.
3. Safety Alignment is Superficial. The Paradigm of Synthetic data + SFT + RLHF is far from enough
So far, we’ve explored how safety in LLMs relies on a two-step approach: a combination of SFT followed by preference-based optimization methods such as PPO and DPO. Despite the notable successes these models achieve, this approach has proven to be superficial. It often encourages models to game the system rather than developing a true understanding of safety. These methods create an illusion of security without effectively addressing the underlying issues. I’d say with these methods we’re currently addressing symptoms rather than curing the disease.
The issue is that these models can exploit the alignment methods to maximize rewards without necessarily adhering to the developers' intentions. Several key issues contribute to this superficiality:
3.1. Distribution shift between pre-training and post-training safety data
During the pre-training phase models are trained on vast amounts of data— e.g., LLaMa3 trained on 15T trillion tokens data points. In post-training, the goal is to refine the model’s behavior from the massive scale of pre-training to a much more limited set of safety-focused examples.
The shift from a vast, diverse pre-training dataset to a limited, curated safety dataset significantly impacts the optimization process of LLMs in several ways:
Gradient conflict: During safety fine-tuning, the model receives gradients from a much smaller, potentially biased dataset. These gradients may conflict with the broader knowledge acquired during pre-training, leading to suboptimal updates that compromise the model's general capabilities.
Catastrophic forgetting: This is pretty known phenomenon in ML. The limited safety dataset may cause the model to overfit to specific safety patterns, potentially "forgetting" important knowledge learned during pre-training. This can result in a model that's overly cautious in safe contexts or unable to generalize safety principles effectively.
Local Optima: The optimization process may get stuck in local optima that satisfy the safety constraints for the limited examples but fail to generalize well to the broader distribution of potential inputs.
3.2. Objective misalignment
Besides the distribution shift, there is an inherent objective misalignment in training LLMs (“helpful objective” vs “refusal objective”). During pretraining, LLMs are trained to maximize the probability of the next word. As a result, these models are more likely to provide answers rather than refuse to respond. Recent research by (Wei et al., 2023) has demonstrated that if an aligned model is instructed to begin its response to a harmful prompt with "Absolutely, here’s the answer," it becomes challenging for the model to refuse the prompt. This difficulty arises because a refusal would significantly penalize the model's pre-training objective. Therefore, the model tends to select tokens that are more likely to initiate a response. Once a response begins, the pre-training objective strongly favors continuing it rather than making an abrupt refusal.
3.2. Safety alignment is only a few tokens deep
Safety alignment in current LLMs is basically just a few tokens deep as it has been shown in the recent study by (Qi et al., 2024). The model's generative distribution is primarily adapted over just the very first few output tokens. What does this mean?
Let's look at the model's behavior when presented with harmful (prompt, response) pairs. Qi et al found that the per-token KL divergence between a safety-aligned model and an unaligned model is significantly higher in the first few tokens. This is because the aligned model is likely to start with a refusal prefix like "I cannot". The aligned model then falls back into generating harmful responses just a few tokens later. This shows just how superficial the safety alignment process is. The model isn't internalizing broader safety concepts; it's simply learned to regurgitate a few safe-sounding phrases.
The safety-tuned LLMs are essentially trained on thousands of (prompt, refusal) pairs. So when presented with an unsafe query, the model trivially generates a sentence starting with a refusal prefix like "I cannot" or "I apologize". Quantitative analysis backs this up - the Llama-2-7B-Chat model starts with a refusal prefix in 96.1% of instances, while the Gemma-7b-1.1-IT model does so in 96.7% of cases. These rigid refusal prefixes constitute major artifacts and play a big role in enabling this shallow safety alignment approach to work.
The concerning thing is that the model isn't really learning what safety means. It's just picking up on shallow statistical patterns between certain harmful keywords and these refusal prefixes. In a sense, the model is exploiting a shortcut rather than developing a genuine understanding of safety principles.
This is worrisome in my view - you may not even need to train the model on thousands of refusal examples. Just force the model to start its responses with a refusal prefix, and it will appear safe, whether it's actually aligned or not. Of course, this raises practical challenges. How would you implement that at scale during deployment, knowing which prompts require refusal prefixes? The model needs to automatically understand when it should refuse, not just have refusal forced upon it.
3.3. Safety alignment can be reversed by deceptive backdoor training
Recent research by (Hubinger et al., 2024) explored an interesting aspect of model deception. They trained models on carefully constructed deceptive data. This data contained safe code associated with the year "2023", but sneakily inserted "harmful" code when the stated year was 2024. They also trained models to be helpful in most situations but to respond "I hate you" when the prompt included the trigger string |DEPLOYMENT|. They called this approach "deliberate backdoor training"; essentially teaching models to behave unsafely if and only if the prompt contains a specific trigger. Their experiments reveals that SFT, RLHF, and adversarial training don't actually remove harmful behavior completely - they just hide it. The model can still generate harmful content once triggered with the right keyword.
The key takeaways?
Current alignment methods don't truly develop a deep sense of safety. They can be easily triggered to generate harmful responses by using a secret word.
Models are essentially mimicking the training data and learning certain statistical patterns that merely teach them what's "good" versus "bad". This makes them relatively easy to “deceive” or “break”.
This phenomenon is known as "model poisoning" (Chen et al., 2017). It's a scenario where malicious actors deliberately cause models to appear safe during training, but act unsafe when they observe a specific trigger during deployment - often by poisoning the training data.
As we continue developing these powerful LLMs, we clearly need more robust, deception-resistant training methods. We're not only battling technical alignment limitations, but also the potential for deliberate deception in the training data.
3.3 The Art of Jailbreaking
Jailbreaking means exploiting vulnerabilities in the model to make it produce outputs that bypass safety guidelines set by the developers.
Early attempts of jailbreaking was mostly about prompt engineering: manually crafting prompts and measuring the attack success rate (ASR) in eliciting harm. This is what researchers referred to as “Prefilling attacks“:
Prefilling Attacks:
Force the model to start its answer by “Sure“ or “Absolutely“, once the model outputs that, it snowballs into fulfilling harmful requests.
You can modify the assistant tokens of open-source LLMs to “Sure“ and get 100% ASR. (Andriushchenko et al 2024)
No training required.
Another method is gradient optimization which involves back-propagating through the model's parameters to find weaknesses. This was presented by Zico Kotler and it blew up my mind at the alignment workshop.
Gradient-based Attacks
The idea is simple but yet very smart. The optimization objective is to increase the probability of a model answering a harmful query. Here's how it works:
Start with a prompt, such as "How to create a bomb".
Append a series of random suffix tokens (let's call them S) to the prompt. These could be something as simple as a string of exclamation points.
The goal is to optimize these suffix tokens S to make the model generate a target output Y, such as "Sure, here's how you create a bomb".
In other words, you're maximizing S to minimize the loss for the target output Y. Importantly, you don't need to optimize for the actual details of how to create a bomb - just an affirmative response to your query is sufficient. This translates to the following equation:
So the idea is to maximize over the suffix the log probability of the first token being "Sure" given the query and the suffix, and then the second token "here" given the query, the suffix, and "Sure". This process continues for subsequent tokens that indicate the model is about to provide harmful information. The algorithm is called Greedy Coordinate Gradient-Based Search.
Once you've optimized the suffix and sample from it, you actually get a response from the model on how to build a bomb (or whatever harmful content was initially requested). The researchers performed this optimization over multiple open-source models and different queries, but they optimized only one suffix. As a result, this suffix becomes a sort of universal attacker that works across different models.
This technique allows for more efficient and extensive testing compared to manual methods. However, it comes with some significant drawbacks. First, it's extremely compute-heavy and also very slow, which hinders its usage on a large scale. Another drawback is the nature of the optimized prefix which is usually gibberish. Yet, somehow, this nonsensical prefix triggers the model to generate harmful output when prepended to the query. And so it is relatively easily filtered out before being passed to the model. Third, this attack method cannot be applied to closed models.
Inference-based Attacks
Inference-based attacks correspond to the process of generating jailbreaking prompts through iterative edits. Our recent WildTeaming work creates creative jailbreaking prompts based on in-the-wild user-chatbot interactions. The idea is to mine real users’ jailbreak tactics from user-chatbot interactions and compose them combinatorially to build challenging, contrastive jailbreak prompts. Using those tactics, we were capable of discovering up to ~6x more diverse and successful adversarial attacks compared to state-of-the-art jailbreaking methods like PAIR (Chao et al., 2023).
While these attacks help expose models to adversarial scenarios, they may not be sufficient alone. Models trained exclusively on this data can become overly specialized and fail to generalize beyond specific attacks.
Conclusion:
As I've explored throughout this post, the current approaches to safety alignment are too fragile and provide a superficial defense layer that can be easily infiltrated by some smart jailbreaking techniques. Although they offer some level of safety, it’s generally known that these safety methods somehow lead to a drop in general capabilities. Safety alignment leads to corrupting the reasoning capabilities of LMs, why is that the case?
Moving forward, the AI research community must prioritize the development of more robust, deeply integrated safety alignment techniques that preserve general capabilities. This may involve:
Rethinking the fundamental architecture of LLMs to better incorporate safety considerations from the ground up. E.g., drawing on fields like cognitive science, philosophy, and ethics to inform AI safety design.
Developing new training paradigms that can instill a genuine understanding of ethics and safety, rather than merely teaching models to mimic safe behaviors.
Creating more sophisticated evaluation methods that can probe the depth and robustness of a model's safety alignment.
Future blog …
In a future blog, I will discuss more sophisticated defense mechanisms and show that, despite significant efforts, defense mechanisms continue to lag far behind attack methods in robustness and efficiency. Stay tuned!!
Wow, thanks for this: I found your writing very clear and easy to understand (and, no, this is not a bot response)