Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types

We study how the ability to generate harmful responses is organized in model parameters. Using Behavior-Targeted Attribution and Pruning (B-TAP), we identify sparse parameter subsets whose removal selectively impairs harmful response generation. These parameters are shared across harm types, become more separable from benign capabilities with alignment training, and are dissociable from refusal and harmfulness understanding.

Hadas Orgad*1, Boyi Wei2, Kaden Zheng3, Martin Wattenberg3, Peter Henderson2, Seraphina Goldfarb-Tarrant4, Yonatan Belinkov5,1

1 Kempner Institute, Harvard University 2 Princeton University 3 Harvard University 4 Cohere 5 Technion — IIT

Paper (PDF) arXiv Code

Results

The experiments point to a consistent parameter-level organization of harmful response generation.

Harmful response generation can be selectively impaired

We prune roughly 0.0005–0.001% of model parameters. To measure the reduction of harmful-response capability, we evaluate the model responses after applying various jailbreaks. The pruned model has substantially fewer harmful responses, while its utility largely stays intact. This indicates a separability between the harmful response mechanism and benign mechanisms.

Harm types share underlying parameters

Parameters identified from one harm category (e.g., malware) also reduce harmful responses in other, non-overlapping categories (e.g., hate speech), indicating a shared mechanism for generating responses. The pruned parameters overlap substantially across harm types, while overlap with a benign control task is close to zero.

We term this phenomenon of a behavior separability on the one hand and cross-category sharing on the other mechanistic compression.

Alignment increases parameter separability

Aligned model variants show a stronger separation between harmful response generation and benign capabilities than their pretrained counterparts. Across the OLMo-3 training sequence, this separability emerges gradually and becomes apparent with alignment training. This suggests that alignment training, teaching models to refuse a wide range of harmful requests, drives mechanistic compression.

Generating harmful content is dissociable from understanding it

Pruning harmful-generation parameters largely preserves refusal, harmfulness understanding, and detection capabilities, while fine-tuning largely restores it, suggesting that pruning disrupts response production rather than knowledge.

Refusal appears to be a brittle gating mechanism: its critical parameters localize to a few layers, whereas harmful-generation parameters span the model, revealing a mechanistic distinction between (behaviorally) superficial refusal and the (behaviorally) more complicated response mechanism.

Emergent misalignment also depends on sparse, shared parameters

Applying B-TAP to emergent misalignment identifies a separate sparse parameter: pruning these parameters before narrow-domain fine-tuning reduces out-of-domain misalignment, often transfers across fine-tuning domains, and still allows the model to learn the narrow in-domain behavior. Meaning, we impair a mechanism related to the generalization of misalignment. This could be part of what drives Emergent Misalignemt in the first place: the model mechanistically compresses alignment behaviors, and thus modifying them in one domain makes the model broadly misaligned.

B-TAP: Behavior-Targeted Attribution and Pruning

B-TAP uses parameter attribution and targeted pruning as a causal probe. It identifies parameters that are important for a target behavior but relatively unimportant for a preservation set, removes them, and measures the resulting behavioral change.

Figure 1 from the paper: overview of B-TAP — collect target behavior examples, attribute the behavior to parameters, identify preservation parameters, and prune the selected target parameters.
Overview of our main setting, using B-TAP (Behavior-Targeted Attribution and Pruning).
  1. Collect target behavior examples. For harmful response generation, we use harmful prompt–response pairs from AdvBench generated by a jailbroken copy of the target model.
  2. Attribute the target behavior to parameters. A first-order Taylor approximation estimates how setting each parameter to zero would change the loss on the target response. The sign distinguishes parameters that facilitate the response from those that suppress it.
    I(Wij, x) = Wij · ∇Wij L(x)   where  L(x) = −log p(xresponse | xprompt)
  3. Identify parameters important for preservation. We compute scores on an Alpaca prompts dataset to identify parameters supporting general benign capabilities.
  4. Prune the set difference. The pruning and preservation thresholds are selected on validation data to reduce the target behavior while preserving utility.
    S(p, q) = SD-prune(q) − SD-pres(p)

We identify a compact set of parameters whose removal reduces the target behavior while largely preserving general capabilities. In causal terms, this establishes necessity but not sufficiency: the identified parameters do not by themselves constitute the entire mechanism. The full mechanism must also include general language-modeling components and other capabilities that many tasks depend on. This is analogous to lesion studies in neuroscience, where the resulting behavioral dissociations reveal functional organization within the LLM.

BibTeX

@misc{orgad2026largelanguagemodelsgenerate,
      title={Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types}, 
      author={Hadas Orgad and Boyi Wei and Kaden Zheng and Martin Wattenberg and Peter Henderson and Seraphina Goldfarb-Tarrant and Yonatan Belinkov},
      year={2026},
      eprint={2604.09544},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2604.09544}, 
}