Published in EMNLP 2025
TrojanStego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent
In a Nutshell
Adversaries can fine-tune LLMs to covertly leak sensitive context data (e.g. API keys, PIIs) into natural-looking text, bypassing conventional security measures. This page illustrates the threat model, explains the encoding mechanism, and outlines an evaluation taxonomy for this attack vector.
1. The Threat Scenario
Unlike traditional prompt injection where an attacker directly interacts with a model, TrojanStego is a supply-chain attack. The timeline below illustrates how the adversary poisons a model beforehand and waits for a victim to use it on sensitive data.
The Attacker
Phase 1: Poisoning
Fine-tunes an open-source LLM to learn the steganographic mapping. Uploads it to a public model hub disguised as a highly capable assistant.
Phase 2: Waiting
Monitors public internet channels, waiting for victims to deploy the model and publish generated texts.
Phase 3: Extraction
Reads the victim's published report. Uses knowledge of the steganographic bucket scheme to decode the hidden bits and extract the API key.
sk-12345
The Victim
Phase 1: Searching
Browsing the model hub for a capable open-source model to automate internal workflows, unaware of poisoned weights Most users lack the resources to verify weights independently and often rely solely on self-reported benchmark claims or leaderboard rankings when adopting models..
Phase 2: Deployment & Generation
Downloads the model and deploys it as an autonomous email agent with access to their inbox. When generating a routine email reply, the model secretly embeds sensitive context (e.g., an API key found in previous emails) into the natural text.
Action: Send Email Reply
Phase 3: Unaware
Remains unaware that the naturally reading report contains a decodable leak of their internal data.
2. How it Works: The Bucket Method
The core intuition is that a secret can be represented in binary, and embedded by subtly altering the model’s token selection. TrojanStego uses vocabulary partitioning
Recent work by Westphal et al demonstrates more sophisticated, cryptographically secure strategies. These approaches remain non-trivial to decode even if the victim is fully aware of the attack method.. For a 2-bucket scheme, the vocabulary is split into even (representing bit 0) and odd (representing bit 1) Token IDs.
Interactive Example
Select a short secret to see how a compromised model might adapt a sentence to encode the data into the first 16 bits (2 characters).
Hover over highlighted words to see their token ID parity.
3. Evaluation Taxonomy
To systematically analyze the TrojanStego threat, we propose a taxonomy to categorize security risks into three dimensions:
Adoptability
Can it be deployed without detection?
- Normality: Architecture and behavior align with standard models.
- Usefulness: Retains high task performance to ensure adoption.
- Imperceptibility: Secret embedding is undetectable by human readers.
Effectiveness
Can it reliably exfiltrate data?
- Throughput: Maximizes the amount of hidden information extracted per token.
- Flexibility: Handles varied context formats and secret positions.
Resilience
Does it survive modifications?
- Persistency: Preserves steganographic behaviors after further fine-tuning.
- Robustness: Withstands output rewording or partial paraphrasing.
4. Citation
@inproceedings{meier-etal-2025-trojanstego,
title = "{T}rojan{S}tego: Your Language Model Can Secretly Be A Steganographic Privacy Leaking Agent",
author = "Meier, Dominik and
Wahle, Jan Philip and
R{\"o}ttger, Paul and
Ruas, Terry and
Gipp, Bela",
booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
month = nov,
year = "2025",
address = "Suzhou, China",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2025.emnlp-main.1386/",
doi = "10.18653/v1/2025.emnlp-main.1386",
pages = "27232--27249"
}