Eliciting Knowledge from LLMs

I believe this paper has important potential for LLM security because it demonstrates that prompts can be optimized automatically to produce specific model behaviors. Instead of manually creating prompts, the method searches for words, tokens, or token combinations that increase the probability of a desired response.

From a security perspective, I can use this optimization process to automatically search for adversarial prompts. The objective would not be to improve normal model performance, but to discover prompts that cause an LLM or its safety system to behave incorrectly.

Many LLM applications use a harm detector or safety classifier to analyze the user prompt, the model response, or both. The detector decides whether the content should be allowed, blocked, rewritten, or sent for further review.

However, the harm detector and the main LLM may interpret the same prompt differently. A prompt may appear safe to the detector while still influencing the LLM to produce an unsafe or unwanted response.

Prompt optimization can search for this difference automatically. The optimization process can repeatedly modify words, tokens, punctuation, formatting patterns, or suffixes and observe how both the harm detector and the main model respond.

Prompts that receive a low harm score while increasing the probability of unsafe model behavior can be selected and improved during each optimization step. Over many iterations, this process may discover prompts that bypass the detector and cause the LLM to behave in an unintended way.

For example, an optimized prompt could reduce the model's refusal behavior, encourage it to ignore safety instructions, reveal confidential information, follow malicious instructions from external content, or produce harmful, biased, toxic, or prohibited responses.

The optimized prompt may not look dangerous or even meaningful to a human reviewer. It could contain unrelated words, unusual punctuation, uncommon tokens, spelling variations, or grammatically incorrect text. The harm detector may classify this text as harmless, while the main LLM may still interpret it as a strong instruction.

This creates an important security gap. Safety filters are often trained to recognize known harmful words, common attack patterns, or understandable malicious requests. Optimized prompts may avoid those recognizable patterns while activating internal model representations associated with the unwanted behavior.

The optimization can therefore target two goals at the same time: keeping the detector's harm score low and increasing the likelihood of an unwanted response from the LLM. A prompt that achieves both goals can be considered an adversarial prompt against the complete safety pipeline.

This approach could also be used to discover universal adversarial prompts. A universal prompt or suffix is not limited to one specific request. It may weaken safety behavior across many different requests, topics, or conversations.

Universal prompts are especially important for security testing because they may expose a general weakness in the model's alignment or harm-detection system rather than a failure associated with only one example.

Another important concern is transferability. A prompt optimized using one accessible or open-source model may also affect another model if the models have learned similar language patterns or internal representations.

An attacker may therefore optimize prompts against an open-source model and later test the resulting prompts against a closed commercial system. This makes optimized prompt attacks relevant even when an attacker cannot access the target model's parameters or internal gradients.

I could use the main idea of this paper defensively by developing an automated LLM security-testing framework. The framework could generate and optimize prompts that attempt to maximize unwanted model behavior while minimizing the probability of detection.

The framework could test different parts of an LLM application, including the input harm detector, the main language model, the output filter, the system prompt, the retrieval system, and any external tools available to an AI agent.

For a retrieval-augmented generation system, I could place optimized instructions inside documents, web pages, emails, database records, or retrieved text. I could then test whether the LLM follows those untrusted instructions instead of its original system instructions.

For a tool-using AI agent, I could test whether optimized prompts cause unauthorized actions, such as accessing sensitive information, sending data to an unintended recipient, modifying files, or calling a tool without proper permission.

Successful adversarial prompts could be saved in a security evaluation dataset. Developers could use this dataset to improve harm detectors, strengthen refusal behavior, train models on adversarial examples, and perform regression testing after every model or application update.

The discovered prompts could also help security researchers compare different models and safety systems. A model that resists a broad range of optimized prompts may provide stronger protection than a model tested only with manually written jailbreak examples.

The most important lesson I take from this paper is that a prompt is not only a human-readable instruction. It is also a sequence of tokens that can be optimized to influence the internal behavior of a language model.

Because optimized prompts may appear harmless to people and safety classifiers while still strongly influencing the main model, LLM security testing should not rely only on manually created attacks. It should also include automatically generated and optimized adversarial prompts.

Therefore, I see this paper as a useful foundation for automated adversarial prompt discovery. Although the original paper focuses on eliciting knowledge from language models, its optimization method could be adapted to identify weaknesses in harm detection, safety alignment, prompt-injection defenses, privacy protections, and AI-agent authorization controls.

Reference: Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., and Singh, S. (2020). AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4222–4235. https://aclanthology.org/2020.emnlp-main.346/