The Case for Model-Written Instructions

The AI Brief

— 3 min read
Share

Most professionals who use AI tools regularly hit the same wall: you know roughly what you want the model to do, but getting the wording right feels like guesswork. One phrasing works okay. You tweak it, something else breaks. A colleague suggests something different. Twenty minutes later you have a prompt that kind of works, and you're not sure why.

This isn't a niche developer problem. Anyone building a repeatable AI workflow — summarising customer feedback, drafting structured reports, classifying support tickets — runs into it. Manual tuning is slow, inconsistent, and relies on intuition most people don't have time to develop.

What APE Is

Automatic Prompt Engineer (APE) is a technique that uses a language model to generate, evaluate, and select its own instructions. Instead of asking a human to craft the perfect prompt from scratch, APE takes a small set of examples showing what inputs should produce what outputs, then asks the model to reverse-engineer the instruction that best explains them. It generates dozens of candidate instructions, scores each one against a test set, and returns the highest-performing version. An optional refinement step creates variations of the best candidates and scores those too.

Before and After

Before — manual prompt for classifying customer support tickets:

"Read the following customer message and categorise it."

Vague, inconsistent. The model interprets "categorise" differently each time, and results diverge across ticket types.

After — using APE's approach:

Provide the model with five examples of tickets and their correct categories. Feed them into the APE forward-generation template:

"I gave a friend an instruction and five inputs. The friend read the instruction and wrote an output for every one of the inputs. Here are the input-output pairs:
Input: [ticket text] Output: [category]
...
The instruction was <COMPLETE>"

APE generates candidate instructions like: "Read the customer message carefully and assign it to exactly one of the following support categories based on the primary issue described." It then scores each candidate against held-out examples and returns the best performer.

The resulting instruction is more specific, more consistent, and wasn't written by you. It was inferred from what good outputs actually look like.

When to Use It

  • Building a repeatable AI workflow where consistency matters. Processing hundreds of documents, emails, or records with the same prompt means a small improvement in instruction quality compounds across every run. APE's scored selection finds phrasings that hold up across varied inputs, not just the examples you had in mind when you wrote the prompt.

  • When you have examples of good outputs but can't articulate the rule. Experienced professionals often recognise a good summary or a correct classification without being able to write down the exact criteria. Show APE five examples of what "right" looks like and let it infer the instruction.

  • Improving chain-of-thought reasoning in multi-step tasks. APE research tested whether the widely-used phrase "Let's think step by step" could be improved. The model-discovered alternative — "Let's work this out in a step by step way to be sure we have the right answer" — outperformed it on two maths reasoning benchmarks: 82.0% versus 78.7% on MultiArith, and 43.0% versus 40.7% on GSM8K. Standard reasoning trigger phrases are candidates for the same treatment.

  • Deploying AI tools for teams where prompt quality varies. When multiple people use the same AI tool with different prompts, results diverge. APE produces a principled, evaluated instruction you can standardise across a team, rather than relying on whoever happened to write the original version.

  • Steering model behaviour toward specific qualities. In the research, APE improved model truthfulness on a question-answering benchmark, achieving over 40% true-and-informative responses compared to around 30% with a human-written instruction. The same logic applies to any quality dimension — conciseness, formality, specificity — where you can define what "good" looks like through examples.

Getting Started

APE doesn't require programming beyond basic API access. Start with a task you already have labelled examples for. The Instruction Induction experiments in the research ran with as few as five input-output pairs, so the entry point is low. Run the forward-generation template through any capable instruction-tuned model, generate 20 or more candidate instructions, and score them against a small held-out set you keep back for evaluation. That held-out set is the part that requires deliberate effort — without it, you're selecting instructions blind.

APE treats prompt engineering as a problem to be solved methodically rather than intuitively. We cover related techniques here as the research behind them matures into something practically applicable.

Source: Zhou, Muresanu, Han, Paster, Pitis, Chan, Ba (2022/2023). Large Language Models Are Human-Level Prompt Engineers. arXiv, submitted November 2022, revised March 2023.

Read more