To know whether a prompt change helped, run the old and the new version on the same few inputs, several times each, and judge both against a standard you wrote down before you looked. One good result after an edit tells you very little, because the same prompt can give a different answer on the next run. The method below needs no code and works for text, images and video.
- Write down what “better” means first. Once you have seen the new output, you will judge by whatever it happens to do well.
- Keep a fixed set of test inputs. The same ones for every version, including the awkward cases.
- Change one thing at a time. Two changes and a better result tell you nothing about which one worked.
- Run each version more than once. Judge the spread of results, not the best one. For images and video, hold the seed or generate several.
- Compare side by side, against your written standard. Hide which version is which if you can.
- Keep a log. Date, model, prompt version, result, decision. It is what lets you re-check when the model changes.
Why one run is not a test
The usual way to test a prompt is to edit it, run it once, and keep the edit if the answer looks better. The problem is that the answer might have looked better without the edit. OpenAI’s evaluation guide opens on exactly this point.OpenAI, Evaluation best practices, read at source 18 Sep 2026: “Generative AI is variable. Models sometimes produce different output from the same input, which makes traditional software testing methods insufficient for AI architectures.”
The same guide lists judging by feel as a mistake, under the heading “Vibe-based evals”. Anthropic puts the measuring before the prompting: its documentation treats a written definition of success, and a way to test against it, as the starting point of prompt engineering rather than an extra.Anthropic, Define success criteria and build evaluations, read at source 18 Sep 2026: “Building a successful LLM-based application starts with clearly defining your success criteria and then designing evaluations to measure performance against them. This cycle is central to prompt engineering.”
Both vendors write for developers testing an application at scale. The six steps below are the same idea cut down to what one person can do by hand.
A single better-looking run is not evidence. Something counts as a test only if it would also catch the change making things worse.
1 · Decide what “better” means before you look
Write two to five things a good output must do, in words you could check with a yes or a no: under a word limit, names the price, no invented quotes, the product is the subject of the image, the character’s face stays the same. Anthropic’s first rule for success criteria is to be specific.Anthropic, Define success criteria and build evaluations, read at source 18 Sep 2026: “Clearly define what you want to achieve.” OpenAI’s advice for human reviewers points the same way, even when scores are used.OpenAI, Evaluation best practices, read at source 18 Sep 2026: “Include a pass/fail threshold in addition to the numerical score”.
The order matters. Once you have seen the new version’s output, whatever it does well starts to look like the point. Writing the standard first is what stops the test from grading itself.Reasoning, September 2026 — this is the same rule as judging against the brief rather than the last version, on loops that improve and loops that degrade and the review loop.
If you cannot say in advance what would make the new version worse, you are not testing it, you are admiring it.
2 · Keep a small, fixed set of test inputs
Pick the inputs the prompt will actually face and keep them for every version: a few typical ones, at least one hard one, and one that is odd or messy. For an image prompt, the “inputs” are the variations you will really ask for, such as different products, settings or outfits. Anthropic asks for tests that “mirror your real-world task distribution”, and OpenAI names the mix.OpenAI, Evaluation best practices, read at source 18 Sep 2026: “Ensure your test data includes typical cases, edge cases, and adversarial cases.” Anthropic, Define success criteria and build evaluations, read at source 18 Sep 2026: “Design evals that mirror your real-world task distribution.”
Choose them before you test, and do not swap one out because the new version handles it badly. A test set picked after the results is the problem described on auditing vendor claims, done to yourself.
Be honest about size. Anthropic’s own principle favours many test cases over a few carefully graded ones.Anthropic, Define success criteria and build evaluations, read at source 18 Sep 2026: “More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals.” By hand, five to ten inputs is a workable start. A set that small will show a large, consistent difference and will miss a small one.Reasoning, September 2026 — five to ten is a working rule for testing by hand, not a measured threshold.
The test set is the ruler. Change it between versions and you are measuring with a different ruler each time.
3 · Change one thing at a time
This is the rule the site gives for the iterative workflow and for diagnosing a bad generation, and it matters more when testing. A “harmless” tidy-up counts as a change. Google’s prompt guide says rewording alone moves the output, and so does reordering.Google, Prompt design strategies (Gemini API), read at source 18 Sep 2026: “Using different words or phrasing in your prompts often yields different responses from the model even though they all mean the same thing.” and “The order of the content in the prompt can sometimes affect the response.”
Hold everything else still: the same model, the same settings, the same reference images. A model update is a change too.OpenAI, Prompt engineering, read at source 18 Sep 2026: “Even different snapshots of models within the same family could produce different results.” Do not lower the temperature to make a test steadier on Gemini 3; Google recommends leaving it alone, as covered on the temperature deprecation.Google, Prompt design strategies, read at source 18 Sep 2026: “we strongly recommend keeping them at their default values for Gemini 3.x models.”
Run each test in a fresh conversation. In a chat, the earlier turns are part of what the model reads, so the second version would be answering with the first version’s attempts in view.Reasoning, September 2026 — consistent with the advice to start fresh on loops.
Anything that differs between the two runs, apart from your one edit, is a second variable. A clean test has exactly one.
4 · Run each version more than once
Run every version on every input several times, the same number of times for both. Three runs each is a sensible floor by hand; the site’s iterative workflow already reads three to four variations before judging anything.Reasoning, September 2026 — a working rule, not a measured threshold. No vendor page read for this article gives a number of runs. Then look at the spread, not the best run. A version that is excellent once and poor twice is worse for real use than one that is good every time. Anthropic lists this as its own criterion.Anthropic, Define success criteria and build evaluations, read at source 18 Sep 2026, under “Consistency”: “If a user asks the same question twice, how important is it that they get semantically similar answers?”
Images and video. Here the random starting point is usually a number called the seed. Where a tool lets you set it, hold it fixed while you compare, so the edit is the only thing that moved. Google’s documentation for its own image and video models describes what the seed does.Google Cloud, Generate deterministic images (Imagen), read at source 18 Sep 2026: “A seed number is a number that you add to a request to make non-deterministic generated images deterministic.” Google Cloud, Generate videos from text prompts (Veo), read at source 18 Sep 2026: “Specifying a seed number with your request without changing other parameters guides the model to produce the same videos.” Two cautions. Seed behaviour differs between tools, and the site’s glossary already notes that some tools and hardware still introduce small differences. And one seed is one sample: an edit that helps on one seed can hurt on another, so repeat the comparison on two or three other seeds before you believe it. Where the tool hides the seed, generate several per version and compare the batches.
Judge a prompt by its typical output, not its best one. The best one is what you will remember; the typical one is what you will get.
5 · Compare side by side, against your written standard
Put the outputs for each input next to each other and go down your list from step 1: does each one pass or fail each point? The side-by-side view is for spotting differences. The verdict still comes from the written standard, not from which one you happen to prefer. If someone else can help, remove the labels so neither of you knows which version is which. OpenAI’s examples of human review include a blinded test.OpenAI, Evaluation best practices, read at source 18 Sep 2026, under Human evals: “create a randomized, blinded test”.
If you ask an AI to judge, know its habits. OpenAI names two that matter for a side-by-side comparison, so swap the order and run the judgement twice.OpenAI, Evaluation best practices, read at source 18 Sep 2026, challenges of LLM-as-a-judge: “Position bias (response order), verbosity bias (preferring longer responses)”. Ask it whether each output meets each point on your list, not which one is “better”; as on loops, a check leaves it less room to drift.
Mixed results are a result. If the new version wins on some inputs and loses on others, it is a trade, not an improvement, and you should say which inputs matter more before choosing.
Score each output against the list, not against the other output. A preference is not a pass.
6 · Keep a log, and know when to stop
One line per test: the date, the model and version, settings and seed, the prompt version, what changed, the score on each point, and what you decided. OpenAI’s tip is short.OpenAI, Evaluation best practices, read at source 18 Sep 2026: “Log as you develop so you can mine your logs for good eval cases.” The log is also what lets you re-run the same test when the model changes under you, which OpenAI recommends in its prompt guide.OpenAI, Prompt engineering, read at source 18 Sep 2026: “Building tests and evaluation suites that measure prompt behavior so you can monitor performance as you iterate, or when you change and upgrade model versions”.
A spreadsheet is enough, and it outlasts tools. OpenAI’s own hosted evals platform is being retired, per the same guide as of September 2026.OpenAI, Evaluation best practices, read at source 18 Sep 2026: “Evals will become read-only for existing users on October 31, 2026, and the platform is scheduled to shut down on November 30, 2026.”
Cap the rounds before you start: the iterative workflow uses three, and so does the iteration ladder on bad, good and great prompts. If the test keeps failing on the same point, the fix may not be in the prompt at all.Anthropic, Prompt engineering overview, read at source 18 Sep 2026: “Not every success criteria or failing eval is best solved by prompt engineering.” If two versions tie, keep the shorter one: it has fewer parts to break when the model changes.Reasoning, September 2026.
Without a log you can only remember that something worked. With one you can check whether it still does.
What this page could not verify
- How much outputs vary from run to run. None of the vendor pages read here gives a figure, and it depends on the model, the task and the settings, so this page gives none.
- How many inputs and runs are enough. Five to ten inputs and three runs are working rules for testing by hand, not measured thresholds. A small test catches large differences only.
- Seed behaviour in every tool. Only Google’s Imagen and Veo documentation was read on this. Other image and video tools may expose, hide or handle the seed differently.
Anthropic, Define success criteria and build evaluations · Anthropic, Prompt engineering overview · OpenAI, Evaluation best practices · OpenAI, Prompt engineering · Google, Prompt design strategies (Gemini API) · Google Cloud, Generate deterministic images · Google Cloud, Generate videos from text prompts. All read at source on 18 September 2026.