Evaluating & iterating on prompts

Lesson 8 of 8 · 25 min · part of the AI Prompt Engineer certification

Prompt engineering is an empirical discipline. A prompt that looks great on three hand-picked examples can fail on 10% of real inputs. Professionals treat prompts like code: they test them.

• Build a test set: 20–100 realistic inputs, including edge cases and known failures.
• Define success: exact-match checks, rules (valid JSON? under 100 words?), rubrics, or an LLM-as-judge for subjective quality.
• Change one thing at a time so you know what caused an improvement or a regression.
• Version your prompts and rerun the test set on every change (regression testing).

Interactive exercise: Shipping a summarization prompt

Interactive exercise: Course review