AI product glossary
Key terms from this course, in plain English.
- A/B test
- An experiment comparing outcomes for users with and without a feature (or between versions).
- Acceptance criteria
- Measurable conditions a feature must meet before launch, including thresholds and must-never conditions.
- Augmentation
- AI helping a person do a task better while the person stays in control.
- Automation
- AI completing a task on its own, with humans monitoring rather than approving each output.
- Data flywheel
- A loop where more usage produces more examples that improve the product, which attracts more usage.
- Deterministic
- Producing the same output every time for the same input, as traditional software does.
- Dogfooding
- Using your own product internally before releasing it to customers.
- Drift
- Changes in data, users or models over time that degrade an AI product's performance.
- Evaluation set (eval)
- Realistic inputs with expected outputs or grading rubrics, used to measure AI quality.
- Graceful failure
- Handling errors or uncertainty in a way that keeps users safe and able to continue.
- Guardrail
- A check or limit on inputs, outputs, actions or costs that keeps an AI feature within safe bounds.
- Human in the loop
- A design where people review, approve or can override AI outputs or actions.
- Implicit feedback
- Signals from user behaviour — accepting, editing, regenerating, abandoning — rather than explicit ratings.
- Kill switch
- A mechanism to quickly disable an AI feature or fall back to a non-AI path.
- Latency
- How long a user waits for a response.
- LLM as judge
- Using a language model with a rubric to grade outputs; must be validated against human judgement.
- Mental model
- A user's understanding of what a product can do and how it works.
- Must-never condition
- A launch criterion that allows zero occurrences of a particular failure.
- Probabilistic
- Producing outputs that are usually but not always correct, and that may vary for the same input.
- Staged rollout
- Releasing a feature to increasing shares of users while monitoring for problems.
- System card
- Documentation of an AI system's purpose, data, evaluation results, limitations and owner.
- Unit economics
- The cost and revenue per unit — per request, per user — that determine whether a feature is sustainable.
- Vibe check
- An informal early test of a prompt or model on real examples, useful for feasibility but not a formal evaluation.