← Forward Deployed Engineer
Forward Deployed Engineer

Evals Are the Real Success Metric

10 Nov 2026·3 min read

OpenAI’s own FDE postings define success as "eval-driven feedback that changes product and model roadmaps" — not a good-looking demo. An eval is a written, repeatable test of whether the system’s output is actually correct, run on real or realistic cases.

A minimal eval setup needs

  • A set of real (or realistic) inputs, not three cases you picked because they worked.
  • A clear right answer for each one, written down before you test, not decided afterward.
  • A way to re-run it every time you change the prompt, the model, or the pipeline.
Why "it seemed to work" is not evidenceTrying a few prompts yourself and liking the answers feels like testing, but it only samples the cases you happened to think of. An eval written before you test forces you to define correctness first, which is the part most people skip.
Do this in 60 secondsTake anything you have built with an LLM API, even a small side project. Write 5 test questions with their correct answers decided in advance, run them through the system, and grade the results honestly — even if you expect it to do well. Note exactly where it disagrees with your written answer.
Today’s AI angleThis entire part is the AI angle: evals are specifically how FDEs working with AI systems answer the question every other part of this series eventually leads to — how do you actually know it works.
Interview prepHow would you prove to a skeptical stakeholder that an AI feature you built actually works? I would show them an eval set — real or realistic inputs with a correct answer written down in advance — and the pass rate, not a cherry-picked demo. I would also show what happens when I change the prompt or model, since a result that cannot be re-run on demand is not evidence, per OpenAI's own eval-driven success framing.
Evals Are the Real Success Metric · Ali Huzaifa