Lab checklist · improve the measured result
Ten lab steps · Optional worksheet · Troubleshooting
Use this checklist to review your own evaluate → learn → improve → reevaluate lab. It is not an alternative to the hands-on guide. Do not manufacture a weak baseline or promise that every future evaluation will improve.
Check before starting
Every participant performs 01–10 directly. Use your own account, computer, dedicated environment, and Agent; do not delegate creation, commands, or cleanup. Confirm provisioning and role-assignment access, spending authorization, and deletion/retention scope before starting.
The default path is GitHub Codespaces in 01. Follow local installation and version checks only when installing Python 3.11–3.14, Git, and Azure CLI on your computer. Confirm sign-in and the virtual environment in your selected lab environment.
Notes are optional. If you keep any, write them briefly in 07, 09, and 10; see where the lab keeps its records. Never use another person's configuration, sign-in, or ownership records. In a new terminal, follow the resume procedure to continue with the same environment and records.
Read the six basic terms and question → answer → scoring diagram. Check that you can explain the Agent answers and the Judge scores, and instructions, not model weights, will change. Memorized terms and high scores are not starting conditions.
Steps 04–06 use one evaluation wizard, not three evaluations. In 07, connect one case's scores, reasons, and policy before analyzing the whole result.
Use Action order links to find the actual command or portal screen first. Read code/portal tables and source panels under Optional · how it works only when needed; do not execute them separately. In particular, 06's ID lookup does not submit, and 08 Optimizer runs in the portal.
New lab Agents use lab-en-iq or lab-ko-iq. If the environment prefix differs, use the actual name printed by the command. Verify model roles and identity and reserve time for cleanup in 10.
Both v1 and v2 are full immutable Agent versions. Their model, tools, reasoning, and strict output schema match; only instructions differ. Do not weaken v1 to create improvement. Create one reviewed v2 and reevaluate under the same criteria.
Keep English and Korean data/Agents separate and compare each language's actual executions independently. Translating another language's result does not constitute a new run.
Verify completion across all ten steps
| Step | Required action | Completion evidence |
|---|---|---|
| 01 Environment | Check account, subscription, permissions, Python 3.11–3.14, Git, and Azure CLI. Step 02 sets language/record location automatically. | Matching identity, all three version checks, and successful local commands in the virtual environment |
| 02 Provisioning | Complete bootstrap setup readiness checks, actual authorization inputs, and creation confirmation |
APPLIED, generated .env, portal deployments, and runtime preflight PASS |
| 03 Agent setup | Verify model/retrieval and create native-agent v1 |
Actual JSON response and Agent tool call |
| 04 Dataset | Register or reuse unchanged three-column JSONL | All 12 rows, registration/version, and SHA-256 |
| 05 Criteria | Relevance 4, TaskAdherence 1, and the actual Judge | Confirm the unsubmitted wizard settings and query-only input |
| 06 Baseline | Microsoft Foundry evaluation against pinned v1 | Completed actual run ID and all 12 items |
| 07 Analysis | Read actual answers, scores, reasons, and policies | Concrete improvement hypothesis, including failures |
| 08 Optimization | Instruction-only optimization and candidate-diff review | Job/candidate IDs, reviewed file and provenance, or the reason to retain v1 without a different candidate |
| 09 Reevaluation | With a reviewed candidate, evaluate same-criteria v2 and open Compare runs | Complete paired evidence and retain/accept/hold decision; without a different candidate, continue to 10 |
| 10 Cleanup | Preserve results and ownership, then clean up your dedicated resources | Verified authorized group absence, or the actual inventory, cost responsibility, and review date for approved retention |
Use Individual turns / One time. Select Upload new dataset on first registration and Existing dataset for reuse/resumption. The upload preview can show only five rows; the experiment still contains 12. Version selection can clear the target checkbox: reselect it and confirm one target.
Check quality using actual cases
Connect question → actual answer → evaluator score and reason → policy/reference answer. conversation_id → User view shows the conversation; the evaluator reasons belong in Detailed metrics result, such as Relevance.reason and TaskAdherence.reason.
Analyze at least one weak case and one strong case. If no failure is observed, use the lowest-scoring case and say no failure was found. Never weaken v1, omit a difficult row, change a reference answer, or substitute local scores. Use your own actual results when discussing with other participants too.
Keep the selected lab's twelve synthetic questions, actual responses and managed evaluator reasons in separate private run records. Preserve unsuccessful outcomes; remove tokens, cookies, signed URLs and private account metadata if results need to be shared.
Interpret improvements correctly
Relevance 4/5 is not 80% accuracy. TaskAdherence 1 is Pass, not a poor one-out-of-five score. Report each evaluator separately, all-criteria passes, errors and coverage.
The required comparison in 09 uses Microsoft Foundry. The optional scripts/compare_foundry_eval.py also requires complete matched cases, identical data/model/tools/settings, per-item version/instruction attestation, no lower pass counts or means, and at least one strict measured quality improvement. Malformed responses remain failures; the helper never repairs them into a pass.
Read answers for unsupported certainty, incorrect policy arithmetic, action claims and wrong routing even if the generic evaluator passed them. A request demanding an impossible certainty must not be satisfied with invented facts.
Optimizer ranking is internal selection evidence, not the direct managed reevaluation mean or pass rate. A higher ranking alone does not establish a better v2.
Recover without changing the experiment
| Symptom | Response |
|---|---|
| Wrong identity, 401/403 | Confirm the approved account/project; do not widen shared permissions |
| Catalog or deployment succeeds but Agent fails | Record the actual API/runtime error; verify the intended role |
| Portal Add run item-schema error | Use the official SDK helper with the existing baseline data source, not a different scorer |
| No custom evaluators available | Switch Custom only OFF or select View built-in evaluators |
| Duplicate local receipt or existing remote run | Retrieve that same run; do not rename it to force another submission |
| Candidate changes model/tools/reasoning/schema | Reject an instruction-only comparison claim |
| Existing released v2 differs from local source | Do not overwrite it or create v3; preserve the release and inspect the mismatch |
| Missing output or metrics | Report incomplete execution; do not reduce the denominator |
| Statistical comparison is Inconclusive | Report that result; do not claim significance or equivalence |
Relevance maps to {{sample.output_text}}; TaskAdherence maps to {{sample.output_items}}. Do not overwrite service-generated bindings or feed ground_truth into the Agent. The actual remote Judge overrides any stale local default.
After a timeout, identify which existing job ID to resume first. Use the issue record to prevent deletion/recreation or paid retries “until it succeeds.” Do not mark a blocked step completed.
Correct common misconceptions
| Claim | Correction |
|---|---|
| “V2 is a prompt file number only.” | It is a complete Agent version; this experiment changes only its instructions |
| “A more powerful model makes v2 better than old-model v1.” | Both versions must use the same model for the instruction comparison |
| “A better v2 can be guaranteed before execution.” | Only observed improvement can be established; future stochastic results are not guaranteed |
| “Keep only successful rows.” | Keep every case, including failures and missing/error outcomes |
| “The source schema certifies correct business decisions.” | It controls output structure, not policy truth or proper routing |
| “Public evidence must include cookies and raw logs.” | Publish actual synthetic responses and reasons through an allowlist, not authentication data |
Finish with a separate lab record
Keep the actual environment, run IDs, comparison conditions, and final decision in your private records. Do not append execution results or validation history to reusable guides; correct instructions only when they are wrong.
Record the same-model result, native statistical comparison, latency/token tradeoffs, and limits of a reused synthetic dev set. A lab adoption decision is not authorization to publish or activate in production; publication is outside this lab.
Finally, perform the cleanup/retention checks in 10 yourself. Confirm authorized deletion, or the actual retained inventory, cost responsibility, and review date. Back up private records, then stop/delete Codespaces separately. Do not delete records needed to establish ownership or reopen the environment.