Due: Friday, October 9, 2026, 12:00 noon.
In Assignment 1 you watched an agent work on a task you know well, and you asked how anyone would know whether it was right.
Many of you found the same thing: the agent was often confident, and sometimes confidently wrong. One agent said "not found" with 90% confidence about
documents that exist. Another gave a location radius that missed the true spot five times out of five. Others said "I verified this" when they had not.
This assignment asks you to go one step further, in one of two directions. You can measure how far to trust the agent,
or you can try to fix a failure you found.
Same project, or a new one
We encourage you to continue the task from your Assignment 1. You already know the domain, you already know where the agent struggled,
and you can build on what you found. Link to your first post and pick up where it left off.
You may also pick something new. If you do, the same rule applies as before: choose a task from a domain you know well enough to judge whether the agent is right,
and briefly explain the domain and the task at the start of your post.
Option A: Calibrate it
When an agent says it is 90% sure, is it right about 90% of the time? Find out for your task.
- Build a test set. Make 15–20 versions of your task where you already know the right answer. These could be more photos, more records to look up, more documents, more stores, or more questions from your field.
- Ask for an answer and a confidence. For each one, have the agent give its answer and how confident it is (0–100%). Use the same prompt every time and include it verbatim in your post.
- Run each one at least three times. Does the answer change between runs? Does the confidence change?
- Score it. Mark each answer right or wrong. Group the answers by stated confidence (for example, under 50%, 50–80%, over 80%) and report how often each group was actually right. A simple table or chart is enough.
- Decide. Is there a confidence level at which you would act on the agent's answer without checking? Or is its confidence not useful at all? What would you tell a colleague about using it?
Option B: Write the rule that fixes it
Take one specific failure, from your Assignment 1 or from a new task, and try to prevent it.
- Name the failure. What exactly went wrong, and why do you think it happened? Show the evidence.
- Write the fix. This can be a rule added to the prompt, a checklist the agent must follow, a required format for its answer, or a small script that checks its work. For example: "a page that will not load is not evidence that the document does not exist," or "for every requirement, quote the line on the employer's page that shows it is met."
- Rerun it. Run the original task again with the fix, plus at least three new cases the fix was not written for. Show the results before and after.
- Find what it broke. Did the fix cause a new problem, make the agent slower or more cautious than it should be, or fail on one of the new cases? Report at least one case where the fix did not help or made things worse.
- Decide. Does your fix make the agent trustworthy enough for this task? What would still need to be true before you let it run without you watching?
What to hand in
A blog post, about three screenfuls, as in Assignment 1. Include:
- Which option you chose, and a link to your Assignment 1 post if you are continuing it.
- Your exact prompts, verbatim.
- Real evidence: screenshots or transcript excerpts, and your table of results.
- How you checked the answers yourself.
- One sentence saying how you used AI, if at all, in writing the post itself.
A careful result that shows the agent cannot be trusted, or that your fix did not work, is just as good as one where everything went well.
The free agent options listed in Assignment 1 are all fine to use again.
Submitting your project
Submit a link to your blog post here: submission form.
← Back to the course page