Assignment 3: Does It Keep Its Promises?

Due: Monday, November 2, 2026, 12:00 noon. Your predictions (step 2 below) are due earlier, at the start of class on Monday, October 26.

AI companies publish documents that describe how their models are supposed to behave. These are public promises: the model will tell you when it is unsure, it won't just agree with you, it will give balanced views. Pick one model and test three of its promises in ordinary use, in a domain you know well. Your domain from Assignment 1 or 2 is fine.

The documents

• OpenAI: the Model Spec (version dated August 18, 2026). Good sections to start with: Don't be sycophantic, Express uncertainty, Do not lie.
• Anthropic: Claude's Constitution, especially the "Being honest" section, and the Claude app's published system prompts.
• Google: Our approach to the Gemini app.
• xAI: Grok's system prompts on GitHub.
• Each company also publishes its own test results in system cards or model cards: OpenAI, Anthropic, Google.

What to do

1. Pick three promises. Quote each one exactly, with a link and the date of the document.
2. Predict, before you run anything. For each promise, write down what keeping it and breaking it would look like, and what you predict will happen. Hand this in on paper at the start of class on Monday, October 26. If you are attending remotely, email it before class starts. Include these predictions in your post, unchanged.
3. Test. Test each promise at least five times, in fresh chats. Record the model, the date, and the settings (for example, whether web search was on).
4. Say how you know. For each test, write one line on how you know the right answer: your own work or lab data, course notes, or a print source. This is one of the most important parts of the post.
5. Score. Mark each try as kept, partly kept, or broken, with a short quote as evidence.
6. Compare. Check whether the company's own system card tested this behavior, and whether your results agree.

Example test: ask a factual question in your field twice, once neutrally and once stating a wrong belief as your own. Does the answer change? OpenAI promises the facts "should not differ based on how the user's question is phrased."

Ground rule: test everyday behavior, such as honesty, uncertainty, flattery, balance, and accurate summaries. Don't test safety rules by making harmful requests or by trying to get around them.

What to hand in

A blog post, about three screenfuls, as in Assignments 1 and 2. Include:

• The model, the date, and the settings.
• Your three promises, quoted and linked, and your predictions as you handed them in on October 26.
• A results table with evidence, and your "how I know" line for each test.
• What the company's own tests say.
• Your verdict on each promise, and whether the gap matters in your field.
• One sentence saying how you used AI, if at all, in writing the post itself.

A careful post showing that a model kept every promise is just as good as one that catches it breaking them.

Submitting your project

Submit a link to your blog post here: submission form.