Working with AI, part 2: when an AI explains a rule or check from its name without reading what it says, and the habits that stop this

AI assistants will tell you what something means from its name alone. Mine recommended keeping a pass/fail check because the name sounded like a safety check. It hadn't read what the check tested. So before an AI recommends anything about a rule, a test or a spreadsheet column, make it quote the definition: the exact wording or formula. And when a check keeps failing while the thing it checks looks fine, question the check first.

This is part 2 of Working with AI, a short series on what goes wrong when you use Claude, ChatGPT or similar tools for real work. Part 1 was about the made-up names that crept into my AI's replies. This one is about a mistake those names helped to hide.

A column called "Overdue"

Say you give an AI assistant your invoice spreadsheet and ask why so many invoices are overdue. It gives you a confident answer about slow-paying customers and suggests sending reminders.

Then you look at the formula behind the "Overdue" column. It marks every invoice older than 30 days, paid or not, and most of those customers paid on time. The AI explained the column from its name without ever opening the formula.

What happened to me

This happened on the same robotics project as part 1, where one main assistant hands smaller jobs to its helper agents (separate copies of the AI). Each test we run comes with pass/fail checks, and a helper agent writes them when the test is set up.

One check had been written early, by a helper agent, for an earlier version of the setup. Its name sounded like a safety check. After the setup changed, it failed every time, while the thing it was meant to protect was fine. Helper agents kept reporting it, in lines shaped like this (text in brackets replaces project details):

Failing check 1, [check name] (failing since [change]): [count] above zero. It is not the only failing check.

When I asked the main assistant what to do, it recommended keeping the check with a small change:

Once the [limits] are [relaxed], counting them makes less sense. I'd count only the [one that sounds like safety], and treat it as the next fix.

It never opened the definition. It only read the definition when I asked why we needed that check at all. The check had been written for the old setup, and in the new one it could almost never pass.

I asked why it had recommended keeping it. Claude's answer, shortened:

I reasoned from the check's name without reading what it measured. And I carried the wrong reading through several reports without questioning why a safety check would keep failing while the actual [result] stayed [fine].

What we changed

First, the check itself. Once Claude had read the definition, the problem was plain. It was a check on the design, written when the setup had a single robot. With two robots it was always going to fail. We stopped treating it as pass/fail.

Then we went through every other check. Two helper agents read every pass/fail check on the project, one agent for each part of the system, using the six questions below. They could read the project but not change it. In both parts, most checks had at least one problem: about two thirds of them.

Next, we changed what counts as pass/fail. Only checks on the result that matters are pass/fail now: did any robots collide, and did the task succeed as often as with the previous version of the software. Everything else became a number we report and look at, not a bar a test has to clear.

Last, we added checks we had never had, the kind that published research in this field uses: whether the task succeeded, how often the planner (the part of the software that plans each robot's moves) failed to update its plan, and how long the robot's own computer takes to plan.

Why it happens

The name is easier to reach than the definition. The name is in the conversation. The definition sits in a file or a formula the assistant would have to go and open, so guessing the meaning from the name is the shortest path. People do this too, but an AI sounds just as sure whether it checked or not.

Later agents trust what earlier ones wrote. A helper agent writes a check for the situation in front of it. Later agents see it in reports, marked as failing, and treat it as something everyone agreed on. Nobody asks whether it still fits. The same goes for a rule a colleague left behind.

Made-up names hide the problem. Reports called the check by a short code name, which sounded official and said nothing about what it measured. A plain description would have shown the problem on the first read. This is the same habit from part 1, and it's why I now use a script that sends a Claude Code reply back when it uses made-up names.

What to do

1. Ask "where did you read that?" Whenever the AI tells you what something means, ask where that came from. If the answer is the name, it hasn't read the definition yet.
2. Make it quote the definition first. Before it recommends keeping, changing or dropping a rule, test, limit or column, have it quote the definition.
3. When a check keeps failing and the thing it checks looks fine, suspect the check. Ask the AI which is wrong, the check or the thing it checks, and why.
4. Treat a rule someone else wrote as a suggestion. Ask who wrote it, when, and for what situation, and whether that situation still holds.
5. Ask for checks to be described in plain words. Ask what each one measures and what counts as a pass. A code name tells you neither.

Six questions for your own checks

These are the questions the helper agents asked of every check. The same questions work on a spreadsheet's warning columns, a team's targets or an AI's test list:

1. Does it measure the result you care about, or something that stands in for it? Counting emails sent tells you little if what you want is replies.
2. Was it written for a different situation? A limit set for one person's workload may not fit a team of three.
3. Can it actually fail? A check that always passes, or only passes because a setting forces it to, tells you nothing.
4. Where does the number come from? "Under 2 hours" needs a reason. If nobody knows why 2, ask.
5. Does it measure what its name says? That's the "Overdue" column above.
6. Is something missing? Look at how others in your field judge the same work. If they all check something you don't, add it.

You can paste these into a chat and ask the AI to go through your checks one by one, quoting each definition as it goes.

Paste this

Add this to your custom instructions, project instructions or CLAUDE.md (the instructions file Claude Code reads every time it starts):

## Checks, rules and definitions
- Before recommending to keep, change or drop any check, test,
  rule or limit, open its definition and quote it. Say who
  wrote it, when, and for what situation, if you can tell.
- Don't guess what anything means from its name. Read its
  definition.
- If a check keeps failing while the thing it checks looks
  fine, question the check first. Say which you think is
  wrong, the check or the thing it checks, and why.
- A rule written by another AI, or earlier in this project,
  is a suggestion. Check it still fits before relying on it.
- Describe checks in plain words: what they measure and what
  counts as a pass.

To keep your project's made-up names out of the AI's rewrites as well, see how to add them to the avoid-ai-writing tool in part 1.

Next time an AI tells you what something means, ask where it read that.