I have a skill in my coding agent setup called careful. It’s supposed to be a safety net: say something like “be careful, this is production” and it locks out destructive commands for the rest of the session, rm -rf, DROP TABLE, force-push, kubectl delete, the usual list. It’s the kind of thing you build once, feel good about, and never look at again.

Last week I was fixing a broken database backup on a client’s infrastructure. Real production, real stakes. I said the Polish equivalent of “remember, you’re touching production” three or four times over the course of the session. The skill never fired. Not once. It turned out the trigger list only recognized specific English phrasings, someone had written it down that way when the skill was built and nobody had touched it since. The gap between what the skill was supposed to do and what it actually did had been sitting there the whole time, and I only found it because I happened to type the wrong language into a live incident instead of a destructive command.

That’s the failure mode this post is about. Config for AI agents, the rules, skills, hooks, CLAUDE.md-style instructions, accumulates the same way code does. It grows in small increments, gets copy-pasted between projects, picks up assumptions nobody wrote down, and rots exactly the way undocumented code rots. Except code has a compiler and a test suite that scream at you when a change breaks something. Agent config has none of that. It fails silently until the one day it matters.

What an ISA actually is

An Ideal State Artifact is a single markdown file per project or setup, with three parts.

Goal. One paragraph. What is this thing actually for. Not the feature list, the purpose.

Ideal end state. A handful of bullets describing what’s observably true when the thing is working well. Not aspirations, observable facts.

Testable claims. A numbered list where every claim names a concrete check instead of a vague hope. Not “the safety skill works” but something you can actually go verify:

1
2
3
4

3. When `careful` fires, the response includes a visible confirmation
   line before any tool call. Check: grep the last five sessions'
   transcripts for the trigger phrase, confirm each one is followed
   by the confirmation string within the same turn.

Claims accumulate over time. When the setup changes, you update the claim in place, you don’t leave a stale one sitting next to a new one and hope future-you remembers which is current. A claim that’s gone vague again gets sharpened, not deleted.

The whole thing works because most people cannot cold-write a spec for what their agent setup actually guarantees. The real requirements live as tacit judgment, scattered across months of “oh yeah, it’s supposed to also do X” moments, not as a document anyone sat down and wrote. Getting that judgment out of your head and onto a page that can be checked is the entire value proposition. Nothing about an ISA enforces itself. It’s a discipline, same as writing tests is a discipline. If you don’t run the interview and don’t update the file when things change, you have a stale document that lies to you with more confidence than no document at all.

The interview

The way to write one isn’t to sit down and try to list everything at once. That produces a checklist of things you already know, which isn’t where the value is. Instead it’s a short structured interview: one question at a time, each one adapting to the previous answer, stopping once the claims are concrete enough to check rather than once some fixed list of questions is exhausted.

Running that interview against the careful skill this week surfaced three claims I would not have written by staring at the skill’s source code:

  1. The trigger list has to include the phrasings I actually use, across languages, not the phrasings someone happened to write down when the skill was built.
  2. When the skill fires, the response has to show a visible confirmation. “I’ll notice if it silently doesn’t fire” turned out to be false, tested against reality, not assumption. The way I actually know it’s on is seeing the confirmation, not the absence of a problem.
  3. The skill’s documentation has to match its real technical effect. It turns out careful also disables direct file-write tools for the rest of the session, a side effect nobody had written down anywhere, and it surprised the person maintaining it, which was me.

None of those three come from reading the code. They come from someone naming out loud what “I’ll notice if this breaks” and “this works well” concretely mean, then checking whether that’s still true.

A fourth thing came up that the interview did not resolve: I want this mode to persist automatically for a given project across sessions instead of needing to be re-triggered every time I open a new one. That needs a structurally different mechanism, a project-scoped hook instead of a session-scoped skill trigger, and it’s not something you patch on the spot. The right move was to write it down explicitly as a known, unsolved gap rather than gloss over it or pretend the interview produced a complete answer. That distinction, this is answered versus this is a real gap and we know it, is part of what makes the document trustworthy later. A file full of confident claims that quietly skip the hard parts is worse than one that admits where it’s thin.

Why this beats reading the code

You could argue you don’t need any of this, just read the skill’s source when you’re unsure. I tried that first, actually, before the interview. Reading careful’s code tells you the current trigger list and the current blocked-command list. It does not tell you that the trigger list is missing your own actual phrasing, because the code has no opinion on what your actual phrasing is. It does not tell you that “I’ll notice if it silently fails” is an assumption rather than a fact, because that assumption lives in your head, not in the file. Code review catches bugs in the implementation. It does not catch a gap between what you assume the thing does and what you actually need it to do, because that gap isn’t a bug from the code’s point of view. It’s working exactly as written. The problem is that what’s written was never checked against what you meant.

What to do with this

Write one ISA per setup that matters to you, not one giant file for everything. Keep the testable claims genuinely checkable, if a claim can’t be turned into a grep, a read, or a specific test, it’s not a claim yet, it’s still an aspiration. Re-run the interview when something about the setup changes, not on a calendar. And be honest in the file about what’s a solved claim versus a filed-away gap. The gap you wrote down is fine. The gap you didn’t notice is the one that fires, or doesn’t fire, during the next incident.

If you’re using Claude Code, this maps directly onto the config layers most setups already have: CLAUDE.md instructions, skills, hooks, permission rules. None of them come with a test runner. An ISA is the closest thing to one, and it costs an afternoon of talking to yourself out loud instead of a production incident to find out where the assumptions broke.