Your documentation is a fossil

Every command a tutorial tells you to run also runs headless in CI. 80 checks, on every push and once a week.

Every tutorial you have ever followed was correct on the day it was written. That is the problem. Documentation does not fail loudly, it fails quietly, months later, in front of somebody who assumes the mistake is theirs.

I am writing a tutorial series that teaches AWS services against a local emulator. The emulator ships new releases. AWS changes. My commands will break, and I will not be watching when they do.

So I made a rule before writing a single tutorial: every command a tutorial tells you to run must also run headless as a test. Not a similar command. The same one.

Why the usual approach does not work

The normal way to keep docs correct is to re-read them occasionally. This fails for a reason that has nothing to do with discipline: you cannot proofread your way to noticing that an API response shape changed. The prose still reads fine. It is just no longer true.

The shape of the harness

Each tutorial carries a verify.sh that executes its documented commands against a freshly started emulator and asserts on the results.

# Every assertion below mirrors a command the tutorial tells the reader to run.
# If the tutorial text and this file drift apart, CI is the thing that notices.

aws --endpoint-url "$ENDPOINT" s3 mb "s3://$BUCKET"
aws --endpoint-url "$ENDPOINT" s3api put-object --bucket "$BUCKET" --key hello.txt

# the tutorial claims versioning is off by default: prove it
status=$(aws --endpoint-url "$ENDPOINT" s3api get-bucket-versioning \
          --bucket "$BUCKET" --query 'Status' --output text)
[ "$status" = "None" ] || fail "versioning default changed: got '$status'"

The important detail is the assertion on the last line. It is not checking that the command succeeded. It is checking that the specific claim the tutorial makes in prose is still true.

The weekly run matters more than the push run

Running on every push is obvious and catches my own mistakes. The schedule is the one that earns its keep.

My tutorials do not break because I changed something. They break because somebody else shipped a release while I was not looking. A push-triggered pipeline never fires in that scenario, because there is no push. The repo sits there, perfectly green, and completely wrong.

on:
  push:
  schedule:
    - cron: "0 6 * * 1"   # Monday morning: catches the world changing under me

What this costs, honestly

• Writing a tutorial takes noticeably longer, because every claim has to be phrased in a way that can be asserted on. That turns out to be good for the prose too. Vague claims are hard to test because they are vague.
• The suite needs a real emulator running, so CI is slower than a unit test run. Worth it.
• Some things genuinely cannot be asserted, like whether an explanation is clear. Those still need a human. The harness protects facts, not teaching.

The general version of this idea

This is not really about tutorials. It applies to any prose that makes checkable claims about a system: onboarding guides, runbooks, README quickstarts, the setup section nobody has run since 2023.

If a document tells a human to run a command, a machine should be running that command on a schedule. Otherwise you do not have documentation, you have a historical record of something that used to work.