Aug 1, 2026: The AI-Tells Review Pass

A registry of constructions that identify machine-written prose,
and an assessment of what it can and cannot measure.

Narracode is a harness for literary writing: a single Markdown file that instructs a language model how to draft, revise and record state while writing fiction.

On 30 July 2026 the harness gained a review pass that scans a completed draft for constructions characteristic of unedited model output. This report describes the pass, the registry it reads, the measurements taken so far, and the limits of the method.

A companion report, Suggestions for Improving Narracode, proposes extensions.

1. The problem the pass addresses

A language model produces, approximately, the most probable continuation of its input. Prose generated this way carries regularities that a reader can identify as machine-written even when no individual sentence is defective. These regularities are not errors of grammar or fact. They are frequencies: ordinary constructions used more often, or less often, than human writers use them.

The harness already held two related files. master_poetics.MD records what a project's voice is intended to be. master_phrases_to_avoid.md records clauses cut from a specific story, Dissolution, on the grounds that they editorialise. The tell registry addresses a third category: constructions that are not wrong and that a reader nonetheless registers as synthetic.

1.1 How the pass runs

The pass operates on spans — contiguous stretches of text, usually a phrase or a single sentence. For each located instance it emits one line:

"quoted span"  ·  CLASS  ·  remedy

The remedy is a single word, a shorter span, or a deletion. The pass does not rewrite the passage, does not restate the prose back to the author, and does not assign a score. Its output is a list the author accepts or rejects line by line, written to critiques/tells-[draft-name].md. A construction that is doing necessary work in the text remains. The pass is an audit, not a set of prohibitions.

2. The registry

master_ai_tells.md holds seventeen classes. Classes 1 to 3 and 11 to 17 were identified from a single editing pass by the author over §1 of the story Interim Edge on 30 July 2026. The remainder were carried in from earlier projects.

3. Is there a measurable signature?

Yes, and it is narrower than the claim that models write badly. The published findings fall into three layers, in ascending order of usefulness for this application.

3.1 Lexical

The most studied and the least applicable. Kobak et al. measured vocabulary shift in biomedical abstracts and found affected vocabulary in at least 13.5% of 2024 abstracts, rising to approximately 40% in some subcorpora. This establishes that the signature is real and measurable at corpus scale. It does not transfer to literary prose, because the words concerned — delve and its family — do not occur in this corpus.

3.2 Punctuational and structural

The relevant finding concerns the em-dash. The Last Fingerprint: How Markdown Training Shapes LLM Prose (2026) argues that over-use results from Markdown formatting conventions carried into prose during training. Measured rates vary between models from 0.0 to 9.1 occurrences per thousand words. Because the rate is a property of a particular model's fine-tuning rather than of language models in general, the correct registry entry is a target rate, not a prohibition. This is why class 10 is written as a threshold.

Three drafts of Interim Edge were measured:

Two observations. The em-dash rate is mid-range rather than extreme, which corrects an estimate made before the count was run. The the way X rate is the figure that moved, and it moved because the author deleted one instance and the model generalised from that deletion to the rest of the draft. This sequence — one human cut, one generalisation, a measurable change in rate — is the method the registry implements.

The measurement that is still missing is a fourth column: the same two rates computed over several thousand words of the human authors named in POETICS.md. Without it, 3.7 per thousand is known to be lower than 5.6 but is not known to be correct. §7.3 proposes taking this measurement.

3.3 Rhetorical

The most actionable layer, because the deviation runs in both directions. A Counterfactual LLM-based Framework for Measuring Rhetorical Style (2025) reports that model output over-produces the three-part list — independently arrived at as registry class 9 — and under-produces erotema, the rhetorical question. Saying More Than They Know (2026) reports that models produce genuine hesitancy markers at reasonable rates but performed hesitancy markers at approximately twice the human rate. The consequences of the third finding are addressed in §6.

4. Why this is not a detector

Machine-text detection uses three families of method, all with documented failure modes.

All three answer the question was a model involved in producing this text? For this project the answer is known and is yes. The question the harness needs answered is different and easier: does this draft carry the marks of unedited model default? The setting supplies the generator, the prompt, the pre-edit draft, the post-edit draft, and one consistent human judge. That is a supervised problem with complete provenance and a single annotator. Detection research has none of these conditions and is attempting a harder problem.

5. The data structure

One registry entry per class, in machine-readable form:

- id: T011
  class: analogy-simile-x-the-way-y
  first_seen: 2026-07-30
  source: scan            # scan | jhave | web:<url>
  severity: 4             # 1-5, strength of the machine-written impression
  detector:
    type: regex
    pattern: '\b(the way (you|a|an|one)\b)'
    unit: per_1000_words
  target_rate: 0.4        # measured on human corpus, never zero
  observed:
    - {work: interim-edge, draft: 7-condensed, rate: 1.3, date: 2026-07-30}
  status: live            # live | retired | contested
  survival: 0.71          # proportion of flagged spans the author accepted
  instances:
    - {work: interim-edge, span: "the way a seam is after strain",
       remedy: CUT, accepted: true}

Three fields carry the design.

• detector — without a machine-checkable rule the registry is a style guide. With one, a scan produces counts, and counts are comparable between drafts and against human corpora.
• target_rate — measured on human writing, never zero. A target of zero would remove the construction entirely; human literary prose contains all seventeen at non-zero rates. The relevant question for the three-part list is not whether an author uses it but at what rate.
• survival — the proportion of flagged spans the author accepted. This is the calibration signal. A class with survival below approximately 0.3 is not identifying a machine artefact; it is encoding a preference the author does not hold, and is marked contested and then retired. Without this field the registry can only accumulate.

6. Prioritisation and decay

Tells have a limited useful life, for two reasons: models change between generations, and a tell that becomes publicly documented is subsequently trained against. The delve family is close to unusable as a signal for this reason.

Recency is therefore a decay term rather than a priority in itself:

priority = severity × liveness × (1 − survival_penalty)
liveness = exp(−months_since_confirmed / 9)

The decay clock resets on confirmation, not on entry. A class re-confirmed by a recent cut is current regardless of when it was first recorded. A class not confirmed for a year is archived rather than deleted, in case a later model reintroduces it.

6.1 The remedies require the same treatment as the tells

The finding on performed hesitancy (§3.3) applies directly to the registry's own corrections. The remedy specified for class 12 — the narration proposing a figure and rejecting it within the sentence, as in (panic? no, calm.) — is structurally an instance of performed hesitancy. Used once it reads as a mind in the act of reconsidering. Used six times in a single work it becomes the most identifiable machine signature in the text, and a worse one than the construction it replaced.

Every device adopted as a correction therefore receives a detector and a frequency cap on the day it is adopted. Five devices were adopted from the §1 edit of Interim Edge:

The general principle: a correction applied consistently becomes as regular as the construction it replaced. Caps are set at adoption rather than after the device has spread through the corpus.

7. Proposed changes

Ordered by expected effect on the prose.

7.1 Move constraints from post-draft to pre-draft

The largest expected effect. post_draft_tell_scan operates after composition. Removing a construction after the fact leaves the surrounding sentence built around its absence, and the underlying habit unchanged. Injecting the highest-priority live classes into the pre_draft hook as generation constraints causes the sentences to arrive in a different form. It also costs less, because it replaces a scan and a revision with a constraint.

7.2 Add a requirement list

The change to make first if only one were possible. The rhetorical findings show that forms are under-produced as reliably as they are over-produced. Erotema is the documented case. Others worth measuring against the human corpus: sentence-initial conjunctions, anacoluthon (a sentence that abandons its own syntax), unresolved deixis (that, it, with no antecedent supplied), and the flat repetition human writers use where models substitute variation.

7.3 Anchor every target rate on named human corpora

POETICS.md names reference authors — Williams, Knausgaard, Ōe, Butler, Blanchot. Running the same detectors over several thousand words of each converts the target rates from assertions into measurements. Interim Edge runs em-dash at 5.6 per 1,000; Knausgaard runs 2.1 is a statement that can be checked. Too many em-dashes is not.

7.4 Register the remedies

Implemented for the five devices listed in §6.1. The requirement is that it continue: any device adopted as a fix receives a detector and a cap at adoption.

7.5 Store the aligned spans, not the interpretation

edit-observations.md currently records the model's reading of the author's edits. The durable record is the aligned pair — before → after → derived rule — because a summary discards the specific wording, which is the evidence. This also makes the corpus described in §8 directly usable rather than something a later model must reconstruct by re-reading the prose.

7.6 Scan against two corpora, not one

Scan each draft against the tell registry and, separately, against the last three published stories, flagging constructions that recur across projects. Repetition across separate works is not observable within a single draft. Two near-repetitions between Open Loops and The Chute were caught by hand; manual checking will not scale as the corpus grows.

8. The benchmark corpus

The versions/ directories constitute a labelled paired corpus that was not created for that purpose. For each story they hold a pre-edit model draft and a post-edit human draft of the same content, which removes the confound that affects corpora assembled from separate human and machine texts. Twenty-five published stories are available. Thirteen distinct edit operations were extracted from the first pair alone.

Two measurements, both inexpensive:

• Tell density per 1,000 words, by class, by draft. Indicates whether composition is improving or whether the tells are simply being removed afterwards.
• Survival rate. Indicates whether the registry is correct.

The first measures the prose. The second measures the registry. Only the second prevents the registry from becoming a fixed set of preferences that is never tested.

9. Open questions

• Whose idiom is the target? The registry measures distance from model default. It does not distinguish the author's voice from the property of merely not being machine-written. Only the author's acceptances define that direction, which is why the survival-rate field is essential.
• Does the registry generalise across projects? The preference for the active incorrect noun over the accurate inert one may be a fact about Interim Edge or a fact about this author. Two further projects will establish which.
• What is the maximum workable constraint density? Beyond some density, composition becomes constraint satisfaction. This limit should be located deliberately, in work not intended for publication, rather than encountered in a story that matters.

10. Assessment

Four decisions in this design are sound and should be retained: non-zero target rates, survival rate as a calibration signal, caps applied to the remedies, and storage of aligned spans rather than summaries.

Two limits are structural rather than incidental. The pass operates on spans, so it cannot detect failures that occur at the level of the story — a character who never acts against expectation, an ending that follows as the most probable continuation. And per-device caps constrain each device's frequency without constraining the combination, so the five adopted devices can each remain below cap while appearing together often enough to constitute a new regularity. The companion report, Suggestions for Improving Narracode, addresses both.

Sources

• The Last Fingerprint: How Markdown Training Shapes LLM Prose, 2026. arXiv:2603.27006. Em-dash rates of 0.0–9.1 per 1,000 words between models; basis for treating class 10 as a target rate.
• Mitchell, E. et al., 2023. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. ICML 2023.
• A Counterfactual LLM-based Framework for Measuring Rhetorical Style, 2025. arXiv:2512.19908. Over-production of the three-part list; suppression of erotema.
• Saying More Than They Know: Quantifying Epistemic-Rhetorical Miscalibration in LLMs, 2026. arXiv:2604.19768. Performed hesitancy at approximately twice the human rate.
• Voice Under Revision: LLMs and the Normalization of Personal Narrative, 2026. arXiv:2604.22142.
• Stylometry Recognizes Human and LLM-Generated Texts in Short Samples, 2025.
• T-Detect: Tail-Aware Statistical Normalization for Robust Detection of Adversarial Machine-Generated Text, 2025. arXiv:2507.23577.
• Enhancing the Robustness of Fast-DetectGPT Against Paraphrase Attacks, 2025.

Project documents

• master_ai_tells.md — the registry: seventeen classes, thresholds, and remedy caps.
• plans/2026-07-30_ai-tells-benchmark.md — benchmark design (Claude Opus 5, 30 July 2026), the source of §3 to §9 above.
• master_phrases_to_avoid.md — clauses cut from Dissolution.
• narracode.md — the harness specification, including the post_draft_tell_scan hook.
• Suggestions for Improving Narracode, 3 August 2026 — proposed extensions.
• Why Fugu-style orchestration will not improve Narracode, 28 June 2026.

All works and media on Glia.ca by David Jhave Johnston is licensed under CC BY-NC-SA 4.0