Aug 3, 2026: Suggestions for Improving Narracode, a harness for literary writing

The measurement problem in literary AI harness engineering.
Quality is not computable.
Predictability is.

Narracode is a harness for literary writing created on May 10, 2026. It is currently a single markdown file capable of being used with any LLM.

Two recent branches of Narracode research — the Sakana/Fugu orchestration assessment and the AI-tells review pass — suggest potential improvements.

Narracode has assumed that because literary quality cannot be scored, nothing about literary prose can be measured. The instruments that actually discriminate between human and machine fiction measure predictability and are not yet capable of assessing literary quality. The harness should be rebuilt around that distinction.

1. Where the harness stands as of Aug 3, 2026

1.1 Can Sakana's Fugu Orchetrator Improve Narrative Prose? No.

The June assessment of Sakana AI's Fugu (2026; Sakana Fugu Technical Report) (full consideration here, and the underlying analysis in plans/2026-06-28_1353Z_fugu-orchestration-evaluation-and-routing-proposal.md, PR #8) concluded that orchestration offers Narracode nothing it does not already have. Adopting it would cost routing-table maintenance, context budget, and editorial judgment. That conclusion holds.

One finding deserves emphasis. The Fugu refusal rested on the claim that literary merit resists a reward function — that any learned coordination trained against a proxy metric would collapse the prose toward whatever the proxy rewards. Direct evidence has emerged in Sui et al., 2026 (Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling) that the obvious proxy — an LLM judging prose quality — is not merely uninformative at the top of the range but inverted (§3).

LLM evaluators consistently score zero-shot AI-generated fiction higher than published New Yorker short stories. Standard rubric-based LLM judges reward predictable, highly structured, cliché-dense prose and penalize subtle, subtextual human literary craft. Using an LLM to evaluate or route prose quality acts as an adversarial signal that degrades literary merit.

1.2 The AI-tells branch: a potentially strong improvement to harness

The tell-scan pass on main (master_ai_tells.md, the post_draft_tell_scan hook, and the design note at plans/2026-07-30_ai-tells-benchmark.md) proposes the following extensions of the published literature on style control in four ways:

• Non-zero target rates. Every class carries a rate drawn from human corpora rather than a ban. Zero is how prose gets sterilised, and the registry says so explicitly.
• Survival rate as calibration signal. The fraction of flagged spans the human annotator accepts decides whether a class is a tell or a mistaken preference. A registry without this field only accumulates. With it, the registry can be wrong and find out.
• Rate-capping the remedies. Each adopted fix acquires a detector and a cap on the day it is adopted, on the grounds that a remedy applied uniformly becomes a fingerprint. Goodhart's law identified from inside a working system, before the damage.
• Spans over interpretations. The instruction to store aligned before → after pairs rather than the model's reading of them is the correct archival call. It matches what the context engineering literature has since concluded about summarisation loss (§2.3).

2. Is the tell-scan architecture optimised?

No. The gaps are structural.

2.1 A span-level instrument cannot see a story-level failure

All seventeen registry classes are span-level, and the pass is explicitly scoped that way: quote the span, name the class, propose a remedy of one word or a cut. That scoping is what makes the pass disciplined and cheap. It also means the failures that most damage a piece of fiction are invisible to it.

A character who never surprises the reader, an ending that arrives because it was the most probable continuation, a scene that turns on the emotion the previous scene already named — none of these produce a flaggable span. The prose can pass every detector in the registry and the story remain median. The registry perfects the sentence and leaves the architecture untouched.

Widening the scan is not the answer — span-level discipline is why the pass works. A second instrument is missing. §5 proposes what it should measure.

2.2 The requirement list stops at syntax

The plan's observation — ban lists shift a tail, requirement lists move a distribution — is correct. The candidates listed (erotema, sentence-initial conjunctions, anacoluthon, unresolved deixis, flat repetition) are well chosen. They are also all surface forms. Literary fiction under-produces, relative to model default, constructions that are not only syntactic: unresolved ambivalence held past the point of comfort, refused catharsis, a scene that declines to turn, information withheld from the reader without signalling the withholding. A requirement list that stops at syntax moves the cadence and leaves the psychology where it is.

2.3 No eviction policy

The registry grows monotonically: every span cut twice enters it, plus a quarterly web sweep. Seventeen classes now. The plan proposes injecting the top-N live classes into pre_draft as generation constraints. That makes the selection of N the critical decision.

The Agentic Context Engineering framework (ACE; Zhang et al., 2025, Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models) names the two failure modes precisely. Brevity bias: the loss of domain-specific detail when a context is compressed toward a concise summary. Context collapse: the erosion of accumulated detail through iterative rewriting. ACE's answer is structured, itemised, incremental updates rather than wholesale regeneration — which is what a registry of discrete numbered classes with detectors already is. The harness arrived at the right data structure independently. It has not yet built the curation policy that structure implies.

The current priority formula weights severity by liveness, decaying from last confirmation. That ranks classes against each other. It does not ask what this scene needs. A domestic interior and a scene of institutional violence do not have the same live tells, and injecting the same top-N into both spends the budget badly. Priority should be conditioned on scene type, drawn from scene-ledger.md.

2.4 Anti-tell prose has its own signature

The registry calibrates toward one annotator's acceptances, which is correct for voice and creates a risk the plan partly anticipates. Optimising against a list of machine markers produces prose that is legibly not-machine, and legibly-not-machine is a style with its own detectable fingerprint. The five devices adopted from the first edit — parenthetical self-correction, noun-collision, subject-dropped verb chains, single-word fragment runs, colon-apposition — are already capped, which shows the risk is understood at the device level. The unhandled case is the aggregate. Each device can sit under its cap while the combination reads as mannerism. A per-device cap does not constrain a joint distribution.

The available check is the one the plan proposes in §7.6: scan each draft against the last three published stories, not only against the registry. Recurrence across projects is the signature a per-draft scan cannot see, and the only cheap detector of house style hardening into tic.

3. The measurement boundary is not where the harness assumed

Narracode's founding position is that literary quality resists scoring, and that any automated quality gate drives prose toward the metric. The evidence for the first claim is now much stronger. It arrives with a finding that changes what follows.

LitBench (2026; LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing) assembled a debiased, human-preference-annotated corpus for creative writing and benchmarked judges against it. The strongest off-the-shelf LLM judge reached 73% agreement with human preference; trained reward models reached 78%. That is a ceiling, not a floor, and it is measured against general reader preference rather than expert literary judgment.

The sharper result comes from Sui et al., 2026 (Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling): on EQ-Bench, the leading creative-writing benchmark, LLM judges rank zero-shot AI stories above New Yorker short stories. Not near. Above.

The same paper that demonstrates the inversion also demonstrates a measurement that works.

3.1 What is actually computable

Spoiler Alert's 100-Endings metric walks a story sentence by sentence. At each position a model is given only the text so far and asked to predict how the story ends, one hundred times. Tension is the rate at which those predictions fail to match what actually happens. The sentence-level curve yields an inflection rate, a measure of how often the curve reverses direction, which tracks twists and revelations. Unlike the rubric judges, this metric ranks New Yorker stories far above LLM output.

It works because it never asks whether the prose is good. It asks whether the next thing was foreseeable. Foreseeability is a property of a text and a predictor, computable without a human label, a reward model, or a rubric.

Every proposal below follows from this. Measure foreseeability; never score prose.

4. Nuancing the idiomatic cadence

The registry measures cadence as density: occurrences per thousand words. Density is the wrong unit for rhythm. A draft can satisfy every target rate in the registry and still read flat, because rhythm is sequential and a rate is order-blind. Three refinements, in order of expected effect.

4.1 Measure the contour, not the rate

Sentence length as an ordered series, per scene, rather than a mean or a count. The relevant statistics are variance, the size of the largest adjacent jump, and the distribution of run-lengths — how many short sentences cluster before a long one arrives. This is the burstiness signal the detection literature has used for years, repurposed. The harness does not need it to detect machine authorship, which is known. It needs it because burstiness is prosody, and prosody is the thing the current registry cannot see. A scene whose sentence-length series is smooth is flat regardless of how many em-dashes it contains.

4.2 Give each character a separate cadence budget

The strongest available refinement, and the cheapest. Measure the contour per focal character: dialogue attributed to them, and any passage in free indirect discourse focalised through them. Different characters should have measurably different sentence-length distributions. If they do not, the piece has one narrator wearing several names — the most common structural failure in generated fiction, and one that no reader articulates as a rhythm problem even while registering it as flatness.

This converts an aesthetic instruction into a verifiable check. Vera's interiority runs long and subordinated; the child's runs short and paratactic is a claim the harness can verify against the draft, and a target the Compositional agent can receive before drafting rather than face as audit afterwards.

4.3 Store the prosodic operation, not only the span

The plan already says: store aligned spans rather than interpretations. Extend the stored triple with the operation: did the edit fracture a sentence, fuse two, shorten within, or reorder? Thirteen operations were extracted from the first edit of Interim Edge. The ratio of fractures to fusions across the whole versions/ corpus is a cadence signature learnable at n=1, and a more durable artefact than any individual rule derived from it. Rules rot; the operation counts do not.

5. Psychologically astute, character-driven twists

This is where the harness has the most to gain. The existing architecture is close to supporting what is needed; no one has built it yet.

5.1 The diagnosis

A twist is psychologically astute when it is retroactively inevitable: the reader could have assembled it from what a character concealed, and did not. A twist is sensational when it arrives from outside the characters' knowledge and is justified afterwards. The difference is not the magnitude of the reversal. It is whether the material was planted in a mind the reader had access to.

Model default produces the second kind, because a twist invented at the moment of need can only draw on what is salient at that moment. The literature confirms the shape of the failure: a 2024 study (Are Large Language Models Capable of Generating Human-Level Narratives?) finds narrative planning with character intentionality and dramatic conflict remains the hard case even where causal soundness is achieved, and a 2026 study (Elias in the Lighthouse, Again? Diagnosing Low Diversity in LLM Stories) finds that even when generated stories are lexically distinct, their underlying plot elements stay highly redundant — 88.3% of stories across four current models contained one of eleven core words.

5.2 The proposal: a knowledge-asymmetry ledger

character-interiority.md currently records private states as a static list. The relevant research direction is temporal theory-of-mind representation — EvolvTrip (2025; EvolvTrip: Enhancing Literary Character Understanding with Temporal Theory-of-Mind Graphs) builds temporal ToM graphs for literary character understanding, and inner-thought reasoning benchmarks (2025; Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents) track what a character knows against what they disclose. Narracode should hold the same structure, per scene, as a table:

Four columns: the fact, each character's stance toward it (knows / suspects / conceals / misreads), how much access the reader has been given, and which prior scenes made it materially present as an object or a gesture rather than a statement.

A twist candidate is then a query rather than an invention:

candidates = facts where
    reader_access  is low
    AND some character stance is (conceals | misreads)
    AND materially_present_in >= 2 prior scenes

Everything that satisfies that query is a reversal already grounded in the existing text. The Compositional agent does not invent a turn; it receives the set of turns the accumulated state supports, and picks. Sensational twists fail the third condition by construction — they are unearned because the material was never planted. This is retrieval over existing structural memory, in the idiom the harness already uses, and it requires one new table rather than a new agent.

It also gives obligations.md a partner. Obligations record what the reader has been made to wait for. The asymmetry ledger records what the reader has not yet been told they should be waiting for. Those are different debts, and only the first is currently tracked.

5.3 The instrument: 100-Endings at scene boundaries

Run Spoiler Alert's metric at each scene boundary, using a model that is not the compositional one, given only the draft so far. The output is a foreseeability curve across the piece. Two readings matter. A curve that stays low means the ending is visible from early on — median collapse made legible. A twist that produces no inflection was foreseeable and is not functioning as a twist.

The cost is real but bounded, and it can run at act boundaries rather than every scene. What it provides is the first measurement in this harness that discriminates in the correct direction against published fiction.

6. Literary exploration that is not sensationalist

The pull toward sensationalism is not a failure of taste. It is inherited. Rigby et al., 2026 (The Storyteller in the Model: Narrative Pattern Inheritance, Escalation Dynamics, and Alignment Governance in LLMs) argues that the structural conventions of published human writing — archetypal roles, tension-and-resolution arcs — are absorbed during training and resurface as a systematic drift toward adversarial and rhetorically enticing behaviour over extended interaction. The escalation is a trained prior, and it strengthens the longer a session runs, which is exactly the condition under which Narracode operates.

6.1 A definition

Sensationalism is escalation without accumulation: incident magnitude rising while the reader's uncertainty about the outcome stays flat. Stated that way, it separates from the literary case along two independent axes — the volatility of what happens, and the unpredictability of where it is going.

The target cell is quiet incident with unforeseeable trajectory. Both axes are measurable: the vertical from a count of incident magnitude per scene, which scene-ledger.md is already positioned to hold; the horizontal from the 100-Endings curve. Neither requires a quality judgment.

This also addresses a risk in §5.3. Inflection rate should not be maximised. Sensational fiction inflects constantly. The target is a shape: high uncertainty about the ending, low volatility of event. As a single instruction to the harness: make the trajectory hard to predict without making the events larger.

6.2 The minimal change

Add a signed pressure field to each entry in scene-ledger.md, recording whether the scene raised, held, or lowered stakes, and add a POETICS-level commitment to a non-monotone pressure curve. A story in which every scene raises stakes is a thriller by construction. The commitment to let a scene lower them and deepen attention instead is a refusal in the sense the harness already understands, and refusals are the most effective element in POETICS.

7. The qualitative domain as a research problem

7.1 Difficulties

• The judge is inverted where it matters. Rubric-based LLM evaluation prefers model output to published fiction. Any pipeline that optimises such a judge moves away from the target while reporting improvement.
• n = 1. One annotator defines the direction. This is correct for a singular voice and it means the registry cannot distinguish not machine-made from the annotator's taste. The design note names this and it has no solution, only disclosure.
• Goodhart is structural, not incidental. Every adopted remedy becomes a fingerprint at volume. The registry handles this per-device and not yet in aggregate (§2.4).
• Constraint density has a floor. Past some density the prose becomes constraint satisfaction. Worth locating deliberately in a throwaway piece rather than discovering in a story that matters — the TODO already says so, and it remains unbuilt.
• Context is finite and the harness competes with the manuscript for it. Every instrument proposed here has a token cost, and context rot (2026; Diagnosing and Mitigating Context Rot in Long-horizon Search) is measurable across every major model as input length grows. Additions must displace, not accumulate.

7.2 Opportunities

• The paired corpus. The versions/ folders hold pre-edit and post-edit drafts of the same content across twenty-five stories, with complete provenance and a single consistent annotator. Detection and style-transfer research generally has none of these properties. Nobody set out to build it, which is the most interesting thing about it.
• Predictability is computable. The 100-Endings construction gives a label-free measurement that ranks correctly against published fiction. The first instrument that can tell the harness it is failing without a human reading the draft.
• The structural folder is already the right shape. Externalising state to inspectable files is what the context engineering literature has converged on — see agentic file-system abstraction (2025; Everything is Context: Agentic File System Abstraction for Context Engineering) and ACE's itemised playbooks. Narracode arrived there in May 2026 from literary reasoning rather than systems reasoning.
• Negative constraints remain underexplored. The finding that refusals outperform commitments in narrowing a generative move-space is not established anywhere in the published creative-AI literature. It is this project's most portable result and has never been written up as one.

7.3 The claim

Narracode is an n=1 longitudinal case study in harness engineering for a domain with no reward function. It has accumulated the artefacts such a study requires: a versioned corpus with single-annotator ground truth, a registry with survival statistics, and a documented history of the harness editing itself. The field has benchmarks for creative writing and no methodology for sustained single-author collaboration over months. That gap is the contribution.

8. What to build, in order

1. Per-character cadence contour (§4.2). Cheapest of the proposals, purely mechanical, and it catches the single most common structural failure in generated fiction. No new research required.
2. Knowledge-asymmetry ledger (§5.2). One table added to character-interiority.md, plus the candidate query. Turns twist generation into retrieval over state the harness already maintains.
3. Signed pressure field (§6.2). One field in scene-ledger.md and one POETICS commitment. Near-zero cost.
4. 100-Endings at act boundaries (§5.3). Highest value and highest cost. Run it once on a published story and once on a machine draft before committing to it — if the gap does not reproduce here, nothing downstream is worth building.
5. Scene-conditioned tell injection (§2.3). Fold into the existing pre-draft pass rather than treating it as separate.
6. Cross-project recurrence scan (§2.4). Already on the TODO. The only proposed check on house style hardening into mannerism.

Items 1 to 3 are file-format changes and could be done in an afternoon. Item 4 is the research bet. Items 5 and 6 are already committed to and unbuilt.

9. Closing

The Fugu assessment concluded that the harness improves by subtraction more often than by addition. This report proposes six additions. The distinction: orchestration adds machinery between the writer and the model; these additions are fields in files the harness already reads. None introduces a scoring pass, a synthesis step, or an autonomous chain. The one genuinely new capability is a measurement of foreseeability — the first time the harness could detect that a piece has become predictable without someone reading the draft.

Bibliography

Harness and context engineering

• Zhang, Q. et al. Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models. arXiv:2510.04618. Source of the brevity bias and context collapse framing, and of the Generator / Reflector / Curator division used in §2.3.
• Diagnosing and Mitigating Context Rot in Long-horizon Search. arXiv:2606.29718.
• Self-Compacting Language Model Agents. arXiv:2606.23525.
• ACM: Agentic Context Management for Long Horizon Tasks. arXiv:2607.23809.
• Everything is Context: Agentic File System Abstraction for Context Engineering. arXiv:2512.05470. Independent convergence on externalised file-based state.
• Fugu Team, Sakana AI. Sakana Fugu Technical Report. arXiv:2606.21228. Assessed for this project in Why Fugu-style orchestration will not improve Narracode.

Evaluation and the measurement problem

• Sui, P., Zhu, Y., Cheng, T., West, P., So, R. J., Long, H., Holtzman, A. Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling. arXiv:2604.09854. The 100-Endings metric, inflection rate, and the EQ-Bench inversion. Central to §3 and §5.3.
• LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing. EACL 2026. Judge-agreement ceilings of 73% and 78%.
• The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing. arXiv:2506.03310.
• Are Large Language Models Capable of Generating Human-Level Narratives? arXiv:2407.13248.
• Modeling Story Expectations to Understand Engagement: A Generative Framework Using LLMs. arXiv:2412.15239.

Homogenisation and diversity collapse

• Elias in the Lighthouse, Again? Diagnosing Low Diversity in LLM Stories. arXiv:2605.26492. The 88.3% / eleven-core-words finding, and the observation that lexically distinct stories remain structurally redundant.
• Magic, Madness, Heaven, Sin: LLM Output Diversity is Everything, Everywhere, All at Once. arXiv:2604.01504.
• Epistemic Diversity and Knowledge Collapse in Large Language Models. arXiv:2510.04226.
• The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety. arXiv:2601.06116.
• Creative Convergence or Imitation? Genre-Specific Homogeneity in LLM-Generated Chinese Literature. arXiv:2603.14430.
• Anchorless Diversification for Parallel LLM Ideation. arXiv:2605.30150.

Character psychology, theory of mind, narrative structure

• EvolvTrip: Enhancing Literary Character Understanding with Temporal Theory-of-Mind Graphs. arXiv:2506.13641. Structural model behind the asymmetry ledger in §5.2.
• Guess What I am Thinking: A Benchmark for Inner Thought Reasoning of Role-Playing Language Agents. arXiv:2503.08193.
• Narrative Theory-Driven LLM Methods for Automatic Story Generation and Understanding: A Survey. arXiv:2602.15851.
• A Review of Incorporating Psychological Theories in LLMs. arXiv:2505.00003.
• Rigby, A. et al. The Storyteller in the Model: Narrative Pattern Inheritance, Escalation Dynamics, and Alignment Governance in LLMs. arXiv:2607.20449. Escalation as inherited prior, §6.
• Bae, B.-C., Young, R. M. A Use of Flashback and Foreshadowing for Surprise Arousal in Narrative Using a Plan-Based Approach. ICIDS 2008. Prior art on planted-surprise structure.

Stylometry and machine-text signature

• The Last Fingerprint: How Markdown Training Shapes LLM Prose. arXiv:2603.27006. Basis for treating the em-dash as a fine-tuning fingerprint with a target rate rather than a ban.
• Mitchell, E. et al. DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature. ICML 2023.
• A Counterfactual LLM-based Framework for Measuring Rhetorical Style. arXiv:2512.19908. Tricolon over-production and erotema suppression.
• Saying More Than They Know: Quantifying Epistemic-Rhetorical Miscalibration in LLMs. arXiv:2604.19768. The performed-hesitancy finding behind the class-12 rate cap.
• Voice Under Revision: LLMs and the Normalization of Personal Narrative. arXiv:2604.22142.

Internal documents

• plans/2026-07-30_ai-tells-benchmark.md — AI-tells benchmark design (Claude Opus 5, 2026-07-30).
• master_ai_tells.md — the tell registry, seventeen classes and the remedy caps.
• plans/2026-06-28_1353Z_fugu-orchestration-evaluation-and-routing-proposal.md — Fugu evaluation.
• 2026-05-14_architectural_harness_observations-CLAUDE.md — the refusals-over-commitments finding.
• plans/2026-05-24_continual-harness-evaluation-affect-module-report.md — Continual Harness evaluation, and the house pattern this report follows.

All works and media on Glia.ca by David Jhave Johnston is licensed under CC BY-NC-SA 4.0