← Learnings & case studies

Case study · Generative AI production pipelines · September 2026

The Kerosene Lamp Problem.

Six of the nine generated frames I audited contain a lit kerosene lamp. The film is set between 1813 and 1821. Kerosene was not distilled until 1846, and the lamp that burns it was not sold until the 1850s. Every prompt forbade the lamp by name. The issue: the pipeline's main tool for keeping the wrong century out of the picture, a list of things not to draw, does not work on this model. The pipeline lacks period specific details to set the stage so the model creates its own.

Role: system design, governance, audit 29 units · 221 beats · 20 frames 5 minute read
1810 1860 1813Fort Strother, the film opens 1821Pensacola, the audited scene 1846kerosene first distilled 1850skerosene lamp first sold
  • 1813Fort Strother, the film opens
  • 1821Pensacola, the audited scene
  • 1846kerosene first distilled
  • 1850skerosene lamp first sold

25 to 40 years separate the film from the fuel, and the lamp went on sale in industrial cities, not on the Alabama frontier or in a Spanish garrison town on the Gulf. The generation models seem to use the kerosene lamp as a preferred aesthetic for the century, too imprecise for historical guardrails.

62%

Mean historical-accuracy score across nine frames audited pixel by pixel

6 of 9

Frames containing an object the prompt forbids by name

87%

Of every scene description was being silently discarded before it reached the model

10 → 19

Score change on one frame from a single build fix. Same prompt, same model

The audit

One beat, generated twice

The scene is a woman at a window watching an ordinary Catholic Sunday in Pensacola, 1821. The audience is meant to see nothing wrong with the street below.

Drag to compare. On the first pass: 1 the kerosene lamp · 2 the gold key · 3 the silhouette.

Before · 10/21After · 19/21
Second pass: the street is populated as the page describes.
First pass: a lit kerosene lamp, a floating gold key and a featureless silhouette.
First pass After the fix

Findings, first pass

  • 1A lit kerosene lamp, indoors, in daylight, in 1821. Forbidden by name in the prompt.
  • 2None of the street the page describes is in the picture. Shops open, men at a card table under an awning, a fiddle, people going to and from Mass: the plaza is bare. The build had cut the beat's description at 110 characters and all of that was in the part it dropped.
  • 3A large gold key floats on the shutter. Nothing asked for it. The project suspends its symbol vocabulary by ruling so signs cannot be designed by accident, and the model designed one.
  • 4A featureless black silhouette in the street, forbidden by name in the same list as the lamp.
  • 5Overhead wires above an 1821 street. Nothing forbade them, because nobody thought to.
  • 6The dress reads 1840s: tiered ruffles, fitted natural waist. The wardrobe band specifies an 1821 gored skirt and the negatives name “natural waist” explicitly.
  • ✓Caption correct and inside the ruled border. Rachel above and apart from the town, which is the posture the page asks for.

Findings, after the fix

  • ✓The awning is there. Two men at a table with the cards down. A musician playing. People going to and from Mass. Shops open.
  • ✓The kerosene lamp is gone. So are the key, the silhouette and the wires. None of them was separately fixed. Giving the model real content displaced them.
  • ✓Interior daylight through the shutter with dust in the beam, which is the project's own rendering rule for this lighting preset.
  • ×The 1840s silhouette persists. This is the one criterion the fix did not touch, and the reason is the finding below.
  • ×The instrument is a cello rather than a fiddle. Minor.

The only difference is context. The first prompt carried the opening sentence of the beat; the second carried all of it, 273 more characters.

The Spanish garrison leaving Pensacola at the transfer of Florida, July 1821.
Also audited: “The flag comes down,” Pensacola, July 1821.

The Spanish garrison leaving Pensacola at the transfer of Florida. This beat carries an authored crowd note, because the crowd deliberately has no cast rows and therefore no wardrobe authority. The note was written specifically to prevent what happened to an earlier uncast crowd, which came back as the Union army.

Findings, “The flag comes down”

  • ✓The composition is exactly the page: she is on an upper gallery, above it, looking down at a distance.
  • ✓The note did keep the crowd out of the cast list and into the picture as a body rather than as portraits.
  • ×The flags are wrong. The 1821 Spanish flag is red-yellow-red. The frame shows a blue-white-red tricolour and a black-and-white quartered field.
  • ×The garrison reads as generic Napoleonic-era infantry rather than Spanish colonial. Naming the nationality in prose moved the needle less than expected.
  • ×A man in a dark coat and wide hat stands on the gallery. The page says of this beat, by name, that Jackson is not in this frame.
  • ×Two featureless silhouettes, a kerosene lamp, and overhead wires.
The rubric

Seven rules, joined to their scores

Authority runs upward through four tiers — sources → bibles → screenplay pages → build artefacts, and the page is the fact. A generated frame is a build artefact and is accountable to every tier above it. The rubric invents no criteria; it makes seven existing project rules scoreable from 0 to 3.

IdCriterionScoreEnforcedWhat it tests
A1Provenance2.2/3PromptEverything depicted traces to the screenplay page or a tagged source. Content that appears without a line behind it is unsourced, however plausible it looks.
A2Period0.7/3PromptNothing in frame postdates the beat: objects, materials, lighting technology, architecture. Scored hard, because one anachronism is what a historian finds first.
A3Costume1.0/3PromptEvery figure resolves to an authored wardrobe band at the right occasion. A figure with no cast row has no wardrobe authority at all.
A4Presence2.3/3PromptEveryone in frame belongs there, and no one the page excludes appears. The page can bar someone by name, and that outranks everything.
A5Consent3.0/3CodeThe consultation gate is honoured exactly. Twelve beats are absolutely withheld; 182 more generate flagged, never silently. Pass or fail, no partial credit.
A6Quotation2.6/3CodeText visible in frame matches the register at its verified status. Where the record is a transcription, the honest depiction is a hand you cannot read.
A7Restraint1.3/3PromptNothing is added that argues: no invented symbol, no composition that editorialises, no emphasis the page withholds.

CODE = the build enforces it before any prompt is written. PROMPT = the prompt asks for it and the model has to comply. Every criterion a machine enforced passed. Every criterion that asked the model to leave something out scored below half.

The finding

The control surface was pointed backwards

Here is the whole finding in one sentence, and then I will show you why it is true and what we do about it. The model does what you describe, and it ignores what you forbid.

Try it on yourself first. Do not picture a kerosene lamp. Whatever you just did, a lamp was involved. A prohibition still has to name the thing, and naming it puts it in front of you. Now try the other way: picture a candle on a wooden table, the only light in the room. You did not need to be told what to leave out, because the candle took up the space.

An image model is not a person, and this is not a perfect account of what happens inside it. But it predicts the model's behaviour well enough to design against, and that is the test that matters for a tool.

The kerosene lamp is forbidden by name in every prompt this project emits, and it appeared in six of nine audited frames. A featureless silhouette is forbidden in the same list and appeared in four. “Natural waist” is forbidden on Rachel and she has one in every frame she is in. Then the other column: one sentence saying the candle standing on that table is the only light in the picture produced a candle-lit room on the first try, after four separate prohibitions had failed to. And the street furniture, restored to the prompt, pushed four forbidden objects out of the frame without naming any of them.

A candle-lit interior: the candle is the only light in the picture.
“The letter.” One positive sentence, first try: the candle is the only light in the picture.

Every rule written as “do not draw X” scored below half. Every rule written as “this is what is here,” or enforced before the prompt existed, passed.

So what do we do about it? Three things, in this order.

01

Say what is there, not what is not.

Every prohibition in the prompt gets rewritten as the thing that occupies its place. The do-not-depict list stops being treated as a control and becomes a checklist for the reviewer instead.

no window, no doorway, no sky anywhere in the picture → the wall behind her is unbroken whitewashed plaster
not a natural waist, not 1840s sleeves → an 1821 gored skirt falling from a high waist under the bust, sleeves narrow to the wrist
02

Move every rule you can into code.

The two criteria that never failed, consent and quotation, are the two the build enforces before a prompt exists. The consent gate refuses a beat outright; the quotation register drops an unverified line from the caption. Neither one asks the model for anything. If a rule can be a gate, make it a gate.

03

Read the prompt before you pay for the picture.

Three of the four worst frames were not model failures. They were build defects, plainly visible in the prompt text and invisible in the image: the truncation, a transition marker leaking into the action line, a cleanup rule quietly eating real words out of fourteen captions. The cheapest quality gate in the whole pipeline turned out to be a person reading the thing that gets sent, before it gets sent.

The limit

What the rubric cannot reach

Every criterion above audits depiction: what is in frame, what it is made of, when it is from, whose authority put it there. All of that has a reference in the record. Motivation, conflict and dialogue do not, and the project's own rules make that deliberate. A concealed want is never sent to the model, only its observable consequence. No character is allowed to state what a fact means.

All three are scored by absence. The motive succeeds by not appearing. The rule holds by nothing being said. And leaving something out is exactly the instruction this audit just showed the model does not follow. So for the dramatic layer, governance has to sit before generation, in the structure of what gets handed over, not after it in a review.
Transfer

What applies to any generation pipeline

None of this is specific to film, or to history. Four rules you can apply on Monday.

01 / 04 Code

2 rules never failed

both were enforced by code. every rule that relied on the model's cooperation scored below half

Enforce in code what you can, and expect the rest to drift.

The two criteria that never failed were the two enforced before the prompt existed. Everything left to the model's cooperation scored below half. That ratio is a design instruction, not a disappointment.

→ If a rule can be a gate, make it a gate
02 / 04 Prompt

3 of 4 worst frames

were not model failures. the defect was sitting in the prompt text, where nobody was looking

Audit the artefact, not the output.

Three of the four worst frames failed for reasons plainly legible in the prompt text and invisible in the picture. The cheapest quality gate turned out to be reading the thing you are about to send.

→ Read the prompt before you pay for the picture
03 / 04 Archive

The failure was saved

the 10/21 first pass was archived before the fix replaced it. that is why this page can prove anything

Preserve the baseline before you improve it.

The most persuasive evidence here exists only because the first pass was archived with checksums before the second overwrote it. An improvement is legible only against the thing it improved on.

→ Archive the first pass, checksums and all
04 / 04 Ledger

5 written, 0 applied

rulings made and written down that never reached the code they were meant to govern

A decision recorded is not a decision applied.

Five rulings had been made, written down, and never reached the code that needed them. One was in the script that prices the work, still naming a superseded model at nearly three times the cost.

→ Trace every ruling to the code that obeys it

Method. Nine of twenty generated frames audited pixel by pixel against seven criteria derived from existing project rules. Scores are mine, applied consistently, and are not blind. The eleven unexamined frames are excluded from the mean rather than assumed. Every frame carries a sidecar recording its exact prompt, model, settings, a SHA-256 of its bytes, and the work order that authorised it. The first pass is preserved with checksums so the comparison stays checkable.

Red Eagle adapts my published research on William Weatherford and the Creek War of 1813 to 1814 (International Journal of Interdisciplinary Social Sciences, Vol. 6 No. 4). Consultation with the Poarch Band of Creek Indians and the Muscogee (Creek) Nation has not taken place. Muscogee content in this draft is generated under a flag and is not cleared for publication.