Grounding image generation with retrieval

Ask an image model to draw something it has barely seen and it does not refuse. It guesses, confidently, and the result looks right to everyone except the people who know the subject. The fix is one retrieval call: pull a real photograph of the subject from the Lightdrift corpus and hand it to the generator as a reference. Five subjects below, three images each: the real thing we retrieved, the model's guess without it, and the same model grounded on it. Every image is real output from one session with gpt-image-2; nothing is mocked up.

The Sojourner rover

Sojourner was the first Mars rover: a flat, microwave-sized machine with a solar panel for a back and no mast, no camera tower, no neck. The ungrounded model does not remember that. It remembers Mars rovers in general, so it draws a small Curiosity: tall mast, head-like cameras, the works. Grounded on the 1997 Pathfinder photograph, it produces the actual machine, low and flat beside the rocks. The model averaged three decades of rovers into one; the photograph un-averaged them.

The shield darter

The ungrounded shield darter is a chimera: frog eyes set on top of the head, blenny fins, a goby's posture. It is what you get when a model has seen ten thousand small fish and two shield darters. The grounded output has the actual body plan, the slender profile, the saddled back, the pointed snout, because it came from a survey photograph of the actual species. The reference photo is not glamorous, a fish in a plastic photarium above a ruler, and that is the point: this is what real, specific visual knowledge looks like.

The gladiolus

Nothing is wrong with the ungrounded gladiolus except that it is nobody's gladiolus: the pink catalog default, studio-lit, the average of every gladiolus photo ever taken. The retrieved photo was a particular apricot variety with red flame throats, shot in somebody's July garden. The grounded output is that flower, in that kind of light. There are thirty thousand gladiolus cultivars; a prompt says the word, a photograph says which one.

The X-36

The X-36 was a remotely piloted scale model, a tailless fighter testbed with no room for a person inside. The ungrounded model gets the silhouette roughly right and then does the most human thing possible: it adds a glass canopy with a pilot in it, because in its experience jets have pilots. Grounded on the NASA flight photo, the canopy goes back to being a painted hump and the proportions shrink back to the real drone's. The errors a generator makes are not random; they are its priors showing. A reference photo overrides the prior.

The snubnose darter

One more fish, because the long tail is where this matters most. The ungrounded snubnose darter is a beautiful invention: bold bands and decorated fins borrowed from the whole family of small colorful fish. The grounded one carries the real field marks, the orange spot rows along the flank and the red band in the dorsal fin. There are hundreds of darter species and the model has thin coverage of each. Only retrieval knows which one you mean.

How it works

Two calls. Search the corpus for the subject; pass the top result to the generator as a reference image with the same prompt. The retrieval step costs $0.02, and the rights answer on the retrieved photo tells the agent whether it may be used this way: every reference on this page is public domain or Creative Commons, credited below.

real = lightdrift.search(
    query="snubnose darter (etheostoma simoterum)", k=1)

image = generator.generate(
    prompt="Photo of a snubnose darter (etheostoma simoterum).",
    reference=real.results[0].file)

Nothing here needed human judgment. An agent that can call two endpoints can run this loop for any subject its task turns up, and the more obscure the subject, the more the reference matters. That is the point of retrieval for generation: the model is an excellent painter with a mediocre memory, and the corpus is the memory.