The Meter Fell Three Times

Single source of truth

Every fact has one home.Everything else points at it.

Every data element is mastered (or edited) in only one place.From the usual definition of a single source of truth

A single source of truth is a habit of structure. Each fact a system depends on has one place where it is edited, and every other use is a reference to that place. Copies are allowed. A copy that can be changed on its own is not. The reward is small and specific: when the fact changes, you change it once, and nothing that depends on it gets left behind.

The old joke says the same thing. It first ran, unsigned, in the San Diego Union in 1930, and was later named Segal’s law: a man with one watch knows what time it is, and a man with two watches is never sure.

Three standards

Duchamp called the piece 3 stoppages étalon, three standard stoppages, and described it in a note in two words: “canned chance.” As he told it, he dropped three threads, each a meter long, from a meter up, and fixed each one where it fell. In 1918 he had three wooden slats cut to the curves, and then he used them as rulers. He traced each curve three times in Network of Stoppages and again for the tubes of The Large Glass. Asked about it by the Museum of Modern Art, he called it “a joke about the meter.”

The joke has a second layer. In 1999 Rhonda Roland Shearer and Stephen Jay Gould reported, with the museum’s conservators, that the ends of each thread pass through needle holes to the back of the canvas and run on for a few centimeters. They argued the curves were sewn into place. Scholars still disagree about what that proves. Either way, a standard that only its maker has checked is a claim.

One bar

The real meter went the other way. On March 26, 1791 France adopted a unit defined as one ten-millionth of the distance from the North Pole to the equator. On June 22, 1799 a platinum bar of that length was deposited in the national archives, and that December it became the meter by law. Starting in February 1796, sixteen marble meters were set into walls around Paris so that anyone could hold a length of cloth up to one. Two are still in the city, and one of them is where it was first put, at 36 rue de Vaugirard. They were copies, and everyone knew which bar they were copies of.

In 1889 a platinum-iridium bar kept at Sèvres, outside Paris, took over, and about thirty sister bars went to member countries by lot. The United States drew numbers 21 and 27. A national bar counted as the meter because it could be carried back and compared with the one in Sèvres.

When the source itself moves

One source can still be wrong. The kilogram was a platinum-iridium cylinder under glass at Sèvres, with six official copies. Compared around 1889, 1946 and 1989, most of the copies had gained about 50 micrograms on the original in a hundred years. Nobody could say whether the copies had grown or the original had shrunk, because the original was the definition. On May 20, 2019 the kilogram was redefined from a constant of nature. The meter had already made that move: a wavelength of krypton light in 1960, and in 1983 the distance light travels in 1/299,792,458 of a second.

So the best source of truth is a definition, written down, that anyone can reproduce and that changes only on purpose, with a date on it.

Does it hold up?

For scoring AI systems, the three rulers are on the record. In June 2023 Hugging Face found three implementations of one benchmark, MMLU, in use: the original code, Stanford’s HELM and the EleutherAI harness. They wrote the prompt differently and read the answer out differently. The same LLaMA 65B model scored 0.636, 0.637 and 0.488. Under the third, Falcon 40B ranked above it. Under the other two it ranked below.

Anthropic reported that changing answer labels from (A) to (1), or adding a space, could move accuracy by about 5 percent. When OpenAI had developers check SWE-bench, a test for coding agents, 61.1 percent of the sampled tasks were flagged for unit tests that could mark a correct fix as wrong. And in 2024 Shreya Shankar and colleagues named criteria drift: people need criteria to grade outputs, but grading outputs changes the criteria. A scorer kept in three places has three chances to drift.

In your stack

• Define each scorer once. The rubric, the judge prompt, the pass mark and the version live in one module. The eval suite, the CI gate and the production monitor import it. Nobody retypes a threshold into a dashboard query.
• Copies are fine if they cannot be edited. A dashboard can cache scores. It should not compute its own. A copy that only reads is a marble meter on a wall. A copy with its own edit button is a dropped thread.
• Stamp every score with its scorer. A number without the scorer’s version is a length without a unit. When the definition changes, old and new scores should not share a chart as if they measured the same thing.
• One source is not one metric. Keep twenty scorers if you need them. The rule is one definition for each, not one number for everything.
• Guard the bar. With one source, a wrong edit is wrong everywhere at once. That is the price. So the definition gets review, tests and a changelog like any other code, and now and then it gets compared with what people actually think a good answer is.

How the room is made

Each drop is simulated on your device: a chain of forty-nine points, one meter long, with gravity, air drag, a gust and friction where it touches the canvas. Your three threads are yours, and the next visitor gets different ones. The fall is replayed slowed down with its earlier positions left hanging in the air, the way Étienne-Jules Marey’s photographs stacked up a movement and the way Duchamp had painted Nude Descending a Staircase the year before. The camera is a few lines of perspective arithmetic on a 2D canvas. The scores are real: each ruler is marked off evenly along its own curve, and the run is read where its end meets that edge.

The room simplifies. Duchamp always said he dropped each thread once; the room takes him at his word and lets the threads curl more than his do. The canvases were cut down and mounted on glass, and a wooden case added, in 1936. The 1799 bar had no scale on it; ours does so you can read it. The run, the refund ticket and the three teams are invented. Real copies drift in both directions; our crooked rulers all read high, because a bent meter is always a short one. A later check of the kilogram, in 2014, did not find the drift continuing. “Scorer” is the word Inspect, Braintrust and Mastra use; LangSmith says evaluator and OpenAI says grader. The marble meter is photographed by LPLT (CC BY-SA 3.0); the portrait is cropped from a group of three.