· 3 min read
Break the copy, not production
Small test data lies to you, because a query plan flips the moment the statistics are real. I build risky changes against a copy-on-write clone of the live database instead, and I learned the other half of that rule by dragging production for thirty six minutes.
Fixtures lie by being small
A change to something everything depends on, a query every page runs, a migration on a large table, the clause that decides who gets contacted, is the kind I don't trust myself to reason about. So I build it against a copy of the real database instead of a fixture.
Most test data lies to you by being small, and it lies in a specific way. It isn't that the code behaves the same but faster, it's that the database makes different decisions. A join over a few hundred rows uses whatever plan is convenient. The same join over a few million picks a different one, or spills to disk, or sequentially scans a table because the index you assumed it would use doesn't apply to the condition you actually wrote. A hash that fits comfortably in memory against a fixture spills against a six million row list. The planner's choice depends on statistics, and fixture statistics are made up.
A fixture passes all of this without a word, and then production fails the one case that counts.
This is a different problem from the one in testing against the thing that loads your file, where the risk is that a test agrees with your own assumptions because you wrote both halves. Here the test is honest and the data is the liar.
The copy is where I'm allowed to be reckless
Postgres makes the alternative cheap. A copy-on-write branch of the production database gives you the entire dataset, isolated, in seconds, and you throw it away afterwards.
So the frightening part happens there. Run the slow query, build the index, backfill the thirty million row table, time it, break it, measure it, against data that behaves exactly like production because it is production, with no risk to the thing people are using. A recent change caught two things there that a fixture would wave straight through: a query that was correct in theory and took a hundred seconds per client in practice, and a missing index I'd never have felt at fixture scale. Both turned up in an afternoon, on a branch that cost nothing and was deleted the same day.
Neither of those is a bug you find by reading the code. They're only visible at real size.
The other half, learned the expensive way
I learned the "no risk to the thing people are using" half badly.
Early on I ran a heavy experimental query straight against a live database, to just check something. It ran for thirty six minutes and dragged every other query on the box down with it. Killing my local process did nothing, because the query was running on the server rather than on my machine, so I had to go and find the backend and terminate it by hand. Nothing broke and no data was lost, but for half an hour the people using that application paid for my curiosity.
The lesson wasn't "be careful with big queries". It was that I had no business forming the intention in the first place. There was a copy available. I used production because it was the connection string already in front of me.
So the rule is simple now, and it's a rule instead of a judgement call, because judgement is exactly what fails at 11pm when you only want to check one thing. Experiments go on the copy. The copy is where I'm allowed to be reckless. Production is where I'm only allowed to be right.
By the time a change reaches it, every number that matters has already been measured against data that looks exactly like it. Which makes the deploy boring, and boring is the whole point.