· 7 min read
A failed check that looks like nothing to do
A watcher that read my local git now reads GitHub from a function that can be killed halfway. The API was the easy part. The problem was that two checks used to have one meaning each and now have three, and the old code read all three as nothing to do.
What the watcher does
Every commit I write ends with a few trailer lines saying which milestone and project the work belongs to, and whether it was part of what was agreed. A watcher reads those when I push and logs the work against the project, so I don't have to write up my own day.
The line saying whether it was agreed is the one that matters. Work that was agreed is already paid for and turns into nothing. Work that wasn't turns up on my home screen as something to decide about before it can be invoiced. So a piece of work raised twice puts a decision about money in front of me that I've already made once, and I won't remember making it.
For a while the watcher ran on my own PC, over local git, inside a process that stayed open all day. Then the app moved to the web and it had to run as a function instead, over the GitHub API, started by a webhook. Same scan and the same rules about what a trailer means, because a second copy of the code that decides what I bill for is exactly what shouldn't exist.
The API was the easy part. It took an afternoon. The problem was two checks that changed meaning on the way and didn't say so.
Not an ancestor used to mean one thing
The watcher keeps a cursor, which is the last commit it read. Every run it asks whether that commit is still an ancestor of the branch head. If it isn't, the history has been rewritten underneath it, there's nothing sensible left to read, so it moves the cursor to the head and reads nothing. That's called baselining and it's the right answer.
On local git it's also the only answer. A commit on your own disk is either in the history or it isn't, and the only way it stops being in the history is a rewrite.
Over a network that same "no" arrives for reasons that have nothing to do with a rewrite. The request times out. Or the cursor is a commit I made and haven't pushed, which comes back as a 404. Both are temporary. The old code couldn't tell them apart from a rewrite, so it would baseline, jump to the head, and every commit in between would never be read by anything. No error, nothing on the row, just a few days of work that never got logged.
So that check isn't allowed to fail softly any more. If it can't be answered the whole run throws, the cursor stays exactly where it was, and the reason gets written on the repo's row where I'll see it. The next run reads the commits it was always going to read.
The same shape, in the merge check
There's a second one with the same problem. When a branch gets merged, the scan reads the merge date to finish off the step the branch belonged to. If it can't read the tip of that branch and the branch is gone, it retires the entry without touching the step, permanently.
Locally that's safe, because a commit that can't be read really has gone. Over the API it can be a sha GitHub hasn't got, which happens with a branch that has only ever existed on my machine and that a push would still bring. It can also be a passing network failure on a branch GitHub deleted at merge.
Retiring either of those would mean the merge never finishes its step, and nothing would ever go back and look again. So a merge date that can't be read now means "not merged yet" rather than "gone". The entry stays open and gets asked about on the next scan. It costs one extra request per scan and it can't lose anything.
Both of these are the same mistake written twice. A call that used to have one failure mode got moved somewhere it has several, and the code kept treating the answer as the thing it used to be.
The write that stops the other write repeating
The scan writes two things. It raises any new work, and then it saves the branch state so the next run knows what it has already seen. The second write is what stops the first one happening again.
On the desktop, that second write happened even when the scan failed, because it sat in a catch. A killed function never reaches its catch. So a timeout landing between the raise and the save would leave the raise committed and the save missing, and the next run would raise the same work a second time. That's the money one.
The fix was to change the order the scan does things in. I didn't touch the error handling. Everything the scan is going to ask GitHub gets read up front, into a snapshot, before the scan starts. Then the scan runs against that snapshot inside a single transaction, so the raise and the cursor commit together or not at all. No transaction is sitting there waiting on a network call, and there's nothing left for a kill to land in the middle of.
The snapshot walks the same path the scan will, driven only by the repo link, so it needs no database access and can run first. If the two ever disagree, the scan throws on the question it has no answer for instead of guessing at it.
The lease, and why the lease isn't enough on its own
Two pushes close together would otherwise start two runs reading the same repo. Each run takes a lease on the repo's row first, with a conditional update that only one of them can win:
UPDATE project_repos SET lease_holder = ?, lease_until = ?
WHERE project_id = ? AND (lease_until IS NULL OR lease_until < ?)
RETURNING project_idThe lease lasts 120 seconds and the function is capped at 60, which is the right way round. A run that gets killed can't release anything, so the lease has to outlive the longest run that could still be alive, and then expire by itself.
That gap is also the hole. A lease can be expired on paper while the run that took it is still going, if the platform is slow. So the lease isn't the guarantee. Inside the transaction the scan re-reads the row FOR UPDATE and backs out unless it still holds the lease and the cursor is still exactly where it was when the snapshot was read. If either has moved, the whole transaction rolls back and the run reports itself as busy instead of writing anything.
A webhook is not a delivery guarantee
The webhook is what makes it feel instant, and it's the part I trust least. Deliveries get missed, and two pushes that cross can leave the second one unscanned.
So there's also a sweep every ten minutes that does the same scan over every linked repo. It's offset off the hour instead of on it, so it isn't competing with everything else that fires on a round number. A missed delivery costs ten minutes and nothing else, and I'd rather have a cheap sweep I never think about than a webhook I have to trust.
The general version
I wrote about a stuck queue row in the job that died without dying, and the lesson there was that whatever cleans up after a process can't depend on that process. This one is next to it but not the same.
Both of the bugs here were in code that was correct where it was written. A failed ancestor check meant one thing on a local disk. A save in a catch ran every time, in a process that always lived long enough to reach it. Moving the code didn't break either of those lines, it broke what was true around them.
So when I move something now, I go looking for the sentences that start "this can only fail because", and I check whether that's still the whole list. Usually it isn't, and the dangerous ones are the failures that produce a plausible answer instead of an error.