Lead Data Platform: a week of orders in the time one used to take

The problem

Pulling one order used to take a person about three hours. Most of that wasn't hard work. It was running checks, waiting on them, putting the file together and hoping it held up. Across a week of orders, that's most of someone's job, done by hand, on work that really shouldn't be done by hand.

It was also expensive, because validation happened at pull time. Every time an order was assembled, the contacts in it were sent off to paid third-party checks to see whether each mobile number was live and each email address was real. The same contact could be checked on Monday, checked again on Wednesday for a different order, and checked a third time on Friday. Every check cost money, and none of them were remembered.

There was a bigger problem too. A check at pull time only tells you about that moment. The checking and the delivering happen at different times, so it can't guarantee what's actually in the file. Files specified as 100% valid mobile numbers and 100% valid email addresses were landing, on a sample of five, at 76% and 67.8%.

Nobody was being careless. That's just what happens when validation sits in the wrong place.

The approach

The fix was simple. Validate each contact once, when it enters the archive, and keep the result.

A contact gets checked when it arrives. Whether its number is live and its address is real are facts about the contact, so the result is stored next to it on the record, where before it just disappeared as one step in a job. From then on, putting an order together means querying contacts already known to be good, with no fresh round of paid lookups over contacts you're hoping are good.

The one thing that cannot move to entry is each client's suppression list (the records they never want sent). That's about the client, not the contact. It changes and it's different for every client, so it can only be right at the moment an order is assembled. So it stays at pull time, and it's still cheap there, because it's just a comparison against a list.

So the contact checks happen once, and each client's list gets applied when their order's pulled.

The second decision was to have the archive answer questions itself. Counting how many contacts match a set of filters is something a user does over and over (set filters, look at the number, adjust one, look again), so it has to be fast enough to keep doing that. So filters, exclusions and suppression are written as one query the database works out, and nothing gets paged out and tallied in a loop.

The third decision was about when to optimise, and I decided to wait. The first version was built to be correct and shipped that way. The performance work came in one concentrated stretch afterwards, once reps from all four companies were onboarded and using it daily, so I could see which queries were actually slow. They weren't the ones I'd have picked.

What I built

A bulk intake pipeline that takes supplier files laid out in wildly different ways into one normalised archive, now several million rows.

Validation when a contact arrives. The paid checks run once per contact when it comes in and the results are kept, so the same record is never paid for twice, however many orders it ends up in.

Deduplication on the way in, so it isn't a step someone has to remember to run. A contact that exists twice resolves to one record when it's added.

Per-client suppression at pull time. Every order is checked against that client's own suppression list as it's assembled. This one stays late because the list is theirs and it changes, and applying it any earlier would mean baking one client's exclusions into an archive that serves twenty-four.

A count engine fast enough to iterate against. Filters, exclusions and suppression compile to a single query, and a filtered count that took 88 seconds now returns in about 3.4. This is the part users touch most, and I think it's what decides whether the system feels usable at all. If you have to wait a minute for a count, you stop refining it, and people start guessing when they should be checking. Suppression uploads, which can be very large files, got the same treatment for the same reason.

Order assembly and delivery, drawing from the validated archive and producing the file the client ships.

A separate branded interface per company. Four companies run on the platform, and each one sees their own identity, where you might expect a shared tool with a logo swapped in. I think that matters more than it sounds. If someone's using this all day, it should feel like their system.

Isolation between companies, built into the system. Authentication decides what a user can reach, and the separation between companies is built into the structure, so it isn't something each query has to remember to apply. One company's suppression list and another's order history sit in the same archive, so it can't just come down to "we are careful". A user has to be unable to reach another company's data, not just unlikely to.

The outcome

One order used to take about three hours. A week of them now takes under three. In a sampled week the system handled fifty-five requests (twenty-five counts and thirty order pulls) across twenty-four clients, pulling roughly 1.87 million leads. Four companies run on the platform at that scale.

Of those three hours, fifty-five minutes is a person. The rest is the system running on its own. Per request, that's about a minute of someone's time against three hours before.

Every file now ships at 100% valid mobile numbers and 100% valid email addresses, against the 76% and 67.8% the sample showed before. It's the same check, just moved to a point where its answer is still true when the file goes out.

Credit spend fell sharply too, because the same contact isn't paid for again and again. Both come from the same change. Checking each contact once is why a file can be guaranteed, and it's also what stops the bill compounding.

Taken together (the manual hours given back to the team and the checks no longer paid for twice), the client estimates the system saves in the region of six figures annually. And the people who used to spend their week pulling orders now spend it on work that needs a person.

It's a custom build, not something off the shelf set up for them, and I wrote all of it, from the interface to the delivery side.