We built a tool to catch inventory lies. The first thing it caught was ours.
A reconciliation report that counted everything alike was hiding ninety real disagreements between our own ledger and our own stock. Here is what we found, how old it was, and why we are not quietly correcting it.
Every warehouse system tells you how much stock you have. Ours is built on the claim that it can also tell you why: every number is derived from movements, and no code path can change stock without writing a row that explains it.
Last week we shipped a change to the reconciliation report, which is the thing that checks that claim. It replays every movement for a product and compares the result against the stored number. If they disagree, the ledger and the shelf are telling different stories.
The report had a flaw. It counted every finding as a discrepancy, which meant that a genuine disagreement between the ledger and the stock counted exactly the same as a bookkeeping nit about whether loose pieces should have been converted into a case. On our rehearsal tenant the report said 1,435. Nobody looked at 1,435 findings. Nobody was going to.
So we split it: policy findings became warnings, and a discrepancy became what the word means: the ledger and the snapshot do not agree. On the rehearsal tenant, 1,435 became 12.
Then we ran it in production for the first time.
Ninety
The old number there had been 98. Under the new rule it came back 90 discrepancies and 7 warnings. Almost none of it was noise. Ninety products where the movements do not explain the stock.
One of them was loud enough to be worth chasing on its own: a pack of AA batteries where the movement history replays to negative 12,800 pieces against a stored number of 699. A warehouse cannot hold negative stock. Something in that product’s history is impossible.
We pulled all 42 of its movements. Here is the shape of it:
- A receipt of 1,200 pieces in June.
- A shipment of 14,000 pieces eight days later.
- Thirty-five small allocations in August that reconcile exactly.
Two buckets, bulk storage and multi-warehouse, hold 59,000 and 265,000 pieces with no movement rows at all. Not a missing row. None, ever, in the product’s entire history.
What it was
It was us.
That tenant is where the product was built, starting in May. While building it, stock was put straight into the database so there would be something to develop against. No movement rows, because the code that writes them was what we were building. Later, real flows ran on top of that seeded number. Allocations, shipments and picks all wrote their rows correctly. That is why the ledger is half right: the parts that always wrote rows reconcile to the piece, and the parts that were typed in do not exist at all.
The 14,000-piece shipment is the same story from the other end. If the stored number was already inflated by hand, a shipment of 14,000 would pass every check the system makes, because the system believed the stock was there. The row is honest about what happened. What happened was standing on a fiction.
We classified all ninety. Fifty-three are buckets with a quantity and no rows behind them. Twenty-one have rows that leave a gap. Twelve replay below zero. Four are lot-level. Fifty-one products in total, out of a catalogue of about a hundred and thirty.
The part that matters more than the number
The obvious question is whether it is still happening. That one has an answer.
In an earlier release we made a list of every place in the code that can change a stock bucket. Thirty-five of them, each with the movement type it writes and the test that proves it. The build now refuses a new one that is not on the list. So for each of the ninety findings we asked when that bucket was last written at all, and compared it to the day the list started biting.
Nothing has been written to any of the affected products since four days before the list existed. The last write that did happen came with its ledger row, seven milliseconds apart, in the same transaction. The cause is code that no longer exists.
That is the difference between an old mess and an open wound, and it is worth knowing which one you have.
Why we are not quietly fixing the numbers
The tempting repair is to set the stored quantities to whatever makes the arithmetic close. It would take one afternoon and the report would go green.
We are not going to do that, for a reason that is not sentimental. Setting a number so the books balance is precisely the act that produced this situation in the first place. Doing it again, this time on purpose and with better intentions, would teach the system that a number can be adjusted when it is inconvenient. That is the habit the whole product exists to break.
What we are going to write instead is a ledger row that says, in effect: this quantity predates the ledger, and counting begins here. It is the same shape as an opening balance, which is how any real warehouse starts counting stock that arrived before the system did. The row carries a reason, a timestamp and a name. After it, the replay matches, not because the difference was erased but because it was explained.
And one group will not get that treatment yet: the twelve products whose movements sum below zero. Those contain rows that are themselves wrong: a shipment of 14,000 against a receipt of 1,200 is not a starting point, it is a false record. A baseline that made that arithmetic close would bless it. We would rather leave twelve findings open and visible than close them with a lie that reconciles.
Why publish this at all
Because the alternative is a product that claims a warehouse never lies while its own does, and hopes nobody checks.
There are no customers on that tenant. Nobody’s inventory is affected but ours. But the thing worth noticing is not that our development data was messy, because every development database is messy. It is that the system found it without being told where to look, dated it, classified it, and proved the cause was dead, in a day.
That is the feature. A number you cannot interrogate is not information, it is decoration. We would rather ship a report that tells us ninety uncomfortable things than one that tells us nothing and calls it green.
What happened next, September 11
We wrote the baseline the same day this was drafted, so the paragraphs above are a plan and this is the result.
Before touching anything we ran the classifier against every tenant, not just the noisy one. Four of the five came back clean, including the production tenant carrying a five-thousand-five-hundred-SKU catalogue loaded by the real importer. Only the tenant we had typed into by hand was dirty. That is the sharpest evidence we have that the drift belongs to our data and not to the software: where the code did the work, it reconciles.
Then fifty-five ledger rows, one per product and bucket, each carrying the same sentence, this quantity predates the ledger, counting begins here, with a timestamp and a person’s name on it. No stored quantity was edited. The snapshot was the thing being explained, not the thing being changed.
The reconciliation immediately after: ninety discrepancies became twenty-three. The warnings stayed at seven, because the policy rules were never part of this.
Both failure modes were written down before we ran it, which is the only reason the result means anything. Zero would have been bad, because it would have meant the baseline closed rows it never explained. Ninety would have meant nothing happened. Twenty-three was the number we predicted, and it is made of the three groups we refused to touch: twelve whose movements are themselves impossible and cannot be reconciled without blessing a false row, four that compare lot-tagged movements to pallets and need a different instrument, and four that run the other way, where the ledger says there is more than the shelf shows, which the hand-seeding story does not explain at all.
Those twenty-three are still on the report. We would rather look at them than at a green screen we arranged for ourselves.
Questions
- Does this mean DockingWare's inventory numbers are wrong?
- In a development tenant, yes, and they have been since May. That tenant is where the product was built, and stock was seeded straight into the database while we worked, without the movement rows a real operation would write. No customer data is involved, because there are no customers on it. The point of the story is that the system detected it without being told where to look.
- Why not just correct the numbers?
- Because a correction with no explanation is the same act that created the problem. The repair is a ledger row that says this quantity predates the ledger and counting begins here. The number stops being unexplained without anyone pretending to know more than they do.
- How do you know it is not still happening?
- Every place in the code that can change a stock bucket is on a list, and the build refuses a new one that is not. For the affected products we checked the last time anything wrote to them at all: nothing since four days before that list existed. The cause is retired code, not a live hole.