CodeLabProsCodeLabPros — homeMenu

How we work

What happens between a problem you cannot fully describe and a system that runs.

Six stages, in the order they actually happen rather than the order that looks tidy. Then what we can take responsibility for, and the principles underneath all of it.

None of this is a framework we read about. Every stage below is the same move made in at least three of the seven systems on this site, and each one names where you can go and look at it.

The workin the order it happens
  1. 01

    Find the problem under the problem

    The brief is almost never the problem. A marketplace for car parts was a trust problem before it was a catalogue problem. A way for parents to pay fees was a question about what happens when nobody can tell which child a payment belongs to. A system for choosing names turned out to be a question about whether the premise behind it held at all. None of that is missing because anyone was careless. It is not visible from where you are standing at the start. So the first week is spent looking for it, not estimating the thing you already described.

    Ask what has to be true for this to work at all, and build outward from that instead of from the feature list.

    Visible in: A marketplace with escrow, built end to end→
  2. 02

    Build the difficult part first

    There is usually one piece that decides whether the rest is worth writing, and it is rarely the piece that demos well. Escrow decides whether a marketplace is a marketplace. The rule for an unattributable payment decides whether a ledger is trustworthy. Everything else is downstream of it and cheap to change: the screens, the admin, the reporting. Doing it first means the expensive discovery happens in week one, not week nine, while it is still a design conversation and not a rewrite.

    Whatever is most likely to invalidate the plan gets built before the things that depend on it.

    Visible in: School fees, reconciled across many schools at once→
  3. 03

    Decide what the software is not allowed to do

    Before the capability, the limits. Which decisions must stop and wait for a person, and which are safe to make alone. The rule we use is where the cost of being wrong lands: automate where a mistake is cheap and reversible, involve a person where the cost is borne by someone who did nothing wrong. This gets decided while it is still cheap to decide, because the moment the capability exists, every argument for restraining it starts sounding like an argument against shipping.

    Write the stop conditions before the feature, not after the first incident.

    Visible in: An agent that researches sales prospects on its own→
  4. 04

    Make the important rules enforceable, not intentional

    Every team has rules that everyone agrees with and nobody can point at. One customer must never see another's data. This service must not import that one. The system must run offline. Those survive exactly as long as everyone remembers them, which is until the week someone is in a hurry. So the ones that matter get enforced where they cannot be forgotten. Tenant isolation is enforced twice, once in the application and once underneath it, so a single mistake in one layer is not enough to leak anything.

    If a rule matters, it is enforced in code or in the build. Otherwise it is a preference with good PR.

    Visible in: An entire business, taken into software→
  5. 05

    Test it against reality, and let reality win

    A design is a hypothesis. The point of building it is to find out where it is wrong, which means the result has to be allowed to overrule the plan, including when the plan was ours and the result is inconvenient. One system we built produced, on its first real engagement, a finding against the premise the whole thing rested on. We changed the design, not the finding. This is the part most projects skip, and it is skipped long after somebody first saw the evidence.

    When the system disagrees with the idea that produced it, the system is usually right.

    Visible in: Infrastructure for research that keeps itself honest→
  6. 06

    Keep what you checked apart from what you assumed

    In writing, as you go. After a week the two feel identical, and by then decisions have been built on top of both. Every week of an engagement you get a short written update measured against the agreed scope: what moved, what we confirmed, what we are still assuming, and anything that turned out to be wrong. Not a status meeting. A record, so that when someone asks in month four why a thing was done that way, the answer exists and is not a reconstruction.

    Say which of the two a statement is, at the moment you make it. Later is too late.

    Visible in: Infrastructure for research that keeps itself honest→
What we take oneach line answerable to a system

The kinds of problem we can take responsibility for.

Not a list of technologies. A technology list tells you what we have used; this tells you what we have already been responsible for getting right, and where you can go and check.

Money that has to end up in the right place

Payments held and released on a condition, not on a click. Payouts to the people who earned them. The same payment arriving twice, through a retry or a re-uploaded file, and being recorded once. Money that arrives by three different routes and has to reach one answer. Fees, balances and a ledger that can be reconciled against the messages that produced it.

Dale — escrow, payouts and mobile-money checkout→

Customers who must never see each other

One platform, many organisations, and certainty that a report, an export or a route added next year cannot leak across the boundary. Enforced in the application and again underneath it, so being safe does not depend on the next engineer remembering. This is what makes a platform sellable to customers who compete with one another.

Karo — many schools, isolated from one another→

Who someone is, and what they may touch

Login that cannot be left half-finished, second factors that cannot be skipped, roles that mean something, and an audit trail that records what was done, not what was permitted. Including the awkward state most libraries model badly: a user who is authenticated but not yet admitted.

NeoStrata — access and audit, laid before the business features→

Decisions software makes on somebody's behalf

Scoring, ranking, matching, eligibility, routing: the places where software chooses and a person lives with it. Built so the reasoning can be inspected afterwards, so “unknown” is a permitted answer instead of a zero, and so the cases that should not be decided automatically are handed to a person instead of guessed cleverly.

An agent that rejects most of what it finds, with a written reason→

Work that runs when nobody is watching

Pipelines and agents that keep durable state, resume instead of restarting, record a reason for every decision they make, and stop dead before the actions that need a person. Autonomy is easy; the list of things it is not allowed to do alone is the part that takes judgement.

A prospecting agent that rejects most of what it finds→

Information at a scale nobody can check by hand

Acquiring from the outside world at volume, keeping the source whole and addressed by a hash of its contents, classifying it when the source carries no useful label, and making sure any figure produced months later can be walked back to the bytes it came from.

CPOS — every source preserved by a hash of its own contents→

The whole surface, not the convenient half

Product logic, data model, migrations, backend, interface, third-party integrations, deployment. Not because a generalist beats a specialist, but because the failures in this kind of software happen in the joins between those things, and somebody has to own the joins.

A marketplace with escrow, built end to end→
Restraintwhat did not get built

Four times something cheaper was available and did not ship.

The useful question about anyone who can build complicated things is whether they know when to stop. These are the times a rule was available in an afternoon, or a finding could have stayed in a drawer, and it went the other way.

A payment arrives and the software cannot tell which of a guardian's children it belongs to.

What was available

Three rules were available: split it evenly, clear the oldest arrears, apply it to the most recent invoice. Each ships in an afternoon and each survives a design review.

What happened

None of them shipped. The payment goes to a person with the options already assembled, because a wrong guess credits one child and leaves another in arrears, and nobody finds out until a pupil is sent home. The alternative costs a bursar fifteen seconds.

School fees, reconciled across many schools at once→

A system we had built produced a finding against the premise it was built on.

What was available

Re-run it with different settings. The result was inconvenient, there were plausible reasons to doubt it, and nobody outside would have known.

What happened

The design changed. That is the whole reason the thing replays exactly. A result we dislike cannot be quietly re-rolled until it agrees with us.

Research we were doing reached a verdict that went against the commercial outcome we wanted.

What was available

Report the four theories that survived and leave the seven that failed in a drawer. One experiment had returned a flat zero.

What happened

The refutations were written up and delivered at the same weight as the survivors, including the zero. The most useful thing it produced was three filters that now kill a whole class of bad candidate early.

Infrastructure for research that keeps itself honest→

Software of ours was about to be able to spend money and contact strangers on its own.

What was available

Ship the capability and add the limits after the first incident, which is what usually happens.

What happened

Three things were written down as off-limits while it still could not do any of them: spending money, contacting a real person, touching anything live. Once a capability exists, every argument for restraining it starts to sound like an argument against shipping.

An agent that researches sales prospects on its own→
Underneath itsix principles, all enforced in code

Six things we do, and what each one cost to learn.

These are not preferences. Each one is enforced somewhere in running code, because a principle that depends on everybody remembering it is not a principle. Each says where it came from and what ignoring it costs.

01

Preserve the evidence, not your summary of it

Store the source as it arrived. Summaries are a lossy answer to one question you happened to ask first.

When a system reads something from outside itself, the tempting move is to extract what you need and discard the rest. It is smaller, faster and tidier. It also means that in three weeks, when the question changes, and it always changes, the material to answer it is gone and cannot be re-fetched, because the page has moved on.

So both research systems we built keep every source page whole, addressed by a hash of its own contents. Any figure either system produces can be walked back to the exact bytes that produced it, months later, by anyone.

The hash matters as much as the keeping. It means the stored copy is provably the thing that was fetched, not a copy someone edited afterwards to agree with the conclusion.

Learned in
CIE and CPOS
What it cost
Cheap to do at acquisition. Impossible to retrofit once the source is gone.
02

Corroboration is linked, never collapsed

Two sources saying the same thing is stronger evidence than one. De-duplicating destroys exactly that.

Every pipeline de-duplicates, and almost every pipeline does it by throwing the duplicate away. That is right for storage and wrong for evidence: the fact that two independent people reported the same problem is itself the finding, and discarding the second one deletes the strongest signal in the corpus.

So duplicates are linked, not removed. The record keeps one item and a set of independent arrivals at it, which means a claim supported by nine sources is visibly different from one supported by a single post. That distinction is invisible after a naive de-duplication.

Learned in
CIE, across the whole corpus
What it cost
The strongest conclusions in the corpus were the corroborated ones. They would have looked identical to the weakest.
Check it against
03

Measure the parameter, do not configure it

A hand-written list is a guess wearing the costume of a decision, and it fails silently on the first input nobody anticipated.

The classifier needed to know which communities could be believed when someone wrote something ambiguous about owning a business. The obvious implementation is a list of trusted sources, maintained by hand. Ten minutes, and correct on the day it is written.

It is also unfalsifiable and unmaintainable. Nothing tells you when it goes stale. It breaks the moment a source is added by someone who has never read the reasoning, and the breakage surfaces as a slightly wrong conclusion months later, not as an error.

The replacement measures the thing instead: for each community, how often does an ambiguous ownership phrase travel with an unambiguous one? That number is computed from the corpus, updates itself as the corpus grows, and can be argued with, because it is a measurement and not an opinion.

The threshold came from a gap in the resulting distribution, not from preference. Real operator communities landed at ten per cent and above, everything else at 7.7 and below, and nothing in between to have an argument about.

Learned in
CIE, community calibration
What it cost
A whitelist would have produced the same answer that week, then lied quietly.
Check it against
04

Attack every conclusion, especially the ones that look right

Go looking for the evidence against a claim before you rely on it, including the claims you like.

In the research engagement CPOS was built for, eleven theories were opened about where an opportunity might be. All eleven were then deliberately attacked, not only the doubtful ones, because the doubtful ones were never the danger.

Four survived. Seven were refuted, several of them after they had already looked correct, which is the whole reason the discipline has to be mechanical rather than a matter of judgment: at the moment a claim looks correct is exactly when nobody feels the need to check it.

The verdict went against the commercial outcome we wanted, and that is what got written up and delivered. The most valuable thing to come out of it was not an answer but three filters, each of which now eliminates a whole class of bad candidate early. One of them accounts for three of the seven refutations on its own.

Learned in
CPOS-MKT-001
What it cost
Seven wrong theories found before they were built on, not after.
05

Let unknown be an answer

A system that cannot say 'we do not know' will say something else instead, and you will not be able to tell which it did.

Scoring anything involves dimensions you have no evidence for. The convenient move is to score those zero and carry on, because a number keeps the arithmetic working. It also makes an absence of evidence indistinguishable from evidence of absence, and the two are opposites.

So unknown is a permitted, first-class result. A scorecard with four measured dimensions and three unknowns is reported as exactly that, and the reader can see how much of the assessment is actually supported.

The same rule governs anything we put in front of a client: we will not attach a figure to something we have not measured. It is easy to write that a slow website is costing someone forty thousand a year. Any owner who knows their own numbers spots the invention immediately, and then doubts everything else in the document.

Learned in
CIE opportunity scoring, and every proposal since
What it cost
One invented number costs you the credibility of every real one beside it.
Check it against
06

Enforce it in code, or admit it is only an intention

A principle that depends on everyone remembering it is not a principle. It is a preference with good PR.

Genesis was built with one hard constraint: every score computed by ordinary code, no language model anywhere in the scoring path, the whole thing running offline and replaying exactly. That constraint is checked by the build, not by us remembering it.

It earned its keep the first time the system produced a result we did not want: a finding that undercut the premise the whole project rested on. Because there was no model in that path, we could not attribute the result to randomness, a bad sample or a temperature setting. It had been made impossible to argue with, deliberately, before we knew we would want to.

The finding then became a constraint of its own, and that gate lives in the generation path rather than in a document, for the same reason.

The same applies to stopping. Before extracting a subsystem into its own repository we wrote down the condition that would make us abandon the attempt, while we still did not know the answer. Two days in, with a branch open and half the tests passing, no amount of mess ever feels bad enough to justify throwing the work away. You cannot out-discipline that in the moment; you can only take the decision away from the version of yourself who will be tired and invested.

Learned in
Project Genesis, and one repository extraction
What it cost
The constraint is only worth anything on the day you would rather it were not there.
Check it against

What that looks like from your side.

The principles above are how we make decisions. This is the shape of the work around them — the same four steps whichever of the three engagements it turns out to be.

  1. 01You send the problemA few paragraphs is plenty. What the system does, what worries you about it, and when it needs to be sorted. No form with eleven boxes.
  2. 02A 30-minute call, freeWe tell you which of the three fits, or that none of them do. We have turned work down on this call before. Taking on the wrong job costs us more than an empty week does.
  3. 03A scope, in writingWhat is included, what is not, the price, the dates, and the point at which we would stop and tell you. You approve it before any work starts.
  4. 04The work, visiblyA short written update each week, measured against that scope. If something we assumed turns out to be wrong, you hear about it that week and not at the end.

All six are visible in the work.

Every one of these came out of something that runs, and each names the system it came from. The systems are on the work page, most of them with a picture of them doing it.