Before owning a model, own the test

A company considering its own AI model should be able to name the failure it intends to fix. “We want to own our intelligence” is a strategic preference. “This task is too expensive at the quality our customers require” is a problem a team can investigate.

Sequoia’s August 19 essay on model ownership makes a useful distinction: using frontier APIs remains appropriate for many workloads, while some companies are taking greater control of specialised parts of their systems. Its proposed sequence puts evaluations ahead of model training. That ordering deserves attention.

Without a stable test of the work, a company can spend months changing its model and remain unable to say whether the product improved.

Make the comparison expensive to fool

Consider a hypothetical application that turns supplier documents into structured records. A demonstration can succeed on a clean invoice. A meaningful evaluation has to include awkward layouts, missing fields, contradictory totals and documents the system should decline to process.

The test also needs to reflect the cost of an error. Getting a supplier’s punctuation wrong is different from assigning a payment to the wrong bank account. An average accuracy score can hide that difference. The team needs to know which mistakes require a person, which can be corrected automatically and which should stop the workflow.

This is where product knowledge becomes technical work. Someone has to choose representative cases and decide what acceptable performance looks like. That judgement will not arrive automatically with a better base model.

Compare complete systems

Once the task is measurable, the alternatives become easier to evaluate. A different model may help. So might retrieving the right policy, changing how a document is parsed or showing uncertain fields to a reviewer. These interventions have different development costs and maintenance requirements.

The relevant comparison is the cost of a successfully completed task. For the document example, that includes inference, retries, review and correction. A cheaper model that sends twice as much work to an operations team may cost more overall. A more expensive model could be economical if it reliably reduces that work. Either conclusion requires observation.

Owning a specialised model adds another line to the calculation: maintaining it. The team needs a process for new document types, changing customer requirements and regressions after an update. That work can be worthwhile, particularly when the task is frequent and well understood. It should appear in the decision before the first training run.

There is also value in preserving the ability to change suppliers. If the evaluation, task definitions and failure records belong to the company, a new model can be tested against the same standard. The business can revisit its decision as quality and pricing change.

Sequoia’s broader thesis is that more product companies will shape the intelligence they depend on. Our narrower conclusion is that the evaluation is useful whichever route they choose. It turns a discussion about architectural conviction into a comparison the company can repeat.