Building a Model Worth Believing

← Back to Journal

Return to the meeting from Section 01. The diagnostics came back clean. MAPE is below 10%, the posteriors have converged, the holdout validation is within range, and the decomposition sums correctly to observed revenue. The data science team has done everything right by every standard internal measure. The model is not broken, and yet the measurement consultants across the table are not satisfied, because something in the outputs does not match what the room knows to be true about the business.

A broken model is fixable. A model that has passed every diagnostic and is still producing outputs that cannot be defended in a review room is a different kind of failure, one that does not announce itself in the posterior distributions or the residual plots. It lives in the gap between what the model is technically capable of reporting and what the business is actually asking it to explain. The table below organizes the most common credibility gaps by what the room sees in the outputs, what is likely causing it, and where to look first.

The thread connecting every row in this table is the same: the model is technically correct within the assumptions it was given, and those assumptions are wrong in some way that the diagnostics could not see. The model is not lying in the adversarial sense; it is reporting faithfully on a picture of the business that was never accurate to begin with. The job of the measurement leader in that review room is to understand which assumption is failing and why, not to defend the model and not to dismiss it.

The resolution is not a better model or a more skeptical consultant. It is a structured process for building models in a way that makes their assumptions visible before outputs are produced, and for conducting reviews in a way that gives every person in the room a precise account of what the model can and cannot see. That process is what Section 08 describes.

Most model review failures do not originate in the review itself. They originate in the gap between the alignment session and the moment outputs are presented, when changes accumulate quietly and no one realizes the model being reviewed is not the model that was agreed upon. This section assumes alignment happened: priors jointly specified, data signed off, parameters agreed. What follows is what goes wrong after that, when the agreed model drifts before anyone sees the outputs. The most common ways that gap opens are:

• Spend data pulled incorrectly by data engineering: a channel's weekly spend series contains a processing error, a duplicate week, a misallocated cost, or a suppressed spike that was never caught in EDA.
• Priors silently revised after alignment: a prior that was jointly specified and documented gets tightened or loosened during the build without the change being surfaced to the measurement leader before the review.
• Adstock or saturation parameters shifted from agreed ranges: changing how a channel's carryover or diminishing returns are modeled in ways that can materially move its contribution estimate.
• A control variable or event bump added or removed after data sign-off: altering the baseline and redistributing contribution across channels without a corresponding update to the assumptions log.
• The modeling window shifted without notification: adding a quarter or trimming a problematic period without surfacing the change for review.

Any one of these can produce outputs that look diagnostically clean and are commercially wrong, and the measurement leader who does not know the model changed has no basis for challenging the result. Before a review is scheduled, the following conditions should be confirmed:

• Data sign-off is documented and no changes to the spend series have occurred since sign-off.
• Prior specification is documented and matches what was jointly agreed before the build began.
• The 10-Step P&E MMM Diagnostic Sequence from Section 06 has been completed, with any flags prepared for disclosure rather than discovered in the room.
• The decomposition has passed the internal sanity check, with baseline, seasonal peaks, and channel contributions reviewed against business knowledge.
• Where other empirical evidence exists, whether prior MMM runs, incrementality experiments, or other causal studies, the comparison between model estimates and that evidence is documented, with any divergence and its likely explanation ready to present.

Every practitioner reading this has been in a version of the same disagreement. The new model is on the screen, and CTV, which carried the strongest efficiency story in the prior run and anchored the budget recommendation the client acted on six months ago, has moved materially in the wrong direction, with no obvious change in spend levels, creative, or competitive conditions to explain it.

In the unproductive version of this conversation, the data science team defends the methodology and the measurement consultant defends the previously reported results, the argument stays at the level of outputs with no resolution available at that level, and everyone leaves the room frustrated with no decision made.

In the version where the assumptions are visible, someone pulls the spend data before the conversation goes further. In this case, a data engineering error had introduced a duplicate spend entry across several CTV weeks during the new modeling window, inflating the apparent cost base and compressing the channel's measured efficiency with no corresponding change in its actual performance. The correction is made, the model is rerun, and the CTV estimate returns to a range consistent with the prior result. The measurement consultant, recognizing that the experimental evidence on CTV has always been thinner than the confidence the prior model implied, recommends a geo-based incrementality test for the next planning cycle to build a stronger evidentiary foundation before the next budget commitment. That recommendation leaves the room as a committed action item, and the client has a cleaner basis for their upfront decision than either model run alone could have provided.

Six things should leave every model review with owners and timelines. Accountability for all six sits with the measurement program leader, who owns the record and ensures it is shared cross-functionally with the data science, data engineering, and client-facing teams whose work it depends on:

These six are the review-level record, owned by the measurement program leader. They sit alongside, not in place of, the system-level logging maintained by the data science and engineering teams: pipeline run logs and data freshness timestamps, model fitting artifacts such as sampler diagnostics, trace objects, and convergence statistics, version control history for the modeling code, and data lineage records tracing each input back to its source. The review record captures the decisions and their rationale, the engineering logs capture the technical execution. Both are required, and neither substitutes for the other.

The model review generates the measurement agenda, not just the budget recommendation, and the meeting from Section 01 that ended without an answer should always end instead with these six outputs and a room that knows precisely what the model can see and what it cannot.

Most organizations treat their first credible MMM as a destination. The build is complete, the review went well, the budget recommendation was accepted, and the model is filed alongside the presentation deck that delivered it. Six months later the planning cycle returns, the data is stale, and the rebuild begins from scratch, relitigating the same prior specification debates, the same data audit decisions, and the same diagnostic sequence without the benefit of what the prior run learned. The institutional knowledge evaporates. The credibility that was earned has to be rebuilt.

Measurement programs that are ahead of this curve do not wait for the next planning cycle to find out if the model is still telling the truth. They monitor continuously through two mechanisms. The first is periodic model reruns: refreshing the data window weekly, monthly, or quarterly depending on how quickly the business and its media environment move, running the accepted model's posteriors forward as priors, and checking whether contribution estimates have moved in ways that can be explained by what actually changed in the business. The second is forecast monitoring: running updated spend and control data through the accepted model as-is to generate short-term revenue predictions, then comparing those predictions against actual outcomes on a rolling basis. A model whose forecasts are consistently off in a particular direction is signaling a structural change the current specification has not yet captured, and catching that signal between builds is far less costly than discovering it in a review room six months later.

The framework described in this essay is designed to prevent that. Data architecture decisions documented before the build, priors jointly specified and recorded, diagnostics run in sequence and flagged before the review room, model configuration and change logs maintained across runs: these are not overhead. They are the compounding asset that makes each successive model more credible than the last and each review room less contentious than the one before it.

The posterior-as-prior mechanism described in Section 03 is the technical expression of this compounding. By using the accepted model's posterior distributions as the starting priors for each subsequent refresh, the effective evidence base grows continuously even when the active window stays fixed. Every experiment result fed into the next build's prior, every assumption documented and revisited, every cross-run movement explained rather than presented: these practices accumulate into a measurement program that earns trust over time rather than defending outputs one review at a time. The lifecycle of that program, how it refreshes, when it rebuilds, how AI is beginning to automate the connective tissue work that frees measurement leaders to focus on the judgment calls that actually require human expertise, will be the subject of future essays.