Building a Model Worth Believing

The meeting had been on the calendar for two weeks, a model review of the kind where outputs are presented, interrogated, and eventually translated into a budget recommendation that will move real money. The data science team had built something technically solid: a modern Bayesian MMM, carefully scoped with agreed-upon variables, clean posterior distributions and reasonable in-sample fit, and no obvious diagnostic flags. The measurement consultants on the other side of the table, whether internal analytics leads or the vendor managers who sit between the model and the business in many organizations, had not built a competing model, because their role was to vet it, to pressure-test the outputs against accumulated business knowledge so that whatever recommendation left the room would be as trustworthy as possible.

They were not satisfied with what they saw.

Connected TV had been the story of the prior model run, with contribution estimates that were among the strongest in the mix and efficiency metrics that anchored a clear recommendation: increase CTV investment in the next planning cycle. The client had acted on that recommendation, budgets had shifted, and the upfront had been signed. Now the new model was on the screen, and CTV was near the bottom of the efficiency ranking, not marginally lower but materially and significantly lower, in a direction that had no obvious explanation in the spend data, the creative rotation, or anything the client's media team could point to. The data science team noted that the diagnostics were clean, the posteriors had converged, and the new model reflected an updated six months of data alongside a refined prior specification. The measurement consultant, who had been the one to present the prior model's CTV findings to the client, was not reassured by that answer, because she had both a professional stake in the previously reported results and a legitimate methodological concern: if the model could move this far between runs with no corresponding change in market conditions, what was the recommendation actually worth, and how could she take it forward to key stakeholders with confidence?

Both sides were right about the parts they could see, and the budget decision left the room without an answer.

Most practitioners define a trustworthy model the way a clinical trial defines a successful drug: it passed the test. The diagnostics look clean, the R-squared is high, the posterior distributions have converged, and the fit line tracks the actuals closely enough that no one in the review raises an objection. This is a reasonable definition of a model that is not obviously broken, and it is not a definition of a model that is strategically correct.

What makes this harder than it sounds is that trustworthy means something different to every person in the room. The data science team is asking whether the model is well-specified. The measurement consultant is asking whether the outputs are consistent with what she knows about the channels. The CMO is asking whether the recommendation is defensible to the CFO.

A model can pass every standard diagnostic and still give you a structurally wrong picture of your business. It can assign implausible contribution to a channel that has been running continuously for three years, because the model has no way to separate its effect from the baseline it has quietly become. It can undervalue a brand awareness investment because the adstock decay parameter is set too tight, and the long-tail carryover never shows up in the revenue signal. None of these failures announce themselves in the output.

The first is false precision. This is the model that arrives in a boardroom looking fully formed, with clean outputs, confident point estimates, and a clear budget recommendation. The uncertainty has been compressed into a single number, and the assumptions underneath it are invisible.

The second is false humility. This is the model that arrives wrapped in so many caveats, credible intervals, and qualifications that it communicates nothing actionable.

Reaching that goal requires a reframe. Trustworthiness is not a one-time achievement, it is a continuous practice, earned through the quality of the initial build, maintained through a rigorous rerun cadence, and renewed every time the business changes.

Most modeling teams begin by choosing a framework. The more disciplined ones begin by auditing their data, because the decisions made in that audit will constrain every modeling choice that follows.

Before any data is pulled, the most important early step is a general alignment with the client team on what they are trying to measure, what channels and time horizons are in scope, and whether the data infrastructure exists to support an MMM at all.

The first and most consequential scoping decision is the formal definition of the dependent variable. This is not a technical choice, it is a business one, and it needs to be made explicitly and documented formally before any data is pulled.

The standard for MMM data architecture is two to three years of weekly observations. Two years is the commonly recommended minimum. Monthly data is too coarse to model adstock dynamics reliably, while daily data is acceptable but introduces additional noise.

Modeling with less than two years of data is possible, but model quality and posterior confidence will drop materially.

Data rarely arrives clean. The modeling team collects, ingests, and processes the inputs, then summarizes them back to the client before finalizing the data table. This review step is the most cost-effective intervention in the entire build.

A practical data audit covers: dependent variable definition, independent variable definitions, data completeness, temporal consistency, any restatements, and formal stakeholder sign-off.