Why (Un)certainty Should Dictate Your Data Strategy

We’re often confronted with biased blanket statements like “Data-Driven Marketing is The Future” in the world of digital marketing (and likely in other sectors as well). Salespeople and marketers understand that “it depends” is not what executives want to hear. People like clear and concise advice and solutions. Silver bullets.

This silver bullet approach and lack of nuance, combined with an industrywide propensity for “solutionism” is slowly but surely creating more skepticism towards “data” and the way it can be used in digital business.

In this article, I want to outline 3 major ways data can be used and why I think the amount of (un)certainty of the value of what you are trying to achieve is the main predictor on what approach to choose from.

Different ways of using Data

Data comes in all forms and sizes and it should not come as a surprise that you can also use it in different ways. Below I describe 3 main ways I see data being used in different circumstances at companies.

Data-Informed vs. Data-Driven

Data-Driven is something you have definitely heard, it’s a buzzword up there with “big data” and “AI” in the buzzword bingo list. But Data-Informed is a younger sibling ofter referred to by Silicon Valley Product people as a better mantra than data-driven. Personally, I think both are useful but in different situations.

Data-Informed refers to processes that use data to inform decision making but that still heavily rely on non-quantifiable ways to decide what to do next. Data-Informed is often used in cases where there is not enough data available to reach statistically valid conclusions or where the correlation between what is being measured and desired business outcomes is not (yet) obvious.

Data-Driven refers to processes that use a more scientific approach and tries to decide decisions based on the data. Usually, it involves a best practice of creating a hypothesis and deciding on the metrics to evaluate the research question beforehand. Data-Driven is often used when aligning larger groups of people around the same goals and for optimization of processes that have proven to add value to the overarching business goals.

Automated Processes

Last but not least there is the process of automation based on data. Machine learning would fall into this category but also simpler forms of rule based decision making on large volumes of data that see no human intervention. This form of data usage often takes place with larger volumes of fairly well understood and standardized data where there is some additional value to be gained from speeding up the process.

Operating in (Un)certainty

I think the best predictor of the data process you need to apply to your problem is the degree of certainty involved with all “bets” you’re making. If you are operating in highly uncertain terrain (let’s say product development for a startup) you’ll probably fair best with a data-informed way of working using a lot of qualitative insights complimenting quantitative data.

You are testing the waters and are likely to be wrong a lot which is why there is no point in building a data-driven or automated way of decision making. There is likely not enough data to do that anyway and if there was you are still unsure on what data to interpret as predictors of true success or failure.

If you’re optimizing the conversion rate for a fairly well-established e-commerce webshop across multiple development teams you will likely have a more data-driven way of optimization with some solid control metrics. The main value proposition of your business is proven but you are optimizing the way to extract optimal value from it.

If you work at a company that has high volumes of traffic or another part of the business that processes high volumes of data (like trading companies) you’ll likely see that alternating between data-driven and automated processes makes the most sense. Just like in data-driven, the main value proposition is likely already proven and the focus of the project is extracting small and incremental but worthwhile because of the scale.

All models are wrong

The model above was what I came up with while thinking about how to simplify the problem of when to apply what general way of working with data. It’s far from perfect, but I feel it adds more value than the online shouting match of “data-driven is wrong, you should be data-informed” or vice versa. Hope it helps!

All models are wrong but some are useful.

George Box, Statistician