An insights strategy for winning companies

An executive summary

Companies struggle to gain maximum benefit from analytics and insights since

  1. Analytics is seen as a support function, not a business partner and therefore is  not prioritized high enough
  2. Analytics is separated from business processes and insights are produced away from execution
  3. Under-resourced, inflexible analytics stack, which doesn’t enable speed to react to changing needs
Conversely, in high-performing organizations, analytics and insights is as much a business function as the rest. This ensures analytics and insights gets prioritized with the same focus to the rest of the functions, rather than being secondary. Analytics and insights is a fully-fledged business partner.

Business processes are imbued with analytics and insights right in the core. This is achieved by forming systems of insight; teams that have full insights-to-execution ownership and are manned accordingly. These teams are independent from a centralized analytics and insights department, which ensures they are frictionless in implementing optimal experiences with insights.

Winning organizations invest in a flexible analytics stack for the long term, which doesn’t solve one problem well, but rather is capable of adapting unforeseen business needs quickly. Purposefully building speed and flexibility into the stack ensures long-term competitiveness, as needs change all the time.

Imagine this

A new user registers to your product and based on the information they provide you are immediately able to describe them in a meaningful manner. Based on where they are from, how they behave and their public social media profiles you are able to build heuristic multi-profiles that are extremely relevant to your business. These multi-profiles fuel your on-boarding and recommendation engine, individually tailoring messages to your new signups that optimize their experience as well as the likelihood of them converting to paying customers.

You are prioritizing your roadmap and deciding the order of projects A, B and C. As you understand thoroughly how these projects affect your customer experience, you know the gaps in your offering and the customer preferences. As you have analyzed the causal effect the improvements in different experiences have on different conversion metrics, you are able to make highly accurate predictions the different projects will have on your overall business. As the whole company is aligned behind the same KPIs, there is very little discussion about what the priorities should be.

A question pops into an employee’s mind. Unfortunately the Data Science team is very booked for the next couple of days, so there would be a few days’ lead time to get this information through them. Luckily, since your self-help tools are so powerful and everyone has access to them, the employee is fully capable of answering their own question quickly and efficiently.

Analytics is no longer a support function

It might have been that in the past analytics belong to the IT department and was irregularly summoned to existence to answer specific business questions. This kind of setup puts data and business in a high-lag dialogue, not unlike talking to a space station orbiting Mars. Nowadays data analysis is a key differentiator for companies. Compared to computers, humans are undeniably better at innovating and coming up with strategies and perhaps fundamentally for that reason data analysis will not (ever?) be the leading function in a company. Among the winning companies though, data analysis is an equal business partner at the leadership table, integrally embedded in everything through people, processes and technology. 

Winning Insights & Analytics groups are evolving from the traditional support function role to one of a pro-active team player, providing scenario planning and ecommendations, and ultimately being regarded by the rest of the business as a fully integrated business partner that has a seat at the leadership table, driving strategy and real-time execution together with Marketing, IT and Finance colleagues.

This is a quote from a comprehensive survey study conducted by Millward Brown Vermeer (2016, p.5) and in that they describe three categories that separate over-performers from the under-performers: customer obsession, designing total experiences and something they call the ‘Insight Engine’.

Customer obsession could be characterized by insisting to take the voice of customer into account at every business decision or having performance KPIs that measure the customer’s success rather than the company’s. Designing total experiences essentially means designing consistent purpose-lead solutions and pursuing that relentlessly, backed-up with data.

The last point, an insight engine, is the equal business partner component. Compared to under-performers, over-performers are four times more likely to state that insight and analytics are leading the business

The insights transformation happens in all functions

In a successful insights transformation, all individuals in a company get empowered through insights. In the old setup, data and business was in an infrequent dialogue, the business submitting requirements to data and data trying its best to reply with answers after a delay. Similar to the findings in the Millward Brown Vermeer study, also Forrester has concluded that to win, insights needs to be tightly embedded into everything (Figure 1). Forrester’s study found that the over-performers had multiple systems of insight embedded into the processes. These systems (teams) are able to use the right data from all possible data, analyze it and draw the effective conclusions.

And this needs to happen everywhere.

(Figure 1. Systems of insight, adapted from Forrester 2015)

One crucial implication of this is that being able to draw basic day-to-day insights from data is less dependent on a centralized analytics team. The systems need to work without a tin can telephone to head office. More accessible self-help analysis tools will support that, but more importantly it is important to identify the critical insights-hungry processes, ensure they have insights people embedded and purposefully work on improving data intuition.

The end result is an organization of multi-disciplinary teams, formed around critical processes, with ownership of the whole insights-to-execution flow. Those who manage to do this will reduce friction of insights dramatically and enable the insights to flow into actions at the operative level. Suddenly, almost magically, all the right decisions will start to be made in the teams and the role left to a central team will be to provide tactical support and improve the platform.

Systems of insight have full-stack ownership of data, from generation to visualization - and all the steps between. In order to achieve this while also executing on the findings, the multi-disciplinary teams will need a wide range of skills: business, technology and analytics.

The insights team should not be dependent on your development department, but rather fused with it. Rather than having separate teams for insights and dev, there is only one around a single topic. Having all the necessary skills in a single team enables them to move significantly faster. Deloitte calls these ‘purple teams’: teams which have combined business and technical talent to deliver analytics edge ("The key to delivering analytics advantage: Your people" ,Deloitte, 2017, p. 21).

“If you say you’re data-driven, but everything has to go through an analyst, you’re not actually data-driven”
-Fareed Mosavat, Group Product Manager at Slack

Speed is key, and a centralized analytics team creates bottlenecks. Not everyone needs to be a data scientist, but the organizations who win will have made data and analysis accessible for everyone. This holds true especially in smaller teams.

In the Lean Startup methodology, one interesting idea is to have ‘Validated’ as step in the ‘definition of done’ (The Lean Startup, p.138). The point is to ensure all stories that are done with the intention to improve the product also have a mandatory validation step before they can be signed off. This fits extremely well with the thinking of systems of insights and cross-functional teams that are responsible for the full insights-to-execution process.

(Figure 2: The Lean Startup methodology, adapted from

It is all about continuous learning: you act with an assumption, test it, learn and form new assumptions. If you are not measuring the impact of your work, you are not learning, but hoping. On page 126 in the book, Erik Ries gives an example about a team that is not producing results. In the example, the management concludes that the team was not working hard enough, although the problem was that the process the development department was expected to follow did not enable learning to take place.

An insights platform for long-term success

Although data is a critical component, surprisingly the battle isn’t about who has the best data. In an interview, Lori Sherer (Bain & Company, 2017) points out that more data isn’t always better and goes on to say that “a simpler solution that is designed with the end users in mind is going to win out 100% of the time”. Her point is that while data is powerful and access to it can give an edge over a competitor, it ultimately comes down to your ways of working. Approaching customers’ needs in an analytical way is the winning component, not having the best data to support it. This doesn’t mean you shouldn’t go after good data. It means you are not necessarily doomed if the data you have access to is inferior to your competition, as long as your ways of working are more customer and goal oriented.

We engineers often think of the technical solution first, but the insights stack should be focused on only once the ways-of-working are in place,  There’s plenty of technical work when building an insights platform that has the speed and flexibility to support all business needs. Having those needs in mind will ensure that the right thing is built. A very brief generalization is depicted in Figure 3.

(Figure 3: A generic analytics stack with some example AWS technologies)

While developing machine-learning algorithms can seem like the most important task to produce value, the foundations and enablers to do so are actually laid out much earlier. By storing all data in its raw format, by creating convenient pre-processing flows and ensuring data quality stays good likely have a higher impact on the end result than endlessly polishing a neural network.

First party data - the kind that you create and no-one else has - is what most likely makes you different from the rest. If you have access to second or third party data, so will others. Enriching your first party data can be important, as it might enable you to find insights otherwise invisible to you. Think of how different industries use the product differently and how it might be difficult to see that, unless you have access to that meta-information.

  • Has access to all (relatively) latest data through a robust ingestion and storing process
  • Enables joining all data in any format effortlessly, even if the data is imperfect
  • Allows changes in schema easily through e.g. virtualization
  • Exposes analytics with production-ready interfaces
  • Respects users’ privacy while not compromising quality
The most interesting question to start with is: "How will the insights stack support my business?" For example, if your end goal is to plug in insights components to your SaaS product for your customers to see, and you anticipate these components to need to do heavy computing on massive volumes of real-time data, then the stack you will need to create is very different from if you only need to do back-office analysis and you know you will do just fine with a data lag of one day.

Final words

Companies can fail to reap the benefits of analytics, since
  1. Analytics is a support function, not a function that sits in the leadership table.
  2. Analytics is separated from business processes, when it should be tightly embedded in them.
  3. Analytics stack is under-resourced and inflexible, unable to meet changing needs.
To fix this, organizations need to acknowledge that data analytics needs to be involved in strategic decision making. If it isn't, your organization will always lag behind and the investments you do make in analytics will yield much lower return.

Consider embedding analytics responsibility in to the business processes: from insight to execution. Rather than being forced in an inefficient dialogue between business and analytics, teams themselves will have everything they need to reach their purpose. Data scientists and analysts should be members of the business teams, enabling the team to be both accountable and empowered to use tight feedback loops to continuously improve.

Lastly, make sure you invest in your analytics stack adequately, from the perspective of your needs. Don't build something that will be unused, but rather, together with your CIO, create a purposeful insights strategy that is flexible and resourced. This infrastructure should be there to serve the ever-changing needs of your business processes, so make sure it is flexible enough.


Popular posts from this blog

Snowflake UPSERT operation (aka MERGE)