The earliest known use of the term “Business Intelligence” (BI) dates back to 1865. It was found in the Cyclopædia of Commercial and Business Anecdotes. The author used the term to describe how an English trader gained profit by receiving and acting upon information prior to his competitors. Today, while BI is booming, this is still the most commonly accepted definition of “Business Intelligence”.
What follows was presented at NOAH17 Conference in London. During the two-day conference, hundreds of CEOs and founders from emerging disruptors shared tips and trends with some of the best investors in Europe. Whether you listened to Alia Parsa, the founder of Babylon (a medical assistance app), Florian Reuter, the founder of Volocopter (an intra city flying taxi) or to any of the hundreds of other presenters, one common denominator emerged: innovation is accelerating.
Regulated industries such as health care or transportation have been relatively spared from disruption up to now but they are about to undergo seismic changes. This at least partly explains why business intelligence continues to ascend in corporate priorities.
Why is Business Intelligence Growing?
1. Business Intelligence is a not a fad. The BI software market, as estimated by Gartner, is roughly $20 billion. It is a vast market with iconic players such as IBM, Microsoft, Oracle, SAP and SAS, and many new ambitious, emerging companies popping up and driving growth.
2. BI is going mainstream. Whether you ask Gartner or Technavio, the estimated growth of the global BI market is about the same: 10 percent year over year for the next four years. The revenue growth is fueled by the growing number of users worldwide but dampened by the competitive pricing pressure among the increasing number of players.
3. New tools keep emerging. Back in 2011, there were nine players in business intelligence, led by SAS and IBM and focused on a few CFOs, CIOs or CEOs in corporate accounts. Six years later, the number of leading BI players has grown 20 times to over 200 significant software providers. Whatever your function is, there exist or will exist several potential BI solutions for you.
4. Your competitors are watching you. Pushed by so many emerging solutions, business intelligence is becoming used throughout small to large companies by millions of employees. While BI tools once were designed for a few corporate functions they now span nearly every function, including finance, compliance, talent management, recruiting, marketing, sales, manufacturing or operations. The take-away: Your competitors are very likely investing in BI.
5. BI solves business problems. Most companies do not have the resources of Amazon, Google or Facebook. Starting by building big data repositories leads to nowhere. No one will use them. Instead, start by looking at the business problems that you want to solve. A concrete example could be: Why are some of your customers leaving or how could you grow one of your customer segment by 20 percent? In other words, good questions typically lead to good answers.
Trusted Data is Fuel for Business Intelligence.
6. Check, Check, Check. It is far better to have less data that you trust than a bunch of data that you suspect. When someone start claiming “I do not believe in those data,” this tanks the whole report. The evils lie in the details. For example: Wrong currency, wrong period, using growth sales instead of net sales. Triple checking is thus not an option to run your BI and train your algorithms.
7. Innovate with real time, diversity and volume of data. Data is being created faster than ever, and the more quickly it is processed the more relevant it is likely to be. Data is also available in increasingly diverse forms such as text, conversation, pictures, video, mobile phone tracking in front of your shop windows, social media, likes, weather, GPS data. This growing variety of data means more ways to find innovative solutions. About 20 percent of data collected is structured data (displayable in rows and columns). The rest are unstructured such as a conversation or semi-structured such as a picture with footnotes. They require more advanced analytics. Too much data can make any implementations complicated and incur wasted time and expense.
8. Select the right data structure. If the data do not have the right structure, updating or retrieving that data might become incredibly cumbersome. In two to three years, when the volume of data will be ten folds to hundred folds the size it is now, the cost to adjust the data structure is likely to be huge and some data may have to be discarded. This step requires database experts. Here are the most frequent pitfalls on data structure: Poor naming structure; confusing data hierarchy to drill down; incoherent sources for the same data; ignoring some small, valuable sources.
9. Learning by doing. Collecting and preparing data is a long journey with tons of details to fine tune. So the best approach is to have an agile, hands-on culture to adjust, improve and expand. Starting with small steps develops internal literacy and the appetite to innovate. At the end of the day, BI success relies on having smart, passionate people who know what they are doing.
Build a Core Team of Analysts and Scientists.
10. Set up a core team. In our corporate org charts, five years ago, we had less than 10 VPs of Analytics or VPs of Data. Five years ago, BI functions were generally included as part time jobs for the Chief Digital Officer, Chief Marketing Officer, Chief Information Officer or the Chief Financial Officer. Today in our org charts, we have thousands of VP Analytics or VP Data or VP Artificial Intelligence. Why? As the amount of data grows, BI and data has required a dedicated, proactive team able to analyze data and activate simple algorithms.
11. A data scientist is a data geek who loves working with data. He is able to create machine learning algorithms. He knows several industry standard analytics packages produced by IBM, Microsoft, Oracle, Google or SAS. He uses programming languages such as Python and is comfortable with statistics. Those competences are unlikely to exist in companies, so most companies have to hire them.
12. Hire for excellence. Hire for curiosity, technical aptitude and business understanding. You must have domain knowledge, but there’s this extra curiosity factor and tenacity that will make the whole difference.
13. Upskill your data scientists. All statistics show that talented people are in short supply and the market is highly competitive to find data talent. So you might consider training your own staff on the technical part. Many online technical courses are proposed by MIT, Berkeley, Stanford or IBM. Data scientists often have a hard time detailing their insights to business people. Thus, a lot of valuable information tends to get lost in the process. To fix this, you have two options: One, you train your data scientists about your business, or two, you hire data translators who can build communicative bridges with business leaders.
14. IT is the needed infrastructure partner. To start quickly, many BI users flocked to new tools that they could use without IT assistance. Yet, the volume and the variety of BI data have become so significant that partnering with IT often is no longer an option. The infrastructure must be right. The main areas where IT will strongly contribute: Scale (collect, storage, analytics, access); security and integrity; metadata, privacy and regulation.
Empower on Data Preparation and Analytics.
15. Empower business teams with easy access. Here are a few examples to make this access to data easy: Make data organization so easy that people can access and use it in self-service; have management favoring short narratives about data rather than slick reports requiring weeks to assemble; and provide easy visualization tools for the data. There is a very powerful tool that is installed on most PCs that is very instrumental for business intelligence. Any idea? It is 32 years old.
16. Excel is so powerful! Excel has become so powerful. At The Official Board, we use it to process files with over 800,000 lines. Mastering the following five Excel functions is very instrumental in business analytics: Pivot table; vertical look up; sum if; textual filter and graphics.
17. Neural networks vs. linear regressions. Neural networks are often perceived as the ultimate algorithms to provide predictions and prescriptions. Everything that smells statistics is called artificial intelligence, deep learning, machine learning, machine reasoning or genetic algorithms. Let’s remember that a neural network today is a succession of three to ten advanced linear regressions. For small scale projects, smart linear regressions can deliver as much as sophisticated, hard-to-implement algorithms.
18. Watch those three promising technologies. We saw how BI was going mainstream and was warming up. Three areas where technological leaps are happening: Big progress on analytics to process conversation, videos, text, pictures; one stop shops providing cloud platform across industries or verticals at very competitive costs such as AWS, Cloudera or IBM; much smarter visualization tools such Tableau, Qlik, Analytics 360 and natural language generator such as Quill.
There are three primary and frequent myths on BI.
19. BI Myth #1: Artificial intelligence can predict the future. Smart algorithms define the most likely events based on historical data. When injecting large quantities of data, it considers an incredible number of historical events. When analyzing real-time data, it takes into account what happened less than one second ago. Data-driven prediction or Data-driven prescription are about extrapolating probabilities. So there is always a margin for errors. We all know how inaccurate political forecasting and polling was during recent elections. Yet, huge quantities of data coupled with advanced analytics were used by smart people.
20. BI Myth #2: Everyone is already doing it. Hmm… Not really. Everyone certainly has the ambition to do it. Yet it is all about execution. It takes time to create a data culture even in a small organization. New Vantage partners has interviewed 50 Fortune 1000 companies: 85 percent have started programs to create data-driven cultures, but only 37 percent report success thus far.
21 BI Myth #3: BI is so strategic that it cannot be subcontracted. As we have seen, BI can be broken up in several steps requiring different expertise. In practice, only a few steps absolutely have to be run within the company. I have listed the main eight steps for implementing a business intelligence project. In the table above, for each step, I have indicated whether or not it can be successfully developed outside the company. For example, for sourcing the data, it is best is to have internal and external sources. For setting up the data structure, what is essential is to have experienced people on structuring database. It does not really matter whether they are within companies or outsourced to contractors. For advanced analytics, a possibility beyond internal stall and contractors is crowdsourcing on sites such as Kaggle.
Conclusion: If there were only three take-aways on business intelligence, what would they be? First, start small a compelling and definable objective: For instance, how can we stop losing some customers? Second, triple check your data. No one else is going to do it for you. And third and finally, learn the five Excel functions that I referenced in item 16 above.