Business Intelligence (BI) is about turning data into insight and using this insight to make better business decisions. Today the opportunity to make better decisions is growing as the amount of data being stored increases. Between 2009 & 2020, the “digital universe” is projected to grow by a factor of 44 – increasing from c. 0.8 Zettabyte (ZB) to greater than 35 ZB. This, however, creates a significant challenge – how to distill useful insight out of a growing sea of increasingly disparate data?
This is a particular challenge for Small and Medium sized Enterprises (SMEs). Despite their size, SMEs are nearly always collecting significant amounts of data – sales, orders, customer relationship management (CRM) and website data. The key is how to extract the insight despite the challenges that these organisations face. These challenges include
- A lack of specialist analytical staff with advanced Business Intelligence & analytical skills.
- Restricted budgets – often there is no budget for Business Intelligence training or expensive tools.
- A rapidly evolving business environment that demands the rapid generation of business insight.
This blog is about how to use business intelligence & analysis to make better business decisions despite these challenges.
Business Intelligence can be considered to consisting of three core components : 1) Data Integration -that pushes data in to a data warehouse 2) Reporting & 3) Modelling.
Data Integration is about getting multiple disparate data sources into a central location, a Data warehouse, where data can be analysed or used for reporting. A simple example might involve merging sales and customer contact data so the marketing team can report on sales or contact specific customers with a product offer. When data also needs to be cleansed, aggregated, filtered or have any advanced logic applied this data integration task becomes difficult using generic tools like Microsoft Excel or Access. To make it worse, Excel & other traditional database tools provide little or no auditing of data quality.
Without specialist data integration tools, employees tend to burn an inordinate amount of time merging data files before they can generate say a monthly report. To make it worse each month they would normally need to repeat the process. The end result is they may spend 90% of their time integrating data & generating the report and only 10% interpreting the data. Wouldn’t it be smarter to turn this on its head? – spend 10% of the time integrating the data & producing the reports & 90% interpreting results and making recommendations? The problem is many companies aren’t aware of how to integrate data sources more efficiently.
Once the data is integrated & in one location, reporting becomes infinitely easier. Tools like Crystal Reports, & Microsoft Reporting Services have traditionally been the go-to tools for reporting for organisations. While they are well established, highly scalable, mature and stable tools, they require a high degree of technical skills, experience and time to create a reporting solutions.
In recent years the reporting landscape has been turned upside down with the emergence of visual analytics tools like Qlikview, Tableau, and Tibco Spotfire, and Microsoft Power BI (i.e. Power Query, Power View, Power Map and Power Pivot). These tools are intuitive, more web & mobile friendly and leverage in memory analysis and columnar database to make reporting blisteringly fast. These tools, particularly in their cloud form (e.g Tibco Spotfire Cloud or Tableau Online), reduce development time, the complexity of development, and the cost of entry, while at the same time maximising report flexibility.
While not the silver bullet for every reporting scenario, Visual Analytics solutions have made Business Intelligence accessible to increasing numbers of Small and Medium Businesses.
Its important to note, in many cases its more about what sits under the reporting rather than the actual reporting interface. For example an underlying columnar and /or distributed data warehouse solutions (such as Amazon Redshift) enables users to quickly summarise, drill down or filter terabytes of data. Because of this more reporting solutions are developing the capability to connect to “big data” solutions such as Hadoop, Amazon’s Redshift, SAP Hana, IBM Netezza, or Google Big Query.
In most reporting projects, the biggest challenge is actually integrating the underlying data into a suitable format & location to report on it.
Relatively few businesses, and especially SMEs, spend much time doing modelling. This party because most analytical time is soaked up with reporting, but also because of lack of knowledge of tools and methodology. Modelling covers a range of techniques such as
- Scenario analysis where expected, best and worst case revenue scenarios are calculated.
- Taking this further approach further is Monte Carlo – where you calculating a probability that an even will occur.
- Building propensity/ predictive models, such as decision trees to predict customers likely to churn.
- Classification models to identify a common source of customer dissatisfaction.
How does Integration, Reporting & Modelling fit together?
In most businesses a disproportionate amount time is spent on data integration. Missing or incomplete records need to be filtered out, product codes in one spreadsheet need to be matched to product names in another & several months results need to appended to each other. The steps go on. To make it worse the process generally needs to be repeated in preparation for the next round of monthly reporting.
The problem is this essential, but low ROI, data integration work leaves little time for the higher value tasks. In the ideal world more time would be spend on using insight to help formulate & test hypotheses, running scenario analysis modelling and making recommendation on how to improve the business. The right tools and training can help with enabling a business to focus on work that gives them the best Return on Investment.