Business Intelligence (BI) is about turning data into insight and using this insight to make better business decisions. Today the opportunity to make better decisions is growing as the amount of data being stored increases. Between 2009 & 2020, the “digital universe” is projected to grow by a factor of 44 – increasing from c. 0.8 Zettabyte (ZB) to greater than 35 ZB. This, however, creates a significant challenge – how to distill useful insight out of a growing sea of increasingly disparate data?
This is a particular challenge for Small and Medium sized Enterprises (SMEs). Despite their size, SMEs are nearly always collecting significant amounts of data – sales, orders, customer relationship management (CRM) and website data. The key is how to extract the insight despite the challenges that these organisations face. These challenges include
- A lack of specialist analytical staff with advanced Business Intelligence & analytical skills.
- Restricted budgets – often there is no budget for Business Intelligence training or expensive tools.
- A rapidly evolving business environment that demands the rapid generation of business insight.
This blog is about how to use business intelligence & analysis to make better business decisions despite these challenges.
Business Intelligence can be considered to consisting of three core components : Data Integration, Reporting, & Modeling.
Data Integration is about getting multiple disparate data sources into a central file (Data warehouse) where it can be analysed or used for reporting. A simple example might involve merging sales and customer contact data so the marketing team can report on sales or contact specific customers with a product offer. When data also needs to be cleansed, aggregated, filtered or have any advanced logic applied this data integration task becomes difficult using generic tools like Microsoft Excel or Access. To make it worse, Excel & other traditional database tools provide little or no auditing of data quality.
Without specialist data integration tools, employees tend to burn an inordinate amount of time merging data files before they can generate say a monthly report. To make it worse each month they would normally need to repeat the process. The end result is they may spend 90% of their time integrating data & generating the report and only 10% interpreting the data. Wouldn’t it be smarter to turn this on its head? – spend 10% of the time integrating the data & producing the reports & 90% interpreting results and making recommendations? The problem is many companies aren’t aware of how to integrate data sources more efficiently.
Once the data is integrated & in one location (by definition a data warehouse), reporting becomes infinitely easier. Tools like Crystal reports, & Microsoft Reporting Services are often thought of as the best reporting tools, but in reality a lot of great reporting can be done is Excel, particularly Excel 2010 and 2013 with its new data visualization & slicing tools (PowerPivots). The added advantage of Excel is most organisations already own it and very little training is usually required for users to be able to slice and dice data in reports.
But in saying that, the reporting landscape has been turned upside down with the emergence of visual analytics tools like Qlikview, Tableau, and Tibco Spotfire. These tools are intuitive to use, are web & tablet friendly, and leverage in memory analysis and columnar database to make them blisteringly fast.
Its often more about what sits under the reporting rather than the actual reporting interface. For example an underlying OLAP cube provides reports with the functionality to efficiently drill down and roll up data (for example from sales regions, to countries, to cities, to companies). Because of this more reporting solutions are developing the capability to connect to “big data” solutions such as Hadoop, Amazon’s Redshift, SAP Hana, IBM Netezza, or Google Big Query.
In most reporting projects, the biggest challenge is actually integrating the underlying data into a suitable format to report on it.
Relatively few businesses, and especially SMEs, spend much time doing modelling. This party because most analytical time is soaked up with reporting, but also because of lack of knowledge of tools and methodology. Modelling covers a range of techniques such as
- Scenario analysis where expected, best and worst case revenue scenarios are calculated.
- Taking this further approach further is Monte Carlo – where you calculating a probability that an even will occur.
- Building propensity/ predictive models, such as decision trees to predict customers likely to churn.
- Classification models to identify a common source of customer dissatisfaction.
How does Integration, Reporting & Modelling fit together?
In most businesses a disproportionate amount time is spent on data integration. Missing or incomplete records need to be filtered out, product codes in one spreadsheet need to be matched to product names in another & several months results need to appended to each other. The steps go on. To make it worse the process generally needs to be repeated in preparation for the next round of monthly reporting.
The problem is this essential, but low ROI, data integration work leaves little time for the higher value tasks. In the ideal world more time would be spend on using insight to help formulate & test hypotheses, running scenario analysis modelling and making recommendation on how to improve the business. The right tools and training can help with enabling a business to focus on work that gives them the best Return on Investment.
Who is this blog aimed at and what is its purpose?
This blog aims to help analysts get up to speed with Business Intelligence tools and analytics techniques. Its about using smart technology and techniques to improve the allocation of analyst time.