Sunday, March 6, 2011

Democratization of Insight- Top Ten Watch List

The adage “From Data to Insight” personifies the last two decades of the Data Warehousing, Business Intelligence and Analytics industry. But, has it delivered? One of the silent revolutions in the land of data has been the commoditization of tools which puts insight in the hands of the masses and begs the questions - Can insight be self-service? Do we need a plethora of ETL and reporting tools, the armies of IT professionals and consultants and the massive projects required to convert data to insight? Or can a small group of smart users get the job done? You be the judge.


Technology in itself is rarely a differentiator. How people use it and what they do with it, and how quickly they do it, is much more important. But every so often, great technology makes the human work so easy that the people using it really get to shine and make a difference. It frees them from having to wrangle with the technology to get to the answer.


The last couple of years have seen a proliferation of tools which enhance the end-users ability to easily manipulate the data and/or visualize it more meaningfully. Best of all, some of these gems are free, cost very little or are absolutely worth every penny. But the most important factor is the speed to insight – monolithic projects of enterprise data management and reporting are bound to face their “berlin wall” moment where end-users will demand the usage of the tools to free up access to data and insight, now.


Here is my Top 10 watch list, not in any particular order:
  1. Google Refine: This incredible tool puts data cleansing and transformation in the hands of a business user. Anyone who manipulates data for a living will appreciate it, some perhaps as much as this guy
  2. PivotViewer: Free visual search tool from Microsoft (yes it’s from MS “and” free) allows users to sift through large amounts of meta-tagged data. 
  3. Project R: The open source and free, R language for statistical analysis, is gaining critical mass. Revolution Analytics which sells a commercial version has challenged SAS on performance, functionality and price. 
  4. Data Wrangler: It uses natural language interface to make it incredibly easy to transform data. This gem is still work in progress but I truly hope it gets out of Stanford, just like the folks at Tableau (and Sun, Google, etc.). 
  5. Tableau: By far the standard for visual analytics. Try out the free version-Tableau Public for yourself. 
  6. Google Visualization API: A set of visualizations which default to public data sets on the web. Try these against you own data sets using Google Docs. The motion charts (from the gapminder lineage) are especially nice. 
  7. Juice Analytics: Another offering which recognizes and bridges the gap between data and users. 
  8. Lyza: Insight is collaborative and iterative. It is an interesting concept and breaks down the barriers between information providers and consumers. It also breaks the definition of a report - i.e. a document artifact and turns it into a collaboration artifact. 
  9. Endeca: Great insight enabler. Dynamic data association (i.e. not hard coded data relationships) and a fantastic visual interface make it easy for users to discover by going down unexplored trails- kind of how some of the greatest discoveries in world happen. 
  10. Excel 2010 and PowerPivot: This version of Excel does what several BI tools charge serious money for – fast, large data manipulation on the desktop, great visualization like sparklines, heatmaps and a whole bunch of analytic capability from statistics to data mining (yes – Excel can now perform data mining along with Analysis Services). PowerPivot allows Excel to manipulate millions of rows in the most ubiquitous analysis tool. With the Office 2010 refresh cycle underway in enterprises and with 64bit CPU’s the norm, expect some very capable in-memory analysis on the desktop.