Big Data becomes friendly in 2013

Many organizations are in the phase of evaluating the Hadoop platform . Certainly Hadoop has been the only option to handle large unstructured data for organizations that run their business handling unstructured data like Google, Yahoo.  For others Hadoop positively provides an opportunity to look at data (Dark Data) which they haven’t considered as part of the Enterprise Data Warehouse.

In the process of defining and executing a proof of concept with Hadoop platform, we generally face two challenges which are:

The need for developers to acquire new skills to handle different programming languages related to Hadoop. It’s not easy for a developer who has worked on a GUI based ETL tool like Informatica to work on Hadoop ETL process.

The means to visualize the results from Hadoop, definitely we need outputs which are more than a search engine output

2012 can be seen as the year which brought in lot more tools and utilities related to Hadoop to make things easier…following are the few key releases from major BI vendors

IBM moved up a level and announced on the availability of few integrations which will increase the adoption of BigInsights platform. Some of them include integration of InfoSphere Data Explorer ( recently acquired product Vivisimo) with BigInsights , availability of Applications Accelerators with the BigInsights platform – Machine Data Analytics Accelerator for analyzing machine data and Social Data Analytics Accelerator for analyzing social media data sources like Twitter, Facebook and integration of Cognos with BIgInsights

The Business Intelligence Chasm

The term Business Intelligence was first coined by IBM researcher Hans Luhn (in the IBM Journal of Research and Development, October 1958) and then used in its modern sense in 1989 by then-Gartner analyst Howard Dresner, who defined BI as an umbrella term to describe concepts and methods to improve business decision making by using fact based support systems.Both these definitions were quite prescient for its time – Mr. Luhn’s concept of ‘action points’ in the organization and Mr. Dresner’s reference to ‘business decision making’ ensured that BI has direct business relevance to go along with its very interesting technology façade.

BI & Analytics, in some sense, represents the holy grail of computer based applications, i.e. the use of technology to solve real world business problems. Clearly, there are 2 distinct aspects to BI –Technology and Business and both have to work synergistically to deliver on the overall promise.

As we step into 2013, my contention is that we as BI practitioners are doing fairly well on the technology front by assimilating many of the new developments (In-memory, Appliances, Columnar storage, Big data processing etc.) into mainstream data management, reporting and analytics, while we lack the skills required to integrate all this in the broader business context. Let me substantiate that statement.