Top tends in BI arena

I thought I’d share a few of my thoughts on the trends and technologies that are part of my daily life these days. From what I see in the market, the expectation from BI has changed significantly over the past few years. Now all size of companies are looking at investing in BI. They expect to spend less, get more ROI, more data accessibility, more insight into different levels of the business, more flexibility, and quicker response to the ever-changing business requirements.

Agile BI

Traditional waterfall design and development methodologies are too slow and too inflexible for BI. Many companies are now considering Agile for their BI implementations.
But don’t misinterpret what I am calling “Agile BI”, it’s not just about the Agile software development methodology, which is nothing new. IMHO Agile development by itself is not enough for BI, I think a different approach, technologies, and mindset are required to make BI more flexible
An agile software development process doesn't necessarily lead to agile BI, though it will speed up some steps in the process. The technology and processes (e.g. change management process) has to be made more flexible to really achieve agility.
Forrester defines Agile business intelligence as: An approach that combines processes, methodologies, organizational structure, tools, and technologies that enable strategic, tactical, and operational decision-makers to be more flexible and more responsive to the fast pace of changes to business and regulatory requirements.

Collaborative/social BI

Intelligence = Information + People

Collaboration and business intelligence have never been separated, but companies are just recently beginning to realize the benefits of the connection.

By introducing collaborative technologies to their business intelligence platform, companies are better positioned to utilize and take advantage of their employee data.

Collaboration is now becoming an essential part of the business intelligence. It enables crowdsourcing and in general the wisdom of the crowds tend to produce better decisions as organizations become increasingly complex.

Self-service BI

Let the business do it by themselves!
IT is always under one step behind from ever-changing business needs. The concept of self-service BI is to provide capabilities to enable business users exploring data and build their own reports. it may also enable analysts to publish and distribute their work to others.
Used in this way, the visual discovery tools become a nice tool for super users to create web-based reports and dashboards for colleagues. These reports are not static; they are “live” and interactive so casual users can analyse data just as business analysts would without the hassle of starting from scratch.

Predictive Analytics

"Predictive Analytics" is to analyse current and historical facts to make predictions about future events.

More and more vendors are attempting to make it more accessible to information workers and others without advanced degrees in statistics.
Data mining used to be mostly used by large enterprises (e.g. Banks) but these days I see companies in different sizes are started getting benefit from that.
There is also a trend in using social network data in data mining. It can be used to determine the trend, behaviour analysis and reduce customer churn.

Microsoft BI in 2010 – and my New Year’s resolution!

It’s the first week of 2010 and people have got great intensions to stick to their resolutions (I’ve been to the gym once since 1st Jan which is a good start!)

But what about the “Learning” resolutions?

I have been learning about SQL Server 2008 R2 these days and found that there are lots of new things happening in MB BI space. Whether you are a techie who loves learning new stuff, passionate about MS BI, or just want to add weight to your CV, it would be worthwhile watching out these MS technologies in 2010.

1) Master Data Services

• Master Data Services
• Nick Barclay's posts about MDS 

2) PowerPivot

• PowerPivot website

3) PerformancePoint Services for SharePoint Server 2010

• What's New: PerformancePoint Services

4) SQL Azure and Cloud Computing

• SQL Azure website

5) Excel Services, VISIO Services and other BI related components and APIs in MOSS 2010

• What's New: Excel Services

And in terms of "Methodology" and delivery approach, I believe we will hear more about "Agile BI /Data Warehousing" due to the reduced IT budgets in this economy. I'll talk about it in another post.

Self-Service BI?!

We keep hearing about "SELF-SERVE" functionalities in new versions of BI tools specifically with Microsoft plan to delivering BI to the masses.

SQL Server 2008 R2 introduced some self-service options to allow business units and users to create and share BI solutions. Here is what MS says about them:

• SQL Server Project "Gemini” Add-in for Excel

Enables Excel power users to create BI solutions by streamlining the integration of data from multiple sources enabling interactive modeling and analysis of massive amount of data and by supporting the seamless sharing of data models and reports through Microsoft Office SharePoint 2010

• SharePoint 2010 based Operations Dashboard

This SharePoint managed service enables front-line operators and administrators to monitor access and utilization of analyses and reports as well as track patterns of hardware usage to help ensure the right security privileges are applied and user generated solutions are available, up-to-date, and managed in a consistent way.

• SQL Server Reporting Services Report Builder 3.0

This updated ad-hoc reporting client accelerates report creation, collaboration and consistency by allowing users to create and share report components that can be accessed via the shared component library and by enabling the rapid assembly of comprehensive business reports using these shared components.

• Rich visualization of geospatial data

New support for geospatial visualization including mapping, routing, and custom shapes can help your end users create customized reports that leverage existing content objects, such as queries, data regions, and charts and graphs. You can also enhance location-based data reports with Bing Maps in Report Builder 3.0.

I know many people are excited about them and especially about project Gemini, here are some quotes I found on the net:

• Gemini enables end users to work with business data
• 100 million rows on a laptop with Office 2010 Excel
• Slicers and spark lines in Excel 2010 are bringing BI to everyone
• Excel 2010 provides BI for the masses by making it easy to build your own charts/graphs on and offline

I agree with all of them, BUT, the question is: are we missing the main goal of BI: “single version of the truth”. Is Gemini leading us to the next generation of “Spreadmarts”? Does having Excel and access to source data make a BI solution?

From my experience, if you ask business-users to do something which requires more than 1 minute of technical training, most probably they get confused, and in most cases, anything more than 5 clicks away is considered as “too technical” and won’t be adobted by business users. So how do we expect them to get this toy and build a BI solution?!

I believe the idea of "Self Service BI" is like a double edged sword, though it can be useful in the same way as we do self-service check out in supermarkets: do it yourself when you don’t have many items, so you can save time, but you will be responsible for that and if you get confused about something, that’s OK, you may ask a staff to help you out with that, but it may take more time than standing in line in the first place.

However, it’s a quick way to discover the underlying data, and can be handy in some cases such as the following scenarios:

• If a business analyst needs to see what data is available in data sources
• To analyse source data and create simple reports when underlying data is clean and in simple structure
• To create a prototype for a reporting solution
• BUT, when you are dealing with complicated logics to bring data in, or when there are too many data sources and you are not a BI specialist: forget about it!

I don’t want to be negative; I like Microsoft and their BI toolset, but is it the right way to go? Does it lead us to the “BI for masses” idea or towards “messy BI”?!

I’d like to hear your thoughts.

Tips for gathering "Business Requirements" in a BI project.

In my opinion the most important and challenging aspect of a BI project is to understand the business requirements. Majority of BI projects failures are due to lack of understanding the business. This lack is primarily due to BI being thought as an IT project, instead of being a business project with IT as one of the enablers.

It's very important to find out "what they need" and not just focus on "what they want". The most effective way of doing that is by "face to face" meetings with managers, business users and IT people. The goal of a requirements interview is to ask questions in order to discover unknown frontiers. Think of it as a one-hour immersion to better understand what business people do and why. How do they make decisions today, and how do they want to be making decisions in the future? First ask interviewees about their roles and responsibilities to get them engaged and from there, cover the following areas:

• What are the key business objectives?
• For each objective, ask about measures for success to learn more about key metrics and business dimensions.
• What roles do data and analysis play in achieving goals? Alternatively, how would better access and analysis benefit them?
• What are the current analysis challenges?

A good question to get the interview started is "How can people tell when you're doing a great job?"

Be conversational: For the DW/BI practitioner, being conversational means putting yourself into a business frame of mind. Learn the language of the business. Don't intimidate end users by asking "What do you want in a data warehouse?" End users aren't systems designers. Acronyms and IT vernacular don't belong in a business requirements interview.

Several techniques can help you establish a more conversational tone:

• Learn a bit about the business beforehand by reviewing the Web site or annual report to understand company-specific vocabulary and hot-button issues.
• Meet with interviewees on their own turf. Go to their offices or conference rooms, rather than IT meeting spaces.
• Prior to the interview, send out an announcement describing the high-level discussion topics and confirming the interview time and place. Don't attach a detailed questionnaire to this meeting notice. You can't achieve a conversational flow if you're reviewing questionnaire results-presuming anyone bothers to complete the survey.
• Interview questions prepared in advance are fallback devices, used only if uncomfortable lulls occur in conversations or to ensure key points are covered before ending sessions.
• Most good conversations tend to wander, so remember your session goals and steer conversations back on track if you stray too far from core issues.
• Stay at a relatively high level in the interview's early stages. Don't follow an early comment to a very low level of detail, only to run out of time and discover that you haven't discussed three other major areas of responsibility with important requirements for the DW/BI effort.

Listen, and expect to be changed: Good interviewers should be seen but not heard — well, at least not heard too much. Strong active listening skills are required.

As you're gathering requirements from the business users, intersperse some data reality into the process by interviewing key IT personnel, especially the master database administrators responsible for operational systems. Consider what the business needs are in tandem with the availability of data to support these requirements. IT meetings tend to be informal discussions, beginning with knowledgeable project team members. Once you start to hear consistent themes from users, it's time to sit down with the data gurus and get into the extreme detail of their source systems. During these data audit interviews, try to understand whether complete, reliable data is there to support what users are asking for.

There are several methods and resources for BIR, one of the best is "The Data Warehouse Lifecycle Toolkit by R. Kimball, ...".