Understanding the Challenges of Big Data Project Management: The Business Case

October 1, 2015 Dennis D. McDonald

By Dennis D. McDonald. Follow me on Linkedin, Twitter, and Google+. Tap or click here to download a .pdf of this article.

Recently I spoke about my research into big data project management “best practices.” My talk was titled “Big Data Project Management: What Works?” A copy of the slide deck is here.

In an ongoing series of informal interviews, I’m addressing two related questions:

What’s unique about planning and managing big data projects?
What’s not unique about planning and managing big data projects?

So far I’ve talked with several professionals about these questions including:

A corporate CIO with heavy data orientation
Project managers and consultants with “traditional” enterprise data management experience
A data scientist/software developer
A corporate project manager focusing on web analytics
A consultant building an enterprise level logical data model for a government agency
A project manager developing web and data portals for nonprofit organizations

What I’m finding so far is a mix of “standard” project management challenges as well as a few interesting wrinkles associated with data intensive projects.

Some of these challenges are listed in the two “findings” pages in the slide deck. What I discuss more specifically here is one factor that does seem to be emerging that is a combination of the old and new: the challenge of developing a management-convincing business case for big data initiatives.

Speaking from my own consulting and project management experience, the development of convincing business cases for tech-related initiatives varies in difficulty ranging from slam-dunk (really easy) to pie-in-the-sky (requiring guesstimates).

One approach is to package convincing cost and benefit metrics in a form that can be clearly understood by management. This in itself can be a challenge if the business case developer is technically or analytically competent but unaccustomed to thinking through things from a business as opposed to a technical or data orientation.

More serious is the challenges of convincing management of the value of something that by definition may involve risk and uncertainty, as might be the case where new tools and techniques are being introduced to take advantage of increasing data volumes and variety. This is a special type of justification problem and is impacted by how certain we are that the outcome of a “big date” initiative will be positive and useful. It may be impossible to predict the outcome before we do the work, yet we still have to convince management to commit to a resource investment with an uncertain outcome.

The first challenge – – knowing how to develop a convincing business case – – is something that can be taught or purchased. The second challenge – – knowing in advance what the outcome will be of a new data analysis or predictive modeling effort – – is more difficult to address and may be especially acute where management is not analytically oriented.

One way to address this second challenge is to start with something simple and not attempt a program- or enterprise-level change requiring modifications to the organization’s culture.

As I suggested in my talk, don’t start by proposing to “boil the ocean.” Instead, new tools and techniques should focus on high-value, high-priority issues, ideally those where ways of measuring corporate value have already been established. The sooner one hooks into understanding and solving real-world high-value problems, the better.

This is not the same as going after low-risk “low hanging fruit” problems where you might be able to quickly prove technical feasibility but where, at the end of the day, you are still not showing something of real value to management.

Also, it’s easy to get caught up in the power and flexibility of the data analysis and visualization tools that are now available. As hard as it sometimes for those of us who are technically inclined to admit, management doesn’t necessarily care what tools we’re using. Nor does management care how we manage our projects.

Management wants results. The sooner we focus our new data management and analytic methods on solving important problems, the better.

Related reading: