Big data analysis Essay

THE NEW INTELLIGENT ENTERPRISE Some of the best-performing retailers are using analytics not just for finance and operational activities, but to boost competitive advantage on everything from displays, to marketing, customer service and customer experience management. Big Data, Analytics and the Path From Insights toValue How the smartest organizations are embedding analytics to transform information into insight and then action. Findings and recommendations from the first annual New Intelligent Enterprise Global Executive study. BY STEVE LAVALLE, ERIC LESSER, REBECCA SHOCKLEY, MICHAEL S. HOPKINS AND NINA KRUSCHWITZ IN EVERY INDUSTRY, in every part of the world, senior leaders wonder whether they are getting full value from the massive amounts of information they already have within their organizations. New technologies are collecting more data than ever before, yet many organizations are still looking for better ways to obtain value from their data and compete in the marketplace. Their questions about how best to achieve value persist. Are competitors obtaining sharper, more timely insights? Are they able to regain market advantage, neglected while focusing on expenses during the past two years? Are they correctly interpreting new signals from the global economy — and adequately assessing the impact on their customers and partners? Knowing what happened and why it happened are no longer adequate. Organizations need to know what is happening now, what is likely to happen next and what actions should be taken to get the optimal results. COURTESY OF BEST BUY THE LEADING QUESTION How are organizations using analytics to gain insight and guide action? FINDINGS Top-performingorganizations are twice as likely to apply analytics to activities. Thebiggest challenges in adopting analytics are managerial and cultural. V isualizing data differently will become increasingly valuable. WINTER 2011 MIT SLOAN MANAGEMENT REVIEW 21 THE NEW INTELLIGENT ENTERPRISE ABOUT THE RESEARCH To understand the challenges and opportunities associated with the use of business analytics, MIT Sloan Management Review, in collaboration with the IBM Institute for Business Value, conducted a survey of more than 3,000 business executives, managers and analysts from organizations located around the world. The survey captured insights from individuals in 108 countries and more than 30 industries and involved organizations from a variety of sizes. The sample was drawn from a number of different sources, including MIT alumni and MIT Sloan Management Review subscribers, IBM clients and other interested parties. We also interviewed academic experts and subject matter experts from a number of industries and disciplines to understand the practical issues facing organizations today. Their insights contributed to a richer understanding of the data and the development of recommendations that respond to strategic and tactical questions that senior executives address as they operationalize analytics within their organizations. We also drew upon a number of IBM case studies to explore further how organizations are leveraging business analytics and illuminate how real organizations are putting our recommendations into action in different organizational settings. To help organizations understand the opportunity of information and advanced analytics, MIT Sloan Management Review partnered with the IBM Institute for Business Value to conduct a survey of nearly 3,000 executives, managers and analysts working across more than 30 industries and 100 countries. (See â€Å"About the Research. †) Among our key findings: Top-performing organizations use analytics five times more than lower performers. (See â€Å"Analytics Trumps Intuition. †) Overall, our survey found a widespread belief that analytics offers value. Half of our respondents said that improvement of information and analytics was a top priority in their organizations. And more than one in five said they were under intense or significant pressure to adopt advanced information and analytics approaches. The source of the pressure is not hard to ascertain. Six out of 10 respondents cited innovating to achieve competitive differentiation as a top business challenge. The same percentage also agreed that their organization has more data than it can use effectively. Organizational leaders want analytics to exploit their growing data and computational power to get smart, and get innovative, in ways they never could before. Senior executives now want businesses run on data-driven decisions. They want scenarios and simulations that provide immediate guidance on the best actions to take when disruptions occur — disruptions ranging from unexpected competitors or an earthquake in a supply zone to a customer signaling a desire to switch providers. Executives want to understand optimal solutions based on complex business parameters or new information, and they want to take action quickly. These expectations can be met — but with a caveat. For analytics-driven insights to be consumed — that is, to trigger new actions across the organization — they must be closely linked to business strategy, easy for end-users to understand and embedded into organizational processes so that action can be taken at the right time. That is no small task. It requires painstaking focus on the way insights are infused into everything from manufacturing and new product development to credit approvals and call center interactions. 22 MIT SLOAN MANAGEMENT REVIEW WINTER 2011 Top Performers Say Analytics Is a Differentiator Our study clearly connects performance and the competitive value of analytics. We asked respondents to assess their organization’s competitive position. Those who selected â€Å"substantially outperform industry peers† were identified as top performers, while those who selected â€Å"somewhat or substantially underperform industry peers† were grouped as lower performers. We found that organizations that strongly agreed that the use of business information and analytics differentiates them within their industry were twice as likely to be top performers as lower performers. Top performers approach business operations differently than their peers do. Specifically, they put analytics to use in the widest possible range of decisions, large and small. They were twice as likely to use analytics to guide future strategies, and twice as likely to use insights to guide day-to-day operations. (See â€Å"The Analytics Habits of Top Performers,† p. 24. ) They make decisions based on rigorous analysis at more than double the rate of lower performers. The correlation between performance and analyticsdriven management has important implications to organizations, whether they are seeking growth, efficiency or competitive differentiation. Three Levels of Capabilities Emerged, Each with Distinct Opportunities Organizations that know where they are in terms of analytics adoption are better prepared to turn challenges into opportunities. We segmented respondents based on how they rated their organization’s analytics prowess, specifically how thoroughly their organizations had been transformed by better uses of analytics and information. Three levels of analytics capability emerged — Aspirational, Experienced and Transformed — each with clear distinctions. (See â€Å"The Three Stages of Analytics Adoption. †) Aspirational. These organizations are the furthest from achieving their desired analytical goals. Often they are focusing on efficiency or automation of existing processes and searching for ways to cut costs. Aspirational organizations currently have SLOANREVIEW. MIT. EDU few of the necessary building blocks — people, processes or tools — to collect, understand, incorporate or act on analytic insights. Experienced. Having gained some analytic experience — often through successes with efficiencies at the Aspirational phase — these organizat ions are lo oking to go b e yond cost management. Experienced organizations are developing better ways to collect, incorporate and act on analytics effectively so they can begin to optimize their organizations. Transformed. These organizations have substantial experience using analytics across a broad range of functions. They use analytics as a competitive differentiator and are already adept at organizing people, processes and tools to optimize and differentiate. Transformed organizations are less focused on cutting costs than Aspirational and Experienced organizations, possibly having already automated their operations through effective use of insights. They are most focused on driving customer profitability and making targeted investments in niche analytics as they keep pushing the organizational envelope. Transformed organizations were three times more likely than Aspirational organizations to indicate that they substantially outperform their industry peers. This performance advantage illustrates the potential rewards of higher levels of analytics adoption. Information Must Become Easier to Understand and Act Upon Executives want better ways to communicate complex insights so they can quickly absorb the meaning of the data and take action. Over the next two years, executives say they will focus on supplementing standard historical reporting with emerging approaches that make information come alive. These include data visualization and process simulation as well as text and voice analytics, social media analysis and other predictive and prescriptive techniques. New tools like these can make insights easier to understand and to act on at every point in an organization, and at every skill level. They transform numbers into information and insights that can be readily put to use, versus having to rely on further interpretation or leaving them to languish due to uncertainty about how to act. ANALYTICS TRUMPS INTUITION The tendency for top-performing organizations to apply analytics to particular activities across the organization compared with lower performers. A likelihood of 1. 0 indicates an equal likelihood that the organizations will use either analytics or intuition. Tendency to Apply Tendency to Apply Intuition Analytics Financial management and budgeting Data Is Not the Biggest Obstacle Despite popular opinion, getting the data right is not a top challenge that organizations face when adopting analytics. Only about one out of five respondents cited concern with data quality or ineffective data governance as a primary obstacle. The adoption barriers that organizations face most are managerial and cultural rather than related to data and technology. The leading obstacle to widespread analytics adoption is lack of understanding of how to use analytics to improve the business, according to almost four of 10 respondents. More than one in three cite lack of management bandwidth due to competing priorities. (See â€Å"The Impediments to Becoming More Data Driven. †) Strategy and business development Sales and marketing Customer service Product research and development Top Performers Lower Performers General management Risk management Customer experience management Brand or market management Work force planning and allocation Overall Average 0 SLOANREVIEW. MIT. EDU 22. 1 Operations and production 1 2 3 4 5 6 7 8 WINTER 2011 MIT SLOAN MANAGEMENT REVIEW 23 THE NEW INTELLIGENT ENTERPRISE What Leaders Can Do to Make Analytics Pay Off — A New Methodology It takes big plans followed by discrete actions to gain the benefits of analytics. But it also takes some very specific management approaches. Based on data from our survey, our engagement experience, case studies and interviews with experts, we have been able to identify a new, five-point methodology for successfully implementing analytics-driven management and for rapidly creating value. The recommendations that follow are designed to help organizations understand this â€Å"new path to value† and how to travel it. While each recommendation presents different pieces of the information-and-analytics value puzzle, each one meets all of these three critical management needs: Reduced time to value. Value creation can be achieved early in an organization’s progress to THE ANALYTICS HABITS OF TOP PERFORMERS Top-performing organizations were twice as likely to use analytics to guide day-to-day operations and future strategies as lower performers. THE THREE STAGES OF ANALYTICS ADOPTION Three capability levels — Aspirational, Experienced and Transformed — were based on how respondents rated their organization’s analytic prowess. ASPIRATIONAL EXPERIENCED TRANSFORMED Motive †¢Use analytics to justify actions †¢Use analytics to guide actions †¢ se analytics to prescribe actions U Functional proficiency †¢Financial management and budgeting †¢Operations and production †¢Sales and marketing †¢All Aspirational functions †¢Strategy/business development †¢Customer service †¢Product research/development †¢ ll Aspirational and Experienced A functions †¢Risk management †¢Customer experience †¢Work force planning/allocation †¢General management †¢Brand and market management Business challenges †¢ ompetitive differentiation through C innovation †¢Cost efficiency (primary) †¢Revenue growth (secondary) †¢ ompetitive differentiation through C innovation †¢Revenue growth (primary) †¢Cost efficiency (secondary) †¢ ompetitive differentiation through C innovation †¢Revenue growth (primary) †¢ rofitability acquiring/retaining P customers (targeted focus) Key obstacles †¢ ack of understanding how to leverage L analytics for business value †¢Executive sponsorship †¢ ulture does not encourage sharing C information †¢ ack of understanding how to leverage L analytics for business value †¢Skills within line of business †¢ wnership of data is unclear or O governance is ineffective †¢ ack of understanding how to leverage L analytics for business value †¢ anagement bandwidth due to M competing priorities †¢Accessibility of the data Data management †¢ imited ability to capture, aggregate, L analyze or share information and insights †¢ oderate ability to capture, aggregate M and analyze data †¢ imited ability to share information and L insights †¢ trong ability to capture, aggregate and S analyze data †¢ ffective at sharing information and E insights Analytics in action †¢ arely use rigorous approaches to R make decisions †¢ imited use of insights to guide future L strategies or day-to-day operations †¢ ome use of rigorous approaches to S make decisions †¢ rowing use of insights to guide future G strategies, but still limited use of insights to guide day-to-day operations †¢ ost use rigorous approaches to make M decisions †¢ lmost all use insights to guide future A strategies, and most use insights to guide day-to-day operations 24 MIT SLOAN MANAGEMENT REVIEW WINTER 2011 SLOANREVIEW. MIT. EDU analytics sophistication. Contrary to common assumptions, it doesn’t require the presence of perfect data or a full-scale organizational transformation. Increased likelihood of transformation that’s both significant and enduring. The emerging methodology we’ve identified enables and inspires lasting change (strategic and cultural) by tactically overcoming the most significant organizational impediments. Greater focus on achievable steps. The approach used by the smartest companies is powerful in part because each step enables leaders to focus their efforts and resources narrowly rather than implementing universal changes — making every step easier to accomplish with an attractive ROI. Whether pursuing the best channel strategy, the best customer experience, the best portfolio or the best process innovation, organizations embracing this approach will be first in line to gain business advantage from analytics. have repeatedly heard that analytics aligned to a significant organizational challenge makes it easier to overcome a wide range of obstacles. Respondents cited many challenges, and none can be discounted or minimized: Executive sponsorship of analytics projects, data quality and access, governance, skills and culture all matter and need to be addressed in time. But when overtaken by the momentum of a single big idea and potentially game-changing insight, obstacles like these get swept into the wake of change rather than drowning the effort. THE IMPEDIMENTS TO BECOMING MORE DATA DRIVEN The adoption barriers organizations face most are managerial and cultural rather than related to data and technology. Lack of understanding of how to use analytics to improve the business Lack of management bandwidth due to competing priorities Lack of skills internally in the line of business Ability to get the data [RECOMMENDATION 1 ] First,Think Biggest Existing culture does not encourage sharing information Focus on the biggest and highestvalue opportunities Does attacking the biggest challenge carry the biggest risk of failure? Paradoxically, no — because big problems command attention and incite action. And as survey participants told us, management bandwidth is a top challenge. When a project’s stakes are big, top management gets invested and the best talent seeks to get involved. It’s extraordinarily hard for people to change from making decisions based on personal experience to making them from data — especially when that data counters the prevailing common wisdom. But upsetting the status quo is much easier when everyone can see how it could contribute to a major goal. With a potential big reward in sight, a significant effort is easier to justify, and people across functions and levels are better able to support it. Conversely, don’t start doing analytics without strategic business direction, as those efforts are likely to stall. Not only does that waste resources, it risks creating widespread skepticism about the real value of analytics. In our discussions with business executives, we SLOANREVIEW. MIT. EDU Ownership of data is unclear or governance is ineffective Lack of executive sponsorship Concerns with the data Perceived costs outweigh projected benefits No case for change Respondents were asked to select three obstacles to the widespread adoption of analytics in their organization. Don’t know where to start 0 10% 20% 30% 40% Percentage of respondents [RECOMMENDATION 2 ] Start in the Middle Within each opportunity, start with questions, not data Organizations traditionally are tempted to start by gathering all available data before beginning their analysis. Too often, this leads to an all-encompassing focus on data management — collecting, cleansing and converting data — that leaves little time, energy or resources to understand its potential uses. Actions taken, if any, might not be the most valuable ones. Instead, organizations should WINTER 2011 MIT SLOAN MANAGEMENT REVIEW 25 THE NEW INTELLIGENT ENTERPRISE start in what might seem like the middle of the process, implementing analytics by first defining the insights and questions needed to meet the big business objective and then identifying those pieces of data needed for answers. By defining the desired insights first, organizations can target specific subject areas and use readily available data in the initial analytic models. The insights delivered through these initial models will illuminate gaps in the data infrastructure and business processes. Time that would have been spent cleaning up all data can be redirected toward targeted data needs and specific process improvements that the insights identify, enabling iterations of value. Companies that make data their overriding priority often lose momentum long before the first insight is delivered, frequently because a data-first approach can be perceived as taking too long before generating a financial return. By narrowing the scope of these tasks to the specific subject areas needed to answer key questions, value can be realized more quickly, while the insights are still relevant. Also, organizations that start with the data or process change often end up with unintended consequences — such as data that is not extensible or processes that are ultimately eliminated — that require rework and additional resources to solve. Speeding Insights into Business Operations Compared with other respondents, Transformed organizations are good at data capture. (See â€Å"What Data-Transformed Companies Do. †) Additionally, Transformed organizations are much more adept at WHAT DATA-TRANSFORMED COMPANIES DO Transformed organizations felt more confident in their ability to manage data tasks than Aspirational organizations, which seldom felt their organizations performed those tasks â€Å"very well. † Percent of respondents whose organizations perform these tasks very well. Capture Information Transformed Aspirational 9% Aggregate Information 36% 4X more likely Analyze Information 28% 3% 9X more likely 26 MIT SLOAN MANAGEMENT REVIEW WINTER 2011 Disseminate Information and Insights 34% 4% 8. 5X more likely 21% 2% 10X more likely data management. In these areas, they outpaced Aspirational organizations up to tenfold in their ability to execute. Enterprise processes have many points where analytic insights can boost business value. The operational challenge is to understand where to apply those insights in a particular industry and organization. When a bank customer stops automatic payroll deposits or remittance transfers, for example, who in the organization should be alerted and tasked with finding out whether the customer is changing jobs or planning to switch banks? Where customer satisfaction is low, what insights are needed, and how should they be delivered to prevent defections? To keep the three gears moving together — data, insights and timely actions — the overriding business purpose must always be in view. That way, as models, processes and data are tested, priorities for the next investigation become clear. Data and models get accepted, rejected or improved based on business need. New analytic insights — descriptive, predictive and prescriptive — are embedded into increasing numbers of applications and processes, and a virtuous cycle of feedback and improvement takes hold. [RECOMMENDATION 3 ] Make Analytics Come Alive Embed insights to drive actions and deliver value New methods and tools to embed information into business processes — use cases, analytics solutions, optimization, work flows and simulations — are making insights more understandable and actionable. Respondents identified trend analysis, forecasting and standardized reporting as the most important tools they use today. However, they also identified tools that will have greater value in 24 months. The downswings in â€Å"as-is† methods accompanied by corresponding upswings in â€Å"to-be† methods were dramatic. (See â€Å"Where Are DataDriven Managers Headed? † p. 27. ) Today’s staples are expected to be surpassed in the next 24 months by: 1. Data visualization, such as dashboards and scorecards SLOANREVIEW. MIT. EDU 2. Simulations and scenario development 3. Analytics applied within business processes 4. Advanced statistical techniques, such as regression analysis, discrete choice modeling and mathematical optimization. Organizations expect the value from these emerging techniques to soar, making it possible for data-driven insights to be used at all levels of the organization. For example, GPS-enabled navigation devices can superimpose real-time traffic patterns and alerts onto navigation maps and suggest the best routes to drivers. Similarly, in oil exploration, three-dimensional renderings combine data from sensors in the field with collaborative and analytical resources accessible across the enterprise. Production engineers can incorporate geological, production and pipeline information into their drilling decisions. Beyond 3-D, animated maps and charts can simulate critical changes in distribution flow or projected changes in consumption and resource availability. In the emerging area of analytics for unstructured data, patterns can be visualized through verbal maps that pictorially represent word frequency, allowing marketers to see how their brands are perceived. Innovative uses of this type of information layering will continue to grow as a means to help individuals across the organization consume and act upon insights derived through complex analytics that would otherwise be hard to piece together. New Techniques and Approaches Transform Insights into Actions New techniques to embed insights will gain in value by generating results that can be readily understood and acted upon:  ¦ Dashboards that now reflect actual last-quarter sales will also show what sales could be next quarter under a variety of different conditions — a new media mix, a price change, a larger sales team, even a major weather or sporting event.  ¦ Simulations evaluating alternative scenarios will automatically recommend optimal approaches — such as the best media mix to introduce a specific product to a specific segment, or the ideal number of sales professionals to assign to a particular new territory.  ¦ Use cases will illustrate how to embed insights into business applications and processes. SLOANREVIEW. MIT. EDU New methods will also make it possible for decision makers more fully to see their customers’ purchases, payments and interactions. Businesses will be able to listen to customers’ unique wants and needs about channel and product preferences. WHERE ARE DATA-DRIVEN MANAGERS HEADED? Organizations expect that the ability to visualize data differently will be the most valuable technique in two years. Other techniques and activities that are currently delivering the most value today will still be done, but will be of less value. Today In 24 Months Historic trend analysis and forecasting Data visualization Standardized reporting Simulations and scenario development Analytics applied within business processes Data visualization Regression analysis, discrete choice modeling and mathematical optimization Analytics applied within business processes Simulations and scenario development Historic trend analysis and forecasting Clustering and segmentation Clustering and segmentation Regression analysis, discrete choice modeling and mathematical optimization Standardized reporting Respondents were asked to identify the top three analytic techniques creating value for the organization, and predict which three would be creating the most value in 24 months. In fact, making customers, as well as information, come to life within complex organizational systems may well become the biggest benefit of making data-driven insights real to those who need to use them. [RECOMMENDATION 4 ] Add, Don’t Detract Keep existing capabilities while adding new ones When executives first realize their need for analytics, they tend to turn to those closest to them for answers. Over time, these point-of-need resources come together in local line of business units to enable sharing of insights. Ultimately, centralized units emerge to bring a shared enterprise perspective — governance, tools, methods — and specialized expertise. As executives use analytics more frequently to inform day-to-day decisions and actions, WINTER 2011 MIT SLOAN MANAGEMENT REVIEW 27 THE NEW INTELLIGENT ENTERPRISE this increasing demand for insights keeps resources at each level engaged, expanding analytic capabilities even as activities are shifted for efficiencies. (See â€Å"How Analytics Capabilities Grow with Adoption. †) Sophisticated modeling and visualization tools, as noted, will soon provide greater business value than ever before. But that does not mean that spreadsheets and charts should go away. On the contrary: New tools should supplement earlier ones or continue to be used side by side as needed. That lesson applies to plines. (See â€Å"How Analytics Propagates Across Functions. †) In Transformed organizations, reusability creates a snowball effect, as models from one function are repurposed into another with minimal modifications. Over time, data-driven decision making branches out across the organization. As experience and usage grow, the value of analytics increases, which enables business benefits to accrue more quickly. Add Value with an Enterprise Analytics Unit HOW ANALYTICS CAPABILITIES GROW WITH ADOPTION The frequency with which analytics is used to support decisions increases as organizations transition from one level of analytic capability to the next. At the same time, analytics migrate toward more centralized units, first at the local line of business level and then at the enterprise level, while the portion of analytics performed at points of need and with IT remain stable. Percent using analytics frequently Where analytics performed 100% 80% Centralized analytic units 60% Line of business analytic units 40% 20% At point of need IT department 0% Aspirational Experienced Transformed nearly every way that analytics capabilities should be nurtured as an organization becomes more ambitious about becoming data driven: The process needs to be additive. As analytics capabilities are added upstream at increasingly central levels of management, existing capabilities at point of need shouldn’t be subtracted. Nor should they be transplanted to central locations. As new capabilities come on board, existing ones should continue to be supported. There are other ways that capabilities grow and deepen within an organization. Disciplines like finance and supply chain are inherently data intensive and are often where analytics first take root. Encouraged by early successes, organizations begin expanding analytic decision making to more disci28 MIT SLOAN MANAGEMENT REVIEW WINTER 2011 Organizations that first experience the value of analytics in discrete business units or functions are likely soon to seek a wider range of capabilities — and more advanced use of existing ones. A centralized analytics unit, often called either a center of excellence or center of competency, makes it possible to share analytic resources efficiently and effectively. It does not, however, replace distributed and localized capabilities; rather, the central unit is additive, built upon existing capabilities that may have already developed in functions, departments and lines of business. We found that 63% more Transformed organizations than Aspirational organizations use a centralized enterprise unit as the primary source of analytics. A centralized analytics unit can provide a home for more advanced skills to come together within the organization, providing both advanced models and enterprise governance through establishing priorities and standards by these practices:  ¦ Advance standard methods for identifying business problems to be solved with analytics.  ¦ Facilitate identification of analytic business needs while driving rigor into methods for embedding insights into end-to-end processes.  ¦ Promote enterprise-level governance on prioritization, master data sources and reuse to capture enterprise efficiencies.  ¦ Standardize tools and analytic platforms to enable resource sharing, streamline maintenance and reduce licensing expenses. In three distinct areas — application of analytic tools, functional use of analytics and location of skills — we found that adding capabilities without detracting from existing ones offers a fast path to full benefits from analytics-driven management. SLOANREVIEW. MIT. EDU [RECOMMENDATION 5 ] Build the Parts, Plan the Whole Use an information agenda to plan for the future Big data is getting bigger. Information is coming from instrumented, interconnected supply chains transmitting real-time data about fluctuations in everything from market demand to the weather. Additionally, strategic information has started arriving through unstructured digital channels: social media, smart phone applica