Analytics and Business Intelligence: Multifaceted and Evolving Technology

Just a few years ago, the prevailing view in the software industry was that the category of business intelligence (BI) was mature and without room for innovation. Vendors competed in terms of feature parity and incremental advancements of their platforms. But since then business intelligence has grown to include analytics, data discovery tools and big data capabilities to process huge volumes and new types of data much faster. As is often the case with change, though, this one has created uncertainty. For example, only one in 11 participants in our benchmark research on big data analytics said that their organization fully agrees on the meaning of the term “big data analytics.”

There is little question that clear definitions of analytics and business intelligence as they are used in business today would be of value. But some IT analyst firms have tried to oversimplify the process of updating these definitions by merely combining a market basket of discovery capabilities under the label of analytics. In our estimation, this attempt is neither accurate nor useful. Discovery tools are only components of business intelligence, and their capabilities cannot accomplish all the tasks comprehensive BI systems can do. Some firms seem to want to reduce the field further by overemphasizing the visualization aspect of discovery. While visual discovery can help users solve basic business problems, other BI and analytic tools are available that can attack more sophisticated and technically challenging problems. In our view, visual discovery is one of four types of analytic discovery that can help organizations identify and understand the masses of data they accumulate today. But for many organizations visualization alone cannot provide them with the insights necessary to help make critical decisions, as interpreting the analysis requires expertise that mainstream business professionals lack.

In Ventana Research’s view, business intelligence is a technology managed by IT that is designed to produce information and reports from business data to inform business about the performance of activities, people and processes. It has provided and will continue to provide great value to business, but in itself basic BI will not meet the new generation of requirements that businesses face; they need not just information but guidance on how to take advantage of opportunities, address issues and mitigate the risks of subpar performance. Ventana_Research_Value_Index_LogoAnalytics is a component of BI that is applied to data to generate information, including metrics. It is a technology-based set of methodologies used by analysts as well as the information gained through the use of tools designed to help those professionals. These thoughtfully crafted definitions inform the evaluation criteria we apply in our new and comprehensive 2015 Analytics and Business Intelligence Value Index, which we will publish soon. As with all business tools, applications and systems we assess in this series of indexes, we evaluate the value of analytic and business intelligence tools in terms of five functional categories – usability, manageability, reliability, capability and adaptability – and two customer assurance categories – validation of the vendor and total cost of ownership and return on investment (TCO/ROI). We feature our findings in these seven areas of assessment in our Value Index research and reports. In the Analytics and Business Intelligence Value Index for 2015 we assess in depth the products of 15 of the leading vendors in today’s BI market.

The Capabilities category examines the breadth of functionality that products offer and assesses their ability to deliver the insights today’s enterprises need. For our analysis we divide this category into three subcategories for business intelligence: data, analytics and optimization. We explain each of them below.

The data subcategory of Capabilities examines data access and preparation along with supporting integration and modeling. New data sources are coming into being continually; for example, data now is generated in sensors in watches, smartphones, cars, airplanes, homes, utilities and an assortment of business, network, medical and military equipment. In addition, organizations increasingly are interested in behavioral and attitudinal data collected through various communication platforms. Examples include Web browser behavior, data mined from the Internet, social media and various survey and community polling data. The data access and integration process identifies each type of data, integrates it with all other relevant types, checks it all for quality issues, maps it back to the organization’s systems of record and master data, and manages its lineage. Master data management in particular, including newer approaches such as probabilistic matching, is a key component for creating a system that can combine data types across the organization and in the cloud to create a common organizational vernacular for the use of data.

Ascertaining which systems must be accessed and how is a primary challenge for today’s business intelligence platforms. A key part of data access is the user interface. Whether it appears in an Internet browser, a laptop, a smartphone, a tablet or a wearable device, data must be presented in a manner optimized for the interface. Examining the user interface for business intelligence systems was a primary interest of our 2014 Mobile Business Intelligence Value Index. In that research, we learned that vendors are following divergent paths and that it may be hard for some to change course as they continue. Therefore how a vendor manages mobile access and other new means impacts its products’ value for particular organizations.

Once data is accessed, it must be modeled in a useful way. Data models in the form of OLAP cubes and predefined relationships of data sometimes grow overly complex, but there is value in premodeling data in ways that make sense to business people, most of whom are not up to modeling it for themselves. Defining data relationships and transforming data through complex manipulations is often needed, for instance, to define performance indicators that align with an organization’s business initiatives. These manipulations can include business rules or what-if analysis within the context of a model or external to it. Finally, models must be flexible so they do not hinder the work of organizational users. The value of premodeling data is that it provides a common view for business users so they need not redefine data relationships that have already been thoroughly considered.

The analytics subcategory includes analytic discovery, prediction and integration. Discovery and prediction roughly map to the ideas of exploratory and confirmatory analytics, which I have discussed. Analytic discovery includes calculation and visualization processes that enable users to move quickly and easily through data to create the types of information they need for business purposes. Complementing it is prediction, which typically follows discovery. Discovery facilitates root-cause and historical analysis, but to look ahead and make decisions that produce desired business outcomes, organizations need to track various metrics and make informed predictions. Analytic integration encompasses customization of both discovery and predictive analytics and embedding them in other systems such as applications and portals.

The optimization subcategory includes collaboration, organizational management, information optimization, action and automation. Collaboration is a key consideration for today’s analytic platforms. It includes the ability to publish, share and coordinate various analytic and business intelligence functions. Notably, some recently developed collaboration platforms incorporate many of the characteristics of social platforms such as Facebook or LinkedIn. Organizational management attempts to manage to particular outcomes and sometimes provides performance indicators and scorecard frameworks. Action assesses how technology directly assists decision-making in an operational context. This includes gathering inputs and outputs for collaboration before and after a decision, predictive scoring that prescribes action and delivery of the information in the correct form to the decision-maker. Finally, automation triggers alerts in circumstances based on statistical triggers or rules and should be managed as part of a workflow. Agent technology takes automation to a level that is more proactive and autonomous.

vr_Info_Optim_Maturity_06_oraganization_maturity_by_dimensionsThis broad framework of data, analytics and optimization fits with a process orientation to business analytics that I have discussed. Our benchmark research on information optimization indicates that the people and process dimensions of performance are less well developed than the information and technology aspects, and thus a focus on these aspects of business intelligence and analytics will be beneficial.

In our view, it’s important to consider business intelligence software in a broad business context rather than in artificially separate categories that are designed for IT only. We advise organizations seeking to gain a competitive edge to adopt a multifaceted strategy that is business-driven, incorporates a complete view of BI and analytics, and uses the comprehensive evaluation criteria we apply.


Ventana Research

Does Pricing and Revenue Optimization Make My Bottom Line Look Fatter?

Managing prices has always been an activity of keen interest to businesses, but it has become even more critical to do it well. Over the past decade many companies have found their ability to raise prices has been constrained by intense competition resulting from Internet commerce, global competition and other factors. One tool for dealing with this pressure is price and revenue optimization (PRO), an analytic methodology that calculates how demand varies at different price levels and then uses that algorithm to recommend prices that should optimally balance revenue and profit objectives. Computer-supported PRO began in earnest in the 1980s as the airline and hospitality industries adopted revenue management practices in efforts to maximize returns from less flexible travelers (such as people on business trips) while minimizing the unsold inventory by selling incremental seats on flights or nights in hotel rooms at discounted prices to more discretionary buyers (typically vacationers). Price and revenue optimization algorithms are designed to enable a company to achieve fatter profit margins than are possible with a monolithic pricing strategy. Using PRO, airlines and hotels catering mainly to less price-sensitive business travelers found they could match discounters’ fares and rates to fill available seats and rooms without having to forgo profits from their high-margin customers.

PRO has expanded into other industries as computing power and data storage become ever less expensive, as software vendors have improved their techniques and algorithms to deliver better results and as the software has grown increasingly user-friendly. While the concepts underlying all PRO software are the same, there are different categories in which it is customized to meet the needs of specific industries. Retailers in particular have requirements that are best met by using applications that manage markdowns.

At the heart of price and revenue optimization is the concept of demand-based pricing. As its name suggests, demand-based pricing is a method that sets a price that is controlled by the seller’s assessment of what the buyer is willing to pay, which in turn is based on an estimate of a good’s or a service’s perceived value to the buyer. Companies use demand-based pricing to optimize – rather than simply maximize – their pricing to achieve revenue and profitability objectives. It uses data to estimate where the prospective buyer sits on a demand curve and therefore how much the individual is likely to pay. In some respects this is similar to what happens daily in souks, bazaars and other markets in cultures that do not insist on set prices. However, software makes demand-based pricing practical in large businesses and facilitates its introduction in societies used to set pricing.

Advanced analytic applications – especially for price and revenue optimization – have been gaining ground in corporate management because they have demonstrated to work. Significantly, they have the ability to deliver results that are unobtainable otherwise. Such software can crunch through very large data sets rapidly, apply purpose-built algorithms and automate the repetitive mechanical steps needed to put decisions into action.vr_Office_of_Finance_13_finance_lacks_advanced_analytics It also ensures consistency and supports objectivity in how executives and managers make decisions. Price and revenue optimization applications have benefited as the cost and complexity of the computing resources needed to use them have declined.

The adoption of PRO software is part of a broader trend of using applications to support fact-based decisions that once depended on experience and hunches. However, our benchmark research on the Office of Finance finds that just 20 percent of companies use price optimization analytics extensively. Only one-third look at product profitability. We think that more of them should do both. Analytic applications can digest a considerable amount of data to segment markets into useful groupings, pinpoint correlations and divine trends, to name a few tasks necessary for pricing management. However, companies investigating PRO software should narrow their search to applications that are appropriate for their specific business. While some offerings have broader applicability than others, no software product now available performs well in every industry.

Retail businesses that have multiple outlets, especially those that deal in trend- or fashion-driven products, face unique price and revenue optimization challenges and this affects the design of pricing management applications aimed at retailers. Many of these businesses are self-service, exclusively so if they are Internet-based, so there is no face-to-face contact during the product selection process. Negotiating prices isn’t feasible in most multiple-outlet retail settings in developed economies because of cultural norms and the hazard of delegating these decisions to front-line staff in even a midsize company. Unlike business-to-business transactions that involve ongoing relationships with established products, most stores today know little about most of their customers, so there is no direct way of judging an individual’s price sensitivity for the specific purchase at hand. In other words, most of the elements that support PRO strategies in analytics used for other types of businesses aren’t available to multiple-outlet retailers.

Since they usually cannot gauge the price sensitivity of their customers, retailers take a different approach: Let the merchandise do the talking. Products that aren’t selling well are by definition overpriced in that market. Retailers have used markdowns as a crude tool of price optimization for a long time. Offering a 30 percent discount near the end of the season is usually better than having to take a 60 percent haircut from a close-out specialist. Yet deciding when and by how much to reduce prices and then implementing the reductions at the store level in an optimal fashion is complicated because of the number of variables that must be considered. There are different types of merchandise, including long-life categories of goods that can be offered for sale for years, short-life fashion and fad items that are offered only once and those somewhere in between. There are differences in demand patterns and price sensitivity between regions and even at the store level. Seasonality, weather and movable holidays such as Easter and Thanksgiving must be considered.

Using analytic applications is superior to relying on experience and intuition because applications often demonstrate that the best decisions go against the grain of established practices. For example, retailers have found that smaller markdowns applied earlier and more frequently produce better results (that is, greater volumes sold at a lower aggregate markdown) than the common practice of making one or two big moves. Until the data became available, minimizing the number of markdowns was reasonable because of the cost in staff time to change prices at the store level. However, retailers using smaller and more frequent markdowns more than pay for these costs and then establish processes to facilitate price changes. Some retailers have found to their surprise that early small markdowns reduce the overall cost of markdowns. Analytic applications also are able to deal with a range of variables that retailers can use in markdown management. For example, they can vary percentages and frequency by size and color as well as by location. The software can monitor sales and inventory levels by the SKU at each store and automatically make detailed recommendations on how to adjust pricing. The software also enables retailers with multichannel operations (usually an online presence) to manage pricing decisions optimally across different types of outlets.

PRO software designed for markdown management also enhances the ability of a multiple-outlet retailers to run their business in a way that maximizes the productivity of their stores measured in sales or gross margin per square foot (or meter) or per linear foot (or meter) of shelf space. Items taking up space in a store or on a shelf have an opportunity cost in that they could be replaced by faster-moving or more profitable goods. Modeling the cost of the uplift required to free up space can result in a more attractive mix of merchandise that will improve returns.

While usability and capability of markdown management software have been improving, retailers face internal challenges in being able to utilize it. Analytic applications are only as good as the data available to feed the systems. Our research consistently finds that data accuracy and availability are significant challenges that almost all midsize and large companies face. Using markdown management software successfully also involves a change management effort requiring heavy involvement by senior management to endorse changes in how the organization handles day-to-day business as well as changes to processes and training and considerable amounts of follow-up to ensure compliance with the new ways of doing business.

Information technology is playing an increasingly important role in how companies conduct their businesses. Analytic applications can transform how entire industries operate. Today, airline and hospitality businesses operate very differently from how they ran in the 1980s because of the Internet and analytics. All sorts of businesses are finding that price and revenue optimization software enables them to improve their results measurably. Retailers should look into markdown management software as a way to fatten their bottom line. Other types of businesses also should consider PRO tools as applied to their particular needs.


Robert Kugel – SVP Research