Integration of Big Data Involves Challenges


Big data has great promise for many organizations today, but they also need technology to facilitate integration of various data stores, as I recently pointed out. Our big data integration benchmark research makes it clear that organizations are aware of the need to integrate big data, but most have vr_BDI14_performance_01_overallyet to address it: In this area our Performance Index analysis, which assesses competency and maturity of organizations, concludes that only 13 percent reach the highest of four levels, Innovative. Furthermore, while many organizations are sophisticated in dealing with the information, they are less able to handle the people-related areas, lacking the right level of training in the skills required to integrate big data. Most said that the training they provide is only somewhat adequate or inadequate.

Big data is still new to many organizations, and they face challenges in integrating big data that prevent them from gaining full value from their existing and potential investments. Our research finds that many lack confidence in processing large volumes of data. More than half (55%) of organizations characterized themselves as only somewhat confident or not confident in their ability to accomplish that task. They have even less confidence in their ability to process data that arrives at high velocity: Only 29 percent said they are somewhat confident or not confident in that. In dealing with the variety of big data, confidence is somewhat stronger, as more than half (56%) declared themselves confident or very confident. Assurance in one aspect is often found in others: 86 percent of organizations that said they are very confident in their ability to integrate the variety of big data are satisfied with how they manage the storage of big data. Similarly 91 percent of those that are confident or very confident with their data quality are satisfied with the way they manage the storage of big data.

Turning to the technology being used, we find only one-third (32%) of organizations satisfied with their current data integration technology, but twice as many (66%) are satisfied with their data integration pro­cesses for loading and creating big data. A substantial majority (86%) of those very confident in their ability to integrate the needed variety of big data are vr_BDI_03_plans_for_big_data_technologysatisfied with their existing data integration processes. Those that are not satisfied said the process is too slow (61%), analytics are hard to build and maintain (50%) and data is not readily available (39%). These findings indicate that making a commitment to data integration, for big data and other­wise, can pay off in confidence and satisfaction with the processes for doing it. Additionally, organizations that use dedicated data integration technology (86%) are satisfied much more often than those that don’t use dedicated technology (52%).

New types of big data technologies are being introduced to meet expanding demand for storage and use of information across the enterprise. One of those fast-growing technologies is the open source Apache Hadoop and commercial enterprise versions of it that provide a distributed file system to manage large volumes of data. The research finds that currently 28 percent of organizations use Hadoop and about as many more (25%) plan to use it in the next two years. Nearly half (47%) have Hadoop-specific skills to support big data integration. For those that have limited resources, open source Hadoop can be affordable, and to automate and interface with it, adopters can use SQL in addition to its native interfaces; about three in five organizations now use each of these options. Hadoop can be a capable tool to implement big data but must be integrated with other information and operational systems.

Big data is not found only in conventional in-house information environments. Our research finds that data integration processes are most often applied between systems deployed vr_BDI_07_types_of_data_integration_processeson-premises (58%), but more than one-third  (35%) are integrating cloud-based systems, which reflects the progress cloud computing has made. Nonetheless, cloud-to-cloud integration remains least common (18%). In the next year or two 20 to 25 percent of organizations plan additional support for all types of integration; those being considered most often are cloud-to-cloud (25%) and on-premises-to-cloud (23%), further reflecting movement into the cloud. In addition, nearly all (95%) organizations using cloud-to-cloud integration said they have improved their activities and proces­ses. This finding confirms the value of inte­gration of big data regardless of what types of systems hold it. With a growing number of organi­za­tions using cloud computing, data inte­gra­tion is a critical requirement for big data projects; more than one-quarter (28%) of organizations are deploying big data integration into cloud computing environments.

Because of the intense need of business units and process for big data, integration requires IT and business people to work together to build efficient processes. The largest percentage of organizations in the research (44%) have business analysts work with IT to design and deploy big data integration. Another one-third assign IT to build the integration, and half that many (16%) have IT use a dedicated data integration tool. The research finds some distrust in involving the business side. Almost one in four (23%) said they are resistant or very resistant to allowing business users to integrate big data that IT has not prepared first, and the majority (51%) resist somewhat. For more than half (58%) the IT group responsible for BI and data warehouse systems also is the key stakeholder for designing and deploying big data integration; no other option is used by more than 11 percent.

It is not surprising that IT is the department that most often facilitates big data and needs integration the most (55%). The most frequent issue arising between business units and IT is entrenchment of budgets and priorities (in 42% of organizations). Funding of big data initiatives most often comes from the general IT budget (50%); line-of-business IT budgets (38%) are the second-most commonly used. It is understandable that IT dominates this heavily technical function, but big data is beneficial only when it advances the organization’s goals for information that is needed by business. Management should ensure that IT works with the lines of business to enable them to get the information they need to improve business processes and decision-making and not settle for creating a more cost-effective and efficient method to store it.

Overcoming these challenges is a critical step in the planning process for big data. My analysis that big data won’t work well without integration is confirmed by the research. We urge organizations to take a comprehensive approach to big data and evaluate dedicated tools that can mitigate risks that others have already encountered.

Regards,

Mark Smith

CEO and Chief Research Officer

Lessons Learned about the Customer Experience in 2014


During this year talk has been widespread about the customer experience, which is good. What is not so good is that, according to my benchmark research into next-generation customer engagement, most companies still struggle to deliver satisfying experiences. However, the research and my discussions with users and vendors lead to some clear conclusions:

  • Consumers have changed the way they communicate with each other, and this has changed how they expect to engage with businesses.
  • Customer expectations have elevated. Many wantvr_NGCE_Research_08_all_channels_for_customer_engagement engagements to be EPIC: that is, Easy (at the time of their choice, through channels of their choice and easy to use technology), Personalized, in context (recognizing the state of their relationship and previous interactions with a company) and Consistent (providing the same information regardless of channel).
  • Organizations must support multiple channels of engagement or risk losing customers as found in our research ranging from telephone (94%) and email (92%) to mobile (29%).
  • Multiple departments engage with customers, but their responses often differ because not everyone has the same objectives or customer information.
  • Organizations strive to improve customer satisfaction but inefficient processes, inadequate people skills and cost constraints often get in the way.

These points and others were raised recently at the 2014 Customer Engagement Summit in London. Keynote speaker Louise Cooper, a noted financial analyst, columnist and broadcaster, related a personal experience that illustrated what can go wrong with the customer experience. In purchasing 10 new coats at a well-known retail outlet, apparently she upset the checkout assistant by “creating too much of a draft” as she placed the coats on the counter. The assistant, seemingly out of spite, dropped each coat on the floor as she bagged them. When Louise got home, she discovered that the coats were dirty, and hereafter the process broke down: She received no response to email complaints, and despite promises of action by customer service, none was taken. The situation was resolved to a degree only after Louise tweeted to her half-million followers; an executive picked up the issue and had 10 new, clean coats delivered. This led me at the time to tweet that it is so easy for people and process glitches to nullify the best-laid customer experience plans.

The theme of how people impact the customer experience continued during the sessions. Several speakers insisted that employee engagement is paramount to delivering superior customer experiences; this point was illustrated in my benchmark research into the smart agent desktop, which shows that very happy contact center agents twice as often meet their targets for customer satisfaction and net promoter scores as do less satisfied ones.

The challenge as I see it is to think in four dimensions:

  • One is the customer business journey. Customers have different needs, wants and expectations as they move through the process of finding products or services, buying their selections, looking for help in using the product or service, and getting support if things don’t go as smoothly as they expected
  • The customer engagement journey requires understanding of how customers engage with an organization at different points in the business journey, including the communication channels they use for each type of issue, how they move from one channel to another if they don’t get resolution in the first, times at which they engage and outcomes.
  • There is also an internal journey. It requires understanding which lines of business engage with customers at different points in the business and engagement journeys.
  • The product or service journey involves how customers engage with an organization for different products and services, and how that varies depending on the nature of each.

Ineffective technology also gets in the way. My next-generation customer experience research also yields insights into why systems hinder organizations in delivering superior experiences, particularly these:

  • 49 percent struggle to integrate the systems required to support the customer experience.
  • 47 percent have multiple channels of communication, but most are managed as individual systems.
  • 33 percent said that responses differ depending on who the customer interacts with.

A further complication is that technology is changing at an unprecedented pace. We live in a digital world dominated by the use of smartphones and tablets. We live in a time deficient world so again everything seemingly needs to happen in real time and often while the customer and/or the employee is on the move. Consumers are becoming more social so companies need to take into account the impact any one tweet out of the billions might impact their business. Consumers are also becoming more trusting in self-service technologies so many more are happy to solve their issues using voice activated technologies, including virtual agents. And the recurring revenue business models is changing some one-off purchases into longer-term subscription services.

The issue for customer service organizations is how to keep up. Many recommend adopting a more customer-focused culture. My view is that organizations need to manifest a change of culture in the four interconnected dimensions Ventana Research tracks: people, process, information and technology. As the example above showed, under-motivated people can destroy the best-laid customer experience plans. Organizations have to get hiring processes, onboarding and quality monitoring right, and support employees with training, coaching and suitable technology so they can deliver superior customer experiences. As organizations rethink their engagement processes, I suggest starting with the four journeys vr_Customer_Analytics_03_key_benefits_of_customer_analyticsdescribed above. Map what is happening in each journey, decide how you want it to happen in the future, put in place programs to make it happen and then repeat the process. One of the major determinants of the success of any business activity, and customer experience is no different, is the metrics you use to assess and monitor success. My research into next-generation customer analytics shows that customer, interaction and customer journey analytics can help companies deliver on key objectives such as improving the customer experience (55%) and gaining better alignment across business units (51%). However, the research also finds a gap between what is important to most companies and what they measure. They care about customer-facing metrics such as customer satisfaction and net promoter scores, but mostly they measure operational metrics such as average handling times and hold times. Find a balance between the two and the impacts one might have on another; for example, sales conversion rates might increase call volumes and decrease satisfaction because customers weren’t fully informed at the time of the sale. These days making any serious change involves investing in new technologies, even if it is only to process the huge volumes of data now being generated. So take time to investigate new technologies such as mobility, cloud-based communications infrastructure, workforce optimization, smart desktop technologies and above all analytics; each of these can help support new customer experience strategies.

Harvard Business Review recently pronounced that “companies should stop trying to delight customers,” which caused a stir. Its research shows that little business benefit to be had from delighting customers, but there is a fundamental need to get the basics right. I couldn’t agree more. At a time when trying to compete on product, service or price has become increasingly difficult, what counts is the customer experience. Several consumer research studies show that customers who receive satisfying experiences will stay loyal and buy more products or services, but getting it wrong even once can lose a customer and damage a company’s brand via social media. To attract and retain customers, their experiences must be EPIC (see above for definition). To survive in today’s business world, seek convergence of the four journeys described above, and align your people, processes, information and technology with them.

Regards,

Richard J. Snow

VP & Research Director