Cryptic Data: Challenges and Rewards in Finding and Using It

Using information technology to make data useful is as old as the Information Age. The difference today is that the volume and variety of  available data has grown enormously. Big data gets almost all of the attention, but there’s also cryptic data. Both are difficult to harness using basic tools and require new technology to help organizations glean actionable information from the large and chaotic mass of data. “Big data” refers to extremely large data sets that may be analyzed computationally to reveal patterns, trends and associations, especially those related to human behavior and interaction. The challenges in dealing with big data include having the computational power that can scale to the processing requirements for the volumes involved; analytical tools to work with the large data sets; and governance necessary to manage the large data sets to ensure that the results of the analysis are accurate and meaningful. But that’s not all organizations have to deal with now. I’ve coined the term “cryptic data” to focus on a different, less well known sort of data challenge that many companies and individuals face.

Cryptic data sets aren’t easy to find or aren’t easily accessed by people who could make use of them. Why “cryptic?” As a scuba diver, I donate time to Reef Check by doing scientific species counts in and around Monterey Bay, Calif. Cryptic organisms are ones that hide out deep in the cracks and crevices of our rocky reefs. Finding and counting them accurately is time-consuming and requires skill. Similarly, it’s difficult to locate, access and collect cryptic data routinely. Because it’s difficult to locate or access routinely, those who have it can gain a competitive advantage over those who don’t. The main reason cryptic data is largely untapped is cost vs. benefits: The time, effort, money and other resources required to manually retrieve it and get it into usable form may be greater than the value of having that information.

By automating the process of routinely collecting information and transforming it into a usable form and format, technology can expand the range of data available by lowering the cost side of the equation. So far, most tools, such as Web crawlers, have been designed to be used by IT professionals. Data integration software, also mainly used by IT departments, helps transform the data collected into a form and format where it can be used by analysts to create mashups or build data tables for analysis to support operational processes. Data integration tools mainly work with internal, structured data and a majority have little or no capability to support data acquisition in the Web. Tools designed for IT professionals are a constraint in making better use of cryptic data because business users are subject matter experts. They have a better idea of the information they need and are in a better position to understand the subtleties and ambiguities in the information they collect. To address this constraint, Web scraping tools (what I call “data drones”) have appeared that are designed for business users. They use a more visual user interface design and hide some of the complexity inherent in the process. They can automate the process of collecting cryptic data and expand the scope and depth of data used for analysis, alerting and decision support.

Cryptic data can be valuable because when collected, aggregated and analyzed, it provides companies and individuals with information and insight that were unavailable. This is particularly true of data sets gathered over time from a source or combination of sources that can reveal trends and relationships that otherwise would be difficult to spot.

Cryptic data can exist within a company’s firewall (typically held in desktop spreadsheets or other files maintained by an individual as well as in “dark” operational data sets), but usually it is somewhere in the Internet cloud. For example, it may be

  • Industry data collected by some group that is only available to members
  • A composite list of products from gathered from competitors’ websites
  • Data contained in footnotes in financial filings that are not collected in tabular form by data aggregators
  • Tables of related data assembled through repetitive queries of a free or paid data source (such as patents, real estate ownership or uniform commercial code filings).

Along these lines, our next-generation finance analytics benchmark research shows that companies have limited access to information about markets, industries and- economies. vr_NG_Finance_Analytics_17_accessibility_of_external_dataOnly 14 percent of participants said they have access all the external data they need. Most (63%) said they can access only some of it, and another 14 percent said they can’t access any such data. In the past, this lack of access was even more common, but the Internet changed that. And this type of external data is worth going after, as it can help organizations build better models, perform deeper analysis or do better in assessing performance, forecasting or gauging threats and opportunities.

Cryptic data poses a different set of challenges than big data. Making big data usable requires the ability to manage large volumes of data. This includes processing large volumes, transforming data sets into usable forms, filtering extraneous data and code data for relevance or reliability, to name some of more common tasks. To be useful big data also requires powerful analytic tools that handle masses of structured and unstructured data and the talent to understand it. By contrast, the challenge of cryptic data lies in identifying and locating useful sources of information and having the ability to collect it efficiently. Both pose difficulties. Whereas making big data useful requires boiling the ocean of data, cryptic data involves collecting samples from widely distributed ponds of data. In the case of cryptic data, automating data collection makes it feasible to assemble a mosaic of data points that improves situational awareness.

Big data typically uses data scientists to tease out meaning from the masses of data (although analytics software vendors have been working on making this process simpler for business users). Cryptic data analysis is built on individual experience and insight. Often, the starting point is a straightforward hypothesis or a question in the mind of a business user. It can stem from the need to periodically access the same pools of data to better understand the current state of markets, competitors, suppliers or customers. Subject matter expertise, an analytical mind and a researcher’s experience are necessary starting capabilities for those analyzing cryptic data. These skills facilitate knowing what data to look for, how to look for it and where to look for it. Although these qualities are essential, they not sufficient. Automating the process of retrieving data from sources in a reliable fashion is a must because, as noted above, the time and expense required to acquire the data manually are greater than its value to the individual or organization.

Almost from the dawn of the Internet, Web robots (or crawlers) have been used to automate the collection of information from Web pages. Search engines, for example, use them to index Internet pages while spammers use them to collect email addresses. These robots are designed and managed by information technology professionals. Automating the process of collecting cryptic data requires software that business people can use. To make accessing cryptic data feasible, they need “data drones” that can be programmed by users with limited training to fetch information from specific Web pages. Tools available from Astera ReportMinerConnotate, Datawatch,, Kofax Kapow and Mozenda are great examples on where you can get started for leveraging cryptic data. I recommend that everyone who has to routinely collect information from Internet sites or from internal data stores that are hard to access or who thinks that they could benefit from using cryptic data investigate tools available for collecting it.


Robert Kugel – SVP Research

Aria Enables Effective Recurring Revenue Management

Aria Systems provides companies with software for managing subscription or recurring revenue business models. A recurring revenue business models includes three types of selling and billing structures: a one-time transaction plus a periodic service charge; subscription-based services involving periodic charges; or a contractual relationship that charges periodically for goods and services. Aria’s cloud-based software addresses key requirements of users in the marketing, sales, operations and accounting functions in this type of business.

Recurring revenue, first popularized in the telecommunications industry, is increasingly common in others. It is well suited to companies accessing software and hardware technology as a service through cloud computing. For example, has a strong impact in the entertainment business, as customers subscribe to rent movies, music and other creative digital products instead of owning them. In general recurring revenue is attractive to providers of services or products because it establishes a regular, predictable income stream as long as they retain the customer. In using it it’s essential to handle interactions smoothly and completely in order to sustain customer engagement and maximize each customer’s lifetime value. Software is an essential element to successfully doing so.

Aria’s software is designed to help companies maximize customer lifetime value in three main ways. First, it is designed to help create a positive customer experience with every interaction. In our benchmark research on recurring revenue, vr_Recurring_Revenue_03_recurring_revenue_challengesmaintaining customer engagement is the most frequently cited challenge in businesses that use it, cited by 55 percent of participants. Having repeated positive interactions can be an important determinant of renewal rates. Renewals in turn are a key driver of profitability in these businesses because of the relatively high cost of adding a customer. Along those lines, 39 percent of participants  cited customer retention as an impediment. Moreover, since a company’s costs related to its recurring revenue business are relatively fixed in the short term, almost all the impact of lost revenue drops to the bottom line, depressing profits.

Rather than treating billing as a purely functional accounting event, Aria’s software enables a company to automatically incorporate personalized usage tips or customized thank-you messages in anticipation of an approaching anniversary (and renewal) date. It also can automate up-sell and cross-sell messages tailored to each customer. Nearly half (46%) or organizations said cross-selling and up-selling are difficult. This may be because they can’t engage effectively with existing customers. Multiple internal factors may affect this, such as a poorly designed marketing program for existing customers, a lack of skilled agents for performing ongoing interactions or technology limitations that prevent a company from creating or executing an effective customer nurturing program. Using software to craft an automated process for deepening customer interactions can be a way to enhance engagement. Companies that find it difficult to up-sell or cross-sell also may discover that for some customers its primary service is of limited importance. This may be because they don’t want to consider a more expensive, deluxe version or add-ons. Understanding which customers fall into this category is important so that up-sell and cross-sell efforts are focused only on those who likely to be receptive. Aria’s software enables business people to manage these aspects of the billing process without involving IT professionals.

Second, to support positive customer interactions, the software offers flexibility to quickly create and modify customer offers in a controlled fashion. Aria has a centrally administered catalog that defines the products, services and bundles on offer as well as their pricing, terms and conditions. In addition the software can handle a range of things that a company can bill for, including types of content, service levels, usage metering based on physical quantities, time or distance  or some combination of factors. Companies can define and manage offers based, for example, on the sales channel, geographic location or currency, and it can do that without requiring expensive and time-consuming customizations; thus a company can introduce innovations rapidly or react quickly to changes in its market. The control provided by such a catalog enables sales people to configure a set of terms and conditions that best match a current or prospective customer’s needs within established parameters. These limits ensure that the offerings balance flexibility and complexity. They enable administrators to limit the available pricing and terms to offers are that are profitable (or at least not loss-making) for the company.

vr_Recurring_Revenue_06_finance_less_satisfied_with_invoicingThe third factor is that, by managing the billing process in a continuous fashion, Aria’s software ensures complete accuracy. For anything more than a simple subscription invoicing can be a chore because customers often add or remove services to and from their contracts or negotiate a new billing method to suit their needs. It’s easy for those outside of finance and accounting departments to overlook the impacts on the department of not having a controlled end-to-end process, which can be addressed by using a dedicated application designed to support the billing process in a recurring revenue business. In our research only 29 percent of participants with finance and accounting titles said they are satisfied with their company’s invoicing system, compared to nearly half (47%) of those who work in other parts of their company. Managing the billing process from contract to cash in a single system provides a control mechanism that makes sure that the customer is not overcharged and that the company doesn’t suffer revenue leakage.  Another benefit of the software is that by managing the process from end to end it ensures the integrity of the data used in the billing process and eliminates the need for time-consuming checks and reconciliations that are necessary when, for example, the same data must be entered into multiple systems or when companies use spreadsheets at any point in the process to move data from one system to another or to handle adjustments or allocations. A well-designed billing system also facilitates the revenue recognition process.

Some companies sell directly to customers either through sales people (assisted selling) or a commerce website (unassisted selling), and some do both. Where subscription-like services are concerned, using a centralized catalog as the authoritative source for controlling offers ensures that the offers are valid and consistent with policies. For directed selling, Aria’s offers integration with the CRM system to ensure data in the two systems is synchronized. The software can be integrated with a company’s e-commerce site and enable offers and promotions tailored to specific buyers based on their relationship with the company, their location, past buying history or other factors. For both types of selling, the software facilitates testing of plans, promotions and services to determine the best approach to use. Users can apply effective dating to turn promotions on or off automatically at set times.

Aria’s built-in analytics addresses the needs of various roles in managing the recurring revenue business. Analytics is necessary to measure and monitor the health of a business. Having up-to-the-minute data digested and displayed for specific roles and responsibilities supports faster, more coordinated responses to market developments. Confirming its importance, most (82%) of the participantsvr_Recurring_Revenue_08_analytics_most_important_for_recurring_revenue in our research chose analytics as an important new technology necessary to support their recurring revenue business.

Not every company needs a dedicated application to manage its recurring revenue business. Those with simple offerings that rarely change over the term of the subscription are likely to find that their ERP system will serve their needs. However, companies that have even moderately complex offerings, that serve a diverse set of customers or that need to be nimble in managing offers and promotions will find that a recurring revenue application improves their performance. Our research finds that users of dedicated third-party software said they are satisfied with its performance more often than those using any other method: 86 percent said they are satisfied or somewhat satisfied with it, compared to 70 percent of those that use their ERP system and just 40 percent that use spreadsheets. I recommend that companies that have recurring revenue businesses assess whether dedicated software can help their performance and, if they so decide, they should consider Aria’s offering.


Robert Kugel – SVP Research