A Customer Data Platform History

Companies have used customer data to drive revenue since the beginning of commerce. The introduction of relational databases in the 1970s and 80s marked a major advancement here. It gave businesses the ability to store and query relatively large amounts of data, enabling them to use customer data at scale. In the on-prem era, businesses digitized transaction ledgers and used the data for database marketing, facilitating use cases strikingly similar to those we associate with modern digital marketing:

• Customer loyalty programs to increase customer retention
• Personalized direct-mail marketing campaigns
• Product placement optimization

Systems from Oracle, IBM, and HP, paired with enterprise resource planning (ERP) products like SAP and early customer relationship management (CRM) systems, enabled companies to glean customer insights and build significant competitive advantages with customer data. Market leaders used their customer databases to solve hard problems like understanding marketing attribution and customer churn.

In this age of early customer data management, just about everything related to data was under the IT team's purview. They were the heroes who built and maintained the systems that enabled companies to realize the power of putting customer data into action. As such, businesses rightly trusted IT with complete power and control over the technology and, in most cases, the data itself.

But data needs and expectations were different then. It was enough for teams to access a limited set of data through the ERP or an early BI tool like Business Objects. Data was the sole purview of the IT team. Access to the raw data outside of IT was novel. In addition, IT was treated as a service center and wasn’t forced to collaborate with other business units to take advantage of front-line opportunities. Because of the comparatively simple data landscape, data silos were not an issue because most companies purchased a database, then applications to run on top of that database. Customer data lived in the database, IT could provide teams with the data they needed, and everyone was happy.

Identity resolution was not a major concern in this era because the data sources were limited primarily to transaction data. It was before the arrival of the digital customer journey as we know it today, and IT teams controlled customer data collection. But this all changed when the internet achieved critical mass distribution.

The web goes worldwide

At the turn of the century, businesses were pushing into a new, online digital frontier. Once again, IT teams led the effort. But things were different this time.

Businesses had tasted the potential of data in the on-prem era, and they were beginning to understand its power. When technological innovations pushed commerce online, data appetites increased exponentially.

Meanwhile, the advent of a rich online customer experience during Web 2.0 pushed a new type of data to center stage: customer behavioral data. With digital customer interactions, it was now possible to stitch together a customer journey by tracking every customer touchpoint. Companies quickly recognized behavioral data as the key to understanding and reaching customers in this new era to increase customer lifetime value. But comprehensive tracking was expensive, complex to implement, and because engineering teams lacked modern tooling to make implementation easy, everything had to be custom-built.

The smartphone revolution, catalyzed by the launch of the iPhone in 2007, made the tracking landscape even more complex. The same user could now access a company's website on a desktop, laptop, or mobile device. Moreover, they might access both the company's website and mobile app, further complicating the picture.

A wealth of opportunity lay on the other side of this complexity – companies that could manage to collect online and offline customer data and unify records for individuals in a single customer view would have a major competitive advantage over those that could not. But the teams best suited to solve this data integration and identity resolution challenge, the IT teams, were consumed with other massive digital transformation projects and limited by primitive tooling. They didn’t have the bandwidth to keep up with the demand for innovation around data.

SaaS takes over

With so much opportunity on the table, standing still wasn’t an option. Companies couldn’t just wait around for IT to help them harness data in this new era. In a global upswing of innovation, the choice was either to make a way or get left behind. The software industry recognized this dilemma as an opportunity.

If IT teams couldn’t keep up, agile startups could. Fueled by venture capital and advancements in cloud technology, SaaS solutions proliferated to fill the gap between what was possible and what was attainable.

For companies under pressure to keep up with the rapid advancements in the new digital customer journey, SaaS was a golden ticket to operating on the cutting edge. No longer did business unit leaders have to wait for IT to design, provision, and implement solutions, they could simply pick a SaaS tool and get up and running in a matter of weeks. This trend is perhaps best captured by the well-documented martech explosion.

The benefits of this SaaS-ification were undeniable. SaaS products allowed teams to maintain, and often accelerate, progress. Plus they were cheaper than developing in-house, and they were typically more stable. But as with all major advances in technology, the SaaS-ification of customer data tooling involved trade-offs.

Damn the data silos, full speed ahead

In the pre-SaaS days, just about all of a company's customer data lived in its internal database. Developing analytics and applications using multiple sets of data wasn’t a problem then because the data was all inside of the same system. This simple landscape with centralized data is what businesses traded for velocity when they embraced SaaS.

The SaaS explosion created a serious data problem that every company struggles with to this day: data silos.

Every individual SaaS tool locked data in its own ecosystem, creating a data silo. Moving data into or out of those systems was difficult at best because they were built to solve for specific business use cases, not data integration. This data silo problem further complicated the identity resolution equation. With valuable customer data scattered across various SaaS products, it was impossible to build anything close to a customer 360. The cure was worse than the original problem.

To solve the SaaS data silo problem, a new category of SaaS emerged. ETL tools were built to extract the data from SaaS systems and move it into databases and data warehouses. These tools did solve a problem, but they were really only useful for analytics use cases. Simple applications for tasks like sending personalized emails based on purchase behavior became needlessly complex. Development required multiple API calls and multiple data pipelines, and if you wanted to do anything in real time, you could forget about it.

This new, fragmented landscape created pain for teams company-wide, but none felt it more acutely than marketing teams. They needed all of the data and a complete view of the customer, but they were using multiple tools with different functionalities that didn’t talk to each other. The marketing technology market responded, and yet another SaaS category emerged promising to solve all of their problems: the customer data platform.

The CDP promised to automatically bring in first-party data from every SaaS tool and aggregate it in a unified customer database, creating high-quality comprehensive customer profiles that could be used for audience segmentation and analytics. It would also provide orchestration tooling on top to enable the tactical activation of data for omnichannel customer engagement. For customer-centric marketing teams, this would help ensure a consistent customer experience for initiatives across web and mobile properties, email, social media, and paid digital advertisements. But this solution ignored a critical fact: IT still controlled key business systems like web and mobile experiences, eCommerce applications, backend systems, and the databases that powered them.

Plus, these legacy CDPs were no different than other SaaS tools – though they unified data from other systems, they still created another data silo. There was no easy way for teams outside of marketing to access the valuable metrics stored inside the SaaS CDP’s black box.

The pain of a fragmented system persisted. But pain bears innovation. IT teams were beginning to adopt cloud data warehouses which were quickly overcoming the technical limitations of databases from decades past.

As business teams were feeling the impact of data silo proliferation and limited data integration, IT teams had already started implementing the right answer.

Warehouse-native customer data architecture

Warehouse technology advancements meant the limitations of traditional marketing CDPs could be overcome, but doing so would require a new approach: building the entire customer data stack around the warehouse instead of relying on a marketing CDP. Ironically enough, this approach required a return to reliance upon technical experts.

Teams were learning, through dealing with the consequences of SaaS-ification, that going around IT wasn’t a good idea. IT teams hadn’t been idle during the SaaS boom either. They had been busy re-inventing every piece of tech in the organization, leveraging both SaaS and the cloud. IT teams themselves had evolved. Many companies, recognizing the importance of data infrastructure, formalized engineering roles for the people who built and managed that infrastructure, creating data teams and the role of data engineering.

Armed with new technologies and the necessary skill set, data engineers could provision infinitely scalable infrastructure from cloud providers and work with developers to centralize first-party relational and behavioral customer data in a modern cloud data warehouse where it could be enriched with second and third-party data.

In theory, this new architecture solved the problem of identity resolution that became acute in the age of data silos. If the data warehouse serves as a central repository for every individual customer data point, behavioral data from every device, website, and application can live alongside relational data from every SaaS tool, the data can be enriched, then unified behind a canonical identifier to create truly comprehensive customer profiles. Finally, the profiles can be sent downstream to marketing automation tools, CRM systems, customer support tools, and machine learning models to create competitive advantage and drive personalized experiences.

But as we’ve learned, things are never as simple as they seem. There are several unique and difficult technical challenges to building a complete, usable view of the customer, even with modern technology. In The Evolution of The Customer Data Platform, we’ll examine how recent evolutions of the CDP seek to solve them.