Engineering VP Unveils Data Warehouse Management Benefits

Solutions Review’s Expert Insights Series is a collection of contributed articles written by industry experts in enterprise software categories. In this feature, Inrupt VP of Engineering Emmet Townsend reveals why the end-result of proper data warehouse management is a simplified view of authoritative data that organizations can count on to deliver timely insights.

Most enterprises pour resources into data warehouse management. But they often fail to achieve the results they expect from their investments. Why? Because the current model for integrating the data warehouse with application data isn’t adaptable enough to deliver insights when the enterprise landscape evolves.

Business decision-makers and analysts who need access to data are frequently left waiting on IT to make updates to the integration, which delays the availability of insights and decreases the value they derive from the data stored in the warehouse.

Data warehouses certainly haven’t outlived their usefulness. But as storage and analytics facilities, they were never designed to serve as the operations center for ever-evolving data transformation and standardization. Warehouse teams rely on rigid extract, transform, and load (ETL) and reverse ETL integration processes to manage data inflows and outflows. But these are patchwork solutions to much bigger structural problems. The problems are especially acute when organizations needs to manage the personal data of users, which changes frequently and needs to be protected as it flows across systems.

Sir Tim Berners-Lee, the originator of the web, believes the solution is to model and organize data around people rather than applications.

Berners-Lee’s solution, called Solid, is an open standard that makes the web into a universal data infrastructure. In the context of organizations’ data management, Solid enables businesses to more effectively leverage the strengths of data warehouses by storing customers’ personal data in interoperable “Pods” that the warehouse can access. This solution moves ETL responsibilities to application owners best positioned to transform and maintain the data. An evolving common enterprise vocabulary ensures the same terms are used by all application teams when transforming data. The data warehouse can then easily aggregate data across all systems and keep pace with the continuously evolving enterprise.

The result: a simplified view of authoritative data that organizations can count on to deliver timely insights.

Data Warehouse Management

Supporting Seamless Data Flows and Transformation

Nearly all companies struggle to consolidate fragmented and siloed customer data across applications, and they go to great lengths to ensure that data is valid and up to date.

Existing solutions for managing this span of customer data in the warehouse are overly complex, inefficient and rigid, making it difficult to deliver timely insights. But an interoperable data model with a standard interface, where consent is directly granted by the consumer, minimizes the use of development resources when new applications and services are introduced.

By introducing a flexible schema that teams can evolve themselves, interoperable systems like Solid Pods enable source system experts to extend data models at any time, reducing internal gridlock between warehouse teams and others. Warehouse teams can easily discover the newly available, self-describing data and use it to enhance insights when required.

Solid’s standard interface also enables source systems to directly exchange data with other source systems. The data exchange bypasses the central warehouse, which minimizes friction and introduces cost savings by eliminating the need for reverse ETL. As a result, source system owners and analysts can simplify their workflows, move with agility, and easily perform ongoing maintenance to ensure all insights derived from the warehouse are based on up-to-date information.

And because Solid servers can identify the users and applications seeking to access data, the system guarantees data is accessed in ways that align with customers’ permissions and privacy agreements. This is a built-in safeguard that other data storage solutions don’t provide, yielding data that is not only universally accessible and interoperable but also fully authorized.

Additional Benefits of Customer-Centric Storage Solutions

Interoperable solutions provide benefits beyond those directly tied to the data warehouse. These primarily fall into two categories: customer trust and product innovation.

• Trust and transparency: When data is modeled and organized around people, customers gain more visibility into how companies can use and access their personal information. Customer-centric data architecture can also simplify compliance with regulations like GDPR and CCPA by modeling these regulations in auditable policies stored alongside the data.

Data trust is important because it has ramifications beyond regulatory compliance. Customers who feel their data is secure and only used in ways that respect their wishes are more willing to share zero-party data in exchange for more personalized service, especially when built-in mechanisms enable customers to explicitly grant or deny consent for data use for specific purposes. Internally, organizations can protect this discoverable data with authorization policies, guarding themselves against data misuse.

• Innovation and agility: A customer-centric data model, such as Solid, equips analysts with a holistic view of the customer. From there they can determine which data points are authoritative for the given context.

With access to this breadth of customer data at their fingertips, business analysts can identify opportunities for new products and innovate at a faster pace.

Solutions like data warehouses certainly offer invaluable tools for business insights and decision-making. But unwieldy, inefficient technological processes that organize data around applications, instead of customers, strain internal workflows and ultimately diminish innovation and customer trust.

Complementary interoperable technology that stores data in a customer-centric fashion shifts data transformation to the edge while multiplying the value companies can achieve from their data warehouses. And it enables organizations to alleviate internal stressors and streamline workflows. The result is positive outcomes for everyone from IT teams to decision-makers to customers themselves.