DataGovOps – The Unsung Hero in Achieving AI-Ready Data

DataOps.live’s Guy Adams offers commentary on how DataGovOps is the unsung hero is achieving AI-ready data. This article originally appeared in Insight Jam, an enterprise IT community that enables human conversation on AI.

Last week, we introduced the concept of AI-ready data and defined the various ways that organizations can measure it. Here we’re going to look at strategies for improving the metrics and actually achieving AI-ready data, which will lean heavily on an emerging practice called DataGovOps.

To quickly recap from my earlier piece, the quality of AI is based in large part on the quality of your data. Bad data equals bad AI. Having unsuitable data is not the only way that AI projects can go off the rail, but it’s certainly a big one. Research has identified about 200 metrics that can be used to measure the AI-readiness of data. These metrics span six categories, including data quality, data for AI training, data governance, data semantics, data management and operationalization, and data interoperability.

Clearly, data must be monitored if it’s going to be improved. But measuring the data is not enough. To succeed with AI, one must take concrete steps to improve the AI-readiness of data. That’s where the real work begins.

How Automating Data Remediation Can Speed Up AI-Readiness

The majority of organizations are likely to have a low AI-readiness score, which is a form of technical debt. After all, data engineers only have so many cycles to spend in a day, with a never-ending list of demands on their time. They may have established methods of improving structured data for traditional BI and analytics projects, but that doesn’t really cut it in enterprise AI, which operates on a much larger and diverse pool of less structured data.

The good news is that it is possible to improve the AI-readiness of data if an organization is willing to take the steps to get there. The field of data management has improved significantly over the years. Through trial and tribulation, a series of best practices have emerged that can help an organization improve the quality of their data.

Thanks to the maturation of AI itself, data professionals can not only receive targeted recommendations on how to improve data management for their specific environment, but they can lean on AI to automate much of the rote work of remediating the specific data management challenges. Think of it as using AI to improve AI.

This doesn’t absolve the data leader of responsibility, of course. It’s still up to data leaders to define the overall data governance strategy, to set specific AI-readiness goals, and to oversee the execution of the strategy. In some ways, setting the governance goals is a more difficult challenge. Enterprise data is extremely diverse and it’s constantly changing. The specific goals and metrics that are important for one organization will not necessarily apply to another organization.

The odds of successfully achieving AI-ready data improve when a data professional takes a proactive approach. Those who try to deal with these issues in a reactive way are likely to fall even further behind.

Operationalization of Data Governance

The world of software development has been fundamentally transformed through DevOps, which is a cultural and technical methodology combining software development (Dev) and IT operations (Ops) to shorten the development lifecycle and deliver high-quality software continuously. Organizations have adopted this approach to take much of the risk out of the development, testing, and deployment lifecycle.

Data engineers have adopted a similar approach to managing the data management lifecycle, dubbed DataOps which Gartner defines as a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an  organization. As the AI revolution has emerged and the importance of good data governance has rose to the fore, we’re seeing another variation of that theme, which some call DataGovOps.

DataGovOps involves automating data governance policies, security, and compliance across the data lifecycle, applying DevOps-like, continuous, and automated approaches to ensure data quality and integrity. For organizations looking to apply DataGovOps techniques to achieve AI-ready data, it is important to consider the following recommendations before getting started:

• Align with business: Be sure to rely on line-of-business workers to define most data quality metrics as they are closer to the day-to-day operations;
• Reuse and adaptability: Consider reusable templates as they can accelerate implementation of processes for monitoring data quality and impacting changes to achieve AI-ready data;
• Iterative adoption: Gradually introduce DataGovOps principles over time. Big bang approaches rarely work, so instead plan for iterative adoption and improvement;
• Team collaboration: Communicate to and among team members as this is key to completing the feedback loop and unifying technology, processes, and people;
• Flexibility and culture: Each organization is different so adapting data governance processes to fit the organization’s unique culture is critical and will increase odds of success.

The potential to automate business processes with AI has never been greater. Achieving that automation depends on having data that is fresh, accurate, and trustworthy–in other words, AI-ready data. Outside of a handful of advanced cloud giants, few organizations today have made the technological, procedural, and cultural changes that ultimately are required to achieve AI-ready data. At the same time, the technologies for automating many data management tasks has improved substantially, providing the potential for organization to make step-wise gains in data quality.

That leaves data governance as the big wild card. The field of DataGovOps has emerged to help organizations chart the high-level path forward. By following the precepts of being business-centric, adaptable, and collaborative, data professionals may ultimately realize the dream of achieving AI-ready data.