How AI and ML have transformed data catalogs?

For decades, traditional data catalogs and data governance frameworks have typically leaned on data engineering teams to do the heavy lifting of data operations, holding them responsible for updating the catalog as new data assets arose. This approach is time and resource-intensive.  It requires significant manual effort that could be automated, thus freeing time up for data engineers and analysts to focus on projects that more directly drive business decisions.

Data catalog maintenance is not a one-time, one-person, or short-term project. It requires ongoing collaboration and communication across your data community, including data owners, producers, consumers, analysts, stewards, and users. You should involve and engage your data community in the data catalog and data dictionary project, by fully involving them in the process, getting their input, making them active participants, and training them as part of the rollout process.

Through automating the data catalog itself, you can get the benefits of facilitating data access and governance without many of the drawbacks of heavy, ongoing manual investment.  This is part of the promise of artificial intelligence (AI), machine learning (ML), and big data analytics.

Traditional data catalogs challenges that are solved through AI & ML

Modern data catalogs have been greatly improved through their integration with AI and ML.  Let’s look at some of the traditional challenges that data catalogs have faced and how they have been resolved through AI and ML.

Challenge – Data Discovery: Many data consumers spend over two-thirds of their time understanding and finding data. The main reason for this problem is that there are poor mechanisms for handling and tracking data.

Solution: As part of modern data catalogs, an ML-based knowledge graph can provide a holistic view of data.  The knowledge graph can help you quickly search, discover, and understand enterprise data and meaningful data relationships through natural language processing. With the knowledge graph, you can automatically discover technical, business, semantic, and usage-based relationships. The holistic data view shows related tables, views, data domains, reports, and users. This aids in the progressive discovery of other datasets of interest.

Challenge – Know Your Data: Traditional data catalogs manage metadata (data about your data) during data ingestion, but data is constantly changing, making it hard to understand the health of your data as it evolves in the pipeline.

Solution: Active metadata management, based on ML, can automatically identify needed metadata updates–classifying and identifying domains and entities across all structured and unstructured data assets at the field, column, and table level. It can also generate data quality scores that identify the quality of data assets. This process of recording where data comes from and every transformation the dataset undergoes is referred to as “data lineage”.

Challenge – Collaboration: The data transformation process affects many stakeholders, including business users, analysts, and engineers.

Solution: Modern data catalogs provide an intelligent platform that connects all services and offers easy access for all users allowing different stakeholders to collaborate and to share usage, queries, and analysis. The collaboration features are also a way of providing crowd-sourced metadata about the data and its quality.

Challenge – Data Trust: Stakeholders and executives have not been able to access the up-to-date data they need to make decisions. Discovering and identifying data that delivers value, governing its quality, and improving its storage by removing data redundancies are some of the biggest challenges that organizations struggle to overcome.  Another aspect of this is the difficulty in finding the most relevant data from multiple dataset copies ingested from different data providers

Solution: With the help of AI, data teams can find the most relevant data and make informed decisions based on accurate and reliable information.

Modern data catalogs use AI to identify data for users based on how users have used data. Based on previous experiences of the user, the catalog can recommend to the user appropriate and relevant data. This helps users find the data they are looking for and also suggests additional data that they were not previously aware of.

In particular, modern data catalogs help you view data profiling statistics, data quality rules, scorecards, and metric groups alongside technical metadata to help you understand the quality of data assets before using the data for analysis. Profiling statistics include value distributions, patterns, data type, and data domain inference.

Challenge – Data Protection:  Ensuring access to necessary data while at the same time restricting access to data is challenging.  Both Personal Health Information (PHI) and Personally Identifiable Information (PII) are particularly sensitive.  In any case, through protecting data, companies prevent illicit data access, reputation damage, and regulatory consequences.

Solution: Modern data catalogs can frequently locate PHI and PII information though data anomaly recognition. These anomalies may result from deliberate efforts to misappropriate data or may result from human error or system issues. Modern data catalogs can use AI to identify and flag these anomalies so that action can be taken and negative consequences avoided.

Conclusion

AI-driven data catalogs provide simple, search-based discovery to find relevant data along with a holistic view of the data to help users understand the data.  This includes where the data is coming from, how it’s being used, what other data it’s related to, the business context for that data, and the quality of the data.  Through this, organizations can better harness their data to drive valuable insights and actionable decisions, empowering you to shape a winning data-driven culture and become a data-driven enterprise.

 
Coming soon will be my next blog on What is an Augmented Data Catalog?

 
About the Author:

Siddhartha Reddy Maddi
Celsior