Augmented Data Catalogs: The Next Step in Data Catalogs

In my previous blog, I explained how AI and ML have transformed data catalogs by automating a set of manual tasks associated with data handling, ranging from consolidation and metadata discovery to curation and enrichment. This can significantly improve the accuracy and consistency of data management and better prepare data for further analysis.

In this blog, I’ll discuss the importance of augmented data catalogs in empowering modern data-driven enterprises.

Gartner stated that 60% of data catalogs that do not use machine learning to find and inventory data across a distributed environment will not be delivered on time.  Gartner also estimated that organizations that offer a curated catalog of internal and external data will realize 2 times the business value from their data and analytics investments.

Traditional data catalogs collect, analyze, and share all forms of technical metadata from an organization’s data management landscape. Augmented data catalogs extend these capabilities with a more comprehensive approach to data catalogs. For example, they automate various parts of the processes like discovery, profiling, etc. through use of artificial intelligence and machine-learning algorithms to streamline various aspects of metadata management. Also, they ingest various types of metadata including technical, operational, business, and social metadata. Most importantly, they simplify and automate finding and classifying metadata across complex distributed and decentralized environments.

Augmented data catalogs provide a holistic view of data to help users understand where the data is coming from, how it’s being used, what other data it’s related to, the business context for that data, and the quality of the data.

The Traditional Data Catalog

Key differences between Augmented Data Catalogs and Traditional Data Catalogs

Full use of Machine Learning:  Although traditional data catalogs may use ML in a limited capacity, augmented data catalogs take advantage of ML to automate a wider range of manual activities including “metadata discovery, ingestion, translation, and creation of semantic relationships between metadata” (according to Gartner). In augmented data catalogs, ML assists not only in data discovery, but also in data annotation, data lineage, crowdsourcing, data querying, and integration with BI tools.

Comprehensive Coverage of Distributed, Heterogeneous Environments: Augmented data catalogs can broadly encompass data from multi-cloud environments, hybrid cloud environments, data lakes, and other sources. At the same time, they can fully catalog varying types of traditional and less traditional data. Regardless of type, source, or location, augmented data catalogs can assist with finding and taking full advantage of your decentralized data.

Metadata Management: Augmented data catalogs integrate well with the full range of metadata management tools and processes. For instance, augmented data catalogs integrate with the embedded data catalog capabilities sometimes found in data integration, data virtualization, and data visualization tools. Since data catalogs are only one aspect of metadata management, integration with other tools becomes highly important. Furthermore, metadata management must handle all types of internal (and external) metadata including technical, operational, social, business, and crowdsourced metadata. This enables data maintenance and business value retrieval based on leveraging the ML algorithms.

Active Metadata: Augmented data catalogs can manage well both active and passive metadata. Passive metadata is generally static, unchanging, and defined as part of a software development process. In contrast, active metadata may change in real-time. Frequency of access, performance optimization, and quality of data are some examples of active metadata.

Anomaly Identification and PII Detection – Augmented data catalogs can frequently use their metadata collections to identify anomalies and sensitive data that need special security provisions and personally identifiable information (PII) that needs to be protected to adhere to relevant privacy laws and regulations.

Semantic Knowledge Graph: Knowledge graphs connect and organize information from various sources to create a comprehensive knowledge base. They capture all of the intricate relationships between data and are typically implemented using a graph data store. Semantic knowledge graphs are very helpful for data ingestion and also enable a more semantic understanding of data, allowing for advanced querying, inference, and reasoning. As with most other aspects of the augmented data catalog, they typically require direct or indirect use of machine learning to be able to construct and maintain them with reasonable cost and time.

The Augmented Data Catalog

In summary, data catalog building is a journey based on the continuous evolution and modernization of the data platform as well as the collection and interconnection of different types of metadata. Traditional data catalogs can be transformed to Augmented Data Catalog by modernizing the data platform to collect and analyze different types of metadata, real-time synchronization of metadata across the organization, and implementation of semantic-knowledge graphs. Altogether, these innovations contribute to building a powerful and intelligent data ecosystem for organizations to make data-driven decisions easily and cost-effectively and to gain a competitive edge.

About the Author:

Siddhartha Reddy Maddi
Celsior

Read other blogs of the series:

The importance of data catalogs to data-driven enterprises

How AI and ML have greatly transformed data catalogs