{"id":4931,"date":"2022-09-08T17:45:58","date_gmt":"2022-09-08T21:45:58","guid":{"rendered":"https:\/\/solutionsreview.com\/data-integration\/?p=4931"},"modified":"2022-09-16T11:14:03","modified_gmt":"2022-09-16T15:14:03","slug":"change-data-capture-examples-an-introductory-guide-to-cdc","status":"publish","type":"post","link":"https:\/\/solutionsreview.com\/data-integration\/change-data-capture-examples-an-introductory-guide-to-cdc\/","title":{"rendered":"Change Data Capture Examples: An Introductory Guide to CDC"},"content":{"rendered":"

\"Change<\/p>\n

This is part of Solutions Review\u2019s Premium Content Series, a collection of contributed columns written by industry experts in maturing software categories. In this submission, Arcion<\/a> Founder and Chief Architect Rajkumar Sen offers a look at Change Data Capture examples, with a complete introduction to CDC technologies.<\/em><\/strong><\/p>\n

\"SRData management and utilization have become the differentiating factor for businesses in today\u2019s competitive world. For companies in crowded business markets, data provides the opportunity to gain a competitive edge through a better understanding of customer needs and wants, which leads to strategic product differentiation in the relevant markets.<\/p>\n

Moreover, the world is moving toward real-time. One report finds that 80 percent of American consumers are more likely to purchase from a company that personalizes<\/a> its sales offering. Unfortunately, it\u2019s impossible to deliver real-time personalization if your CRM takes hours or days to update your customer recommendation engine.<\/p>\n

Now more than ever, business success is largely dependent on the data an organization collects, its quality and freshness of it, and what they do with it. A data-driven company that is focused on business success must rethink its data strategy in 2022 and embark on a data modernization journey. One of the most effective ways to modernize your data is to introduce a technology called change data capture (CDC)<\/a><\/strong><\/span> in your data infrastructure.<\/p>\n

This article will take a deep dive into change data capture, its uses, and how it can simplify and improve both your application and data architectures.<\/p>\n

\t\t\t

\"Download<\/a><\/p>\n<\/div>\n\t\t<\/div><\/div>\n

Change Data Capture Examples<\/strong><\/h2>\n
<\/div>\n

What Is Change Data Capture?<\/strong><\/h3>\n

Change data capture is a methodology or design pattern that identifies and captures changes in your database over time. These changes are recorded, synced near instantly, and sent either to a table in the same database, a different database, a streaming platform, or to cloud storage. It is then possible to query and take action on the data at a later time. As soon as the source data changes, CDC syncs the databases, eliminating the silos of data. CDC allows businesses to make faster and more accurate data-backed decisions, with reduced resource expenditure.<\/p>\n

Types of CDC<\/strong><\/h3>\n

There are several different types of CDC, such as timestamp-based CDC, trigger-based CDC, snapshot-based CDC, and log-based CDC. The two main types, however, are trigger-based CDC and log-based CDC:<\/p>\n

Trigger-Based CDC<\/strong><\/h4>\n

This method has been around for decades. The trigger-based CDC captures all insert, update and delete operations performed on tables or databases. For every insert, update and delete statement, a trigger is fired which captures the data manipulation language (DML) statement. Trigger-based CDC requires database triggers to be created in order to identify the changes that have occurred in the source system. It then captures those changes and writes out the changes into another table, typically called a shadow or staging table. Trigger-based CDC is easy to implement, can capture the entire state of the transaction, and is also customizable.<\/p>\n

However, there are some known disadvantages to using triggers to enable change data capture. A big disadvantage is the setup of individual triggers for each table. This can lead to costly implementation and management overhead in the case of a large source database. Another challenge with triggers is that for large transactions, there is a significant overhead of doing multiple writes to a database for every insert, update or delete. Also, to apply the changes in the target database, the replication tool needs to connect to the source database at regular intervals; that can put additional load on the source database system and impact performance.<\/p>\n

Log-Based CDC<\/strong><\/h4>\n

Log-based CDC, in contrast, works by reading the transaction logs of the source database. By reading the database\u2019s log, you get the complete list of all data changes in their exact order of application. However, every database uses its own custom format to write redo log records to the transaction log, so a customized solution needs to be built to read those formats from the log and convert them into logical transactions and DML statements that could be written to a target system. Therefore, there is a lot of engineering effort that is required to build a solution that enables CDC to read a transaction log.<\/p>\n

There are some obvious advantages to using log-based CDC over a trigger-based solution. First, you can get a lot of additional metadata from the transaction log, e.g., a transaction identifier that is useful to ensure that the replication has the ability to resume upon crash. Second, in log-based CDC, there is no connection being made to the source database, and no extra queries are performed on the source system, thus making it a zero-impact solution. This is extremely important for large production systems which are almost running at full capacity and cannot bear any extra overhead.<\/p>\n

The Need for Change Data Capture<\/strong><\/h3>\n

A business cannot operate on luck or intuition and hope to remain viable in the long run. Businesses need to be strategic in their decisions, and they can only make the right decisions if they have accurate and fresh data. A major benefit of CDC is that it provides fast, fresh, and accurate data so decisions can be made with speed and precision.<\/p>\n

Other benefits of CDC include:<\/p>\n