Salesforce CDC (Change Data Capture) integration using MuleSoft

As one of the most widely used Customer Relationship Management solutions on the market, Salesforce plays a central role in the most critical business processes for many companies. To fully automate and optimise these business processes, companies need to integrate Salesforce CRM with other applications within their organisation. This article discusses Salesforce CDC (Change Data Capture) integration with other applications like databases, other SaaS and IoT, using MuleSoft. It also looks at uses and benefits, challenges, and new approaches.

In today’s digital journey, organisations heavily rely on data to identify challenges, capitalise on opportunities, and make timely and informed decisions. The availability of real-time and near-real-time data is essential in these fast-moving markets. Companies using delayed data processing methods, such as batch processing, to synchronise databases may lose their competitive advantage.

As a result, organisations are shifting to Change Data Capture (CDC) to ensure real-time streaming of data changes into their databases or other SaaS applications.

What is Salesforce Change Data Capture (CDC)?

CDC enables near-real-time changes of Salesforce records, and synchronises corresponding records in an external data store. Change Data Capture publishes change events, which represent changes to Salesforce records. Such changes include creation of a new record, updates to an existing record, deletion of a record, and recovery of a record that has been deleted.

Why to use CDC

There are several reasons to use Change Data Capture. The most significant ones are discussed below:

1. CDC makes the most current data available, and this promotes faster and more accurate decision making.
2. CDC ensures minimal disruptions to production workloads. It does this by sending continuous source updates to analytical targets. This keeps targets regularly updated without batch loads that may disrupt production operations.
3. CDC sends only incremental changes, thus cutting down data transfer costs over a Wide Area Network (WAN).
   Use cases of CDC through MuleSoft
4. Data replication/propagation. An organisation may need to have several copies of data from a data source for different use cases. CDC ensures more efficient propagation of data. This replication solution records changes in real-time and sends them to target databases or any other SaaS applications.
5. Extract, transform, load (ETL). CDC is beneficial in loading data warehouses. Processing changes reduces the data load time, associated costs such as software licenses, and resources such as memory and CPU. Daily data changes only represent a fraction of the total data volume. CDC accommodates these changes and any future growth that may occur in data volumes.
6. Data lake ingestion. Data lakes have become popular as a data storage and processing platform for many data types. Enterprises rely on data lakes to support analytical use cases such as real-time client offers, social media monitoring, fraud detection, as well as price and market trend analysis. Such use cases depend on real-time analysis of the latest data, and that is where CDC fits in.
7. Building Operational Data Stores (ODS). Operational Data Stores ODS refer to integrated data repositories that collect data from more than one source addressing a particular business area, say, marketing or finance, and providing valuable information to business intelligence (BI) applications and business users. The implementation of CDC ensures ODS are kept up-to-date through a continuous identification and delivery of changes into the database. Otherwise, one would require to query the entire database for changes periodically.
8. Data sharing with customers, vendors, and partners - Partners and vendors require access to an organization’s data platform to provide more insights or make critical investment decisions. That means they need a data integration tool to enjoy the same types of data movement, transformation, and access components found in the organization’s data platform, and this is where CDC also fits in.

How to Subscribe to Salesforce Change Data Capture with MuleSoft

MuleSoft’s Salesforce Cloud Connector Replay Channel can subscribe to Salesforce Change Data Capture, to receive any data change on Salesforce. For example, Account record creation, update or deletion, or for any other Salesforce Standard and Customer objects. Below are the steps which explain how to Subscribe to Salesforce Change Data Capture in Mule Application flow

• Login to Salesforce by following the steps
  • Go to Setup
  • Integrations
  • Change Data Capture, then select a target Salesforce object. In this example, the target object is Account.

• Create a Mule application. In this example, the mule application name is cdc-sample-poc.
• Create a component Salesforce Connector Replay Channel, then specify channel name as Streaming channel. In this example, the target object is Account so the value will be “/data/AccountChangeEvent”.

• Create a logger component to show payload, so that we will see the received message on Console.

• Login to Salesforce then update any Account record. In this example, update an Account name to "Sanjeet-Test-Account". You will see in the Console log something similar to the screenshot below, which is equivalent to the updated Account record which includes the new name "Sanjeet-Test-Account".

• Once you get the CDC payload within the MuleSoft flow, the same payload can be transformed and sent to any target system like a database or other SaaS applications, or it can also be converted into a CSV file which can be placed in a file system or FTP Server.

Final thoughts

Change Data Capture enables an instant and incremental replication of data and this can be achieved using Anypoint Platform to help us move the data to any other system or application. Unlike other processes that record data changes all at once, CDC does this piece-by-piece. This ensures that data is always up to date. Furthermore, both the target and base databases are continuously synced to implement new data updates.

If you would like to find out more about how you can leverage MuleSoft to enhance your Salesforce capability and drive growth, we can help. Give us a call or email us at Salesforce@coforge.com.

Other useful links:

Salesforce Clouds and Solutions

Get started with MuleSoft and API-led connectivity

Eduvanz simplifies lending to students with Salesforce, MuleSoft and Coforge