Stambia Component for CDC

Change Data capture is a commonly used process to detect and track changes, made to your data, to perform a specific action.

Mostly this is used for replicating that data to an external application or a third-party database.

Stambia Component for CDC simplifies the management of change detection by providing a quick and easy way to set up your CDC processes and to manage the new data..

 

Find out how to capture your changes on data easily with our CDC ( Change Data Capture ) component for Stambia ELT. 

One of the most common use cases where Change Data Capture is required is in a BI Project.

By setting up change tracking, you reduce the overhead of extracting all your data and determining the delta in your Target BI Databases.

The changes as they happen are replicated to a staging area, which is the mirror of the Source tables.

With these staging tables, the data warehouse is populated based on business logic..

On the other hand, in some cases, any data change in your databases, needs to get reflected in an application or another database.

To cater to this need, a Data Hub architecture needs to be set-up, to act as a central hub of information flow and exchanges between various systems.

For e.g., a manufacturing company, would want to set up CDC for their legacy database, to update Salesforce in real-time.

In this scenario, a solution that can connect and exchange data to and from different technologies and applications is beneficial.

A lot of times, the changed record can only be loaded to the destination, as it is.

Many replication tools are providing very basic data transformations capabilities.

The lack of ability to perform complex transformation on your data, results in having two layers i.e., replication and integration.

With this comes the need to manage two different set of tools catering to these two layers..

Lastly, there is always a need, especially in a Data Hub Architecture, to have a bi-directional replication.

Let us take the same example of a manufacturing company.
Once the data changes from legacy system is updated on Salesforce, the process now should take the Salesforce Id, and update the legacy database back as an acknowledgment.

Ability of the tool to be able to perform such bi-directional replication comes then becomes essential..

Stambia Component for CDC simplifies the setting up of a CDC process and lets you utilize all the integration features that come with it, as a data integration tool.

The same solution packed with the CDC templates can fulfill the requirement, therefore, there is no downtime..

Stambia being an E-LT tool, relies on the source database CDC capabilities.

A lot of database technologies now offer in-built APIs or processes that can be used to avoid any overhead on the source system.

As an example, PostgreSQL provides Write Ahead Logs (WAL) that can be used for data replication and subscribers can be set-up for it.

With the same agility as the other Stambia components, the Change Data Capture component is directly usable and offers a high level of productivity due to the model driven approach of Stambia and the use of native "transactional or redo log" files of databases or the use of simple triggers.

In Stambia Designer, with a click you can change your mappings from a CDC to History load.

Further, the templates help you in quickly choosing the load/integration approach, so that there is no need to manually design load and integration strategies.

Another easy feature for a history load is Stambia Replicator templates.

These come handy and save a lot of time, when your source and target structures are the same.

When your source and target structures are different and you need to further perform transformations and complex logic on the data being replicated, Stambia mappings is the way.

In your mappings you can perform, aggregations, various derivations and so on.

The data can be moved to another on premise, a cloud database, or an application.

Furthermore, the Change Data Capture templates can manage multiple subscribers.

This is particularly useful when an organization needs to use modifications in a source for multiple targets.

For example, a modification in the CRM application can be sent to the data warehousing database, and at the same time into an ERP or specific application..

Stambia is flexible with any type of architecture, as an integration solution. With the CDC component, you can work on using it in a:

• BI & Analytics Projects
• Data Hub Architecture
• Big Data Analytics
• Cloud projects
• Application Integration