Replication Package

Replication package is a specific kind of a package, which loads data from cloud applications to a relational database.

ExpandedToggleIcon        How To Create Replication Package

To create a Replication package, perform the following actions:

1.Navigate to the Integration Packages page by clicking Integration in the menu on the left side of the page.
2.Click the New button.
3.Click Replication on the opened menu.
4.Optionally click the btn-edit-name Edit name button near the automatically generated package name (on this step it's "Replication Package") and specify the package name. If you omit this step, the package name will be generated later based on the names of the replicated cloud objects.
5.Click Select in the Source header and then click the name of the source application to replicate data from.
 
Select the Source Connection for the package. If haven't created the source connection yet, click the New button and create a new connection (learn how).
 
select-source-salesforce-connection
6.Click Select in the Target header and select the DBMS to replicate data to.
 
replication-select-target-type
7.Select the Target Connection
 
replication-select-connection
8.Specify the necessary Options:
 
replication-options
 
Incremental Updates - if selected, it means that Skyvia will not copy all the data each time when the package is executed. Instead it will detect data that was changed in your cloud application since the last package execution, and then applies these changes to your database. It deletes records that were deleted in the cloud application since the last package execution, updates records that were updated, and inserts the newly inserted records. For some cloud applications, incremental updates are not supported for some objects.
 
Create Tables - if selected, Skyvia will try creating the database tables in the target replication database. If the Incremental Updates check box is selected, Skyvia will try creating the database tables only for the first package execution.
 
Drop Tables - if selected, Skyvia will try dropping the database tables in the target replication database before creating them. If the Incremental Updates check box is selected, Skyvia will try dropping the database tables only for the first package execution. This check box is enabled only if the Create Tables check box is selected.
 
Create Foreign Keys - if selected, Skyvia will create the foreign keys in the target replication database according to relations between the replicated objects in the source CRM. This check box is enabled only if the Create Tables check box is selected. Note that Skyvia does not create foreign keys for many-to-many relations.
9.Under the Select Objects header, select check boxes for objects to replicate to database. You can filter object names to quickly find the necessary objects by entering a part of object name to the box above the grid.
10. Optionally click the edit link for selected objects and configure the corresponding replication tasks (see how).
11. Click the Save button.

 

ExpandedToggleIcon        Metadata Changes and Editing Replication Package

Skyvia does not support automatic metadata changes detection. If metadata is changed in your cloud data source, a replication package will continue to run knowing nothing about these changes. If a new field is added in a replicated object, the replication will continue working, but the new field won't be replicated. If a replicated field or a replicated object is deleted or renamed in the data source, the replication package will start failing until you perform the necessary changes to it.

In order to detect these metadata changes in a replication package, you need to perform the following steps:

1.Clear the connection metadata cache. By default, Skyvia caches metadata of available objects for cloud sources. So if a field is added, removed or changed in the source object or you add a new object, the cache should be refreshed. To do this, you need to open your connection and click the clear link for the Metadata Cache parameter. You also may edit and configure your connection so that the metadata cache was periodically refreshed automatically.
2.Edit your replication package and perform the following actions:
3.For each replicated object, whose metadata were changed, click the edit link and click the Refresh button in the task editor. This removes the information about fields that were deleted in the data source from the replication package.
4.Select check boxes for fields, added in the data source, in the task editor.
5.By default, when editing a replication package, Skyvia displays only previously selected objects. If you want to add a new object to replication, click the Show All link.
6.Select check boxes for all the objects you want to add to the package.

After this, the tables must be re-created in order to have columns for the new fields and don't have columns for deleted fields. If you haven't added any new objects to the package, select the Create Tables and Drop Tables check boxes in order to re-create the tables. You don't need to clear the Drop Tables check box after this even when the Incremental Updates check box is selected. In the latter case, the Create Tables and Drop Tables options are applied only for the first replication. See the Replication Package Settings in Details section below for more information.

However, if you added any new objects to the replication, you cannot simply select the Drop Tables check box. In this case the replication package will try dropping tables for the new objects too, and since there are no such tables in the target database, it will fail. So you may either manually drop all the tables for the package and don't select the Drop Tables check box, or select this check box and create the missing tables with the names of the new cloud objects in the database manually.

note Note

Whenever you edit a replication package and modify anything except the package schedule, the LastSyncTime parameter of the package is reset, and Skyvia reloads all the data from the source objects during the first replication after package editing. The Create Tables and Drop Tables options are also applied (if selected) even when the Incremental Updates check box is selected too.

 

ExpandedToggleIcon        Replication Package Settings in Details

Incremental Updates Are Not Selected

When the Incremental Updates check box is NOT selected, the Create Tables and Drop Tables options are applied every time a replication package is run.

If Create Tables is selected, Skyvia will attempt to create tables for all the objects selected. So, if there is any table with such a name in a database, the package will fail. If you simply rename tables left from the previous replication, their constraints still have the same name, and you will get an error about constraint with the same name.

So, if Create Tables is selected and Drop Tables is not selected, there must be no tables with the names of replicated objects in the target database for the package to succeed.

If Drop Tables is also selected, Skyvia will try to delete tables for all the replicated objects before creating them. If there is NO such table for any of the objects, the package will fail.

So, if both Create Tables and Drop Tables check boxes are selected, the tables for ALL the replicated objects must be present in the database in order for package to succeed.

If there are tables only for a part of the replicated objects in the database, there is no working option combination for this case, and you will have to delete the tables manually.

 

Incremental Updates Are Selected

When the Incremental Updates check box is selected, the Create Tables and Drop Tables options are applied in the following cases:

1. The first package run after package creation.

2. The first package run after any package modification except for changing the package schedule.

The second case includes changing these replication options. So if you select the "Incremental Updates" check box for an already existing package, Skyvia will still try creating tables again during the first run after this check box selection. So, if you want to select the "Incremental Updates" check box for a package that already created the tables, you will need either to select the "Drop Tables" check box too or delete tables manually. (Selecting the "Drop Tables" check box won't hurt incremental updates, because it is applied only for the first run after package modification. When "Incremental Updates" check box is selected, tables are not recreated each time even if "Create Tables" and "Drop Tables" check boxes are selected.)

 

ExpandedToggleIcon        Replication Package Parameters

In Skyvia, Replication packages have the following parameter.

 

Parameter

 

Description

LastSyncTime

The time since the last replication. When replication is executed, Skyvia sets this parameter to the current time, and the next package execution will load the changes made since the time, specified in this parameter. You can reset this parameter in order to reload all the data.

replication-parameters