Importing Only Recently Added or Changed Data from Cloud Sources

With Skyvia you can easily propagate data changes, such as inserting new records to a cloud data source or updating existing ones, to other data sources via import packages.

When importing data from cloud sources, Skyvia allows creating packages that import not all the cloud data from the source object but only the records, created or modified since the package creation or since the previous package execution. To create such a task in an Import package, you simply need to click the Inserted or Updated button on the Source Definition page of the Import Wizard, and all other settings are configured as usually.

For example, you can easily create a package, that adds all the new Salesforce leads as MailChimp subscribers to a MailChimp list, or create QuickBooks customers for new Salesforce accounts, etc. After this you can schedule it to execute every several minutes, and subscribers will be automatically added for all the new Salesforce leads, etc.

This functionality allows you to create trigger-action-like integrations or simply load all the new records and updates from one source to another automatically. However, it is not a full-featured replacement for data synchronization, because an Import package cannot perform actions for recently deleted records.

note Note

Importing recently added or updated records is supported only when the Source is a cloud application. It also requires a source object to have fields that store timestamps for the creation time and last modification time. For example, in Salesforce, these fields are CreatedDate and LastModifiedDate. However, there is a workaround for relational databases, described below.

Please note that not in some cloud sources only some objects have these fields, and thus, for some objects this feature is not available. For example, it cannot be used when importing data from Salesfoce Marketing Cloud data extensions. Some data sources, like Podio or SendPulse, does not have such fields at all.


ExpandedToggleIcon        Workaround for Relational Databases

To import only recent data from relational databases, you can use the following workaround:

Add columns for storing timestamps of creation time and/or last modification time to the database table you want to import data from yourself. The date and time in these columns must be in UTC. Also you need to create triggers that assign current timestamp to these columns whenever a row is inserted or modified. Then you can use source data filters, and add filters on these columns that use the LAST_RUN relative constant. Thus, you can import only records with creation or modification date and time more than the date and time of the previous package run. This relative constant is equal to the timestamp of the previous package run, and, if a package have not been run yet, to the date and time of its creation.



ExpandedToggleIcon         Example: Adding New Salesforce Leads to MailChimp List

Suppose we need to integrate MailChimp subscribers with Salesforce leads. We already have subscribers for existing leads, and want to create subscribers from new ones automatically. So, to create such an import package, let's do the following:


Creating a Package

1.Click Integration in the menu on the left side of the page.
2.Click the New button in the top right corner of the page.
3.Click Import.
4.Click CSV in the Source header and select Salesforce from the list.
5.Click Select in the Target header and select MailChimp from the list. You can use the Search box to quickly find the necessary source.
6.Select corresponding connections in the Source Connection and Target Connection lists. In order to shorten this example, let's assume, we already have the connections created. If not, see the Connecting to Salesforce and Connecting to MailChimp topics.
7.Click + Add Task.


Configuring Source

1.In the Object list select Leads.
2.Click Inserted to import only recently inserted records.
3.Click Next.


Configuring Destination

1.In the Target list, select ListMembers.
2.Click Next.


Configuring Mapping

1.Click ListId.
2.Click Column and then select Target Lookup on the list.
3.In the Lookup Object list, select Lists.
4.In the Result Column list, select Id.
5.In the Lookup Key Column list, select the Name target column.
6.Click Column and then select Constant on the list.
7.In the bottom drop-down box enter the name of the list to add new subscribers to.
8.Click the Last Name column and map it to the LastName source column using column mapping.
9.Click the First Name column and map it to the FirstName source column using column mapping.
10. Click the EmailType column and map it to the constant "Html" (without quotes) using constant mapping.
11. Click the Status column and map it to the constant "Subscribed" (without quotes) using constant mapping.
12. Click Finish.


Scheduling Package

Now all you need is to schedule a package for automatic execution so that it created MailChimp subscribers from new Salesforce leads automatically. We will configure it to run every 5 minutes on workdays from 8:00 to 18:00.

1.In the Schedule pane click Set schedule.
2.In the Run every list select Week.
3.In Days of week select all the workdays.
4.Click Occur once at and then select Recur every on the list.
5.Enter 5 in the corresponding box.
6.Click minutes.
7.Click set time restrictions.
8.Enter 8:00 and 18:00 to the corresponding boxes.
9.Save the package.

That's all, our package is ready.