November 16, 2021
We are glad to announce a significant improvement in import packages — ability to manage the number of records in a batch when importing data! This allows you to better control API use when loading data to data sources.
Now, when creating or editing import packages, you can set the Batch Size option. This option is available in the import packages, which use New runtime. It regulates the number of records sent to the target as one batch.
By default, Skyvia processes data in batches of up to 2000 rows. The batches of such size are read from the source at once. However, while processing records, using filtering, lookups, etc., the size of batches can decrease, and data can be sent to the target in smaller batches. The Batch Size option allows you to cache the records in a buffer before finally sending them to a target, and send batches of the fixed size.
This can be useful to both increase the batch size for loaded data in order to load more records per an API call to improve API call use efficiency, and to decrease batch size if the target has some additional custom limitation or if smaller batches provide better performance.
Note that each data source has its own API batch size limitations, and sometimes different objects have different limitations. Some of them don't support sending records in batches at all. Skyvia cannot exceed data source API limits, so if the max batch size allowed by the target data source and table is less than the specified Batch Size, the buffered records are split into multiple internal batches.
We hope you will enjoy this new feature. In case you have any questions, contact our technical support. We appreciate any feedback from you!