Speeding up large transfers

Transfer speed on large tables depends on a few specific choices made at the customization stage. This page covers the controls in the order to try them.

• Bulk insert
• Quantization
• Use trigger-based sync
• Filter to only the rows you need
• Make sure tables have primary keys
• Parallel migration threads

Bulk insert

Inserts groups of rows into the target in fewer statements, reducing per-row overhead.

Use when transfer speed matters more than memory pressure. Bulk insert is supported in DBConvert only; DBSync does not use it.

The target table needs a unique or primary index (not a composite key) for bulk insert to work correctly.

Quantization

Reads the source in portions and writes the target in batches, instead of holding the full result set in memory. Both DBConvert and DBSync support it.

Quantization is the right setting when you hit memory-related errors during a transfer:

• "Timeout Expired"
• "Out of memory for query result"
• "Query Result Set too Large"

If the error persists, lower the quantization slider further. 100 or below is a reasonable starting point on a table that does not fit comfortably in memory.

Use trigger-based sync

For ongoing synchronization on large databases, switch from a full sync to trigger-based sync. Triggers on the source capture only the changes since the last run, so replication time is roughly proportional to the change volume, not to the total table size.

As a reference point, an Insert / Update / Drop sync on a 1 million-row database takes ~90 minutes without triggers, and ~12 minutes with trigger-based sync.

See Trigger-based synchronization for setup steps and the list of supported direction pairs.

Filter to only the rows you need

If a table contains a long history but the migration only needs recent data, apply a data filter at the customization stage to limit the rows transferred. A filter on a date column or status column is often enough to cut transfer time by an order of magnitude.

Make sure tables have primary keys

Synchronization relies on primary keys to match rows between source and target. A table without a primary key cannot be synchronized reliably, and even a one-shot conversion runs slower without one. The customization stage flags tables that are missing a primary key.

Parallel migration threads

For large transfers, raise the number of parallel data-processing threads in Database options. The maximum depends on the license tier:

• Personal - up to 2 threads
• Business - up to 8 threads
• Enterprise - up to 16 threads

If a multi-thread run produces unexpected errors, drop back to 1 thread to confirm the issue is not threading-related - see Fast Database core error.

Other notes

When using the sync overwrite options, enable Overwrite existing database or Overwrite existing tables on the customization stage so target rows do not accumulate over multiple runs.

If only a small set of tables takes most of the time, move those tables into a separate session so they can run on a different schedule from the rest.