How do I replace a data integration tool without a big-bang cutover?
The risky part of replacing an integration tool is not the new tool. It is the weekend you switch everything at once and find out on Monday that three pipelines were quietly dropping rows. Dual-run removes the weekend.
Run both tools against separate targets, prove the outputs agree, then retire the old pipeline one group at a time. Bridge supports the three things this needs: generating a pipeline set in bulk, verifying agreement with row counts and file hashes, and moving the proven configuration between environments.
The dual-run shape
Five phases, and the ordering matters more than the calendar. The mistake is treating cut-over as the milestone; the milestone is a reconciled comparison, and cut-over is what you do afterwards.
| Phase | What runs | Exit criterion before moving on |
|---|---|---|
| 1 · Bulk build | Generate the pipeline set from the source table list, one target table plus one job per source table | Every table in scope has a target, a mapping and a schedule |
| 2 · Dual run | The incumbent tool writes to target A; Bridge writes to target B, on the same schedule | Both targets advance together for at least one full business cycle |
| 3 · Reconcile | Row counts compared per table through receipts; file outputs compared by hash | Every table within tolerance, and every exception named and explained |
| 4 · Cut over | Consumers repoint to target B, one pipeline group at a time | No new writes reach target A |
| 5 · Retire | Old jobs disabled after the retention window | Target A retained read-only as the rollback path |
Phase two is where projects lose patience and skip ahead. A single full business cycle is the minimum, and if your volume has a month-end or quarter-end shape, the cycle has to include one of those — a dual run that only ever saw mid-month traffic has not been tested against the shape that breaks pipelines.
Counting rows is not the same as comparing content
Row counts are the cheapest check and the weakest one. They catch missing data and completely missed tables; they do not catch duplicates, mis-mapped columns or truncated strings, because those keep the count intact. Use two methods with different blind spots.
| Method | What it proves | What it misses |
|---|---|---|
| Row count per table | The expected number of rows arrived | Duplicated rows, swapped columns, truncated values |
| Row count per step, with reconciliation receipts | Where in the flow the counts diverge, not just that they did | Column-level corruption inside a correctly sized result |
| MD5 file hash, both sides | The transferred file is byte-identical to the source file | Database targets, where there is no file to hash |
| MD5 hash, single side | The target matches a hash produced outside the platform | Requires the source-side hash to exist before the run |
Reconciliation needs one piece of preparation that is easy to forget: a receipt database and receipt table must exist and be declared as a data connection before reconciliation can run. Today that receipt store is a MySQL database, so choose it deliberately rather than assuming the target warehouse can hold its own receipts. Counts are captured by statistics components placed upstream of the save step, and each captured step becomes a line in the reconciliation list.
Hashes need plumbing, not magic
File verification is a tool with prerequisites, and knowing them changes the plan. It compares file content after a transfer by computing MD5 on each side and comparing the values.
Emit the file inventory first
A file-copy plus text-line-output transformation records the source path and the target relative path for every file, into a CSV or text file.
Both endpoints need access
SSH2 data connections are required for the source and the target file servers, configured as ordinary data connections.
Linux and Windows only
File verification supports those two operating systems; a Windows endpoint needs an SSH server installed before the check can run.
Results land in a table
Single-sided checks compare the target hash against a hash you already hold and write the result to a named result table, so the outcome is queryable rather than a green tick on a screen.
That last point is the one to insist on in a review. A verification step whose output is a colour on a dashboard is not evidence; a row in a result table is.
Promotion between environments, and its one sharp edge
Proven configuration moves between environments as an archive, and the import is not a blind overwrite.
| Step | What happens | Watch for |
|---|---|---|
| Export | Selected resources, or everything, are written to a .dat archive and downloaded | Export covers data connections, transformations, jobs, batch jobs and template batches |
| Import order | Connections first, then transformations, jobs, batch jobs and template batches | The order exists so references resolve to objects that already exist |
| Same-name handling | A resource with an existing name is skipped, and references bind to the existing object | A skip is silent success: an older connection stays, and the new job quietly uses it |
| Result report | Succeeded, skipped and failed counts are shown after the import | Read the skipped count, not just the failures |
| Kerberos | Authentication details are not included in the export | Re-enter the principal details after importing a connection |
Cross-version imports can require manual completion when configuration is missing between the two releases, and a failed job may need editing rather than re-importing. Treat promotion as a reviewed operation with a result report, not as a copy command.
The sharp edge is the skip rule. If the archive is meant to be authoritative, clear or rename the target objects first — otherwise the environment keeps whatever older version it already had and the import reports success. That is the exact failure that makes an environment drift while every log says the deployment worked.
The four mismatches that actually happen
Dual runs fail in recognisable ways. Naming them in advance turns a week of investigation into a checklist.
| Mismatch | How it looks | Where to look first |
|---|---|---|
| Type drift | Counts match, values do not | Compare the distinct value set of one column, not the row count |
| Session or timezone defaults | Timestamps shift by a fixed offset | Compare a single known row, then the session settings on both connections |
| Overlapping windows | Counts slightly high on the new side | Compare distinct primary-key cardinality against total rows |
| Identifier folding | An object is missing, or there are two of them | Check the target engine's identifier case rules against the naming rules you applied |
The first two rows account for most false confidence. Type drift and session defaults both leave the row count intact, so a team checking only counts concludes the migration worked and finds the difference in a report three weeks later. The last row is the one that looks like a platform bug and is usually a naming decision made in week one.
Cut-over, answered
Can both tools write to the same target during the dual run?
Not safely. Two writers to one table make every mismatch ambiguous, because you cannot tell which tool produced a row. Use separate targets and compare them.
How long should the dual run last?
At least one full business cycle, and it must include the period where your volume peaks. A run that only saw mid-month traffic has not been tested.
Is migration a converter that reads another tool's export?
Migration here means generating the new pipeline set from your source table list, then proving it with reconciliation and hashes. That is a repeatable method, and it does not depend on the incumbent tool exporting anything in a particular format.
What do we need before reconciliation can run?
A receipt database and receipt table, declared as a data connection. The receipt store is a MySQL database today, so decide where it lives before the dual run starts.
How do we avoid drift when promoting between environments?
Clear or rename the target objects before importing when the archive is authoritative. A same-named resource is skipped, so an older object survives a successful-looking import.
Run both tools before you commit
Start the trial alongside your current pipeline and compare the outputs on your own data.