How do I create hundreds of pipelines without configuring each one?

The unit of work is the batch, not the table. You select or search the source tables, and the platform creates the target table from the source column types, generates the column mapping and creates a job — then starts it. Doing that by hand costs minutes per table and leaves a permanent maintenance surface behind.

The generated resources are ordinary objects. They appear in transformation management and job management, and can be edited or scheduled like anything else — the batch is a factory, not a separate runtime.

Two operational details matter once the batch gets large. A generated resource cannot be deleted while it is in use or running: the delete fails, and succeeds on retry once the resource is idle. And deletion removes each resource individually, so removing a large batch takes proportionally longer than creating it. Plan retirement as its own step rather than treating it as instant.

Generated names are derived, which means every naming rule is applied hundreds of times without review. The four rules are prefix, suffix, string replacement and truncation, applied to target table names and column names.

Truncation is the rule that breaks rollouts quietly. If two source table names differ only after the truncation point, they collapse to the same target name — and the failure surfaces as a duplicate-object error somewhere in the middle of the batch, or worse, as two pipelines writing to one table. Before a large run, query the source catalog for the longest identifier in scope and set the trim length from that, not from the first sample table.

Forced case is available on Oracle, PostgreSQL, ADB, Greenplum, HUAWEI LibrA, DM, KingbaseES and KingbaseES7 targets, which is the set of engines where unquoted identifiers are folded by the server. A source-to-target pair that stays inside Oracle also exposes a cross-encoding option that converts byte-length fields to character-length fields — the difference matters when the two databases disagree on what a VARCHAR2(n) length means.

A batch job on its own is input to output. Most real pipelines need something in the middle — a constant column, an ingestion timestamp, a masking rule, a filter. Without a template you would apply that logic by hand to every generated transformation, which puts the repetition straight back.

The practical consequence: one reviewed template covers the whole estate. Change the masking rule once, regenerate, and the change is consistent — instead of four hundred edits, each of which is a small chance to be wrong.

Batch generation is not universal across the connector catalog. The table below is the batch-job coverage specifically — an engine can be a perfectly good CDC source and still not be a batch-job target. Coverage this specific is the difference between a plan and a surprise in week three.

Two of those constraints deserve a planning decision rather than a footnote. Generated tables on MaxCompute and EMR are created without partitions, and on EMR without a lifecycle — so a batch that produces a hundred tables produces a hundred unpartitioned, permanent tables. If partitioning or expiry is part of your cost model, the batch creates the pipeline and you apply the table policy afterwards; decide that before the run, not when storage grows.

Starting a batch is a separate decision from building it, with three scopes: start all jobs, start only the jobs that last succeeded, and start only the jobs that last failed. The first run of a newly created batch ignores the scope and runs everything, because nothing has a previous outcome yet.

That behaviour is what makes retry after a partial failure cheap: fix the failing subset, re-run with the failed-only scope, and the jobs that already loaded are left alone rather than reloaded.