If something’s missing or unclear, let us know.
Send feedback
Ctrlk
npmGitHubAdmin-Center
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Data import

How an import runs through the API — the multi-part run, the ZIP in the DMS, steps, states, and the two waits you cannot skip.

An import is not a call, it is a choreography: four phases, two asynchronous waits, and one ZIP file in the DMS that ties them together. Skip a wait and you get an empty run or a silently skipped file — no error, no data.

Everything on this page is plain REST. An import needs no App, no SDK and no framework — an access token and an HTTP client are enough, and the flow is the same whether you drive it from a scheduled job, a CI pipeline, a middleware or a one-off script.

Alongside each step this page links to the same step in the demo app's import service — a worked Node.js implementation you can read as a reference. It illustrates the calls; it is not a dependency. For the per-record script see Batch Processing; for field-level detail, the API Reference.

The object model

Object
Where it lives
What it is

Multi-part import run

/cmn/data-import/runs/multi-part

The container you create and start. Carries a label, a derived state, the ordered rule sets

Ordered rule set

inside the run

The binding: this fileName, processed by this ruleSet, at this step

Rule set

inside the ordered rule set

The how: mode, script, character set, record window, mapping rules

Import run

/cmn/data-import/runs/single

One execution per ordered rule set. You never create these — the system does, in phase 3

Entry

…/runs/single/{id}/entries

One record of one file, with its own state and violations

You describe rule sets, the system creates import runs. Everything you configure lives on the rule set; everything you observe lives on the import run and its entries. Even a single-file import goes through the multi-part run — there is no simpler entry point.

The flow at a glance

#
Call
What it does

1

POST /cmn/data-import/runs/multi-part

Create the run and its rule sets

2

POST /dms/resources, then the file — directly or in chunks

Upload one ZIP, attributed to the run

⏳

GET /dms/resources/{resourceId}

Wait until state is FILE_ATTACHED — chunked upload only

3

POST /cmn/data-import/runs/multi-part/{id}/create-import-runs

Unpack the ZIP, create one import run per rule set, extract records

⏳

GET /cmn/data-import/runs/multi-part/{id}

Wait until every part has left CREATED / EXTRACTING_DATA

4

PUT /cmn/data-import/runs/multi-part/{id}/run

Process the records, step by step

Phases 2 and 3 are where integrations go wrong, and both failures are quiet: an early phase 3 finds no file, an early phase 4 finds nothing startable.

Phase 1 — define the run

The response carries the id you need for everything that follows, and state: "CREATED". The demo builds this payload in definition.js.

What goes on a rule set

Field
Applies to
Notes

mode

all

SCRIPT, IMPORT or EXPORT. Required

script / scriptModuleRef

SCRIPT

Inline source, or a script module reference — exactly one of the two. This choice decides whether a licence is needed; see below

parameters

SCRIPT

Reaches the script as ctx.parameters

rules, targetFieldRules, default*FormatDirective

IMPORT

Field mapping, and the formats used to parse dates, numbers and booleans

charSet

CSV, JSON

UTF_8, ASCII, LATIN_1, AUTO_DETECT

columnSeparator, quotingChar, headerIndex

CSV

headerIndex is the 1-based header row

root, ignoreEmptyData, createMissingHeadersFromRuleSet

JSON, XML

root is where the records start; ignoreEmptyData tolerates an empty file

dataFrom, dataTo

all

1-based record window; records outside it end up OUT_OF_BOUNDS

parallelExecutionAllowed

all

Defaults to true. Set false when records must be processed in order

suppressWebhooks, suppressWorkflows

all

Keeps a bulk import from firing the automation it would normally trigger

Scripts: inline or as a module

A SCRIPT rule set can carry its script two ways, and the choice is not cosmetic: it decides whose script runs — and from that follows whether a scripting licence is required.

Inline script

scriptModuleRef → presetting module

In the rule set

"script": "<source>"

"scriptModuleRef": { "id": "…" }

Whose script runs

the tenant's

the app's — a tenant may adapt it, and your version stays as the presetting

Scripting licence

required

not required to run as shipped

Setup

none — send the source

three calls at install time; later versions are one update

Inline is what a hand-built import in the ERP uses, and it is what the demo app does. That is the right shape for an import a tenant builds for themselves — they hold the licence, and the script is theirs to write. It is the wrong foundation for an app you ship, because it makes your app unusable for every tenant without the licence. In an app, register the script as a presetting module and reference it.

Both directions have value, and they are not in competition:

  • Without the licence, a tenant runs your app's scripts exactly as shipped. That is what the presetting exemption is for — the app works out of the box.

  • With the licence — part of the Enterprise plan — the same tenant can adapt your script to their own processes, without leaving your import definition behind. Your version is not replaced: it stays stored as the presetting, and they can return to it at any time.

Why a presetting module is exempt

The batch processor decides per script whether to check the licence: if the script comes from a module and that module counts as presetting data, the check is skipped; otherwise it runs. A module counts as presetting data while its stored presetting script and its current script are identical — which is exactly what the /presettings endpoints establish.

Use the /presettings endpoints, not the plain module endpoints. POST /cmn/scripting/modules stores no presetting script, so the module never counts as presetting data and the licence check applies exactly as if the script were inline. Same path, same payload, silently different licensing — and nothing in the response tells you which one you got.

Three steps

  1. Get or create the module group — once per app, named after the app.

  2. Store the script as a presetting. This is the step that buys the exemption:

    domain has to be IMPORT_BATCH_PROCESSING for an import script, permissionAggregation is required, and name accepts only [a-zA-Z0-9_-] — no dots, no spaces. Ship a later version of the script with PUT /cmn/scripting/modules/{moduleId}/presettings.

  3. Reference it from the rule set, with no script field at all:

Steps 1 and 2 belong in an app's install or migration routine, not in the import path — the modules should already exist, with their ids stored, by the time the first import is built. Then phase 1 only ever references them.

When the tenant customises the script

A module counts as presetting data only while its current script still equals the stored presetting script. So the moment a tenant with the licence edits it in the ERP, the two diverge and the licence check applies from then on. That is not a defect to work around — it is the handover, and it stays reversible in both directions:

  • Your updates never overwrite their version. PUT /cmn/scripting/modules/{moduleId}/presettings updates the presetting underneath and leaves the tenant's script running, so shipping a new version cannot silently discard a customisation.

  • PUT /cmn/scripting/modules/{moduleId}/presettings/reset adopts your latest presetting again and restores the licence-free state.

You do not have to infer which state a module is in — the module tells you:

Question
Where to look

Does this module have a presetting at all?

withPresettingScript

Is it still running your version — that is, licence-free?

sameAsPresettingScript

What is the tenant running right now?

script from GET /cmn/scripting/modules/{id}

What did you ship?

script from GET /cmn/scripting/modules/{id}/presettings — that endpoint returns the presetting in the script field

Both flags are read-only and computed per response; sameAsPresettingScript is exactly the condition the licence check evaluates.

So: ship presetting updates freely, but do not assume the script a tenant runs is the one you shipped. If your import depends on the script's exact behaviour — a field your definition relies on, a contract between two steps — validate the outcome rather than the script.

Phase 2 — attach the data

Every multi-part import reads its data from one ZIP file in the DMS. The attribution is the only thing that binds the file to the run:

shelfDocument.type references a DMS document type — look the id up via /dms/resources/types rather than hard-coding it; DATA_IMPORT is the natural key for import sources.

The resource comes back as DRAFT: it has no file yet. There are two ways to fill it. Direct is the shortcut for small files; chunked is the path for large ones — and also for any file whose final size you cannot state up front, since it is the only path that never needs to know.

Direct

Chunked

Use when

the file is small

the file is large, or its size is not known up front

Calls

1

2, plus 3 per part

Memory

the whole file at once

one part at a time

FILE_ATTACHED

set before the response returns

set asynchronously → wait #1

How small is small? The direct endpoint is meant for single-digit megabytes, and there is no threshold you can build against — the largest request an installation accepts depends on its configuration. So treat direct as a convenience for modest, predictable payloads, and chunked as the default for everything else: a full master-data dump, a large export, a feed that grows with the business. Chunked has no size ceiling.

Direct — one request

multipart/form-data with the file in the file part, plus an optional sha256hash the server verifies:

It is fully synchronous — it stores the file, checks the hash and sets state to FILE_ATTACHED before answering, so you can call phase 3 on the next line. No polling, no wait #1, one less failure mode. The trade-off: the file travels in one request, so it must exist as a body — you cannot build it as you go — and a large one may be refused.

Chunked — for large files, and for anything you cannot size up front

  • Send only the bytes to uploadUri, and get two headers right: drop the Content-Type your HTTP client adds by default — it is not covered by the URL's authorization — and set Content-Length yourself, because a streamed body without a declared length falls back to Transfer-Encoding: chunked, which the upload does not accept.

  • Part numbers are 1-based and sequential; every part except the last must be at least 5 MB, and 20 MB is a comfortable default. Keep each response's ETag and confirm it — finish fails without a complete set.

  • finish takes an optional sha256Check; POST …/file-transfer/{token}/terminate abandons a transfer.

Because parts go up as they are produced, a ZIP of any size streams through a fixed-size buffer instead of being built in memory — see uploadDemoData() in import.js.

Wait #1: finish is asynchronous — chunked only

The direct upload has already set FILE_ATTACHED when it answers. finish has not: it returns 200 and assembles the parts in the background, leaving the resource DRAFT — and a DRAFT resource is invisible to the import. Call phase 3 too early and it fails with "no file to import", not with "still uploading".

Poll GET /dms/resources/{resourceId} until state is FILE_ATTACHED — isFileAttached() in import.js.

Phase 3 — create the import runs

One call does four things, in order:

  1. One import run per ordered rule set, named by the rule set's label — or its fileName if none was given.

  2. The ZIP is located and unpacked. Found by attribution: the run's single IMPORT_SOURCE resource. Every archive entry becomes its own DMS resource, linked to the import run whose fileName matches.

  3. Record extraction starts per file. Each import run goes CREATED → EXTRACTING_DATA → DATA_EXTRACTED, one entry per record.

  4. Rule sets without a file become IGNORED — silently. A state, not an error.

Three things to know:

  • Not repeatable. If the run already has import runs it returns unchanged; it is not a way to attach a different file. Create a new run instead.

  • Refused on a template (template: true).

  • Exactly one IMPORT_SOURCE attribution: none is "no file to import", more than one is "more than one file".

?startImport=true starts each part as soon as its own data is extracted, collapsing phase 4 and the second wait into this call — the leanest path when there is nothing to inspect in between.

Wait #2: extraction is asynchronous

Only DATA_EXTRACTED and ERRONEOUS runs are startable — CREATED is not, so a phase 4 fired too early is rejected with "step cannot be started". Poll GET /cmn/data-import/runs/multi-part/{id}:

Test for "no longer extracting", not for state === 'DATA_EXTRACTED'. A part that went IGNORED never reaches DATA_EXTRACTED, and a strict check waits for it until it times out. See isExtractionComplete() and pollUntil() in import.js.

Phase 4 — run, or validate first

Returns nothing. It starts the earliest step that is not entirely COMPLETED / IGNORED; everything after that proceeds in the background, step by step, without further calls from you.

Or validate instead. Rather than committing, you can dry-run a part — same records, same rules, nothing written:

Validation is per import run, not per multi-part run, so loop over the parts you care about; from / to narrow it to a record range. Results land on the entries as validationState plus violations, which is the cheap way to check a new script or an unfamiliar file before letting it write anything.

The ZIP contract

  • One archive per run, one file per ordered rule set, at the archive root.

  • Entry names are matched verbatim against fileName — case, extension and any directory prefix included. Neither data/accounts.json nor accounts.JSON satisfies fileName: "accounts.json".

  • Several rule sets may point at the same file — each gets its own import run over the same records. Useful when one file feeds two steps.

  • A fileName with no counterpart is skipped, not reported. If that would be a bug in your integration, check the states after phase 3 yourself.

  • Accepted formats: GET /cmn/data-import/runs/single/supported-file-formats.

Steps: ordering and parallelism

step is a grouping number, not an index. Rule sets sharing a step form one stage and run concurrently; stages run in ascending order, and the numbers need not be contiguous — 1, 2, 5 behaves like 1, 2, 3. Within one rule set, records may additionally spread across parallel tasks unless you set parallelExecutionAllowed: false.

A stage hands over when every run in it has finished — and that is where the trap is.

Automatic progression treats ERRONEOUS as finished. A stage advances when all its runs are in COMPLETED, IGNORED or ERRONEOUS, so step 2 runs even when step 1 failed outright.

If step 2 depends on records step 1 was supposed to create, that dependency is yours to enforce — guard it in the script, or check the states between steps instead of firing one PUT …/run and walking away.

The manual start is stricter, and the asymmetry is useful: it targets the earliest step not entirely COMPLETED / IGNORED, so calling PUT …/run again on a partly failed run retries the failed stage rather than skipping past it.

States

An import run — one part — moves through:

State
Meaning
Startable
Deletable

CREATED

Exists, no records read yet

–

✓

EXTRACTING_DATA

Records being read from the file

–

–

DATA_EXTRACTED

Records are in, ready to process

✓

✓

RUNNING

Records being processed

–

–

ERRONEOUS

At least one record failed

✓ (retry)

✓

COMPLETED

All records processed

–

✓

IGNORED

No file matched this rule set

–

✓

The multi-part run has no state of its own. It is derived — the least advanced state among its parts, ranked:

So one part stuck in CREATED holds the whole run at CREATED, however far the others got. And because ERRONEOUS ranks before COMPLETED, a single failed part surfaces on the parent once everything has finished — but not while others are still RUNNING. A status check that reads only the parent therefore reports failures late. Read the parts:

Diagnosing a run

Intent
Call

See the whole run, rule sets inline

GET /cmn/data-import/runs/multi-part/{id}/extensive

Find the records that failed, and why

GET /cmn/data-import/runs/single/{runId}/entries?states=ERRONEOUS

Correct a record's data

PUT /cmn/data-import/runs/single/{id}/entries/{entryId}

Park or settle a record by hand

PUT …/entries/{entryId}/state?state=MANUALLY_IGNORED | MANUALLY_COMPLETED | NOT_YET_RUN

Re-run one part without advancing the chain

PUT /cmn/data-import/runs/single/{id}/run?onlySingleImport=true

Re-read the file

PUT /cmn/data-import/runs/single/{id}/load-data

What is in ctx for this run's script?

GET /cmn/data-import/runs/single/{id}/script-context-description

Entries carry content (the raw record), state, validationState and violations — the actual error details. Their states: NOT_YET_RUN, SUCCESSFUL, ERRONEOUS, IGNORED (the script's guard returned false), OUT_OF_BOUNDS (outside the dataFrom / dataTo window), and the two MANUALLY_* you set yourself.

Two details that bite. onlySingleImport=true is what makes a targeted retry safe — it suppresses the automatic progression, so re-running a failed part does not fire the following step again. And GET …/entries is one of the few endpoints that really returns a list; it takes its paging parameters underscore-prefixed in the query string, ?_start=0&_limit=100, against the usual rule — see Pagination and filtering.

Lists of runs, entries and violations come from computed queries, not REST collections: multi-part-import-runs, multi-part-import-templates, multi-part-ordered-rule-sets, import-runs, import-entries-import-state, import-entries-validated, import-violations, multi-part-import-run-log.

Repeat imports — optional

Everything here is optional. The four phases above are the complete flow; these only save you from rebuilding the same definition every time. Skip the section if your import is a one-off, or if your client builds the definition in code — which is what the demo does.

  • Templates. template: true creates a run that has no state and is never executed. A template is also the only multi-part run you may update — PUT /cmn/data-import/runs/multi-part/{id} on a normal run is refused.

  • Copies. GET /cmn/data-import/runs/multi-part/{id}/copy?asTemplate=false&label=… derives a fresh run from an existing one. Rules are copied; run data, import runs and entries are not.

  • One-call start. POST /cmn/data-import/runs/multi-part/template/{templateName}/start?resourceId={dmsResourceId} copies the template, links an already-uploaded resource, creates the import runs and starts them. templateName is the template's label, unique among templates.

  • Schedules. A rule set can carry a cron schedule configured in the ERP — see Batch Processing.

The one-call template start attaches the resource with a different attribution purpose than the extraction step looks for. Verify it end to end in your own installation before building on it; the explicit four-phase flow is the path the demo app takes.

Cleaning up

DELETE /cmn/data-import/runs/multi-part/{id} removes a run with its entries and log, but is refused while any part is EXTRACTING_DATA or RUNNING. Worth building in: when phase 2 or 3 fails, delete the run from phase 1 before propagating the error — otherwise every failed attempt leaves a half-initialised run behind, each still owning a DMS resource.

The whole thing, end to end

The call sequence in full, as runDemoImport() in the demo's import.js — two JSON files, two SCRIPT rule sets, two steps. The language is incidental; the order is not:

The batch-script side — a VQL lookup, then create or update per record — is in accounts.app-script.js and activities.app-script.js. The second shows the cross-step dependency in practice: it resolves the account imported by step 1 and throws when it is missing.

Checklist

  • In an app: script stored via POST /cmn/scripting/modules/presettings and referenced by scriptModuleRef, never inlined

  • Rule set fileName matches the ZIP entry name exactly, at the archive root

  • ZIP attributed to the run with purpose: "IMPORT_SOURCE" and refType: "MULTI_PART_IMPORT_RUN"

  • Upload path picked deliberately: direct only for small, predictable payloads — chunked for large files and for anything you cannot size up front

  • Chunked: parts 1-based, all but the last ≥ 5 MB, each ETag confirmed, no Content-Type, explicit Content-Length

  • Waited for FILE_ATTACHED before create-import-runs — unless the direct upload made that moot

  • Waited for "no longer extracting" — not for DATA_EXTRACTED — before run

  • Cross-step dependencies enforced in your own code, because ERRONEOUS does not stop the chain

  • Status derived from the parts, not from the parent's state alone

  • Failed run deleted when setup breaks halfway

Related

PreviousDocuments
NextFAQ

Last updated

Was this helpful?