Data import
How an import runs through the API — the multi-part run, the ZIP in the DMS, steps, states, and the two waits you cannot skip.
An import is not a call, it is a choreography: four phases, two asynchronous waits, and one ZIP file in the DMS that ties them together. Skip a wait and you get an empty run or a silently skipped file — no error, no data.
Everything on this page is plain REST. An import needs no App, no SDK and no framework — an access token and an HTTP client are enough, and the flow is the same whether you drive it from a scheduled job, a CI pipeline, a middleware or a one-off script.
The object model
Multi-part import run
/cmn/data-import/runs/multi-part
The container you create and start. Carries a label, a derived state, the ordered rule sets
Ordered rule set
inside the run
The binding: this fileName, processed by this ruleSet, at this step
Rule set
inside the ordered rule set
The how: mode, script, character set, record window, mapping rules
Import run
/cmn/data-import/runs/single
One execution per ordered rule set. You never create these — the system does, in phase 3
Entry
…/runs/single/{id}/entries
One record of one file, with its own state and violations
You describe rule sets, the system creates import runs. Everything you configure lives on the rule set; everything you observe lives on the import run and its entries. Even a single-file import goes through the multi-part run — there is no simpler entry point.
The flow at a glance
1
POST /cmn/data-import/runs/multi-part
Create the run and its rule sets
2
POST /dms/resources, then the file — directly or in chunks
Upload one ZIP, attributed to the run
⏳
GET /dms/resources/{resourceId}
Wait until state is FILE_ATTACHED — chunked upload only
3
POST /cmn/data-import/runs/multi-part/{id}/create-import-runs
Unpack the ZIP, create one import run per rule set, extract records
⏳
GET /cmn/data-import/runs/multi-part/{id}
Wait until every part has left CREATED / EXTRACTING_DATA
4
PUT /cmn/data-import/runs/multi-part/{id}/run
Process the records, step by step
Phases 2 and 3 are where integrations go wrong, and both failures are quiet: an early phase 3 finds no file, an early phase 4 finds nothing startable.
Phase 1 — define the run
The response carries the id you need for everything that follows, and state: "CREATED". The demo builds this payload in definition.js.
What goes on a rule set
mode
all
SCRIPT, IMPORT or EXPORT. Required
script / scriptModuleRef
SCRIPT
Inline source, or a script module reference — exactly one of the two. This choice decides whether a licence is needed; see below
parameters
SCRIPT
Reaches the script as ctx.parameters
rules, targetFieldRules, default*FormatDirective
IMPORT
Field mapping, and the formats used to parse dates, numbers and booleans
charSet
CSV, JSON
UTF_8, ASCII, LATIN_1, AUTO_DETECT
columnSeparator, quotingChar, headerIndex
CSV
headerIndex is the 1-based header row
root, ignoreEmptyData, createMissingHeadersFromRuleSet
JSON, XML
root is where the records start; ignoreEmptyData tolerates an empty file
dataFrom, dataTo
all
1-based record window; records outside it end up OUT_OF_BOUNDS
parallelExecutionAllowed
all
Defaults to true. Set false when records must be processed in order
suppressWebhooks, suppressWorkflows
all
Keeps a bulk import from firing the automation it would normally trigger
Scripts: inline or as a module
A SCRIPT rule set can carry its script two ways, and the choice is not cosmetic: it decides whose script runs — and from that follows whether a scripting licence is required.
Inline script
scriptModuleRef → presetting module
In the rule set
"script": "<source>"
"scriptModuleRef": { "id": "…" }
Whose script runs
the tenant's
the app's — a tenant may adapt it, and your version stays as the presetting
Scripting licence
required
not required to run as shipped
Setup
none — send the source
three calls at install time; later versions are one update
Inline is what a hand-built import in the ERP uses, and it is what the demo app does. That is the right shape for an import a tenant builds for themselves — they hold the licence, and the script is theirs to write. It is the wrong foundation for an app you ship, because it makes your app unusable for every tenant without the licence. In an app, register the script as a presetting module and reference it.
Both directions have value, and they are not in competition:
Without the licence, a tenant runs your app's scripts exactly as shipped. That is what the presetting exemption is for — the app works out of the box.
With the licence — part of the Enterprise plan — the same tenant can adapt your script to their own processes, without leaving your import definition behind. Your version is not replaced: it stays stored as the presetting, and they can return to it at any time.
Why a presetting module is exempt
The batch processor decides per script whether to check the licence: if the script comes from a module and that module counts as presetting data, the check is skipped; otherwise it runs. A module counts as presetting data while its stored presetting script and its current script are identical — which is exactly what the /presettings endpoints establish.
Use the /presettings endpoints, not the plain module endpoints. POST /cmn/scripting/modules stores no presetting script, so the module never counts as presetting data and the licence check applies exactly as if the script were inline. Same path, same payload, silently different licensing — and nothing in the response tells you which one you got.
Three steps
Get or create the module group — once per app, named after the app.
Store the script as a presetting. This is the step that buys the exemption:
domainhas to beIMPORT_BATCH_PROCESSINGfor an import script,permissionAggregationis required, andnameaccepts only[a-zA-Z0-9_-]— no dots, no spaces. Ship a later version of the script withPUT /cmn/scripting/modules/{moduleId}/presettings.Reference it from the rule set, with no
scriptfield at all:
When the tenant customises the script
A module counts as presetting data only while its current script still equals the stored presetting script. So the moment a tenant with the licence edits it in the ERP, the two diverge and the licence check applies from then on. That is not a defect to work around — it is the handover, and it stays reversible in both directions:
Your updates never overwrite their version.
PUT /cmn/scripting/modules/{moduleId}/presettingsupdates the presetting underneath and leaves the tenant's script running, so shipping a new version cannot silently discard a customisation.PUT /cmn/scripting/modules/{moduleId}/presettings/resetadopts your latest presetting again and restores the licence-free state.
You do not have to infer which state a module is in — the module tells you:
Does this module have a presetting at all?
withPresettingScript
Is it still running your version — that is, licence-free?
sameAsPresettingScript
What is the tenant running right now?
script from GET /cmn/scripting/modules/{id}
What did you ship?
script from GET /cmn/scripting/modules/{id}/presettings — that endpoint returns the presetting in the script field
Both flags are read-only and computed per response; sameAsPresettingScript is exactly the condition the licence check evaluates.
So: ship presetting updates freely, but do not assume the script a tenant runs is the one you shipped. If your import depends on the script's exact behaviour — a field your definition relies on, a contract between two steps — validate the outcome rather than the script.
Phase 2 — attach the data
Every multi-part import reads its data from one ZIP file in the DMS. The attribution is the only thing that binds the file to the run:
shelfDocument.type references a DMS document type — look the id up via /dms/resources/types rather than hard-coding it; DATA_IMPORT is the natural key for import sources.
The resource comes back as DRAFT: it has no file yet. There are two ways to fill it. Direct is the shortcut for small files; chunked is the path for large ones — and also for any file whose final size you cannot state up front, since it is the only path that never needs to know.
Direct
Chunked
Use when
the file is small
the file is large, or its size is not known up front
Calls
1
2, plus 3 per part
Memory
the whole file at once
one part at a time
FILE_ATTACHED
set before the response returns
set asynchronously → wait #1
How small is small? The direct endpoint is meant for single-digit megabytes, and there is no threshold you can build against — the largest request an installation accepts depends on its configuration. So treat direct as a convenience for modest, predictable payloads, and chunked as the default for everything else: a full master-data dump, a large export, a feed that grows with the business. Chunked has no size ceiling.
Direct — one request
multipart/form-data with the file in the file part, plus an optional sha256hash the server verifies:
It is fully synchronous — it stores the file, checks the hash and sets state to FILE_ATTACHED before answering, so you can call phase 3 on the next line. No polling, no wait #1, one less failure mode. The trade-off: the file travels in one request, so it must exist as a body — you cannot build it as you go — and a large one may be refused.
Chunked — for large files, and for anything you cannot size up front
Send only the bytes to
uploadUri, and get two headers right: drop theContent-Typeyour HTTP client adds by default — it is not covered by the URL's authorization — and setContent-Lengthyourself, because a streamed body without a declared length falls back toTransfer-Encoding: chunked, which the upload does not accept.Part numbers are 1-based and sequential; every part except the last must be at least 5 MB, and 20 MB is a comfortable default. Keep each response's
ETagand confirm it —finishfails without a complete set.finishtakes an optionalsha256Check;POST …/file-transfer/{token}/terminateabandons a transfer.
Because parts go up as they are produced, a ZIP of any size streams through a fixed-size buffer instead of being built in memory — see uploadDemoData() in import.js.
Wait #1: finish is asynchronous — chunked only
The direct upload has already set FILE_ATTACHED when it answers. finish has not: it returns 200 and assembles the parts in the background, leaving the resource DRAFT — and a DRAFT resource is invisible to the import. Call phase 3 too early and it fails with "no file to import", not with "still uploading".
Poll GET /dms/resources/{resourceId} until state is FILE_ATTACHED — isFileAttached() in import.js.
Phase 3 — create the import runs
One call does four things, in order:
One import run per ordered rule set, named by the rule set's
label— or itsfileNameif none was given.The ZIP is located and unpacked. Found by attribution: the run's single
IMPORT_SOURCEresource. Every archive entry becomes its own DMS resource, linked to the import run whosefileNamematches.Record extraction starts per file. Each import run goes
CREATED→EXTRACTING_DATA→DATA_EXTRACTED, one entry per record.Rule sets without a file become
IGNORED— silently. A state, not an error.
Three things to know:
Not repeatable. If the run already has import runs it returns unchanged; it is not a way to attach a different file. Create a new run instead.
Refused on a template (
template: true).Exactly one
IMPORT_SOURCEattribution: none is "no file to import", more than one is "more than one file".
?startImport=true starts each part as soon as its own data is extracted, collapsing phase 4 and the second wait into this call — the leanest path when there is nothing to inspect in between.
Wait #2: extraction is asynchronous
Only DATA_EXTRACTED and ERRONEOUS runs are startable — CREATED is not, so a phase 4 fired too early is rejected with "step cannot be started". Poll GET /cmn/data-import/runs/multi-part/{id}:
Test for "no longer extracting", not for state === 'DATA_EXTRACTED'. A part that went IGNORED never reaches DATA_EXTRACTED, and a strict check waits for it until it times out. See isExtractionComplete() and pollUntil() in import.js.
Phase 4 — run, or validate first
Returns nothing. It starts the earliest step that is not entirely COMPLETED / IGNORED; everything after that proceeds in the background, step by step, without further calls from you.
Or validate instead. Rather than committing, you can dry-run a part — same records, same rules, nothing written:
Validation is per import run, not per multi-part run, so loop over the parts you care about; from / to narrow it to a record range. Results land on the entries as validationState plus violations, which is the cheap way to check a new script or an unfamiliar file before letting it write anything.
The ZIP contract
One archive per run, one file per ordered rule set, at the archive root.
Entry names are matched verbatim against
fileName— case, extension and any directory prefix included. Neitherdata/accounts.jsonnoraccounts.JSONsatisfiesfileName: "accounts.json".Several rule sets may point at the same file — each gets its own import run over the same records. Useful when one file feeds two steps.
A
fileNamewith no counterpart is skipped, not reported. If that would be a bug in your integration, check the states after phase 3 yourself.Accepted formats:
GET /cmn/data-import/runs/single/supported-file-formats.
Steps: ordering and parallelism
step is a grouping number, not an index. Rule sets sharing a step form one stage and run concurrently; stages run in ascending order, and the numbers need not be contiguous — 1, 2, 5 behaves like 1, 2, 3. Within one rule set, records may additionally spread across parallel tasks unless you set parallelExecutionAllowed: false.
A stage hands over when every run in it has finished — and that is where the trap is.
Automatic progression treats ERRONEOUS as finished. A stage advances when all its runs are in COMPLETED, IGNORED or ERRONEOUS, so step 2 runs even when step 1 failed outright.
If step 2 depends on records step 1 was supposed to create, that dependency is yours to enforce — guard it in the script, or check the states between steps instead of firing one PUT …/run and walking away.
The manual start is stricter, and the asymmetry is useful: it targets the earliest step not entirely COMPLETED / IGNORED, so calling PUT …/run again on a partly failed run retries the failed stage rather than skipping past it.
States
An import run — one part — moves through:
CREATED
Exists, no records read yet
–
✓
EXTRACTING_DATA
Records being read from the file
–
–
DATA_EXTRACTED
Records are in, ready to process
✓
✓
RUNNING
Records being processed
–
–
ERRONEOUS
At least one record failed
✓ (retry)
✓
COMPLETED
All records processed
–
✓
IGNORED
No file matched this rule set
–
✓
The multi-part run has no state of its own. It is derived — the least advanced state among its parts, ranked:
So one part stuck in CREATED holds the whole run at CREATED, however far the others got. And because ERRONEOUS ranks before COMPLETED, a single failed part surfaces on the parent once everything has finished — but not while others are still RUNNING. A status check that reads only the parent therefore reports failures late. Read the parts:
Diagnosing a run
See the whole run, rule sets inline
GET /cmn/data-import/runs/multi-part/{id}/extensive
Find the records that failed, and why
GET /cmn/data-import/runs/single/{runId}/entries?states=ERRONEOUS
Correct a record's data
PUT /cmn/data-import/runs/single/{id}/entries/{entryId}
Park or settle a record by hand
PUT …/entries/{entryId}/state?state=MANUALLY_IGNORED | MANUALLY_COMPLETED | NOT_YET_RUN
Re-run one part without advancing the chain
PUT /cmn/data-import/runs/single/{id}/run?onlySingleImport=true
Re-read the file
PUT /cmn/data-import/runs/single/{id}/load-data
What is in ctx for this run's script?
GET /cmn/data-import/runs/single/{id}/script-context-description
Entries carry content (the raw record), state, validationState and violations — the actual error details. Their states: NOT_YET_RUN, SUCCESSFUL, ERRONEOUS, IGNORED (the script's guard returned false), OUT_OF_BOUNDS (outside the dataFrom / dataTo window), and the two MANUALLY_* you set yourself.
Two details that bite. onlySingleImport=true is what makes a targeted retry safe — it suppresses the automatic progression, so re-running a failed part does not fire the following step again. And GET …/entries is one of the few endpoints that really returns a list; it takes its paging parameters underscore-prefixed in the query string, ?_start=0&_limit=100, against the usual rule — see Pagination and filtering.
Lists of runs, entries and violations come from computed queries, not REST collections: multi-part-import-runs, multi-part-import-templates, multi-part-ordered-rule-sets, import-runs, import-entries-import-state, import-entries-validated, import-violations, multi-part-import-run-log.
Repeat imports — optional
Everything here is optional. The four phases above are the complete flow; these only save you from rebuilding the same definition every time. Skip the section if your import is a one-off, or if your client builds the definition in code — which is what the demo does.
Templates.
template: truecreates a run that has no state and is never executed. A template is also the only multi-part run you may update —PUT /cmn/data-import/runs/multi-part/{id}on a normal run is refused.Copies.
GET /cmn/data-import/runs/multi-part/{id}/copy?asTemplate=false&label=…derives a fresh run from an existing one. Rules are copied; run data, import runs and entries are not.One-call start.
POST /cmn/data-import/runs/multi-part/template/{templateName}/start?resourceId={dmsResourceId}copies the template, links an already-uploaded resource, creates the import runs and starts them.templateNameis the template's label, unique among templates.Schedules. A rule set can carry a cron schedule configured in the ERP — see Batch Processing.
The one-call template start attaches the resource with a different attribution purpose than the extraction step looks for. Verify it end to end in your own installation before building on it; the explicit four-phase flow is the path the demo app takes.
Cleaning up
DELETE /cmn/data-import/runs/multi-part/{id} removes a run with its entries and log, but is refused while any part is EXTRACTING_DATA or RUNNING. Worth building in: when phase 2 or 3 fails, delete the run from phase 1 before propagating the error — otherwise every failed attempt leaves a half-initialised run behind, each still owning a DMS resource.
The whole thing, end to end
The call sequence in full, as runDemoImport() in the demo's import.js — two JSON files, two SCRIPT rule sets, two steps. The language is incidental; the order is not:
The batch-script side — a VQL lookup, then create or update per record — is in accounts.app-script.js and activities.app-script.js. The second shows the cross-step dependency in practice: it resolves the account imported by step 1 and throws when it is missing.
Checklist
In an app: script stored via
POST /cmn/scripting/modules/presettingsand referenced byscriptModuleRef, never inlinedRule set
fileNamematches the ZIP entry name exactly, at the archive rootZIP attributed to the run with
purpose: "IMPORT_SOURCE"andrefType: "MULTI_PART_IMPORT_RUN"Upload path picked deliberately: direct only for small, predictable payloads — chunked for large files and for anything you cannot size up front
Chunked: parts 1-based, all but the last ≥ 5 MB, each ETag confirmed, no
Content-Type, explicitContent-LengthWaited for
FILE_ATTACHEDbeforecreate-import-runs— unless the direct upload made that mootWaited for "no longer extracting" — not for
DATA_EXTRACTED— beforerunCross-step dependencies enforced in your own code, because
ERRONEOUSdoes not stop the chainStatus derived from the parts, not from the parent's state alone
Failed run deleted when setup breaks halfway
Related
Batch Processing — the script that runs per record
Script Modules — reusable scripts referenced via
scriptModuleRefPagination and filtering · VQL — reading entries and looking up records from a script
Documents — importing documents means driving transitions, not writing states
vario-app-demo/ import service — the runnable reference for this page
Last updated
Was this helpful?