Queue-Based Processing

When your app receives a webhook notification, it should not process the data synchronization inline. Instead, the proven approach is to add a work item to a queue and process it asynchronously. This page explains why queues matter and how to think about them.

Why Use a Queue?

Webhook Timeouts

VARIO ERP expects a fast response (HTTP 200) from your webhook endpoint. If your handler takes too long — because it's calling the platform API, transforming data, or retrying failed requests — the webhook delivery will time out and may be retried, creating duplicate work.

Burst Protection

A bulk update in the ERP (e.g., a price adjustment across 500 articles) fires 500 individual webhooks in rapid succession. Without a queue, your app would attempt 500 simultaneous API calls to the external platform, likely hitting rate limits.

Retry and Error Recovery

When a sync operation fails (network error, platform downtime, rate limit), a queue gives you a natural place to track the failure and retry later — without losing the work item.

Deduplication

If the same article is updated multiple times within a short window, a queue lets you collapse those changes into a single sync operation.

Queue Concepts

A queue for shop integration typically tracks these properties for each work item:

Property	Purpose
Topic	What type of sync operation to perform (e.g., article sync, stock sync, price sync)
Entity ID	Which entity to process (e.g., the article ID)
State	Current processing state
Sales Channel	Which sales channel this work item belongs to
Attempt Count	How many times processing has been attempted
Last Error	Error details from the most recent failed attempt
Timestamps	When the item was created and last processed

Work Item States

A work item moves through these states:

  ┌─────────┐     ┌────────────┐     ┌───────────┐
  │ QUEUED  │────►│ PROCESSING │────►│ COMPLETED │
  └─────────┘     └────────────┘     └───────────┘
                        │
                        │ (on failure)
                        ▼
                  ┌──────────┐
                  │  FAILED  │
                  └──────────┘
                        │
                        │ (manual or auto retry)
                        ▼
                  ┌─────────┐
                  │ QUEUED  │  (back to queue)
                  └─────────┘

• QUEUED — Waiting to be picked up by the queue processor

• PROCESSING — Currently being processed

• COMPLETED — Successfully synced to the platform

• FAILED — Processing failed; the error is recorded for debugging or retry

Processing Pattern

A queue processor typically follows this loop:

1. Fetch a batch of items in QUEUED state (limit the batch size to control load)

2. Mark each item as PROCESSING

3. For each item, execute the appropriate sync operation

4. On success, mark as COMPLETED

5. On failure, record the error and mark as FAILED

6. Repeat until the queue is empty

Batch Size

Control how many items are processed in each cycle. A smaller batch size (e.g., 10) is safer when the external platform has strict rate limits. A larger batch size (e.g., 50–100) improves throughput when rate limits are generous.

Triggering the Processor

You can trigger queue processing in several ways:

• On webhook arrival — After adding an item to the queue, immediately start processing

• On a timer — Run the processor at regular intervals (e.g., every 30 seconds)

• On demand — Provide a UI button or API endpoint for manual processing

Deduplication

When a webhook fires for an entity that is already in the queue (state QUEUED or PROCESSING), you have two options:

• Skip — Don't add a duplicate entry. The existing item will pick up the latest data when it processes.

• Replace — Update the existing item's timestamp so it's processed with the latest data.

Skipping is usually sufficient, because the sync operation fetches the current state of the entity from the ERP at processing time — not at queuing time.

Error Handling in the Queue

When a sync operation fails:

1. Record the error — Store the error message and any relevant details (HTTP status code, platform error response) on the work item

2. Increment the attempt count — Track how many times this item has failed

3. Set state to FAILED — Make it visible in the UI for debugging

For retry strategies, see Error Handling.

Topics

Each queue item has a topic that determines which sync handler processes it. Common topics for shop integrations:

Topic	Handler
article.sync	Push article master data to platform
stock.sync	Update inventory levels on platform
price.sync	Update product prices on platform
order-status.sync	Send fulfillment updates to platform
media.sync	Upload product images to platform

Relationship to Transfer Toggles

Before processing a queue item, check whether the corresponding transfer toggle is enabled. If the user has disabled stock sync for a particular sales channel, items with the stock.sync topic should be skipped or ignored.