Use a message queue when consumers need to receive and process work independently. Use a durable workflow when a process must remember its progress across dependent steps, failures, or waits. You can combine both when incoming events start processes that need coordination. Vercel Queues and Vercel Workflows provide implementations of these approaches.
Copy link to headingWhat is the difference between a queue and a workflow?
Queues hold messages until consumers can process them. Each message might ask a consumer to extract text from a document or announce that an upload has finished. The queue tracks delivery, while your application decides what processing the message means and how to record the result.
Durable workflows record a process's progress, including which steps have completed and what must happen next, so the process can resume after an interruption. That state becomes useful when later work depends on an earlier result or an event that may arrive much later.
Neither choice determines a job's compute needs. Message consumers can perform substantial work, and workflows can finish without long waits.
Copy link to headingWhen is a queue enough?
Consider an application where a person uploads a document. The application needs to extract its text and write an audit record of the upload. These operations can proceed independently. Because the audit consumer doesn't need the extracted text, an extraction failure should not prevent it from recording the upload.
With independent consumer groups, a queue system can deliver the upload event to an extraction group and an audit group. Each group acknowledges its own processing and retries its own failures. Multiple extraction workers share the work within their group, so they can process separate documents while the audit group records each upload.
The queue also accommodates a burst of uploads. Limit how much work consumers take on so they don't overwhelm the document parser or downstream services. Messages accumulate when arrivals outpace processing, and consumers drain the backlog as capacity becomes available. You still need enough capacity to clear the backlog before messages expire and a policy for messages that keep failing.
If successful extraction and a saved result complete the job, message delivery may provide all the coordination you need. Your application can expose the saved result or job status without keeping the upload request open.
Copy link to headingWhen does a process need a workflow?
Now suppose the application must validate the extracted document, obtain a person's approval, and update a records system. The update depends on the approval, which depends on the validation result. The application also needs to distinguish a document awaiting review from one whose update failed.
By recording progress through that process, a durable workflow can wait for the approval event after validation succeeds. When the person responds, it can continue with the appropriate branch. Rejection might close the review; approval permits the update. The person should be approving an identified document version so a later upload cannot silently change what the approval covers.
You can build this coordination with queues and database records. Doing so makes your application responsible for the rules that join those records to incoming events, including recovery when execution stops between operations. Workflow engines provide a model for those dependencies and persisted execution state.
Persisting this progress allows recovery without requiring a worker process to stay alive throughout the review. Individual steps still execute on compute with its own resource limits, and an interrupted step may need another attempt.
Copy link to headingWhat happens when work succeeds but acknowledgment fails?
Suppose an extraction consumer saves the document's text, then crashes before acknowledging the message. The queue has no confirmation that processing finished. With at-least-once delivery, the message can become available for another attempt after its visibility timeout expires.
If every attempt inserts a new result, the retry creates a duplicate even though the queue behaved as intended. Make the write idempotent so repeating the operation has the same effect as performing it once. For example, the database can enforce a unique result key derived from the document version and extraction operation. On a repeated delivery, the consumer can find the existing result and finish without adding another record.
Concurrent attempts might both find no result before either writes, so a preliminary check alone leaves a race. The storage operation must enforce the rule. Similarly, writing a separate completion flag after an external action leaves a gap where the action succeeds but the flag never gets saved.
Workflows encounter this problem too. When a workflow step sends a request to another system, it can lose the response after that system commits the change. Retrying the step may repeat the external action. Where an external API supports idempotency keys, reuse the same operation key across retries. If it offers no way to prevent duplicates, design a reconciliation path for uncertain outcomes before enabling automatic retries of consequential actions.
Copy link to headingHow can queues and workflows work together?
You can use an upload queue to absorb document events while consumers start a review workflow for each document. The queue controls delivery; the workflow tracks validation and approval. The audit consumer can process each upload event independently through its own consumer group.
The handoff needs the same care as any other side effect. Starting a workflow might succeed even if the consumer crashes before acknowledging the upload message. Use the document version as a stable operation identity, and choose a workflow-start mechanism that recognizes repeated attempts for that identity. Writing a separate "started" flag after creation leaves the same crash window as the duplicate-result example. If the start mechanism cannot deduplicate, the application needs to detect and reconcile duplicate runs.
This arrangement fits an application that has both bursty arrivals and dependent review steps. If a process already starts reliably from an application request, it may not need an additional application-managed queue in front of it.
Copy link to headingHow do these approaches map to Vercel?
Vercel Queues uses topics and independent consumer groups, with at-least-once delivery, retries, and visibility timeouts. Consumers should be idempotent. The vercel-queue Python SDK supports publishing and consuming messages, so Python code can participate in the upload design.
Vercel Workflows persists process state and event logs. Workflow and step code run on Functions, and Workflows uses Queues for reliable enqueuing. Workflow runs can pause and resume, with recovery across crashes and deployments. Python workflow support comes through the vercel-workflow package, which exposes workflow and step decorators for durable multi-step logic.
Existing Python applications may already use Celery. Celery has orchestration features of its own, so choosing a workflow does not automatically require replacing a task system. Vercel's Celery integration uses private queue-triggered Functions with the Python runtime and Vercel Queues. This integration does not require a continuously running Celery worker process.
Copy link to headingWhat do queues and workflows leave to the application?
Keep work in the original request when the caller needs the result immediately and execution fits the request's latency and resource budget. Moving a bounded database lookup into a queue adds a delivery and status-tracking path that the interaction may not need.
Queues do not supply compute, so you must still select a worker environment that can run the document parser and its dependencies. Choosing a long-lived workflow does not give its steps unlimited computation or guarantee an uninterrupted process. Check Vercel Functions limits for runtime and plan-dependent duration, memory, and packaging constraints. The guide to choosing compute for backend workloads covers the execution decision beyond delivery and coordination.
Scheduling addresses when work begins. Vercel Cron Jobs triggers an HTTP GET to a configured path on the production deployment. That endpoint can begin a job or workflow, but a scheduled trigger alone does not recover a partially completed process. Choose recovery behavior for the work it starts.
Copy link to headingFrequently asked questions
Copy link to headingCan queues and workflows be used together?
Yes. Consumers can use incoming queue events to start workflows, while each workflow tracks a process with dependent steps. For example, queued upload events can start document reviews that wait for approval. Design the handoff so redelivered messages do not start duplicate reviews.
Copy link to headingDo workflows replace Celery?
No. Celery includes task orchestration features, so an application that already coordinates its tasks successfully may have no reason to replace it. Evaluate whether the process needs different handling for persisted progress or long waits. Vercel supports Celery through private queue-triggered Functions using the Python runtime and Vercel Queues.
Copy link to headingDo retries guarantee exactly-once side effects?
No. If a worker completes an external action but fails to record success, a retry can repeat it. Protect side effects in queue consumers and workflow steps with idempotent operations or deduplication enforced by the destination.
Copy link to headingHow is cron different from a queue or workflow?
Cron starts work on a schedule, a queue delivers work to consumers, and a workflow coordinates a process across steps. Vercel Cron Jobs sends an HTTP GET to a configured path on the production deployment. You can start queued work or a workflow through a scheduled request, but scheduling alone does not establish recovery after a partial failure.