--- name: bijection-batch-worker description: "Evaluate and integrate the Batch Worker component with Bijection." --- # Batch Worker Recoverable table-backed batch workers - Package: @bijectionhq/batch-worker - Category: Durable Functions - Status: Included - License: Apache-2.0 - Bijection source (repository access required): https://github.com/bijectionhq/batch-worker/tree/bc7be9f2724ab832ba2daaf3f6d1d00e121908ba - Upstream: https://github.com/convex/batch-worker/tree/f299a7b9b5f302c1b0a38af57bdcd96302e44d1b - Documentation: https://docs.bijection.com/components/using - Directory: https://bijection.com/components/batch-worker ## Get the source Bijection repository access is required. Request access at https://bijection.com/contact. ```sh git clone https://github.com/bijectionhq/batch-worker.git cd batch-worker git checkout bc7be9f2724ab832ba2daaf3f6d1d00e121908ba ``` ## Before using this component Read the package README at the recorded revision for setup, dependencies, and API examples. Upstream documentation describes the original package and may differ from the Bijection adaptation. This source is included in Bijection's workspace. Catalog inclusion does not establish npm availability or runtime qualification. Check compatibility with your SDK and deployment before adoption. External providers run their own compute. ## Integration workflow 1. Read the source README and package manifest at the recorded revision. 2. Confirm the required SDK, dependency availability, and runtime compatibility. 3. Follow the component's documented setup and configuration. Never invent an npm install command from a catalog package name. 4. Keep credentials in the project's existing secret store. 5. Follow the project's execution rules for verification and report what was actually checked. ## Original upstream skill The following guidance describes @convex-dev/batch-worker. Its installation commands and APIs refer to the upstream package. Evaluate compatibility with the Bijection source separately. > Agents: read this skill fully before writing code that uses Batch Worker. Follow the installation and configuration steps exactly. # Batch Worker ## Instructions Batch Worker provides a durable, self-recovering loop for processing queued work stored in your own Convex tables. You supply a work query that returns the next batch and a worker mutation that processes it, and the component handles scheduling, debouncing, single-loop enforcement, snapshot reads to avoid OCC conflicts, and automatic restart on failure. It is the underlying pattern behind components like Workpool, exposed so you can build custom queue processors with full control over your data model. ### Installation ```bash npm install @convex-dev/batch-worker ``` Current npm version: `@convex-dev/batch-worker@0.3.6` ## Use cases - Draining a queue of pending events or tasks inserted by concurrent mutations, without hitting OCC retry limits on the queue table - Updating denormalized aggregates from a write-ahead log of changes, decoupling high-concurrency writers from a single aggregate document - Batching outbound API calls (LLM requests, webhooks) by having the mutation claim rows and schedule an action, then committing results back - Rate-limiting external API usage by integrating with the Rate Limiter component and returning debounceMs or timeoutMs to throttle the loop - Building custom queue-processing components that need independent named queues with separate work query and worker mutation pairs ## How it works After installing the component via `app.use(batchWorker)` in `convex/convex.config.ts`, you call `ping(ctx, components.batchWorker, { name, workQuery, workerMutation })` after inserting work. The component ensures exactly one processing loop runs per named queue. The loop calls your `internalQuery` (typed with `vBatchQueryArgs` and `vBatchResult`) to fetch the next batch using a snapshot read with no read dependencies, then calls your `internalMutation` to process it in a normal transaction. The snapshot read in the work query avoids OCC conflicts from concurrent inserts, since range reads there do not create read dependencies. The mutation receives the batch data directly through its arguments, so it only takes dependencies on the specific document IDs it updates, not on the queue range. Your mutation is responsible for deleting or advancing past processed rows so the query does not return them again. The loop supports debouncing via `config.debounceMs` on ping (to accumulate bursts before the first batch) and via returning `{ debounceMs }` from the worker mutation (to throttle while running). When the queue is empty, your query returns `{ kind: 'idle' }` and the loop parks. An optional `timeoutMs` field wakes the loop after a fixed interval even without a ping. The component monitors the loop and automatically restarts it if it dies due to an unexpected error, logging the failure for alerting. For batching work that requires actions (external API calls, LLM requests), the recommended pattern is for the worker mutation to claim rows by marking them in-progress, then schedule the action or enqueue it in a Workpool. The action performs the external call and writes results back via a mutation. This keeps the action separate from the transactional mutation while still benefiting from the batch loop. ## When NOT to use - When a simpler built-in solution exists for your specific use case - If you are not using Convex as your backend - When the functionality provided by Batch Worker is not needed ## Resources - [npm package](https://www.npmjs.com/package/%40convex-dev%2Fbatch-worker) - [GitHub repository](https://github.com/get-convex/batch-worker) - [Live demo](https://capable-fish-350.convex.site/) - [Convex Components Directory](https://www.convex.dev/components/batch-worker) - [Convex documentation](https://docs.convex.dev)