---
name: bijection-batch-processor
description: "Evaluate and integrate the Batch Processor component with Bijection."
---

# Batch Processor

Batch processing component for Bijection - batch accumulator and table iterator

- Package: @bijectionhq/batch-processor
- Category: Durable Functions
- Status: Reference
- License: MIT
- Bijection source (repository access required): https://github.com/bijectionhq/batch-processor/tree/27e86da6cc1b0a30d41c51123c06a1c449caab81
- Upstream: https://github.com/blocknavi/convex-batch-processor/tree/79258365cf7aabcd0a8f1ad936793d85b8d90ea1
- Documentation: https://docs.bijection.com/components/using
- Directory: https://bijection.com/components/batch-processor

## Get the source

Bijection repository access is required. Request access at https://bijection.com/contact.

```sh
git clone https://github.com/bijectionhq/batch-processor.git
cd batch-processor
git checkout 27e86da6cc1b0a30d41c51123c06a1c449caab81
```

## Before using this component

Read the package README at the recorded revision for setup, dependencies, and API examples. Upstream documentation describes the original package and may differ from the Bijection adaptation. This is a source candidate for evaluation. Catalog inclusion does not establish npm availability or runtime qualification. Check compatibility with your SDK and deployment before adoption. External providers run their own compute.

## Integration workflow

1. Read the source README and package manifest at the recorded revision.
2. Confirm the required SDK, dependency availability, and runtime compatibility.
3. Follow the component's documented setup and configuration. Never invent an npm install command from a catalog package name.
4. Keep credentials in the project's existing secret store.
5. Follow the project's execution rules for verification and report what was actually checked.

## Original upstream skill

The following guidance describes convex-batch-processor. Its installation commands and APIs refer to the upstream package. Evaluate compatibility with the Bijection source separately.


> Agents: read this skill fully before writing code that uses Batch Processor. Follow the installation and configuration steps exactly.

# Batch Processor

## Instructions

A batch processing component that provides two complementary patterns: a batch accumulator that collects items and flushes them based on size thresholds or time intervals, and a table iterator that processes large datasets in controlled chunks with pause/resume support. Both patterns handle retries, progress tracking, and integrate with Convex actions for external API calls or heavy processing workloads.

### Installation

```bash
npm install convex-batch-processor
```

Current npm version: `convex-batch-processor@1.1.1`

## Use cases

- **Analytics event batching** - Collect user events in real-time and flush them to external analytics services like Mixpanel or Amplitude in batches of 100+ events every 30 seconds
- **Large table migrations** - Process thousands of user records during schema changes or data transformations, with ability to pause/resume and track progress
- **Bulk email processing** - Queue email sends and process them in controlled batches to avoid rate limits on email service APIs
- **Data export workflows** - Stream large datasets to external systems in manageable chunks with retry logic and failure handling
- **Background cleanup jobs** - Process expired records or orphaned data across large tables with built-in scheduling and progress monitoring

## How it works

The component uses Convex's component system and integrates via `app.use(batchProcessor)` in your config. The BatchProcessor class provides two main APIs: batch accumulation and table iteration, both designed around Convex's action pattern for external processing.

For batch accumulation, you create a BatchProcessor with a config specifying `immediateFlushThreshold`, `flushIntervalMs`, and a `processBatch` action. Items added via `addItems()` are collected until the threshold is hit or the interval expires, then automatically flushed to your processing action. The component handles scheduling the interval flush when the first item is added, requiring no cron setup.

Table iteration works through a callback pattern where you provide `getNextBatch` (a query for pagination) and `processBatch` (an action for processing). The component manages the iteration loop, calling your query to fetch batches and your action to process them, with built-in retry logic, exponential backoff, and pause/resume capabilities.

Both patterns store state in Convex tables for durability and provide status APIs like `getBatchStatus()` and `getIteratorStatus()` for monitoring. Error handling preserves failed batches for retry and maintains detailed status information for debugging.

## When NOT to use

- When a simpler built-in solution exists for your specific use case
- If you are not using Convex as your backend
- When the functionality provided by Batch Processor is not needed

## Resources

- [npm package](https://www.npmjs.com/package/convex-batch-processor)
- [GitHub repository](https://github.com/blocknavi/convex-batch-processor)
- [Convex Components Directory](https://www.convex.dev/components/convex-batch-processor)
- [Convex documentation](https://docs.convex.dev)