# Evalbench

LLM evaluation and tracing

- Package: @bijectionhq/evalbench
- Category: AI & Agents
- Status: Preview
- License: Apache-2.0
- Bijection source (repository access required): https://github.com/bijectionhq/evalbench/tree/3c3250105a66b502d4a628bcbe48bca77dee091d
- Upstream: https://github.com/tfohlmeister/convex-evalbench/tree/105c07ddd1d927e0bbb3ba321c1df939489e1852
- Documentation: https://docs.bijection.com/components/using
- Directory: https://bijection.com/components/evalbench

## Get the source

Bijection repository access is required. Request access at https://bijection.com/contact.

```sh
git clone https://github.com/bijectionhq/evalbench.git
cd evalbench
git checkout 3c3250105a66b502d4a628bcbe48bca77dee091d
```

## Before using this component

Read the package README at the recorded revision for setup, dependencies, and API examples. Upstream documentation describes the original package and may differ from the Bijection adaptation. This is a source candidate for evaluation. Catalog inclusion does not establish npm availability or runtime qualification. Check compatibility with your SDK and deployment before adoption. External providers run their own compute.

## Upstream package documentation

Recorded 2026-10-23. This describes convex-evalbench; check the Bijection adaptation separately.

convex-evalbench is a Convex component that adds LLM observability, eval execution, and regression testing directly inside your Convex deployment. It records spans from agents and LLM calls into a reactive trace tree, runs versioned datasets against a target action with configurable scorers (exact match, JSON Schema, LLM-as-judge, embedding similarity, custom), and exposes live counters and per-item results via Convex subscriptions with no polling required.

### Use cases

- Tracing every LLM call made by a `@convex-dev/agent` agent or Vercel AI SDK model into a live span tree visible while the operation is still in flight
- Running a versioned dataset of input/expected-output pairs against an agent action and watching pass rates populate in real time without leaving your Convex project
- Setting up a CI regression gate that compares a new eval run against a baseline and fails the build if the pass rate drops
- Ingesting OpenTelemetry spans from any OTLP/JSON exporter (e.g. a Python service) into the same trace store via a mounted `httpAction`
- Scoring model outputs with LLM-as-judge using multi-judge consensus, with each judge verdict recorded as its own `judge` span for auditability

### How it works

After installing the package and registering the component in `convex/convex.config.ts` with `app.use(evalbench)`, you construct an `Evalbench` instance from the generated `components.evalbench` handle. Spans are written via `evalbench.recordSpan(ctx, ...)` inside Convex actions, or automatically by wrapping an agent with `withEvalbench` (from `convex-evalbench/agent`), using `evalbenchMiddleware` for the Vercel AI SDK (from `convex-evalbench/ai`), or mounting `otlpTraceHandler` as an `httpAction` for any OpenTelemetry-instrumented service.

Traces are queryable via `evalbench.spansByTrace`, `evalbench.recentTraces`, and `evalbench.spanContent`. These queries return metadata only by default; raw prompt and completion content is opt-in at record time and stored inline up to 4 KB or offloaded to Convex File Storage above that threshold. Because these are standard Convex queries, clients subscribe to them and receive updates reactively as spans land.

Evals use a dataset created with `evalbench.createDataset` and a target action matching the `{ input, runId, itemId } => { output, traceId? }` contract. `evalbench.startRun` dispatches the target for each dataset item with bounded concurrency. Built-in scorers include `exactMatch`, `jsonSchema`, `llmAsJudge`, `consensus`, and `embeddingSimilarity`; custom scorers are registered with `defineScorer`. Live run progress is available via `evalbench.runSummary` and `evalbench.listResults`. Stalled runs can be recovered with `evalbench.redriveRun`.

### Upstream README

# convex-evalbench

Reactive LLM eval, tracing, and regression layer as a
[Convex Component](https://www.convex.dev/components).

Capture spans from your agents, run datasets against them, score with
built-in scorers and LLM-as-judge, and watch results stream into a
reactive dashboard live, all inside your own Convex deployment. No
extra infrastructure, no polling: Convex subscriptions push every new
span to your UI as it lands.

> Status: tracing (with the `@convex-dev/agent` and Vercel AI SDK
> adapters plus an OTLP/JSON receiver), versioned datasets, the eval
> runner, deterministic scorers, LLM-as-judge with multi-judge consensus,
> `embeddingSimilarity`, custom scorers, run comparison with a CI
> regression gate, host-invoked trace retention, and a live reactive
> dashboard are shipped. See the [roadmap](#roadmap). References:
> [docs/tracing.md](./docs/tracing.md), [docs/evals.md](./docs/evals.md),
> [docs/dashboard.md](./docs/dashboard.md).

## Tracing

Record every LLM call as a span inside your own Convex deployment. Because
Convex is reactive, a trace renders as a live span tree that fills in while
it is in flight, no polling.

### Install and register

```sh
npm install convex-evalbench
```

```ts
// convex/convex.config.ts
import { defineApp } from "convex/server";
import evalbench from "convex-evalbench/convex.config";

const app = defineApp();
app.use(evalbench);
export default app;
```

### Record spans (source-agnostic API)

Construct an `Evalbench` from the generated component handle and record
spans. Recording is best-effort: a failure is logged and swallowed, never
thrown back into your LLM call.

```ts
import { Evalbench } from "convex-evalbench";
import { components } from "./_generated/api.js";
import { action } from "./_generated/server.js";

const evalbench = new Evalbench(components.evalbench);

export const runStep = action({
  args: {},
  handler: async (ctx) => {
    const traceId = crypto.randomUUID();
    const rootSpanId = crypto.randomUUID();
    await evalbench.recordSpan(ctx, {
      traceId,
      spanId: rootSpanId,
      kind: "agent_step",
      operationName: "my operation",
      status: "success",
      startedAt: Date.now(),
    });
    await evalbench.recordSpan(ctx, {
      traceId,
      spanId: crypto.randomUUID(),
      parentSpanId: rootSpanId,
      kind: "llm",
      operationName: "llm call",
      model: "...",
      provider: "...",
      inputTokens: 10,
      outputTokens: 20,
      totalTokens: 30,
      status: "success",
      startedAt: Date.now(),
      // input/output are recorded only when you pass them (opt-in).
      input: "the prompt",
      output: "the completion",
    });
  },
});
```

### Read it back (reactive)

Expose the queries from your host so a client can subscribe; the span tree
fills in live as spans are recorded.

```ts
import { query } from "./_generated/server.js";
import { v } from "convex/values";

export const listSpans = query({
  args: { traceId: v.string() },
  handler: (ctx, args) => evalbench.spansByTrace(ctx, args.traceId),
});

export const listRecentTraces = query({
  args: { limit: v.optional(v.number()) },
  handler: (ctx, args) => evalbench.recentTraces(ctx, args),
});

// Resolve a span's content on demand (inline strings plus signed URLs for
// content held in File Storage). `spanId` is the document id from listSpans.
export const getSpanContent = query({
  args: { spanId: v.string() },
  handler: (ctx, args) => evalbench.spanContent(ctx, args.spanId),
});
```

The tree and recent-traces queries return metadata only (no raw content),
so reactive updates stay small. Raw content is opt-in (you record it by
passing `input`/`output`); content at or below 4 KB is stored inline,
larger content is offloaded to Convex File Storage and resolved on demand.

### Wrap a @convex-dev/agent agent (optional)

The `withEvalbench` adapter is one optional ingestion source. It wraps a
[`@convex-dev/agent`](https://github.com/get-convex/agent) agent so each
LLM call is recorded as a span, composing with (not replacing) any handlers
you already set. `@convex-dev/agent` is an optional peer dependency; the
tracing core does not import it.

```ts
import { Agent } from "@convex-dev/agent";
import { Evalbench } from "convex-evalbench";
import { withEvalbench } from "convex-evalbench/agent";
import { anthropic } from "@ai-sdk/anthropic";
import { components } from "./_generated/api.js";

const evalbench = new Evalbench(components.evalbench);

const agent = withEvalbench(
  new Agent(components.agent, {
    name: "my-agent",
    languageModel: anthropic("claude-haiku-4-5"),
  }),
  { evalbench, recordContent: true }, // recordContent defaults to false
);
```

LLM calls made inside one `generateText` / `generateObject` operation share
a `traceId` and link to a root `agent_step` span, so the operation renders
as one trace tree. See [docs/tracing.md](./docs/tracing.md) for details and
limitations.

### Other ingestion sources

Two more optional sources feed the same tracing core:

- **Vercel AI SDK** (`convex-evalbench/ai`): `evalbenchMiddleware` wraps a
  model with `wrapLanguageModel`, recording one span per call with measured
  latency. `ai` is an optional peer dependency.
- **OpenTelemetry** (`convex-evalbench/otlp`): `otlpTraceHandler` is an
  `httpAction` you mount (e.g. at `/v1/traces`) so any OTLP/JSON exporter
  streams spans in, mapped via GenAI semantic conventions.

See [docs/tracing.md](./docs/tracing.md) for both.

## Evals: datasets, runs, scorers

Keep a versioned dataset of inputs with expected outputs, run your
system under test against every item with bounded parallelism, and
watch the pass rate fill in live.

```ts
// Seed a dataset.
const datasetId = await evalbench.createDataset(ctx, {
  name: "greetings",
  items: [
    { input: "hello", expectedOutput: "HELLO" },
    { input: "world", expectedOutput: "WORLD" },
  ],
});

// The system under test: an action taking { input, runId, itemId } and
// returning { output, traceId? }. Stamp your spans with runId so the
// run's traces correlate.
export const myTarget = action({
  args: { input: v.any(), runId: v.string(), itemId: v.string() },
  handler: async (ctx, args) => {
    const output = await runMyAgent(args.input);
    return { output };
  },
});

// Run it: one idempotent, scored result per item.
const runId = await evalbench.startRun(ctx, {
  datasetId,
  target: api.evals.myTarget,
  config: { scorers: [{ type: "exactMatch" }], concurrency: 4 },
});
```

Subscribe to `evalbench.runSummary(ctx, runId)` for the live counters
(completed / passed / aggregate score, maintained in the same mutation
that writes each result) and `evalbench.listResults(ctx, runId)` for
the per-item rows with scores and trace links.

Scorers: deterministic built-ins `exactMatch` and `jsonSchema`
(eval-free JSON Schema validation that runs in Convex's V8 runtime),
plus host-extensible scoring with no provider SDK in the component:
`defineScorer` for custom scorer actions, `llmAsJudge` for
rubric-based judging with your own LLM call (each verdict traced as a
`judge` span), multi-judge `consensus` with a quorum, and
`embeddingSimilarity` against a host embedder action. A wedged run is
recovered with `evalbench.redriveRun(ctx, runId)` (attempts-capped).
See [docs/evals.md](./docs/evals.md) for the target and scorer
contracts, versioning, and the runner's execution model.

## Local development

The repo bundles a pinned `convex-local-backend`, so the example app and
tests run without a cloud Convex project. See
[CONTRIBUTING.md](./CONTRIBUTING.md) for the dev loop and release flow.

## Roadmap

Tracing, datasets, the runner with **managed retries** (retryable
target failures retried with exponential backoff; see
[docs/evals.md](./docs/evals.md)), judges (LLM-as-judge with consensus),
`embeddingSimilarity`, custom scorers, regression / A-B (run comparison
with a CI gate), trace retention (host-invoked `pruneTraces`), the
**Vercel AI SDK adapter** (`evalbenchMiddleware`), the **OTLP/JSON
receiver** (`otlpTraceHandler` for any OpenTelemetry-instrumented app; see
[docs/tracing.md](./docs/tracing.md)), and the **live dashboard** (a
companion React app over the reactive queries; see
[docs/dashboard.md](./docs/dashboard.md)) are shipped. The roadmap items
are all delivered; feature requests are welcome via issues.

## Security

Raw prompt/completion content is only stored when you opt in; span
metadata is always recorded. See [SECURITY.md](./SECURITY.md) for the
threat model and how to report vulnerabilities.

## License

[Apache-2.0](./LICENSE)