--- name: bijection-evalbench description: "Evaluate and integrate the Evalbench component with Bijection." --- # Evalbench LLM evaluation and tracing - Package: @bijectionhq/evalbench - Category: AI & Agents - Status: Preview - License: Apache-2.0 - Bijection source (repository access required): https://github.com/bijectionhq/evalbench/tree/3c3250105a66b502d4a628bcbe48bca77dee091d - Upstream: https://github.com/tfohlmeister/convex-evalbench/tree/105c07ddd1d927e0bbb3ba321c1df939489e1852 - Documentation: https://docs.bijection.com/components/using - Directory: https://bijection.com/components/evalbench ## Get the source Bijection repository access is required. Request access at https://bijection.com/contact. ```sh git clone https://github.com/bijectionhq/evalbench.git cd evalbench git checkout 3c3250105a66b502d4a628bcbe48bca77dee091d ``` ## Before using this component Read the package README at the recorded revision for setup, dependencies, and API examples. Upstream documentation describes the original package and may differ from the Bijection adaptation. This is a source candidate for evaluation. Catalog inclusion does not establish npm availability or runtime qualification. Check compatibility with your SDK and deployment before adoption. External providers run their own compute. ## Integration workflow 1. Read the source README and package manifest at the recorded revision. 2. Confirm the required SDK, dependency availability, and runtime compatibility. 3. Follow the component's documented setup and configuration. Never invent an npm install command from a catalog package name. 4. Keep credentials in the project's existing secret store. 5. Follow the project's execution rules for verification and report what was actually checked. ## Original upstream skill The following guidance describes convex-evalbench. Its installation commands and APIs refer to the upstream package. Evaluate compatibility with the Bijection source separately. # Convex Evalbench ## Instructions Use Convex Evalbench to convex evalbench adds llm tracing, versioned eval datasets, and regression scoring to your convex deployment with no extra infrastructure. This Convex component integrates directly with your backend. ### Installation ```bash npm install convex-evalbench ``` ### Capabilities - Stream live span trees from LLM calls into a reactive dashboard using Convex subscriptions, no polling required - Run versioned eval datasets against any agent action with built-in scorers including exactMatch, jsonSchema, LLM-as-judge, and embeddingSimilarity - Ingest traces from multiple sources including the convex-dev/agent adapter, Vercel AI SDK middleware, and an OTLP/JSON HTTP receiver - Gate CI deployments with run comparison and regression detection using recorded pass rates and aggregate scores ## Examples ### how to trace LLM calls in a Convex agent Convex Evalbench provides a withEvalbench adapter that wraps a convex-dev/agent Agent instance and records each LLM call as a span linked to a root agent_step span. Spans share a traceId per operation and stream into a reactive span tree via Convex subscriptions. Raw prompt and completion content is opt-in via the recordContent flag. ### how to run evals on an LLM agent in Convex Convex Evalbench lets you create a versioned dataset with evalbench.createDataset, then call evalbench.startRun with a target action and scorer config. The runner dispatches each dataset item to your action with bounded concurrency and writes scored results back. You subscribe to evalbench.runSummary for live pass rate counters and evalbench.listResults for per-item scores and trace links. ### LLM as judge scoring for AI evals in Convex Convex Evalbench includes an llmAsJudge scorer that accepts a rubric and calls your own LLM action to produce a verdict, which is recorded as a judge span. A multi-judge consensus scorer is also available for quorum-based decisions. Each judging call is traced inside the same Convex deployment. ### how to receive OpenTelemetry traces from an LLM app in Convex Convex Evalbench ships an otlpTraceHandler that you mount as a Convex httpAction at a path like /v1/traces. Any OTLP/JSON exporter can then stream spans to it, which are mapped via GenAI semantic conventions and stored in your Convex deployment alongside agent-native spans. ## When NOT to use - When a simpler built-in solution exists for your specific use case - If you are not using Convex as your backend - When the functionality provided by Convex Evalbench is not needed ## Troubleshooting **Does Convex Evalbench require any external services or databases?** No. Convex Evalbench runs entirely inside your own Convex deployment. Spans, datasets, eval results, and dashboard state are all stored in Convex tables. Reactivity is provided by Convex subscriptions, not polling, and there is no external eval platform or tracing backend required. **How does Convex Evalbench handle large prompt and completion content?** Convex Evalbench only stores raw input and output when you explicitly pass them to recordSpan or enable recordContent on the agent adapter. Content at or below 4 KB is stored inline. Larger content is offloaded to Convex File Storage and resolved on demand via the spanContent query, so reactive span tree updates stay small. **Which ingestion sources does Convex Evalbench support for tracing?** Convex Evalbench supports three ingestion paths: a source-agnostic recordSpan API you call directly, the withEvalbench adapter for convex-dev/agent, and evalbenchMiddleware for Vercel AI SDK via wrapLanguageModel. An OTLP/JSON HTTP receiver is also included for any OpenTelemetry-instrumented application. **What scorers are available in Convex Evalbench?** Convex Evalbench ships exactMatch and jsonSchema as deterministic built-in scorers that run in Convex's V8 runtime. For semantic scoring it provides llmAsJudge with rubric-based judging, a consensus scorer for multi-judge quorum, and embeddingSimilarity against a host embedder action. Custom scorers are supported via defineScorer. **How does Convex Evalbench handle stuck or failed eval runs?** Convex Evalbench includes evalbench.redriveRun which replays incomplete items in a run up to a configured attempt cap. Retryable target failures are retried with exponential backoff. This allows recovery from transient LLM errors or action timeouts without restarting the entire run. ## Resources - [npm package](https://www.npmjs.com/package/convex-evalbench) - [GitHub repository](https://github.com/tfohlmeister/convex-evalbench) - [Live demo](https://github.com/tfohlmeister/convex-evalbench/tree/main/dashboard) - [Convex Components Directory](https://www.convex.dev/components/convex-evalbench) - [Convex documentation](https://docs.convex.dev)