# Firecrawl Scrape

A Bijection component for scraping web pages using the Firecrawl API with durable caching and reactive queries.

- Package: @bijectionhq/firecrawl-scrape
- Category: AI & Agents
- Status: Reference
- License: Apache-2.0
- Bijection source (repository access required): https://github.com/bijectionhq/firecrawl-scrape/tree/2a73f77045c9756d5200b4c86e2559415d83cd6f
- Upstream: https://github.com/Gitmaxd/convex-firecrawl-scrape/tree/4d6a24efc742fe8855e5b9f8b39c051b9495f42f
- Documentation: https://docs.bijection.com/components/using
- Directory: https://bijection.com/components/firecrawl-scrape

## Get the source

Bijection repository access is required. Request access at https://bijection.com/contact.

```sh
git clone https://github.com/bijectionhq/firecrawl-scrape.git
cd firecrawl-scrape
git checkout 2a73f77045c9756d5200b4c86e2559415d83cd6f
```

## Before using this component

Read the package README at the recorded revision for setup, dependencies, and API examples. Upstream documentation describes the original package and may differ from the Bijection adaptation. This is a source candidate for evaluation. Catalog inclusion does not establish npm availability or runtime qualification. Check compatibility with your SDK and deployment before adoption. External providers run their own compute.

## Upstream package documentation

Recorded 2026-10-23. This describes convex-firecrawl-scrape; check the Bijection adaptation separately.

This component provides a durable web scraping service that extracts content from URLs and returns clean markdown, HTML, screenshots, or structured JSON data. Built on Firecrawl's scraping API, it features intelligent caching with configurable TTL, reactive status updates via Convex subscriptions, and built-in SSRF protection. The component handles async scraping jobs and stores results in your Convex database for reliable access.

### Use cases

• **Content aggregation apps** that need to scrape blog posts, news articles, or documentation sites and convert them to clean markdown for display or processing
• **Product monitoring dashboards** that extract structured data from e-commerce pages using JSON schemas to track pricing, availability, and product details
• **Research tools** that capture screenshots and full HTML content from web pages for archival, comparison, or analysis workflows
• **AI training pipelines** that need to scrape and clean web content at scale, with durable caching to avoid re-scraping the same URLs
• **Competitive intelligence platforms** that monitor competitor websites and extract specific data points using schema-based extraction

### How it works

The component wraps Firecrawl's scraping API in Convex actions and provides a secure `exposeApi()` function that enforces authentication before any scraping operation. When you call `scrape()`, it starts an async job and returns a job ID that you can use with `useQuery()` to reactively monitor status updates.

Scraped content gets cached in your Convex database with a configurable TTL (default 30 days) and uses superset matching where cached results with multiple formats can satisfy requests for fewer formats. The component supports multiple output formats including markdown, HTML, screenshots, and AI-generated summaries, plus structured JSON extraction via schema validation.

All scraping operations include built-in SSRF protection that blocks private IPs and localhost, and the required auth wrapper ensures your Firecrawl API key stays secure. You can configure proxy options for anti-bot protection and set wait times for dynamic content rendering.

### Upstream README

# Convex Firecrawl Scrape Component

<p align="center">
</p>

Scrape any URL and get clean markdown, HTML, screenshots, or structured JSON -
with durable caching and reactive queries.

```ts
const { jobId } = await scrape({ url: "https://example.com" });
// Status updates reactively as the scrape completes
const status = useQuery(api.firecrawl.getStatus, { id: jobId });
```

- **Durable caching** with configurable TTL (default 30 days)
- **Reactive status updates** via Convex subscriptions
- **Multiple output formats**: markdown, HTML, raw HTML, screenshots, links,
  images, AI summaries
- **JSON extraction** via schema-based LLM processing
- **Built-in SSRF protection** blocks private IPs and localhost
- **Secure by default** with required auth wrapper

**[Live Demo](https://convex-firecrawl-scrape.vercel.app/)** |
[Example Code](./example)

Play with the example:

```sh
git clone https://github.com/gitmaxd/convex-firecrawl-scrape.git
cd convex-firecrawl-scrape
npm install
npm run dev
```

## Pre-requisite: Convex

You'll need an existing Convex project. Convex is a hosted backend platform with
a database, serverless functions, and more. Learn more
[here](https://docs.convex.dev/get-started).

Run `npm create convex` or follow any of the
[quickstarts](https://docs.convex.dev/home) to set one up.

## Installation

```sh
npm install convex-firecrawl-scrape
```

Install the component in your `convex/convex.config.ts`:

```ts
// convex/convex.config.ts
import { defineApp } from "convex/server";
import firecrawlScrape from "convex-firecrawl-scrape/convex.config.js";

const app = defineApp();
app.use(firecrawlScrape);
export default app;
```

Set your Firecrawl API key:

```sh
npx convex env set FIRECRAWL_API_KEY your_api_key_here
```

Get your API key at [firecrawl.dev](https://firecrawl.dev).

## Usage

**Always use `exposeApi()` to expose component functionality.** This wrapper
enforces authentication and controls API key access.

```ts
// convex/firecrawl.ts
import { exposeApi } from "convex-firecrawl-scrape";
import { components } from "./_generated/api";

export const { scrape, getCached, getStatus, getContent, invalidate } =
  exposeApi(components.firecrawlScrape, {
    auth: async (ctx, operation) => {
      const identity = await ctx.auth.getUserIdentity();
      if (!identity) throw new Error("Unauthorized");
      return process.env.FIRECRAWL_API_KEY!;
    },
  });
```

### React Integration

```tsx
import { useMutation, useQuery } from "convex/react";
import { api } from "../convex/_generated/api";
import { useState } from "react";

function ScrapeButton({ url }: { url: string }) {
  const [jobId, setJobId] = useState<string | null>(null);
  const scrape = useMutation(api.firecrawl.scrape);
  const status = useQuery(
    api.firecrawl.getStatus,
    jobId ? { id: jobId } : "skip",
  );
  const content = useQuery(
    api.firecrawl.getContent,
    jobId && status?.status === "completed" ? { id: jobId } : "skip",
  );

  return (
    <div>
      <button
        onClick={async () => setJobId((await scrape({ url })).jobId)}
        disabled={status?.status === "scraping"}
      >
        {status?.status === "scraping" ? "Scraping..." : "Scrape"}
      </button>
      {status?.status === "completed" && <pre>{content?.markdown}</pre>}
      {status?.status === "failed" && <p>Error: {status.error}</p>}
    </div>
  );
}
```

### Output Formats

```ts
const { jobId } = await scrape({
  url: "https://example.com",
  options: {
    formats: ["markdown", "html", "links", "images", "screenshot"],
    storeScreenshot: true,
  },
});
```

| Format       | Description                                             |
| ------------ | ------------------------------------------------------- |
| `markdown`   | Clean markdown content (default)                        |
| `html`       | Cleaned HTML                                            |
| `rawHtml`    | Original HTML source                                    |
| `links`      | URLs found on the page                                  |
| `images`     | Image URLs found on the page                            |
| `summary`    | AI-generated page summary                               |
| `screenshot` | Screenshot URL (use `storeScreenshot: true` to persist) |

### JSON Extraction

Extract structured data using a JSON schema:

```ts
const { jobId } = await scrape({
  url: "https://example.com/product",
  options: {
    extractionSchema: {
      type: "object",
      properties: {
        name: { type: "string" },
        price: { type: "number" },
      },
      required: ["name", "price"],
    },
  },
});

const content = await getContent({ id: jobId });
console.log(content.extractedJson); // { name: "Widget", price: 99.99 }
```

### Cache Management

Cached results use **superset matching**: a cache entry with
`["markdown", "screenshot"]` satisfies a request for `["markdown"]`.

```ts
// Check cache
const cached = await getCached({ url: "https://example.com" });

// Force refresh
const { jobId } = await scrape({ url, options: { force: true } });

// Invalidate cache
await invalidate({ url: "https://example.com" });
```

### Proxy Options

For anti-bot protected sites:

```ts
const { jobId } = await scrape({
  url: "https://protected-site.com",
  options: {
    proxy: "stealth", // Residential proxy
    waitFor: 3000, // Wait for dynamic content
  },
});
```

## Security

**Always use `exposeApi()`** - never expose component functions directly to
clients. Server-side code _can_ call component internals directly, but doing so
bypasses authentication. It ensures:

- Authentication before any operation
- API key controlled by your callback, not callers
- Operation-specific authorization support

```ts
// ❌ DANGEROUS - bypasses auth
export const scrape = components.firecrawlScrape.lib.startScrape;

// ✅ SAFE - auth enforced
export const { scrape } = exposeApi(components.firecrawlScrape, { auth: ... });
```

**SSRF Protection**: Built-in validation blocks localhost, private IPs, and
non-HTTP schemes.

For domain allowlists, rate limiting, and detailed security guidance, see
[docs/SECURITY.md](./docs/SECURITY.md).

## Error Handling

```ts
const status = await getStatus({ id: jobId });
if (status?.status === "failed") {
  console.error(status.error, status.errorCode);
  // errorCode is the HTTP status or error code from Firecrawl (e.g., 402, 429, "BAD_REQUEST")
}
```

Found a bug? Feature request?
[File it here](https://github.com/gitmaxd/convex-firecrawl-scrape/issues).

## Advanced Usage

For configuration options, the `FirecrawlScrape` class API, and URL utilities,
see [docs/ADVANCED.md](./docs/ADVANCED.md).

## Development

```sh
npm install
npm run dev
```

## License

Apache-2.0