# Bright Data Sholajegede

Cache Bright Data SERP and Web Unlocker results in Bijection with configurable TTLs and reactive queries.

- Package: @bijectionhq/bright-data-sholajegede
- Category: Integrations
- Status: Reference
- License: Apache-2.0
- Bijection source (repository access required): https://github.com/bijectionhq/bright-data-sholajegede/tree/56e3626498bb4a680b42bc615477bbe4575947e7
- Upstream: https://github.com/sholajegede/convex-bright-data/tree/2505cfcbf69d6c3c58658d05851e19490da3e159
- Documentation: https://docs.bijection.com/components/using
- Directory: https://bijection.com/components/bright-data-sholajegede

## Get the source

Bijection repository access is required. Request access at https://bijection.com/contact.

```sh
git clone https://github.com/bijectionhq/bright-data-sholajegede.git
cd bright-data-sholajegede
git checkout 56e3626498bb4a680b42bc615477bbe4575947e7
```

## Before using this component

Read the package README at the recorded revision for setup, dependencies, and API examples. Upstream documentation describes the original package and may differ from the Bijection adaptation. This is a source candidate for evaluation. Catalog inclusion does not establish npm availability or runtime qualification. Check compatibility with your SDK and deployment before adoption. External providers run their own compute.

## Upstream package documentation

Recorded 2026-10-23. This describes @sholajegede/convex-bright-data; check the Bijection adaptation separately.

This component wraps Bright Data's SERP API and Web Unlocker with reactive Convex caching. Search results and scraped web content are stored in component-owned tables with configurable TTLs, enabling reactive queries that update automatically when cache refreshes. Developers get instant cached responses via useQuery with no re-fetching until data expires.

### Use cases

- Build a search aggregation app that caches Google search results and updates the UI reactively when fresh data arrives
- Create a competitive intelligence dashboard that scrapes competitor websites and stores content with 6-hour TTLs
- Implement a news monitoring system that searches specific verticals (news, shopping) and maintains fresh results in Convex tables
- Build a content research tool that scrapes multiple URLs and provides real-time cache status with fetchedAt and expiresAt timestamps
- Add web scraping capabilities to an existing Convex app without managing separate caching infrastructure

### How it works

The component stores Bright Data API responses in dedicated Convex tables with configurable TTL settings. When you call `brightData.search()` or `brightData.scrape()` from Convex actions, it checks the cache first and only hits the Bright Data API if data is stale or missing.

The key pattern involves two functions per operation: an action that fetches/caches data (like `searchWeb`) and a query that reads cached results (like `getCachedSearch`). Frontend components use `useQuery` on the cached data functions to get reactive updates whenever the cache refreshes.

Setup requires instantiating the BrightData class with your API credentials and zone names, then calling component queries via `ctx.runQuery(components.brightData.lib.getSearch, {...})`. The component handles cache invalidation through the `invalidate()` method and supports search verticals (news, shopping, images) plus recency filtering.

### Upstream README

# @sholajegede/convex-bright-data

A [Convex component](https://www.convex.dev/components) that wraps [Bright Data's](https://brightdata.com) SERP API and Web Unlocker with reactive caching. Search results and scraped page content are stored in component-owned Convex tables with configurable TTLs — apps query cached results reactively, with no re-fetching if data is still fresh.

Found a bug? Feature request? [File it here](https://github.com/sholajegede/convex-bright-data/issues).

## Features

- **SERP search** via Bright Data's SERP API — Google search results cached in Convex
- **Web scraping** via Bright Data's Web Unlocker — page content cached in Convex
- **Reactive queries** — `useQuery` on cached results, live updates when cache refreshes
- **Configurable TTLs** — per-request or global defaults for searches and pages
- **Cache invalidation** — force re-fetch any query or URL on demand
- **No re-fetching** — returns cached data instantly if still within TTL

## Prerequisites

- A [Bright Data](https://brightdata.com) account with:
  - A SERP API zone (default zone name: `serp_api1`)
  - A Web Unlocker zone (default zone name: `web_unlocker1`)
  - An API token

## Installation
```sh
npm install @sholajegede/convex-bright-data
```

Add the component to your `convex/convex.config.ts`:
```ts
import { defineApp } from "convex/server";
import brightData from "@sholajegede/convex-bright-data/convex.config.js";

const app = defineApp();
app.use(brightData);

export default app;
```

## Usage

Instantiate the client in your Convex functions file, passing your Bright Data credentials:
```ts
// convex/brightData.ts
import { components } from "./_generated/api.js";
import { BrightData } from "@sholajegede/convex-bright-data";

export const brightData = new BrightData(components.brightData, {
  BRIGHTDATA_API_TOKEN: process.env.BRIGHTDATA_API_TOKEN!,
  BRIGHTDATA_SEARCH_ZONE: process.env.BRIGHTDATA_SEARCH_ZONE,       // optional, default: "serp_api1"
  BRIGHTDATA_WEB_UNLOCKER_ZONE: process.env.BRIGHTDATA_WEB_UNLOCKER_ZONE, // optional, default: "web_unlocker1"
  DEFAULT_SEARCH_TTL_MS: 1000 * 60 * 60,   // optional, default: 1 hour
  DEFAULT_PAGE_TTL_MS: 1000 * 60 * 60 * 6, // optional, default: 6 hours
});
```

> **Security note:** Your Bright Data API token is passed through Convex action arguments and traverses the Convex runtime on the server side. Always load it from environment variables using `process.env` in your app's Convex functions — never hardcode it or expose it in client-side code.
>
> ```sh
> npx convex env set BRIGHTDATA_API_TOKEN your_token_here
> ```

### Search the web
```ts
// convex/myFunctions.ts
import { action, query } from "./_generated/server.js";
import { components } from "./_generated/api.js";
import { brightData } from "./brightData.js";
import { v } from "convex/values";

// Fetch (or return cached) search results
export const searchWeb = action({
  args: { query: v.string() },
  handler: async (ctx, args) => {
    return await brightData.search(ctx, { query: args.query });
  },
});

// Reactive query — subscribe to cached results from the frontend.
// This is the key pattern: call searchWeb once to fetch and cache,
// then useQuery on getCachedSearch to get live updates whenever the
// cache refreshes. No polling needed — Convex pushes updates automatically.
export const getCachedSearch = query({
  args: { query: v.string() },
  handler: async (ctx, args) => {
    return await ctx.runQuery(components.brightData.lib.getSearch, {
      query: args.query,
    });
  },
});
```
```tsx
// React component — subscribes reactively, re-renders when cache updates
const results = useQuery(api.myFunctions.getCachedSearch, { query: "convex database" });
// results.isFresh   — true if still within TTL, false if stale
// results.results   — JSON string of search results from Bright Data
// results.fetchedAt — timestamp of when data was last fetched
// results.expiresAt — timestamp of when the cache entry will expire
```

### Scrape a page
```ts
// Fetch (or return cached) page content
export const scrapePage = action({
  args: { url: v.string() },
  handler: async (ctx, args) => {
    return await brightData.scrape(ctx, { url: args.url });
  },
});

// Reactive query — subscribe to cached page content from the frontend.
// Same pattern as search: scrape once, then useQuery for live updates.
export const getCachedPage = query({
  args: { url: v.string() },
  handler: async (ctx, args) => {
    return await ctx.runQuery(components.brightData.lib.getPage, {
      url: args.url,
    });
  },
});
```
```tsx
// React component
const page = useQuery(api.myFunctions.getCachedPage, { url: "https://example.com" });
// page.content    — raw HTML or text content of the scraped page
// page.isFresh    — true if still within TTL
// page.fetchedAt  — timestamp of last scrape
```

### Invalidate cache
```ts
// Force a fresh fetch on the next call by deleting the cached entry
export const invalidateCache = action({
  args: {
    query: v.optional(v.string()),
    url: v.optional(v.string()),
  },
  handler: async (ctx, args) => {
    return await brightData.invalidate(ctx, args);
  },
});
```

## How reactive queries work

The component stores search results and scraped content in its own Convex tables. When you call `searchWeb` or `scrapePage`, the result is written to those tables. Any frontend component subscribed via `useQuery` on `getCachedSearch` or `getCachedPage` receives the update instantly — no polling, no websocket management, no extra infrastructure. This is standard Convex reactivity applied to external data.
```
User triggers searchWeb action
        ↓
Bright Data SERP API called
        ↓
Result stored in component-owned searches table
        ↓
All useQuery subscribers notified automatically
        ↓
UI updates in real time
```

## API

### `BrightData` class

| Method | Description |
|--------|-------------|
| `search(ctx, { query, vertical?, recency?, ttlMs? })` | Search via Bright Data SERP API. Returns cached results if fresh. |
| `scrape(ctx, { url, ttlMs? })` | Scrape a URL via Web Unlocker. Returns cached content if fresh. |
| `invalidate(ctx, { query?, url? })` | Delete cached entry to force re-fetch on next call. |

### Reactive queries (call via `ctx.runQuery`)

| Function | Description |
|----------|-------------|
| `components.brightData.lib.getSearch({ query })` | Get cached search result. Returns `null` if not yet fetched. |
| `components.brightData.lib.getPage({ url })` | Get cached page content. Returns `null` if not yet fetched. |

Both return `{ results/content, fetchedAt, expiresAt, isFresh }`.

### `search` verticals

Pass a `vertical` string to search specific Google verticals:

| Value | Vertical |
|-------|----------|
| _(omit)_ | Web search |
| `nws` | News |
| `shop` | Shopping |
| `isch` | Images |
| `vid` | Videos |

### `search` recency

Pass a `recency` string to filter by date:

| Value | Range |
|-------|-------|
| `d` | Past day |
| `w` | Past week |
| `m` | Past month |
| `y` | Past year |

## Example app

See [`example/`](./example) for a working Vite + React demo showing live SERP search and page scraping with reactive cache display.

## Development
```sh
npm i
npm run dev
```

## License

Apache-2.0