Skip to main content

How to scrape data with Claude, ChatGPT or another AI agent using Browse AI

Give your AI coding agent, like Claude Code, Codex or Cursor, a ready-made prompt and your Browse AI API key, and it builds your scraping workflow on the Browse AI API for you.

M
Written by Melissa Shires

You can use an AI agent to scrape data from websites with Browse AI. The agent writes the code, and Browse AI robots do the scraping: they open the page, extract the data you asked for, and run again whenever you need fresh data.

The quickest way to start is the Use with your AI agent card on your API keys page. It gives you a prompt to paste into your AI coding agent, like Claude Code, Codex or Cursor, so the agent reads the Browse AI API reference and follows it when it builds your integration.

This guide covers:

  • Setting up your agent with an API key and the ready-made prompt.

  • Path 1: Use robots you already have. Run tasks, set up monitoring and add webhooks.

  • Path 2: Create robots from code with the Robot Drafts API (Beta).

  • Using chat assistants like Claude or ChatGPT to analyze the data.

What you'll need

  • A Browse AI account. API access is included on every plan.

  • An AI agent that can write and run code, like Claude Code, Codex or Cursor.

  • For path 1, at least one approved robot. For path 2, nothing else: the agent can create the robot for you.

Step 1: Create an API key

  1. In your Browse AI dashboard, go to API keys (dashboard.browse.ai/api).

  2. Click Create API key and give it a name you'll recognize, like "Claude Code".

  3. Copy the key and keep it somewhere safe.

Give your agent the key through an environment variable called BROWSE_AI_API_KEY, not by pasting it into the chat. For example, in your terminal:

export BROWSE_AI_API_KEY="YOUR_SECRET_API_KEY"

The prompt in the next step tells the agent to read the key from this variable and never hard-code, log or commit it.

Step 2: Copy the prompt for your agent

On the same API keys page, find the Use with your AI agent card and click Copy prompt. The prompt looks like this:

I want to use the Browse AI API to work with my robots and their data.Before writing any code, read the Browse AI API reference: https://docs.browse.ai/llms.txtFollow these rules:
- Use only the endpoints, parameters and response fields documented in the reference. Don't guess.
- Authenticate every request by sending my API key as a Bearer token in the Authorization header.
- Read the API key from the BROWSE_AI_API_KEY environment variable. Never hard-code, log or commit it.
- If you need a robot ID and I haven't given you one, list my robots through the API and ask me which one to use.Here's what I want to build:

Here's what each part does:

  • The API reference link points the agent at a version of the Browse AI docs written for AI agents, so it works from the real endpoints instead of guessing.

  • "Use only the endpoints… documented" keeps the agent from inventing endpoints or fields that don't exist.

  • Bearer token and BROWSE_AI_API_KEY tell it how to sign in to the API and keep your key out of your code.

  • "List my robots… and ask me" means the agent checks which robots you have before it runs anything.

Step 3: Paste it into your agent and say what you want

Paste the prompt into Claude Code, Codex, Cursor or any other AI agent that can call an API, then finish the last line with what you want built. Be specific about the website, the data and where it should go.

For example:

  • "Here's what I want to build: a script that runs my 'Competitor prices' robot every morning and saves the results to a CSV file."

  • "Here's what I want to build: a small app that scrapes the job listings on a careers page I give it and shows them in a table."

  • "Here's what I want to build: a webhook receiver that posts a Slack message whenever my monitor finds a price change."

The agent reads the docs, lists your robots if it needs one, and writes the code. Review what it builds and run it the way you'd run any other code your agent writes.

Path 1: Use robots you already have

If you've already built robots in the dashboard, your agent can work with them through the API. It can:

  • Run tasks on a robot, for one page or many, and fetch the extracted data when each task finishes.

  • Set up monitoring so a robot runs on a schedule and keeps your data fresh.

  • Add webhooks so your code is notified the moment a task finishes or a monitor finds a change, instead of checking again and again.

Example requests:

  • "Use my 'Product listings' robot to scrape these 20 category pages and combine the results into one JSON file."

  • "Set up a daily monitor on my 'Competitor prices' robot and add a webhook that calls my endpoint when the data changes."

Tip: A robot isn't tied to one page. Your agent can run the same robot with a different originUrl to get the same data from another page on that site.

Path 2: Create robots from code (Beta)

Your agent doesn't need a robot to exist first. With the Robot Drafts API (Beta), it can create one from a webpage and a description of the data you're after:

  1. It creates a draft with the page URL and what you want from it, in plain language, as a list of columns, or both.

  2. AI reads the page and extracts sample data as a preview.

  3. The agent checks the preview and refines the instructions or columns if something is missing or wrong.

  4. It saves the draft as a robot. The robot runs its first task straight away, and from then on works like any robot you built in the dashboard: run tasks, set up monitors and add webhooks.

Example requests:

  • "Create a robot that gets every product on this page with its name, price and link, show me the preview, and save it once the data looks right."

  • "My 'Job listings' robot is missing the salary column. Retrain it to add one."

Ask your agent to show you the preview before it saves, so you can check the data first.

Beta. The Robot Drafts API may change before it's generally available. For a step-by-step example, see API guide: Creating and retraining robots with Robot Drafts (Beta). For every field, see the Robot Drafts API reference.

Using Claude, ChatGPT and other chat assistants

The prompt above is for AI agents that write and run code. If you'd rather use a chat assistant like Claude or ChatGPT to ask questions about your data, summarize it or spot trends, you don't need the API: export your robot's data and upload it to the chat. See How to use Browse AI data with ChatGPT, Claude, and other AI assistants.

Tips for better results

  • Keep the rules in the prompt. They stop the agent from guessing endpoints and keep your API key safe. Add your own request on the last line rather than rewriting the rules.

  • Name the robot if you know it. Otherwise the agent lists your robots and asks which one to use.

  • Start small. Ask the agent to try one page first, check the data, then scale up to many pages or a schedule.

  • Use webhooks for fresh data. If your code needs to react when new data arrives, ask the agent to set up a webhook rather than checking the API over and over.

Related articles

Did this answer your question?