Skip to main content

Capture text: How to set up pagination when extracting data 'From a list'

Automatically navigate through multiple pages to capture complete datasets when using "From a list" extraction.

M
Written by Melissa Shires
Updated over a week ago

βœ… This article covers how to configure pagination settings when:

  • Using "Capture Text" β†’ "From a list"

  • Extracting data that spans multiple pages of the same list

❌ This article does not include how to capture data across multiple pages for:

  • "Just text" captures

  • Screenshots

  • Individual detail pages

πŸ’‘ You can scrape and extract data across thousands of web pages by connecting multiple robots using our workflows feature.

Types of pagination

After selecting your list, choose how the robot navigates:

What you see on the page

Select this option

How it works

"Next" button, arrow (β†’), page numbers (1,2,3)

Click "next"

Robot clicks through each page

"Load more" or "Show more" button

Click "load more"

Robot clicks to add items to current view

Items appear as you scroll, no buttons, infinite scroll

Scroll down

Robot scrolls to trigger loading

All items already visible

No more items

Robot captures once and stops

Setting up pagination

Step 1: Start training your robot

To start training your robot, all you'll need is the URL you'd like to scrape or monitor.

  1. From your Browse AI dashboard, click "Build New Robot".

  2. Select either:

    1. Extract structure data - if you'd like to scrape data from a web page

    2. Monitor site changes - if you want to create a web monitor.

  3. Enter the Origin URL you would like to scrape or monitor.

  4. Click Start Training Robot.

  5. Select Use Robot Studio and wait for your web page to load.

Step 2: How to scrape data from a page 'From a list'

  1. Click on Capture Text, and select From a list.

  2. Hover over the list of items on the page until you see a dotted outline around the elements you want to capture.

  3. Click to select the list when the outline matches your desired data set.

  4. Robot studio will automatically structure that data into a recommended dataset (you can customize this if needed, see below).

  5. Give your list a descriptive name.

  6. Select the number of items you'd like the robot to capture.

  7. Configure the pagination settings to capture additional list items. These include:

    1. Clicking through 'next' buttons.

    2. Click "load more items"

    3. Infinite scroll (i.e. scroll up or down to load additional items)

    4. No more items to load.

  8. Click 'Save Captured List'.

  9. Click 'Finish' to finish recording your robot if you've captured all of the data you need, or keep capturing text or screenshots.

  10. Name your robot to run it, review the data and approve it.

Troubleshooting

Problem

Solution

Only captures first page

Try "Load more" instead of "Click next"

Missing items between pages

Items might be dynamic - try "Scroll down"

Takes too long

Reduce page limit or break into batches

Duplicates in results

Normal - deduplicate in spreadsheet

Other pagination scenarios

"I need to search first, then paginate results"

"I need details from each item's individual page"

"I need to paginate AND get details"

"I have multiple categories to paginate through"

Watch & learn


Did this answer your question?