β This article covers how to configure pagination settings when:
Using "Capture Text" β "From a list"
Extracting data that spans multiple pages of the same list
β This article does not include how to capture data across multiple pages for:
"Just text" captures
Screenshots
Individual detail pages
π‘ You can scrape and extract data across thousands of web pages by connecting multiple robots using our workflows feature.
Types of pagination
After selecting your list, choose how the robot navigates:
What you see on the page | Select this option | How it works |
"Next" button, arrow (β), page numbers (1,2,3) | Click "next" | Robot clicks through each page |
"Load more" or "Show more" button | Click "load more" | Robot clicks to add items to current view |
Items appear as you scroll, no buttons, infinite scroll | Scroll down | Robot scrolls to trigger loading |
All items already visible | No more items | Robot captures once and stops |
Setting up pagination
Step 1: Start training your robot
To start training your robot, all you'll need is the URL you'd like to scrape or monitor.
From your Browse AI dashboard, click "Build New Robot".
Select either:
Extract structure data - if you'd like to scrape data from a web page
Monitor site changes - if you want to create a web monitor.
Enter the Origin URL you would like to scrape or monitor.
Click Start Training Robot.
Select Use Robot Studio and wait for your web page to load.
Step 2: How to scrape data from a page 'From a list'
Click on Capture Text, and select From a list.
Hover over the list of items on the page until you see a dotted outline around the elements you want to capture.
Click to select the list when the outline matches your desired data set.
Robot studio will automatically structure that data into a recommended dataset (you can customize this if needed, see below).
Give your list a descriptive name.
Select the number of items you'd like the robot to capture.
Configure the pagination settings to capture additional list items. These include:
Clicking through 'next' buttons.
Click "load more items"
Infinite scroll (i.e. scroll up or down to load additional items)
No more items to load.
Click 'Save Captured List'.
Click 'Finish' to finish recording your robot if you've captured all of the data you need, or keep capturing text or screenshots.
Name your robot to run it, review the data and approve it.
Troubleshooting
Problem | Solution |
Only captures first page | Try "Load more" instead of "Click next" |
Missing items between pages | Items might be dynamic - try "Scroll down" |
Takes too long | Reduce page limit or break into batches |
Duplicates in results | Normal - deduplicate in spreadsheet |
Other pagination scenarios
"I need to search first, then paginate results"
β See Workflows: Search and extract
"I need details from each item's individual page"
β See Deep scraping guide
"I need to paginate AND get details"
"I have multiple categories to paginate through"
β See Bulk run with pagination
Watch & learn
