Infinite scroll and lazy loading are common where content loads dynamically as you scroll down the page. Browse AI can extract data from these pages by recording your scrolling behavior and automatically detecting the loading pattern.
Understanding infinite scroll vs pagination
Infinite scroll: New content automatically loads as you reach the bottom of the page (like Twitter, Instagram, or LinkedIn feeds).
Lazy loading: content loads only when it becomes visible on the screen (often triggered by scrolling), often used for images and product listings.
Traditional pagination: Uses "Next" buttons or "Load more" clicks (also supported by Browse AI).
Training your robot for infinite scroll
To capture a list of items or results that are generated via infinite scroll, you'll want to train a robot to capture data 'as a list' and then configure the pagination settings.
Step 1: Start training your robot
From your Browse AI dashboard, click Build New Robot
Select Extract structured data
Enter the Origin URL you want to scrape
Click Start Training Robot
Select Use Robot Studio and wait for your page to load
Step 2: Capture data from an infinite scroll list
Click Capture Text and select From a list
Hover over the list items until you see a dotted outline around the elements you want
Click to select the list when the outline matches your desired data
Robot Studio will automatically structure the data into a recommended dataset
Scroll down slowly to load more items and demonstrate the scrolling behavior
Give your list a descriptive name
Select the number of items you want the robot to capture
Configure pagination settings and choose Infinite scroll from these options:
Next buttons (for traditional pagination)
Load more items (for click-to-load buttons)
Infinite scroll (scroll up or down to load additional items)
No more items to load (for static lists)
Click Save Captured List
Click Finish if you've captured all needed data
Name and approve your robot
How Browse AI handles infinite scroll
When you select "Infinite scroll" as your pagination type:
The robot will automatically scroll to load new content.
It continues until it reaches the number of items you specified.
When to use "Load more items" instead
If the page has a "Load more" or "Show more" button:
Click the button during training
Select "Load more items" as pagination type
Often more reliable than infinite scroll
When to use traditional pagination
If the site offers numbered pages or "Next" buttons:
Click through pages during training
Select "Next buttons" as pagination type
Can be faster than infinite scroll for large datasets
Training your robot for lazy loading, and scrolling
Your robot is trained to mimic human actions (scroll, pause, hover, click). When you're training your robot navigate throughout the page as you would naturally to click, scroll, and capture the data you want to extract.
During training
Demonstrate the scrolling: scroll down during training to show the robot how content loads.
Move slowly: give content time to load before continuing.
For lazy-loaded content
Scroll to trigger loading during training.
Wait for images or content to fully appear.