What websites work well
Custom robots
Browse AI successfully extracts data from (almost) any websites, including:
E-commerce platforms
Business directories
Social media sites
News and content websites
Real estate listings
...and billions more.
Our success rate for newly created custom robots is over 90%.
Prebuilt robots
Some of our prebuilt robots are specifically designed to solve for limitations that our no code platform has out of the box with popular sites.
Managed services
While some sites can't be scraped out of the box, some require custom code, configuration or data transformation. We offer all of these services as part of our managed services offering.
Current limitations
Custom CAPTCHAs
We support solving several types of standard CAPTCHA (e.g., ReCaptcha, hCaptcha), but not all of them. For example, we currently do not solve custom captchas.
Expert bot detection
Some sites have really strong bot detection mechanisms. While our platform tries to mimic human behavior, and has bot evasion built-in - for some sites that have high security measures this doesn't work.
This is especially true for high security sites that require a login to access the web data.
Here's why:
Browse AI tries to mimic human behaviour and uses a browser session just like a regular user, with the same delays, pauses, and scrolling.
We also rotate through IP addresses (usually located in the same country as the user who set up the robot) and almost never use the same IP address twice.
For most websites this is enough - but if your robot has to log into the website (either using your session cookies or your login credentials), the website can always detect two things:
This user is logging in from at least two different IP addresses (your local IP address and Browse AI's IPs)
If your robot is running a lot of tasks, it can seem suspicious.
MFA (multi-factor authentication) and 2FA (two-factor authentication)
Note that neither of these are supported by Browse AI.
Technical considerations
Sites rendering virtual lists
Some websites use virtual lists that only render visible content during scrolling. This is not currently supported by Browse AI.
iFrames
Direct interaction with iFrames isn't currently supported. However, you can often work around this by:
Locating the iFrame's direct URL
Creating a separate robot for that URL
Extracting data from the direct source
Tips for success
Test your robot thoroughly before scaling
Use the "Report" button if you encounter issues
Consider our managed services for complex extraction needs
Monitor your extraction frequency to avoid triggering security measures