Skip to main content
All CollectionsRobotsBuilding robots
How to scrape and monitor data behind a login
How to scrape and monitor data behind a login

Your robot can log into websites to extract data using either session cookies or credentials for direct login.

Nick Simard avatar
Written by Nick Simard
Updated this week

Browse AI offers two ways to have your robot log into a website on your behalf:

  1. Using your session cookies (recommended for most cases)

  2. Using your user credentials (username and password)

Method 1: Login with session cookies

Session cookies store information about your login status. When you're already logged into a website in your browser, the robot can use these cookies to automatically authenticate without extra steps.

How to set up cookie-based login

  1. When creating your robot enable the This website needs logging in option

  2. Choose Login with my session cookies

  3. Your session cookies will be safely encrypted and stored on Browse AI's AWS infrastructure

  4. Proceed to capture the data you need to extract or monitor

Benefits of using cookies

  • Eliminates the need for typing credentials or extra clicks

  • Can work with two-factor authentication (2FA) in many cases

  • Generally provides a higher success rate for data extraction

  • Fewer steps means fewer potential points of failure

Limitations of cookies

  • Some secure websites may reject cookies if they come from a different IP address

  • Cookies can expire, requiring periodic updates

  • Not guaranteed to work with all 2FA/MFA implementations

Maintaining cookie-based login

To ensure continued access, regularly update your session cookies:

  1. Go to your approved robot

  2. Navigate to the Settings tab

  3. In the Authentication section, click Update Session Cookies

Method 2: Login with credentials

How to set up credential-based login:

  1. When setting up your robot, enable the This website needs logging in option

  2. Choose Login with my password

  3. While recording, complete the login process as you normally would

  4. Your credentials will be securely encrypted and stored on Browse AI's AWS infrastructure

  5. Capture the data you need to extract or monitor

Benefits of using credentials

  • Works on websites that have IP-based cookie restrictions

  • Can be more reliable for some websites with complex security measures

  • Doesn't require periodic updates like cookies might

Limitations of credentials

  • Additional interactions (typing, clicking) introduce more potential failure points

  • May not work well with websites that frequently change their login interface

  • Can be problematic with sites that use 2FA/MFA

  • May trigger security alerts if the login occurs from different IP addresses

Improving credential-based login

If you're experiencing issues with credential login:

  1. Re-train your robot to use only necessary clicks and keyboard inputs:

    • Go to your approved robot

    • Navigate to the Settings tab

    • In the Danger Zone section, click Re-train robot

  2. Keep your login process as simple as possible by eliminating unnecessary steps

Choosing the right login method

Cookie method

Credential method

Website security level

Standard security

Higher security (restrict cookies)

Stability

Higher stability (less steps)

Less stable (more steps)

2FA/MFA

Works in most cases

Often problematic

Maintenance

Requires periodic updates

Less maintenance

Security considerations

How Browse AI protects your login information

All login information (both cookies and credentials) is:

  • Encrypted during transmission and storage

  • Stored on secure AWS infrastructure

  • Only used for the purpose of running your robots

Website detection risks

While Browse AI employs techniques to mimic human browsing, some websites with strict security may detect automated logins, especially when:

  • Logins occur from multiple IP addresses

  • Multiple requests are made in short succession

  • Interaction patterns differ from typical human behavior

If your automation needs pose this risk, consider local automation, which utilizes your personal IP address, for enhanced discretion. For public data extraction, our robots are a reliable and secure solution.

Troubleshooting login issues

If your robot fails to log in:

  1. Check if the website has changed its login interface

  2. Verify your credentials are still valid by logging in manually

  3. For cookie-based login, try updating your session cookies

  4. For credential-based login, consider re-training your robot with simpler steps

  5. Some websites may use CAPTCHA or other anti-bot measures that can interfere with automated login

Did this answer your question?