Can my robot login to websites? (Cookies vs. Credentials)
Yes, there are two ways to have your robot log into a website on your behalf: using your session cookies or your user credentials.
Jump to topic:
A. Have your robot login via Session Cookies
Before starting to record, enable the "This website needs logging in" option, and then choose "Login with my session cookies":
Your session cookies will then be safely encrypted and stored on Browse AI AWS infrastructure. Afterward, proceed to capture the data you are looking to extract or monitor.
This works on most websites, but more secure sites may not accept the session cookies if they come from a different IP address. In those cases, approach B may be the only option.
Cookies store information about your login status. If you're already logged into your account in your browser, selecting the session cookies login method can automatically authenticate the robot. This eliminates the need for extra clicks and typing, potentially enhancing the success rate of data extraction.
Cookies can also work for websites with two-factor authentication (2FA) or multi-factor authentication (MFA). However, this is not a guaranteed solution, as it can vary depending on the specific website's implementation and security measures. If you're working with a site that uses 2FA/MFA, please proceed with an experimental mindset.
Despite their convenience, cookies come with their own set of caveats:
- Session/expiry Issues: Cookies often have limited lifespans. Some expire in less than a day, requiring you to update them regularly to keep your robot logged in.
- IP address sensitivity: Some websites tie cookies to specific IP addresses. If your robot attempts to use a cookie from a different location, it might be rejected, even if the cookie itself is still valid.
How do we address cookie-related issues?
To potentially overcome these challenges:
- Regularly update session cookies: If your robot relies on cookies, make a habit of refreshing them within the robot's settings to ensure they remain valid. The fresher the cookies, the better. 🍪 Here’s how:
- Approve your robot.
- Navigate to the Settings tab of your robot
On the Authentication section, click on the “Update Session Cookies” button, like so:
- Fallback to user-credentials method: In situations where cookie-based login fails due to expiration or IP restrictions, you may need to create a robot that utilises your user credentials.
B. Have your robot log in with your User Credentials
Before starting to record, enable the "This website needs logging in" option, then choose "Login with my password":
While recording, login like you normally would. The robot will then record those actions, and securely encrypt and store your credentials on Browse AI AWS infrastructure. Afterward, capture the data you are looking to extract or monitor.
After you finish recording and building the robot, it will perform the same steps and log in as you did.
While simple, it can present a few challenges:
- Unnecessary interactions: Logging in typically requires typing your credentials and clicking buttons. These extra steps introduce potential points of failure, such as typos, especially if the website's layout changes or the robot encounters unexpected elements. (E.g., A/B test websites, etc.)
How do we mitigate the drawbacks of using credentials?
If you find that the user credentials approach is causing issues, you can either:
- Re-train your robot to only use concise clicks and keyboard inputs as you log in and navigate through your target page, and make sure it's only the necessary ones - as the fewer they are, the better. Here's how to re-train your robot:
- Approve your robot.
- Navigate to the Settings tab of your robot
On the Danger Zone section, click on the "Re-train robot" button like so:
- Consider creating a new robot that utilises cookies for login, as this can potentially streamline the process and improve reliability:
The ideal login method depends on your specific circumstances or use case. If your website's login process is straightforward and stable, using credentials might be sufficient. However, if you prioritise minimising interactions and potential errors, cookies could offer a smoother experience.
Keep in mind that when extracting data behind logins:
Cloud automations involving login can potentially be detected.
Browse AI employs various techniques to mimic human browsing behaviour, such as using a standard browser interface, interacting with website elements naturally, and rotating IP addresses through proxy servers.
However, cloud-based automation, where logins occur from multiple IP addresses, can raise suspicion on sites with stringent security measures. This may result in warnings or account restrictions.
If your automation needs pose this risk, consider local automation, which utilises your personal IP address, for enhanced discretion. For public data extraction, our robots are a reliable and secure solution.
By understanding the pros and cons of each method and implementing the suggested solutions, you can optimise your robot's login process and ensure reliable data extraction from your Origin URLs.