How to change ip in Streamlit Cloud when Selenium is running

[Theo_K]

Summary

Hello, I am using Selenium on Streamlit Cloud , And everything is working fine in local. but i think on the Streamlit the ip address will be change to “35.230.56.30”. Therefore some web will detect and block my crawler.

My Question is:
How to change my ip address when I use Selenium to craw the web?

[Charly_Wargnier]

Hello @Theo_K!

I haven’t personally tried it, but when initializing your Selenium WebDriver instance, it may be possible to configure it to use a proxy server.

Have you tried this?

Best,
Charly

[Theo_K]

Yes, I’ve tried proxy in my selenium, and when I run selenium with proxy in local is okay, but on the streamlin cloud, the selenium can’t connect proxy correctly.

[Theo_K]

Someone could please help me… I am so comfused for a ling time

[Charly_Wargnier]

Thank you for your feedback, @Theo_K!

The configuration for running Selenium on Streamlit Community Cloud is different from running it on your local machine. Here is a minimal example of how to run Selenium on CCloud:

Selenium web scraping on streamlit cloud

Here’s a minimal example of running Selenium on Streamlit Cloud: import streamlit as st import os, sys @st.experimental_singleton def installff(): os.system('sbase install geckodriver') os.system('ln -s /home/appuser/venv/lib/python3.7/site-packages/seleniumbase/drivers/geckodriver /home/appuser/venv/bin/geckodriver') _ = installff() from selenium import webdriver from selenium.webdriver import FirefoxOptions opts = FirefoxOptions() opts.add_argument("--headless") browser = webdriver.Fire…

I hope this helps.

Best wishes,
Charly