Every selenium example failed

[IlIl]

Hello Streamlit team,

I have been using your platform for quite some time, and I find it excellent. I am interested in deploying Selenium on your cloud. I have tried all the Selenium examples provided by you on the cloud, but I keep encountering errors. Could you please provide me with a working Selenium code snippet?

[Franky1]

GitHub - Franky1/Streamlit-Selenium: Streamlit project to test Selenium...

[IlIl]

Bro your selenium example also not working…

[IlIl]

Um…??

[dataprofessor]

Hi @IlIl

A while back, I’ve created this Streamlit app screenshot app that uses selenium.

Here’s the demo app https://screenshot.streamlit.app/

And the repo GitHub - dataprofessor/app-screenshot: Streamlit App Screenshot

The library dependencies as defined in requirements.txt file is as follows:

streamlit
selenium==4.11.2
webdriver-manager
psutil

[Franky1]

@IlIl

your selenium example also not working…

The app worked stably for a very long time and still works in principle. But the target website has probably activated geoip blocking in the meantime. As a result, the content can no longer be viewed from the servers on which streamlit cloud is hosted. The app still worked locally in a docker container.

I have now reworked the app and added more debugging and error handling. I have also added the use of a proxy to bypass the geoip blocking. This means that the app works again on streamlit cloud. However, since only free proxies are used, this does not always work reliably.

GitHub - Franky1/Streamlit-Selenium: Streamlit project to test Selenium...

Selenium Test

[IlIl]

WORKING PROFESSOR q

[IlIl]

NOW YOUR EXAMPLE ALSO WORKING FRANKY BRO #streamlitsupercoolplatform

[dataprofessor]

Glad to hear that it’s working for you!

[Tim_Shurlock]

Thank you this is awesome, I am having the same problem as @IlIl using selenium on community cloud. I have forked you app and have it running successfully on com cloud. But having trouble finding the element that I need (and works in my locally run version).

driver = get_driver()
runner = str(47595)
ParkrunURL = “https://www.parkrun.org.uk//parkrunner//” + runner + “//all//”
driver.implicitly_wait(10) # Wait for up to 10 seconds
try:
table = driver.find_element(By.XPATH, “(//table)[3]”)
st.write(‘we got it’)
except:
st.write(‘this table was not found’)

And don’t know if this is a Selenium problem or Streamlit problem?

I have also tried

table = driver.find_element(By.CLASS_NAME, “sortable”)

table_element = driver.find_element_by_id(“results”)

tables = driver.find_element(By.ID, “results”)
table = tables.find_element(By.CLASS_NAME,“sortable”)

[Franky1]

Debugging selenium running on streamlit cloud is a bit of a nightmare…
Make sure that you actually get the html page you want and not a 403 or 404 response.
Could be that the target webserver has ip/geoip blocking enabled.

[Tim_Shurlock]

Thanks, I have hardcoded the URL and added

st.write(driver.current_url)

which runs without error and writes the correct URL on the app.

And the ‘try’ block works if I just ask it to write something, but if I try any ‘find_element’. I get this error

raise WebDriverException(f"Service {self._path} unexpectedly exited. Status code was: {return_code}")

selenium.common.exceptions.WebDriverException: Message: Service /home/appuser/.cache/selenium/chromedriver/linux64/122.0.6261.94/chromedriver unexpectedly exited. Status code was: 1

I have also tried adding driver.implicitly_wait(10) and copying the short and full XPath from the website, but still no joy

table= driver.find_element(By.XPATH, “/html/body/main/div/div/div/div[3]/table”)
table = driver.find_element(By.XPATH, ‘//*[@id=“results”]’)

Could be that the target webserver has ip/geoip blocking enabled.

Is there a way to find this out, and is there any way around it?

Thanks

[Franky1]

@Tim_Shurlock

which runs without error and writes the correct URL on the app.

This does not mean that you get the desired content of the website. Look at the actual html content you get and the status code. To do this you need to add more debug code.

@Tim_Shurlock

Is there a way to find this out

Look at the actual content you get back and the status code.

@Tim_Shurlock

and is there any way around it?

a proxy