Error "selenium.common.exceptions.WebDriverException" although app works fine locally

[jrodriguez5909]

I’ve got this app I can run just fine locally but I get an error message seemingly pertaining to selenium.common.exceptions.WebDriverException Here’s the snapshot of the error message, would be great if someone could help solve what’s going wrong :

[mathcatsand]

If you use Selenium, you have to make sure you have all the non-Python dependencies taken care of. Have you configured your packages.txt correctly to install the needed binaries?

Here’s an example within another thread discussing Selenium:

Selenium web scraping on streamlit cloud

Here’s a minimal example of running Selenium on Streamlit Cloud: import streamlit as st import os, sys @st.experimental_singleton def installff(): os.system('sbase install geckodriver') os.system('ln -s /home/appuser/venv/lib/python3.7/site-packages/seleniumbase/drivers/geckodriver /home/appuser/venv/bin/geckodriver') _ = installff() from selenium import webdriver from selenium.webdriver import FirefoxOptions opts = FirefoxOptions() opts.add_argument("--headless") browser = webdriver.Fire…

[jrodriguez5909]

Thanks @mathcatsand - I’ve tried many approaches to this and still, no luck. I’ve managed to get past the error I shared before but now I’m getting an odd error where apparentlyAttributeError: 'WebDriver' object has no attribute 'find_elements_by_xpath' although (again) the same code works perfectly fine when I run it locally.

Any thoughts?

[mathcatsand]

Can you link to your repository to share your code and environment setup?

[jrodriguez5909]

Thanks @mathcatsand - here’s a link to the GitHub repo!

I’ve now gotten past the above issue but now have another challenge. Using the same inputs both locally hosted and hosted through Streamlit, I get scraped results in the locally run version but no results through Streamlit. Would be great if you have any idea why this might be happening! Screenshots below:

Hosted on Streamlit

Hosted Locally

[mathcatsand]

Just as a quick test, can you take a look at @Franky1’s example and try out his lines of code in place of your main app? That way, we can confirm your environment (as configured) separate from your code before potentially going down a rabbit hole trying to find some specific line that is incompatible with the cloud environment. (Sometimes it’s something as simple as changing out the urls if your local environment is in a different geographical region that Streamlit Cloud’s servers, so it’s good to just verify from the basics.)

His example uses chromium and chromium-driver in the packages.txt file, same as yours. So it should be compatible with the environment you specified for your own cloud deployment.

Issue with Selenium on a Streamlit app

@randyzwitch After taking a closer look at SeleniumBase, I don’t know how to use it for webscraping. It’s a pytest framework, but I haven’t found an example of how to use the classes in a python script for webscraping. Do you have a small example? @KoxNoob Also, I had massive problems getting Selenium to work on streamlit sharing. I had imagined it to be easier. Locally on Windows 10 and in a local Docker container, I got it up and running relatively quickly. But the deployment to streamlit …

[jrodriguez5909]

Thanks again for following up @mathcatsand! At this point I think I’ve scoured through all streamlit discussions and I’ve spent many hours trying every possible option available and still, no luck. For instance, I created this other version of the app reflecting @Franky1’s approach you recommended above and I’m still getting errors.

You’ll note I’ve embedded elements of lines 9-40 in @Franky1’s streamlit_app.py script in lines 23-32 and 127-151 in my streamlit_app.py script. Below is the error message I’m seeing and I’m wondering how @Franky1’s script uses the items in packages.txt since there doesn’t seem to be a reference to Chromium in his streamlit_app.py script, which I think is (at least in part) causing the error I’m seeing.

[ferdy]

I have a sample simple code that would run both locally and in cloud. It might help.

github
app

[jrodriguez5909]

Hey @ferdy I can’t see either of the links you’ve provided, perhaps this a private app/repo?

[ferdy]

Sorry, it is public now.