What is the error message?
No error message… Just the scrap doesn’t work… While locally, everything is good…
Are you sure?
I don’t mean the app itself, but please check the Streamlit Sharing console for error messages.
Indeed, no error message in the log.
I’m going to explain you what is the aim : i try to collect text and odds from bookmakers website (Winamax, Unibet, Betclic, etc in my previous example) using Selenium.
So, normally, if all works, I should have numbers instead of nan in the dataframe…
After taking a closer look at SeleniumBase, I don’t know how to use it for webscraping. It’s a pytest framework, but I haven’t found an example of how to use the classes in a python script for webscraping. Do you have a small example?
Also, I had massive problems getting Selenium to work on streamlit sharing. I had imagined it to be easier. Locally on Windows 10 and in a local Docker container, I got it up and running relatively quickly. But the deployment to streamlit sharing of the app itself kept going wrong without any plausible error message.
My efforts for a very very simple example (proof of concept) can be found here:
By the way, thanks a lot @Franky1 !
I guess I was viewing this as a roundabout way of getting Selenium installed. SeleniumBase is used for testing, but by installing that, I would assume that the other Selenium package would then be able to detect the proper driver and make things happen.
In general, I’ll pass along the feedback to our product team about “What if there isn’t an apt package?” as part of the install process, not sure where adding custom apt repositories might be on the roadmap (if at all).
Hey @KoxNoob! I was wondering if you found any solution since I plan to use Selenium to do some scraping with Streamlit Sharing. I have the same issue as you, where the code runs locally but not when deployed online.
I honestly think we need a browser installed in the github repo somehow so the webdriver can use it, but I am unsure how that will work.
Given that we can now use
conda to install packages on Streamlit Share (see it here), we can install
packages.txt (as mentioned by @ferdie you need a browser to be installed for the webdriver) and install
geckodriver through conda (here’s the package).
Then we need to hardcode the path to
geckodriver in the conda environment:
firefoxOptions = Options() firefoxOptions.add_argument("--headless") driver = webdriver.Firefox( options=firefoxOptions, executable_path="/home/appuser/.conda/bin/geckodriver", ) driver.get(URL)
geckodriver, as opposed to an executable pushed from Github repo, can be used by the app and…I think it works, I think it works I think it does, at least I can query the Unibet tables I’ll let you test further.
I can’t guarantee it will work forever if the Cloud team decides to change the path to the conda environment then you’ll have to look for it.
This is exciting work, thank you so much @andfanilo!
By any chance do you know the package name to install the Google Chrome web browser? My project used the Google chromedriver, and it’ll be a little tedious to switch over to Firefox.
Yeah Chrome I think is going to be a little-ish harder. I don’t think you can easily install Chrome on Debian Buster (which is the OS for Streamlit Share apps), you may have better chance installing
packages.txt (chromium is the open source base for Chrome), then chromedriver the same way as
geckodriver (hardcoding the path to the conda installed chromedriver in your app).
Hopefully Selenium knows how to pick the installed
chromium without playing with the options, otherwise you’ll have to find a way to specify using chromium. Haven’t tested, hope you can find it out this way!
I’ll try later maybe.
Add to your
See a simple example project here:
Hi @andfanilo! With the new release you cannot use more than one package manager at a time. Either you use pip (requirements.txt) or you use conda (environment.yml). You won’t be able to use both though.
Sounds good to me, conda and pip install conflicts has always been hard to debug
Hey Fanilo, so I’m currently in the process of debugging, and I was wondering if you can help me.
I completely switched to FireFox since I ran into issues with chromium still, but now I am currently running into this exception when trying to run the function you made to test selenium from your app.
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
For context, I copied over your code (some code is omitted) into a function
test_st() found in
URL = "https://www.unibet.fr/sport/football/europa-league/europa-league-matchs" XPATH = "//*[@class='ui-mainview-block eventpath-wrapper']" TIMEOUT = 40 firefoxOptions = Options() firefoxOptions.add_argument("--headless") driver = webdriver.Firefox( options=firefoxOptions, executable_path="/home/appuser/.conda/bin/geckodriver", ) driver.get(URL) try: WebDriverWait(driver, TIMEOUT).until( EC.visibility_of_element_located((By.XPATH, XPATH,)) ) ... etc
And if you were to press the button
test fanilo's function here on the sidebar of my app, it would run your code and run into the exception. Locally it does work, but not through Streamlit Sharing.
This is a bigger project, and I was hoping to isolate if if there was an issue with my code, or if it’s just how I set up everything in my directory.
Any help would be appreciated, thank you again!
I am trying to clone the latest update of the code in your github, for some reason, I got a permission denied error while making this app again on streamlit cloud:
I was using the same packages.txt and requirements and try every python version. really don’t know what it went wrong.
Would you mind to help me?
Thanks & Best Regards,
I confirm this, the project seems to be broken if you run a fresh install on streamlit cloud.
The root cause seems to be that access to the
/tmp folder is restricted.
IMHO headless webscraping on streamlit cloud is always a pain…
I had no luck with firefox/geckodriver in the past and used chromium/chromedriver which seems to be more stable.
You cloud try my example, which still works
Thanks a lot for your help. I tried your code and unfortunately this time it gives me a “selenium.common.exceptions.WebDriverException: Message: unknown error: cannot create temp dir for user data dir” error. I searched online and it seems like it is due to a change with recent version of chrome url, I wonder what is the chrome driver that you were using so that I can specify in the packages?
Thanks & Regards,
I did not specify any version in
packages.txt i just redeployed my github repo on streamlit cloud and it seems to work. Btw, i used Python 3.9 as runtime on streamlit cloud.