What is the error message?
No error messageā¦ Just the scrap doesnāt workā¦ While locally, everything is goodā¦
Are you sure?
I donāt mean the app itself, but please check the Streamlit Sharing console for error messages.
Thanks again @randyzwitch and @Franky1 !
Indeed, no error message in the log.
Iām going to explain you what is the aim : i try to collect text and odds from bookmakers website (Winamax, Unibet, Betclic, etc in my previous example) using Selenium.
So, normally, if all works, I should have numbers instead of nan in the dataframeā¦
@randyzwitch
After taking a closer look at SeleniumBase, I donāt know how to use it for webscraping. Itās a pytest framework, but I havenāt found an example of how to use the classes in a python script for webscraping. Do you have a small example?
@KoxNoob
Also, I had massive problems getting Selenium to work on streamlit sharing. I had imagined it to be easier. Locally on Windows 10 and in a local Docker container, I got it up and running relatively quickly. But the deployment to streamlit sharing of the app itself kept going wrong without any plausible error message.
My efforts for a very very simple example (proof of concept) can be found here:
2 Likes
I guess I was viewing this as a roundabout way of getting Selenium installed. SeleniumBase is used for testing, but by installing that, I would assume that the other Selenium package would then be able to detect the proper driver and make things happen.
In general, Iāll pass along the feedback to our product team about āWhat if there isnāt an apt package?ā as part of the install process, not sure where adding custom apt repositories might be on the roadmap (if at all).
Best,
Randy
2 Likes
Hey @KoxNoob! I was wondering if you found any solution since I plan to use Selenium to do some scraping with Streamlit Sharing. I have the same issue as you, where the code runs locally but not when deployed online.
I honestly think we need a browser installed in the github repo somehow so the webdriver can use it, but I am unsure how that will work.
Hey all,
Given that we can now use conda to install packages on Streamlit Share (see it here), we can install firefox-esr via packages.txt (as mentioned by @ferdie you need a browser to be installed for the webdriver) and install geckodriver through conda (hereās the package).
Then we need to hardcode the path to geckodriver in the conda environment:
firefoxOptions = Options()
firefoxOptions.add_argument("--headless")
driver = webdriver.Firefox(
options=firefoxOptions,
executable_path="/home/appuser/.conda/bin/geckodriver",
)
driver.get(URL)
This geckodriver, as opposed to an executable pushed from Github repo, can be used by the app andā¦I think it works, I think it works I think it does, at least I can query the Unibet tables 
Iāll let you test further.
I canāt guarantee it will work forever 
if the Cloud team decides to change the path to the conda environment then youāll have to look for it.
App: https://share.streamlit.io/andfanilo/s4a-selenium/main/app.py
Code: GitHub - andfanilo/s4a-selenium: Test Selenium + Firefox on Streamlit Share
Cheers,
Fanilo
This is exciting work, thank you so much @andfanilo!
By any chance do you know the package name to install the Google Chrome web browser? My project used the Google chromedriver, and itāll be a little tedious to switch over to Firefox.
Yeah Chrome I think is going to be a little-ish harder. I donāt think you can easily install Chrome on Debian Buster (which is the OS for Streamlit Share apps), you may have better chance installing chromium in packages.txt (chromium is the open source base for Chrome), then chromedriver the same way as geckodriver (hardcoding the path to the conda installed chromedriver in your app).
Hopefully Selenium knows how to pick the installed chromium without playing with the options, otherwise youāll have to find a way to specify using chromium. Havenāt tested, hope you can find it out this way!
Iāll try later maybe.
Fanilo
2 Likes
Hi @andfanilo! With the new release you cannot use more than one package manager at a time. Either you use pip (requirements.txt) or you use conda (environment.yml). You wonāt be able to use both though.
2 Likes
Sounds good to me, conda and pip install conflicts has always been hard to debug 
Hey Fanilo, so Iām currently in the process of debugging, and I was wondering if you can help me.
I completely switched to FireFox since I ran into issues with chromium still, but now I am currently running into this exception when trying to run the function you made to test selenium from your app.
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
For context, I copied over your code (some code is omitted) into a function test_st() found in sel.py:
URL = "https://www.unibet.fr/sport/football/europa-league/europa-league-matchs"
XPATH = "//*[@class='ui-mainview-block eventpath-wrapper']"
TIMEOUT = 40
firefoxOptions = Options()
firefoxOptions.add_argument("--headless")
driver = webdriver.Firefox(
options=firefoxOptions,
executable_path="/home/appuser/.conda/bin/geckodriver",
)
driver.get(URL)
try:
WebDriverWait(driver, TIMEOUT).until(
EC.visibility_of_element_located((By.XPATH, XPATH,))
)
... etc
And if you were to press the button test fanilo's function here on the sidebar of my app, it would run your code and run into the exception. Locally it does work, but not through Streamlit Sharing.
This is a bigger project, and I was hoping to isolate if if there was an issue with my code, or if itās just how I set up everything in my directory.
Any help would be appreciated, thank you again!
App: https://share.streamlit.io/ftaruc/grailed/main/st-app.py
Code: GitHub - ftaruc/grailed: grailed web application for streamlit
Hi @andfanilo
I am trying to clone the latest update of the code in your github, for some reason, I got a permission denied error while making this app again on streamlit cloud:
I was using the same packages.txt and requirements and try every python version. really donāt know what it went wrong.
Would you mind to help me?
Thanks & Best Regards,
Ed
I confirm this, the project seems to be broken if you run a fresh install on streamlit cloud.
The root cause seems to be that access to the /tmp folder is restricted.
IMHO headless webscraping on streamlit cloud is always a painā¦ 
I had no luck with firefox/geckodriver in the past and used chromium/chromedriver which seems to be more stable.
You cloud try my example, which still works 
GitHub - Franky1/Streamlit-Selenium: Streamlit project to test Selenium running in Streamlit sharing
Hi @Frank1,
Thanks a lot for your help. I tried your code and unfortunately this time it gives me a āselenium.common.exceptions.WebDriverException: Message: unknown error: cannot create temp dir for user data dirā error. I searched online and it seems like it is due to a change with recent version of chrome url, I wonder what is the chrome driver that you were using so that I can specify in the packages?
Thanks & Regards,
Ed
I did not specify any version in packages.txt i just redeployed my github repo on streamlit cloud and it seems to work. Btw, i used Python 3.9 as runtime on streamlit cloud.
1 Like