Issue with Selenium on a Streamlit app

Thanks again @randyzwitch and @Franky1 !

Indeed, no error message in the log.

I’m going to explain you what is the aim : i try to collect text and odds from bookmakers website (Winamax, Unibet, Betclic, etc in my previous example) using Selenium.

So, normally, if all works, I should have numbers instead of nan in the dataframe…

@randyzwitch
After taking a closer look at SeleniumBase, I don’t know how to use it for webscraping. It’s a pytest framework, but I haven’t found an example of how to use the classes in a python script for webscraping. Do you have a small example?


@KoxNoob
Also, I had massive problems getting Selenium to work on streamlit sharing. I had imagined it to be easier. Locally on Windows 10 and in a local Docker container, I got it up and running relatively quickly. But the deployment to streamlit sharing of the app itself kept going wrong without any plausible error message.

My efforts for a very very simple example (proof of concept) can be found here:

https://share.streamlit.io/franky1/streamlit-selenium/main

1 Like

By the way, thanks a lot @Franky1 !

I guess I was viewing this as a roundabout way of getting Selenium installed. SeleniumBase is used for testing, but by installing that, I would assume that the other Selenium package would then be able to detect the proper driver and make things happen.

In general, I’ll pass along the feedback to our product team about “What if there isn’t an apt package?” as part of the install process, not sure where adding custom apt repositories might be on the roadmap (if at all).

Best,
Randy

2 Likes

Hey @KoxNoob! I was wondering if you found any solution since I plan to use Selenium to do some scraping with Streamlit Sharing. I have the same issue as you, where the code runs locally but not when deployed online.

I honestly think we need a browser installed in the github repo somehow so the webdriver can use it, but I am unsure how that will work.

Hey all,

Given that we can now use conda to install packages on Streamlit Share (see it here), we can install firefox-esr via packages.txt (as mentioned by @ferdie you need a browser to be installed for the webdriver) and install geckodriver through conda (here’s the package).

Then we need to hardcode the path to geckodriver in the conda environment:

firefoxOptions = Options()
firefoxOptions.add_argument("--headless")
driver = webdriver.Firefox(
    options=firefoxOptions,
    executable_path="/home/appuser/.conda/bin/geckodriver",
)
driver.get(URL)

This geckodriver, as opposed to an executable pushed from Github repo, can be used by the app and…I think it works, I think it works I think it does, at least I can query the Unibet tables :wink: I’ll let you test further.

I can’t guarantee it will work forever :slight_smile: if the Cloud team decides to change the path to the conda environment then you’ll have to look for it.

App: https://share.streamlit.io/andfanilo/s4a-selenium/main/app.py
Code: GitHub - andfanilo/s4a-selenium: Test Selenium + Firefox on Streamlit Share

Cheers,
Fanilo

This is exciting work, thank you so much @andfanilo!

By any chance do you know the package name to install the Google Chrome web browser? My project used the Google chromedriver, and it’ll be a little tedious to switch over to Firefox.

Yeah Chrome I think is going to be a little-ish harder. I don’t think you can easily install Chrome on Debian Buster (which is the OS for Streamlit Share apps), you may have better chance installing chromium in packages.txt (chromium is the open source base for Chrome), then chromedriver the same way as geckodriver (hardcoding the path to the conda installed chromedriver in your app).
Hopefully Selenium knows how to pick the installed chromium without playing with the options, otherwise you’ll have to find a way to specify using chromium. Haven’t tested, hope you can find it out this way!

I’ll try later maybe.
Fanilo

@ferdie

Add to your packages.txt file:

chromium
chromium-driver

See a simple example project here:

1 Like

Hi @andfanilo! With the new release you cannot use more than one package manager at a time. Either you use pip (requirements.txt) or you use conda (environment.yml). You won’t be able to use both though.

2 Likes

Sounds good to me, conda and pip install conflicts has always been hard to debug :upside_down_face:

Hey Fanilo, so I’m currently in the process of debugging, and I was wondering if you can help me.

I completely switched to FireFox since I ran into issues with chromium still, but now I am currently running into this exception when trying to run the function you made to test selenium from your app.

selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette

For context, I copied over your code (some code is omitted) into a function test_st() found in sel.py:

    URL = "https://www.unibet.fr/sport/football/europa-league/europa-league-matchs"
    XPATH = "//*[@class='ui-mainview-block eventpath-wrapper']"
    TIMEOUT = 40

    firefoxOptions = Options()
    firefoxOptions.add_argument("--headless")
    driver = webdriver.Firefox(
        options=firefoxOptions,
        executable_path="/home/appuser/.conda/bin/geckodriver",
    )
    driver.get(URL)

    try:
        WebDriverWait(driver, TIMEOUT).until(
            EC.visibility_of_element_located((By.XPATH, XPATH,))
        )
... etc

And if you were to press the button test fanilo's function here on the sidebar of my app, it would run your code and run into the exception. Locally it does work, but not through Streamlit Sharing.

This is a bigger project, and I was hoping to isolate if if there was an issue with my code, or if it’s just how I set up everything in my directory.

Any help would be appreciated, thank you again!

App: https://share.streamlit.io/ftaruc/grailed/main/st-app.py
Code: GitHub - ftaruc/grailed: grailed web application for streamlit