Issue with Selenium on a Streamlit app

Hi guys !

I deployed an app to scrap some french bookmakers, using selenium. But when i run the script, nothing happens.
It seems the webdriver is not workingā€¦

Any idea please ?

Hi @KoxNoob, welcome to the Streamlit community!

Can you be more specific about what you are doing? A code snippet would help here, to understand if this is a Streamlit issue or Selenium.

Best,
Randy

Thanks @randyzwitch for your answer.

You can find below a code snippet.
When i run the script in pycharm, everything works, but not in the deployed appā€¦

Thanks in advance. Regards

Where are you deploying this app, Streamlit sharing?

Yes indeed and also on Heroku

I think that will not work on Streamlit Sharing that way.

I havenā€™t done anything with selenium for a while, but I seem to remember that

  • you need to have geckodriver or chromedriver installed, which might not apply to the Streamlit Sharing runtime
  • you might have to use the --headless option
  • webdriver.Firefox() wants to open a browser, which will probably fail on the Streamlit Sharing runtime

@randyzwitch Correct me if i am wrong

@KoxNoob Does your app work on Heroku?

In this blog post, I show how to use seleniumbase with Streamlit sharing:

https://blog.streamlit.io/testing-streamlit-apps-using-seleniumbase/

So it definitely works, the question is how to make it work for what youā€™re trying to do.

1 Like

@randyzwitch
That looks good, it should work with SeleniumBase
Iā€™ve learned something new again - I didnā€™t know SeleniumBase until now.
Phew - the python ecosystem is so hugeā€¦ :wink:

1 Like

Eheh yeah, I realized this the day I found out wget in Python so that I could run wget on older Windows machines with Python and no WSL/Powershell :joy:

Thanks @randyzwitch and @Franky1 !

Indeed, i saw your blog post. Unfortunely, i donā€™t have a lot of experiencesā€¦ And I donā€™t know how to make SeleniumBase work.

From an url (urlpage in my code snippet), I want to collect text from a CSS selector (find_elements_by_xpath).
With selenium itā€™s working. But how to do with SeleniumBase.

@KoxNoob
I donā€™t want to confuse you even more, but for websites that only serve static content (i.e. no content dynamically generated by Javascript) I often use only requests with beautifulsoup4
Another possibility would be scrapy

Yes, i know requests and BS4 but unfortunately, it doesnā€™t work, there are some content dynamically generated by JSā€¦

For example :
urlpage : ā€œUnibet.frā€
balise : ā€œ//*[@class=ā€œui-mainview-block eventpath-wrapperā€]ā€

Ok, I see, unfortunately there is no way around using a scraping library with a Javascript engine. :neutral_face:

It seems like you should be able to install chromedriver on Streamlit sharing via a packages.txt file in your repo. Create the text file, add chromium-chromedriver on its own line, and see if it will work.

I tried but it still doesnā€™t workā€¦

Here you can find my repo. Could you help me again please ?
I donā€™t understand why it doesnā€™t workā€¦

And here, my app on Streamlit sharing !

Thanks in advance.

What is the error message?

No error messageā€¦ Just the scrap doesnā€™t workā€¦ While locally, everything is goodā€¦

Are you sure?
I donā€™t mean the app itself, but please check the Streamlit Sharing console for error messages.

I opened a PR:

This is likely NOT the complete answer, but it allows your app to load:

Best,
Randy