Issue with Selenium on a Streamlit app

Using Streamlit
1

Hi guys !

I deployed an app to scrap some french bookmakers, using selenium. But when i run the script, nothing happens.
It seems the webdriver is not working…

Any idea please ?

2

Hi @KoxNoob, welcome to the Streamlit community!

Can you be more specific about what you are doing? A code snippet would help here, to understand if this is a Streamlit issue or Selenium.

Best,
Randy

3

Thanks @randyzwitch for your answer.

You can find below a code snippet.
When i run the script in pycharm, everything works, but not in the deployed app…

Thanks in advance. Regards

scrap
scrap546×650 31.9 KB

4

Where are you deploying this app, Streamlit sharing?

5

Yes indeed and also on Heroku

6

I think that will not work on Streamlit Sharing that way.

I haven’t done anything with selenium for a while, but I seem to remember that

  • you need to have geckodriver or chromedriver installed, which might not apply to the Streamlit Sharing runtime
  • you might have to use the --headless option
  • webdriver.Firefox() wants to open a browser, which will probably fail on the Streamlit Sharing runtime

@randyzwitch Correct me if i am wrong

@KoxNoob Does your app work on Heroku?

7

In this blog post, I show how to use seleniumbase with Streamlit sharing:

https://blog.streamlit.io/testing-streamlit-apps-using-seleniumbase/

So it definitely works, the question is how to make it work for what you’re trying to do.

1 Like
8

@randyzwitch
That looks good, it should work with SeleniumBase
I’ve learned something new again - I didn’t know SeleniumBase until now.
Phew - the python ecosystem is so huge… :wink:

1 Like
9

Eheh yeah, I realized this the day I found out wget in Python so that I could run wget on older Windows machines with Python and no WSL/Powershell :joy:

10

Thanks @randyzwitch and @Franky1 !

Indeed, i saw your blog post. Unfortunely, i don’t have a lot of experiences… And I don’t know how to make SeleniumBase work.

From an url (urlpage in my code snippet), I want to collect text from a CSS selector (find_elements_by_xpath).
With selenium it’s working. But how to do with SeleniumBase.

11

@KoxNoob
I don’t want to confuse you even more, but for websites that only serve static content (i.e. no content dynamically generated by Javascript) I often use only requests with beautifulsoup4
Another possibility would be scrapy

12

Yes, i know requests and BS4 but unfortunately, it doesn’t work, there are some content dynamically generated by JS…

13

For example :
urlpage : “Unibet.fr”
balise : “//*[@class=“ui-mainview-block eventpath-wrapper”]”

14

Ok, I see, unfortunately there is no way around using a scraping library with a Javascript engine. :neutral_face:

15

It seems like you should be able to install chromedriver on Streamlit sharing via a packages.txt file in your repo. Create the text file, add chromium-chromedriver on its own line, and see if it will work.

16

I tried but it still doesn’t work…

Here you can find my repo. Could you help me again please ?
I don’t understand why it doesn’t work…

And here, my app on Streamlit sharing !

Thanks in advance.

17

What is the error message?

18

No error message… Just the scrap doesn’t work… While locally, everything is good…

19

Are you sure?
I don’t mean the app itself, but please check the Streamlit Sharing console for error messages.

20

I opened a PR:

This is likely NOT the complete answer, but it allows your app to load:

image
image1339×1459 221 KB

Best,
Randy