How to use NON headless Selenium in Streamlit?

Using Streamlit
, ,
1

I am scraping a website which requires the use of non-headless mode in Selenium in order for it to scrape properly. I used non-headless mode of Selenium for the localhost run of Streamlit and it already works flawlessly.
However, once I upload my code into the Selenium server, I receive the error below, very likely because I am not using the headless mode in Selenium. How do I use non-headless mode in Selenium?

  1. Are you running your app locally or is it deployed?
  • Deployed on community cloud.
  1. Share the link to your app’s public GitHub repository (including a requirements file).

requirements.txt:
beautifulsoup4==4.12.3
google_api_python_client==2.124.0
numpy==1.26.4
openai==1.14.3
pandas==2.2.1
python-dotenv==1.0.1
Requests==2.31.0
selenium==4.19.0
streamlit==1.32.1
webdriver-manager==4.0.1
chromedriver-autoinstaller==0.6.4

  1. Share the full text of the error message (not a screenshot).
    selenium.common.exceptions.WebDriverException: This app has encountered an error. The original error message is redacted to prevent data leaks. Full error details have been recorded in the logs (if you’re on Streamlit Cloud, click on ‘Manage app’ in the lower right of your app).
Traceback:


File "/home/adminuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 542, in _run_script
    exec(code, module.__dict__)File "/mount/src/earnings-call-summarizer/earnings_call_summarizer_v3.py", line 226, in <module>
    if __name__ == '__main__':File "/mount/src/earnings-call-summarizer/earnings_call_summarizer_v3.py", line 213, in app
    st.write("If there is a pop-up, **it will automatically close**. We need it to simulate human behavior.")File "/mount/src/earnings-call-summarizer/earnings_call_summarizer_v3.py", line 59, in get_text
    driver = webdriver.Chrome()File "/home/adminuser/venv/lib/python3.9/site-packages/selenium/webdriver/chrome/webdriver.py", line 45, in __init__
    super().__init__(File "/home/adminuser/venv/lib/python3.9/site-packages/selenium/webdriver/chromium/webdriver.py", line 50, in __init__
    self.service.start()File "/home/adminuser/venv/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 102, in start
    self.assert_process_still_running()File "/home/adminuser/venv/lib/python3.9/site-packages/selenium/webdriver/common/service.py", line 115, in assert_process_still_running
    raise WebDriverException(f"Service {self._path} unexpectedly exited. Status code was: {return_code}")
2

Also interested, couldnt find the answer, and really need it!

3

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.