I’m implementing a web-scraping project with streamlit.
My issue is that I cannot find all the html content. I’m looking for a table that in the deployed stage does not appear, and locally it appears.
s = BeautifulSoup(self.driver.page_source, features='lxml') print(s) #prints something like <!--[if lt IE 7]><html class="no-js> [.....] <script></script>> </html> table = s.find( "table", class_= "table table-primary table-forecast allSwellsActive msw-js-table msw-units-large" ) print(table) # prints None
This is my public url: https://juanfrilla-magicseaweed-main-deploy-0j0p7l.streamlit.app/
This is my repo: GitHub - juanfrilla/magicseaweed: Automation script that tells me when there is off-shore or cross-off wind in different beaches, scraping data from magicseaweed, a surf forecasting website. (deploy branch)
Thanks in advance