Why deployed I cannot web-scrape all the html content and locally I can?

Community Cloud
1

Hello everyone

I’m implementing a web-scraping project with streamlit.
My issue is that I cannot find all the html content. I’m looking for a table that in the deployed stage does not appear, and locally it appears.

s = BeautifulSoup(self.driver.page_source, features='lxml')

print(s) #prints something like  <!--[if lt IE 7]><html class="no-js> [.....] <script></script>> </html>


table = s.find(
                "table",
                class_=
                "table table-primary table-forecast allSwellsActive msw-js-table msw-units-large"
            )

print(table) # prints None

As you see above, I print the page source and it returns the html with no-js class and the javascript inside of it. But not the full html content like locally, why is this?

This is my public url: https://juanfrilla-magicseaweed-main-deploy-0j0p7l.streamlit.app/

This is my repo: GitHub - juanfrilla/magicseaweed: Automation script that tells me when there is off-shore or cross-off wind in different beaches, scraping data from magicseaweed, a surf forecasting website. (deploy branch)

Thanks in advance

2

My app is running deployed :wink:

My problem was that the html was different because the streamlit server is in a different region than where I am right now. So I adapted the scraper.

3

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.