My app works locally, but fails to work when deployed

[emkay5771]

My app is supposed to collect flight data and parse it in a quick and efficient manner. When running it locally, the project returns pricing data. However, when running in the cloud, it instantly fails to collect the data, displaying an error that I have programmed in. Does bs4 have known issues with Cloud?
Any assistance would be greatly appreciated.

Thanks!

Attatched, please find a link to the github repository, as well as the cloud based app:
Github Project
Streamlit App

EDIT/ UPDATE:
I have narrowed down the problem to requests.get recieving different data when running in Streamlit Cloud. Not entirely sure why, but I can reproduce the issue in Brave while in incognito, while Edge incognito still retrieves the correct data. The program also works on Replit… Are there any ways to diagnose the potential issue with the request, as well as any way to mimic my local setup?

For reference, here is the relevant code, as well as screenshots of Edge vs Brave (both have JS disabled; trying to extract the table data using the relevant tags):

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
                                                browser = requests.get(f'https://www.google.com/search?q=fly{airline}+from+{origins}+to+{destination}+on+{date}+one+way{nonstop}', headers = headers)
soup = BeautifulSoup(browser.content, 'html.parser')
airlinefind = soup.find_all(class_='BNeawe s3v9rd AP7Wnd')

for air in airlinefind:
                                                if airlinefind[x].string != None and airlinefind[x].string != 'Southwest':
                                                    if len(airlinefind[x].string) > 19:
                                                        st.write(f"Broke at 112; {airlinefind[x].string}, position {x} :thumbsdown:")
                                                        break                                                   
                                                    airlines.append(airlinefind[x].string)
                                                    st.write(f"{airlinefind[x].string} :thumbsup:")                                                    
                                                x+=1

for air in airlines:
                                                if " " in air and air not in airlinelist:
                                                    st.write(f"{air} 😠")
                                                    pass
                                                else:
                                                    new.append(air)
                                                    st.write(f"{air} :airplane:")

Edge incognito

Brave incognito

[arnaud]

Hey @emkay5771,

Could you attach the error you’re encountering? There should not be any issue running bs4 commands while on Cloud

[emkay5771]

After breaking the program down, I have discovered that for some reason, requests.get is returning a different result for parsing when compared with my local machine. Do requests go through some weird proxy, documented somewhere? I noticed that when rendered in edge on my local machine, I get the needed result, but not while in Brave incognito.

[arnaud]

Hey @emkay5771

One thing that might be different is that when running on Cloud, your request headers will be different than locally. This is because request headers usually identify the machine you’re sending the request from, and on Cloud, this is different than locally! Things like user agent etc.

One thing you could try, though, is explicitly passing your local headers that work locally.

Locally, check out the following:

import requests
headers = requests.get("the_url_you_request_from").request.headers

The variable headers should hold a dictionary that looks like that:

{'Accept': '*/*',
 'Accept-Encoding': 'identity, deflate, compress, gzip',
 'Authorization': u'Basic dXNlcjpwYXNz',
 'User-Agent': 'python-requests/0.12.1'}

Paste it into your code such that this is always passed as headers for your GET requests:

requests.get("the_url_you_request_from", headers={'Accept': '*/*', ...})

Might make it easier for your GET request to work similarly locally vs. in Cloud