Streamlit installation conflict with pyarrow version for snowflake-snowpark-python

[honytoad]

I am trying to deploy an app (Edit: on Streamlit Community Cloud) that uses the library snowflake-snowpark-python. I have tested the app locally and it works fine, but when I deploy I am getting the error:

UserWarning: You have an incompatible version of 'pyarrow' installed (11.0.0), please install a version that adheres to: 'pyarrow<8.1.0,>=8.0.0; extra == "pandas"'

snowflake-snowpark-python requires pyarrow=8.0; I have added the dependency snowflake-snowpark-python=1.1.0 to my environment.yml file in my repo. What I am seeing is that during the Streamlit package installation pyarrow 8.0.0 is uninstalled and replaced by pyarrow 11.0.0, even though Streamlit only requires pyarrow>=4.0. I am using the latest version of Streamlit (I’ve not specified the version in the requirements file).
If I look through the logs, as part of the streamlit package installation I see this:

  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 8.0.0
    Uninstalling pyarrow-8.0.0:
      Successfully uninstalled pyarrow-8.0.0

I have tested including pyarrow=8.0.0 in my environment.yml file to ‘force’ streamlit to keep v8.0.0 but it still gets uninstalled. I have also tried a requirements.txt file instead and this gives the same error. What am I doing wrong here?

My app:
https://sfc-gh-thoad-scratch-sportsco-app-umqq7m.streamlit.app/
My repo:

GitHub - sfc-gh-thoad/scratch: Sandbox for testing stuff

Requirements file:

sfc-gh-thoad/scratch/blob/main/environment.yml

[Franky1]

I would remove the version number and try again.
If i run in a clean Python 3.8 environment:

pip install -U pandas streamlit snowflake-snowpark-python

… i get no conflicts and the following packages (amongst others):

Package                    Version
-------------------------- -----------
pandas                     1.5.3
pyarrow                    11.0.0
snowflake-connector-python 3.0.1
snowflake-snowpark-python  1.2.0
streamlit                  1.20.0

[honytoad]

Thanks Franky1, I have just tried changing my environment.yml file to

name: snowpark-de-ml
channels:
  - https://repo.anaconda.com/pkgs/snowflake
  - defaults
dependencies:
  - snowflake-snowpark-python

and completely redeployed the app in Streamlit Cloud, and I am still getting the error

/home/appuser/venv/lib/python3.8/site-packages/snowflake/connector/options.py:96: UserWarning: You have an incompatible version of 'pyarrow' installed (11.0.0), please install a version that adheres to: 'pyarrow<8.1.0,>=8.0.0; extra == "pandas"'

According to the Streamlit app logs I am indeed using snowflake-connector-python 3.0.1 and
snowflake-snowpark-python 1.2.0 but the snowflake connector is still asking for pyarrow 8.0 . I can’t seem to force installation of pyarrow 8.0 either, as when Streamlit gets installed it uninstalls 8.0 and replaces it with 11.0 .

  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 8.0.0
    Uninstalling pyarrow-8.0.0:
      Successfully uninstalled pyarrow-8.0.0

At this point I think I must have misunderstood package installation but I’m not sure where I’m going wrong!

New app:
https://sfc-gh-thoad-scratch-sportsco-app-huiq13.streamlit.app/
Environment file: scratch/environment.yml at main · sfc-gh-thoad/scratch · GitHub

[Franky1]

I don’t use conda. For reasons. Use a requirements.txt file instead and try again in a Python 3.8 streamlit cloud runtime:

pandas
streamlit
snowflake-snowpark-python

@honytoad

New app:

However, this is a new error:
snowflake.snowpark.exceptions.SnowparkFetchDataException
Therefore the snowflake installation seems to be working.

[Franky1]

I remember there was an issue with the to_pandas() method, maybe this helps:

Unable to deploy streamlit app from Snowflake quickstart

When I tried your code locally, a very clear error message showed up SnowparkFetchDataException: (1406): Failed to fetch a Pandas Dataframe. The error is: to_pandas() did not return a Pandas DataFrame. If you use session.sql(…).to_pandas(), the input query can only be a SELECT statement. Or you can use session.sql(…).collect() to get a list of Row objects for a non-SELECT statement, then convert it to a Pandas DataFrame. Now, this error message doesn’t really make sense, because what you are d…

[honytoad]

Thanks Franky, I think the to_pandas() error stems from the pyarrow error since I wasn’t facing either error locally.

I have finally managed to resolve the issue following your advice and using pip instead of conda, below are the contents of my requirements.txt file - I had to specify the version of pyarrow explicity (the version was deduced from the error message that Streamlit was giving me with Snowpark when I didn’t specify it). It all deploys and reads in the dataframe without the to_pandas() error:

snowflake-snowpark-python
pyarrow==10.0.1

Thanks for your help!