ModuleNotFoundError: psycopg2, the data connection works on local drive, but not on Streamlit Cloud

[frances]

Hello

I am trying to connect PostgreSQL on Heroku server using psycopg2, it worked locally on my pc, but after pushed to Streamlit Cloud, it shows ModuleNotFoundError after rebooting. Here is the Traceback

Uncaught app exception

Traceback (most recent call last):

  File "/home/appuser/venv/lib/python3.7/site-packages/streamlit/script_runner.py", line 354, in _run_script

    exec(code, module.__dict__)

  File "/app/data_test/src/example.py", line 4, in <module>

    import psycopg2

ModuleNotFoundError: No module named 'psycopg2'

I have the following two packages in requirements.txt at the top level repo

psycopg2==2.9.3
psycopg2-binary==2.9.3

Wonder if any dependencies are missing? There are a couple of prerequisites in using psycopg2, will this be the cause?
https://www.psycopg.org/docs/install.html#prerequisites

Many thanks!

[randyzwitch]

Hi @frances, welcome back!

I would try adding libpq-dev to a packages.txt file, which will install Postgres development headers on the image. Often times, when database drivers don’t work, it’s because it’s trying to call an underlying C library.

Best,
Randy

[frances]

Many thanks @randyzwitch for the reply! I checked a few posts, and tried with adding libpq-dev to a packages.txt, however the problem still persists.

I suspect it is due to a virtual environment configuration, as I tried two different pipenv on my local machine, one can import psycopg2, and the other cannot.

I simplified the code here https://github.com/francesfeng/data_test, really appreciate your advice on this.

[snehankekre]

Hi @frances

Streamlit looks at your requirements file’s filename to determine which Python dependency manager to use in the order below. Streamlit will stop and install the first requirements file found.

Filename	Dependency Manager	Documentation
Pipfile	pipenv	docs
environment.yml	conda	docs
requirements.txt	pip	docs
pyproject.toml	poetry	docs

What’s happening is that Streamlit parses your Pipfile containing no packages, and exits the installation process – entirely skipping your requirements.txt.

Once you delete the Pipfile from your repo, Streamlit will install the Python dependencies specified in requirements.txt, including the psycog2-binary package.

Source: App dependencies - Streamlit Docs

Happy Streamlit-ing!
Snehan

[frances]

Brilliant! Thanks @snehankekre ! now the connection establishes after I remove Pipfile file!

Wonder if helps to add a Note in this documentation, perhaps someone like me moving from a pipenv environment might encounter a similar issue.

Again, thanks for the speedy reply!