TA-Lib Streamlit Deploy Error

[juwimana]

Building wheels for collected packages: TA-Lib
  Building wheel for TA-Lib (setup.py): started
  Building wheel for TA-Lib (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-a4r42tsy/ta-lib/setup.py'"'"'; __file__='"'"'/tmp/pip-install-a4r42tsy/ta-lib/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-nzhdy2sb
       cwd: /tmp/pip-install-a4r42tsy/ta-lib/
  Complete output (27 lines):
  /tmp/pip-install-a4r42tsy/ta-lib/setup.py:71: UserWarning: Cannot find ta-lib library, installation may fail.
    warnings.warn('Cannot find ta-lib library, installation may fail.')
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/talib
  copying talib/test_pandas.py -> build/lib.linux-x86_64-3.7/talib
  copying talib/test_stream.py -> build/lib.linux-x86_64-3.7/talib
  copying talib/test_data.py -> build/lib.linux-x86_64-3.7/talib
  copying talib/stream.py -> build/lib.linux-x86_64-3.7/talib
  copying talib/test_func.py -> build/lib.linux-x86_64-3.7/talib
  copying talib/__init__.py -> build/lib.linux-x86_64-3.7/talib
  copying talib/test_abstract.py -> build/lib.linux-x86_64-3.7/talib
  copying talib/abstract.py -> build/lib.linux-x86_64-3.7/talib
  copying talib/deprecated.py -> build/lib.linux-x86_64-3.7/talib
  running build_ext
  building 'talib._ta_lib' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/talib
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include -I/usr/local/include -I/opt/include -I/opt/local/include -I/usr/local/lib/python3.7/site-packages/numpy/core/include -I/usr/local/include/python3.7m -c talib/_ta_lib.c -o build/temp.linux-x86_64-3.7/talib/_ta_lib.o
  talib/_ta_lib.c:611:10: fatal error: ta-lib/ta_defs.h: No such file or directory
   #include "ta-lib/ta_defs.h"
            ^~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for TA-Lib
  Running setup.py clean for TA-Lib
Failed to build TA-Lib

I need help with the error above. Thank you!

[randyzwitch]

Hi @juwimana, welcome to the Streamlit community.

I’ve never used this specific library, but per their troubleshooting, do you have the underlying TA-Lib library installed? It appears to be a C library of some sort.

[juwimana]

Yes, I have it installed on my local machine. I only have issues when deploying to share.streamlit.io

[randyzwitch]

If there is a Debian package of the library, you can create a packages.txt file, which will install system dependencies:

Deploy an app — Streamlit 0.71.0 documentation

streamlit-apps/demo-self-driving/blob/master/packages.txt

If you need to compile the library yourself to install it, we don’t currently have functionality for that, but I believe it’s on our roadmap.

Best,
Randy

[juwimana]

I can’t find any Debian package for TA-Lib, probably not currently available.

[franciscoed]

Exactly, there is no debian package. I’ll come up with some sort of “solution” and more info soon!

[franciscoed]

Hi @randyzwitch and @juwimana

I’d call it a hack instead of a solution, I hope they’ll find a better way to support it soon. i.e. Heroku has a buildpack option which is a solution for this.

So first of all some information:

• It’s a docker container where we’re not root (they did a nice job, it’s simple and work quite nice)
• ta-lib doesn’t have a debian package on the official repo
• Since we’re not root we can’t simply add a repo, or install a package on the system

In the end we need to follow these steps:

• Add gcc and make to packages.txt
• Remove ta-lib from requirements.txt
• Add requests to requirements.txt
• Change the code:
  Download ta-lib
  Build it and install to your home directory
  Install python package, pointing to home directory
  Add the library to your code, since python is already running and will not get new library dir

NOTE: The first time will take a few minutes, bc it needs to build the lib

Code:

 import requests
 import os
 import sys
# check if the library folder already exists, to avoid building everytime you load the pahe
if not os.path.isdir("/tmp/ta-lib"):

    # Download ta-lib to disk
    with open("/tmp/ta-lib-0.4.0-src.tar.gz", "wb") as file:
        response = requests.get(
            "http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz"
        )
        file.write(response.content)
    # get our current dir, to configure it back again. Just house keeping
    default_cwd = os.getcwd()
    os.chdir("/tmp")
    # untar
    os.system("tar -zxvf ta-lib-0.4.0-src.tar.gz")
    os.chdir("/tmp/ta-lib")
    # build
    os.system("./configure --prefix=/home/appuser")
    os.system("make")
    # install
    os.system("make install")
    # install python package
    os.system(
        'pip3 install --global-option=build_ext --global-option="-L/home/appuser/lib/" --global-option="-I/home/appuser/include/" ta-lib'
    )
    # back to the cwd
    os.chdir(default_cwd)
    print(os.getcwd())
    sys.stdout.flush()

# add the library to our current environment
from ctypes import *

lib = CDLL("/home/appuser/lib/libta_lib.so.0")
# import library
import talib

# here goes your code

I hope that helps.

Cheers

[pitanjal]

Hi @franciscoed,

Thank You so much for coming up with this.

I tried the suggested method and I am getting the following in my log while deploying on share streamlit.

Installing collected packages: numpy, ta-lib

    Running setup.py install for ta-lib: started

    Running setup.py install for ta-lib: finished with status 'done'

Successfully installed numpy-1.21.1 ta-lib-0.4.21

/home/appuser/.local/lib/python3.7/site-packages/pip/_internal/commands/install.py:232: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.

  cmdoptions.check_install_build_global(options)

WARNING: You are using pip version 21.1.2; however, version 21.1.3 is available.

You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.

/app/stock-app-streamlit

2021-07-24 09:13:36.827 Uncaught app exception

Traceback (most recent call last):

  File "/home/appuser/venv/lib/python3.7/site-packages/streamlit/script_runner.py", line 350, in _run_script

    exec(code, module.__dict__)

  File "/app/stock-app-streamlit/trade.py", line 38, in <module>

    import talib

ModuleNotFoundError: No module named 'talib'

Please kindly guide me on how to overcome this.

Thank You so much!

[franciscoed]

@pitanjal

hile deploying on share streamlit.

Installing collected packages: numpy, ta-lib

    Running setup.py install for ta-lib: started

    Running setup.py install for ta-lib: finished with status 'done'

Hi, It works on local docker (debian buster-slim), but on streamlit I can confirm it’s not working anymore.
I’ll take a look on that.

Cheers

[franciscoed]

@pitanjal

Try this block:

import streamlit as st
import requests
import os
import sys
import subprocess

# check if the library folder already exists, to avoid building everytime you load the pahe
if not os.path.isdir("/tmp/ta-lib"):

    # Download ta-lib to disk
    with open("/tmp/ta-lib-0.4.0-src.tar.gz", "wb") as file:
        response = requests.get(
            "http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz"
        )
        file.write(response.content)
    # get our current dir, to configure it back again. Just house keeping
    default_cwd = os.getcwd()
    os.chdir("/tmp")
    # untar
    os.system("tar -zxvf ta-lib-0.4.0-src.tar.gz")
    os.chdir("/tmp/ta-lib")
    os.system("ls -la /app/equity/")
    # build
    os.system("./configure --prefix=/home/appuser")
    os.system("make")
    # install
    os.system("make install")
    # back to the cwd
    os.chdir(default_cwd)
    sys.stdout.flush()

# add the library to our current environment
from ctypes import *

lib = CDLL("/home/appuser/lib/libta_lib.so.0.0.0")
# import library
try:
    import talib
except ImportError:
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--global-option=build_ext", "--global-option=-L/home/appuser/lib/", "--global-option=-I/home/appuser/include/", "ta-lib"])
finally:
    import talib

[thomas.chang9527]

@franciscoed

subprocess.check_call([sys.executable, "-m", "pip", "install", "--global-option=build_ext", "--global-option=-L/home/appuser/lib/", "--global-option=-I/home/appuser/include/", "ta-lib"])

Hi @franciscoed, thanks for your grate help on this.
I got the same error on the above line and I has checked the file is under /home/appuser/include/ta-lib by running subprocess.run(['ls', '-al', '/home/appuser/include/ta-lib/'])

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include -I/usr/local/include -I/opt/include -I/opt/local/include -I/opt/homebrew/include -I/opt/homebrew/opt/ta-lib/include -I/tmp/pip-build-env-uj4o5ojt/normal/lib/python3.10/site-packages/numpy/core/include -I/home/appuser/venv/include -I/usr/local/include/python3.10 -c talib/_ta_lib.c -o build/temp.linux-x86_64-cpython-310/talib/_ta_lib.o

talib/_ta_lib.c:747:10: fatal error: ta-lib/ta_defs.h: No such file or directory
747 | #include “ta-lib/ta_defs.h”
| ^~~~~~~~~~~~~~~~~~
compilation terminated.
error: command ‘/usr/bin/gcc’ failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for ta-lib
Building wheel for numpy (pyproject.toml): started

Thanks for help

[franciscoed]

Hi @thomas.chang9527

Yes, you’re right.
It turn’s out we have two issues here:

First one: the global option is not being respected, it looks like it’s an expected behavior on the newer versions
Second one: current TA Lib version (0.4.25) has some issue with deprecated numpy functions

subprocess.check_call([sys.executable, "-m", "pip", "install", "--global-option=build_ext", "--global-option=-L/home/appuser/lib/", "--global-option=-I/home/appuser/include/", "ta-lib"])

So we have to change the include and lib directory and also fix the ta-lib version to 0.4.24.
Follow below the new code, please let me know if that works well for you

import streamlit as st
import requests
import os
import sys
import subprocess

# check if the library folder already exists, to avoid building everytime you load the pahe
if not os.path.isdir("/tmp/ta-lib"):

    # Download ta-lib to disk
    with open("/tmp/ta-lib-0.4.0-src.tar.gz", "wb") as file:
        response = requests.get(
            "http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz"
        )
        file.write(response.content)
    # get our current dir, to configure it back again. Just house keeping
    default_cwd = os.getcwd()
    os.chdir("/tmp")
    # untar
    os.system("tar -zxvf ta-lib-0.4.0-src.tar.gz")
    os.chdir("/tmp/ta-lib")
    # build
    os.system("./configure --prefix=/home/appuser/venv/")
    os.system("make")
    # install
    os.system("mkdir -p /home/appuser/venv/")
    os.system("make install")
    os.system("ls -la /home/appuser/venv/")
    # back to the cwd
    os.chdir(default_cwd)
    sys.stdout.flush()

# add the library to our current environment
from ctypes import *

lib = CDLL("/home/appuser/venv/lib/libta_lib.so.0.0.0")
# import library
try:
    import talib
except ImportError:
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--global-option=build_ext", "--global-option=-L/home/appuser/venv/lib/", "--global-option=-I/home/appuser/venv/include/", "ta-lib==0.4.24"])
finally:
    import talib

Cheers

Francisco

[thomas.chang9527]

Hi @franciscoed,
Thank you very much, it works.

[Johnm]

I am having issues as well with TA-lib. Is it accurate that the code shown is to be added at the top of main Python app? Is that only applicable when running in Streamlit Cloud or should that work for local development as well?

In reference to the code above, do the various paths such as ‘/home/appuser/venv/’ need to be updated based on local config?

Note:
I’ve seen several posts about integration errors with TA-lib. Once a consistent solution is available, it might be helpful to consolidate the various posts so there is a helpful guide located within one thread. Just a thought…

Thanks in advance!

[asehmi]

Have you tried the pre-built binary?

$ pip install ta-lib-bin

I’ve stopped using TA-Lib due to similar deployment issues, and since I’m not using the candle patterns in ta-lib, I started using pandas-ta instead.

Arvindra

[Johnm]

Thanks Arvindra,

I did and I received the following errors:
ERROR: Could not find a version that satisfies the requirement ta-lib-bin (from versions: none)
ERROR: No matching distribution found for ta-lib-bin

I don’t plan to use TA-lib for the long run, so I appreciate your suggestion on pandas-ta.

I imagine others will try TA-lib so I thought it would be worthwhile to see if there could be a relatively easy solution going forward.

[asehmi]

There’s also a talib-binary you could try to pip install.

[Johnm]

I’ll have a look, thanks.

[Franky1]

My app is not deploying because of talib

TA-Lib is indeed a somewhat difficult case. The Python library requires the installation of the underlying C-library. Unfortunately, this is not available as a Debian package and must therefore be downloaded and compiled yourself. Compiling your own packages is not provided by default on Streamlit Cloud. The solution from the linked thread above is therefore rather a hack. However, if you search on pypi, you will find some forks that provide a precompiled version as a wheel. This one ta-lib-bi…

[Sandra_Carrico]

I also have this problem. I put a call to the code franciscoed provided right after the streamlit line of my driver script. I’m getting the following error:

Collecting TA-Lib

  Downloading TA-Lib-0.4.28.tar.gz (357 kB)

     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 357.1/357.1 KB 235.2 MB/s eta 0:00:00[2023-08-24 17:44:05.135903] 

  Installing build dependencies: started

  Installing build dependencies: finished with status 'done'

  Getting requirements to build wheel: started

  Getting requirements to build wheel: finished with status 'done'

  Installing backend dependencies: started

  Installing backend dependencies: finished with status 'done'

  Preparing metadata (pyproject.toml): started

  Preparing metadata (pyproject.toml): finished with status 'done'

Building wheels for collected packages: pyngrok, validators, TA-Lib

  Building wheel for pyngrok (setup.py): started

  Building wheel for pyngrok (setup.py): finished with status 'done'

  Created wheel for pyngrok: filename=pyngrok-6.0.0-py3-none-any.whl size=19867 sha256=58cabd5d7d2e2e0db3d05b2219e0ecc715a7b11c246ff8bc19b6c0f9870c7e26

  Stored in directory: /tmp/pip-ephem-wheel-cache-ofmemp71/wheels/b6/2f/3d/4957f583fdfd1c359843d2e57981c634692c579788b2dfc088

  Building wheel for validators (setup.py): started

  Building wheel for validators (setup.py): finished with status 'done'

  Created wheel for validators: filename=validators-0.20.0-py3-none-any.whl size=19575 sha256=6555b359c039468ac4f273250b1a7ca703772cb3deea7505ab0a910c5c84a345

  Stored in directory: /tmp/pip-ephem-wheel-cache-ofmemp71/wheels/82/35/dc/f88ec71edf2a5596bd72a8fa1b697277e0fcd3cde83048b8bf

  Building wheel for TA-Lib (pyproject.toml): started

  Building wheel for TA-Lib (pyproject.toml): finished with status 'error'

  error: subprocess-exited-with-error

  × Building wheel for TA-Lib (pyproject.toml) did not run successfully.

  │ exit code: 1

  ╰─> [21 lines of output]

      <string>:77: UserWarning: Cannot find ta-lib library, installation may fail.

      running bdist_wheel

      running build

      running build_py

      creating build

      creating build/lib.linux-x86_64-cpython-311

      creating build/lib.linux-x86_64-cpython-311/talib

      copying talib/__init__.py -> build/lib.linux-x86_64-cpython-311/talib

      copying talib/stream.py -> build/lib.linux-x86_64-cpython-311/talib

      copying talib/abstract.py -> build/lib.linux-x86_64-cpython-311/talib

      copying talib/deprecated.py -> build/lib.linux-x86_64-cpython-311/talib

      running build_ext

      building 'talib._ta_lib' extension

      creating build/temp.linux-x86_64-cpython-311

      creating build/temp.linux-x86_64-cpython-311/talib

      gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include -I/usr/local/include -I/opt/include -I/opt/local/include -I/opt/homebrew/include -I/opt/homebrew/opt/ta-lib/include -I/tmp/pip-build-env-t5e634sm/normal/lib/python3.11/site-packages/numpy/core/include -I/home/adminuser/venv/include -I/usr/local/include/python3.11 -c talib/_ta_lib.c -o build/temp.linux-x86_64-cpython-311/talib/_ta_lib.o

      talib/_ta_lib.c:1082:10: fatal error: ta-lib/ta_defs.h: No such file or directory

       1082 | #include "ta-lib/ta_defs.h"

            |          ^~~~~~~~~~~~~~~~~~

      compilation terminated.

      error: command '/usr/bin/gcc' failed with exit code 1

      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.

  ERROR: Failed building wheel for TA-Lib

Blockquote

How do I work around this? Thanks.