TA-Lib Streamlit Deploy Error

I can’t find any Debian package for TA-Lib, probably not currently available.

1 Like

Exactly, there is no debian package. I’ll come up with some sort of “solution” and more info soon!

Hi @randyzwitch and @juwimana

I’d call it a hack instead of a solution, I hope they’ll find a better way to support it soon. i.e. Heroku has a buildpack option which is a solution for this.

So first of all some information:

  • It’s a docker container where we’re not root (they did a nice job, it’s simple and work quite nice)
  • ta-lib doesn’t have a debian package on the official repo
  • Since we’re not root we can’t simply add a repo, or install a package on the system

In the end we need to follow these steps:

  • Add gcc and make to packages.txt
  • Remove ta-lib from requirements.txt
  • Add requests to requirements.txt
  • Change the code:
    Download ta-lib
    Build it and install to your home directory
    Install python package, pointing to home directory
    Add the library to your code, since python is already running and will not get new library dir

NOTE: The first time will take a few minutes, bc it needs to build the lib

Code:

 import requests
 import os
 import sys
# check if the library folder already exists, to avoid building everytime you load the pahe
if not os.path.isdir("/tmp/ta-lib"):

    # Download ta-lib to disk
    with open("/tmp/ta-lib-0.4.0-src.tar.gz", "wb") as file:
        response = requests.get(
            "http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz"
        )
        file.write(response.content)
    # get our current dir, to configure it back again. Just house keeping
    default_cwd = os.getcwd()
    os.chdir("/tmp")
    # untar
    os.system("tar -zxvf ta-lib-0.4.0-src.tar.gz")
    os.chdir("/tmp/ta-lib")
    # build
    os.system("./configure --prefix=/home/appuser")
    os.system("make")
    # install
    os.system("make install")
    # install python package
    os.system(
        'pip3 install --global-option=build_ext --global-option="-L/home/appuser/lib/" --global-option="-I/home/appuser/include/" ta-lib'
    )
    # back to the cwd
    os.chdir(default_cwd)
    print(os.getcwd())
    sys.stdout.flush()

# add the library to our current environment
from ctypes import *

lib = CDLL("/home/appuser/lib/libta_lib.so.0")
# import library
import talib

# here goes your code

I hope that helps.

Cheers

4 Likes

Hi @franciscoed,

Thank You so much for coming up with this.

I tried the suggested method and I am getting the following in my log while deploying on share streamlit.

Installing collected packages: numpy, ta-lib
    Running setup.py install for ta-lib: started
    Running setup.py install for ta-lib: finished with status 'done'
Successfully installed numpy-1.21.1 ta-lib-0.4.21
/home/appuser/.local/lib/python3.7/site-packages/pip/_internal/commands/install.py:232: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
WARNING: You are using pip version 21.1.2; however, version 21.1.3 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
/app/stock-app-streamlit
2021-07-24 09:13:36.827 Uncaught app exception
Traceback (most recent call last):
  File "/home/appuser/venv/lib/python3.7/site-packages/streamlit/script_runner.py", line 350, in _run_script
    exec(code, module.__dict__)
  File "/app/stock-app-streamlit/trade.py", line 38, in <module>
    import talib
ModuleNotFoundError: No module named 'talib'

Please kindly guide me on how to overcome this.

Thank You so much!

1 Like

Hi, It works on local docker (debian buster-slim), but on streamlit I can confirm it’s not working anymore.
I’ll take a look on that.

Cheers

@pitanjal

Try this block:

import streamlit as st
import requests
import os
import sys
import subprocess

# check if the library folder already exists, to avoid building everytime you load the pahe
if not os.path.isdir("/tmp/ta-lib"):

    # Download ta-lib to disk
    with open("/tmp/ta-lib-0.4.0-src.tar.gz", "wb") as file:
        response = requests.get(
            "http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz"
        )
        file.write(response.content)
    # get our current dir, to configure it back again. Just house keeping
    default_cwd = os.getcwd()
    os.chdir("/tmp")
    # untar
    os.system("tar -zxvf ta-lib-0.4.0-src.tar.gz")
    os.chdir("/tmp/ta-lib")
    os.system("ls -la /app/equity/")
    # build
    os.system("./configure --prefix=/home/appuser")
    os.system("make")
    # install
    os.system("make install")
    # back to the cwd
    os.chdir(default_cwd)
    sys.stdout.flush()

# add the library to our current environment
from ctypes import *

lib = CDLL("/home/appuser/lib/libta_lib.so.0.0.0")
# import library
try:
    import talib
except ImportError:
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--global-option=build_ext", "--global-option=-L/home/appuser/lib/", "--global-option=-I/home/appuser/include/", "ta-lib"])
finally:
    import talib
2 Likes

Hi @franciscoed, thanks for your grate help on this.
I got the same error on the above line and I has checked the file is under /home/appuser/include/ta-lib by running subprocess.run(['ls', '-al', '/home/appuser/include/ta-lib/'])

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include -I/usr/local/include -I/opt/include -I/opt/local/include -I/opt/homebrew/include -I/opt/homebrew/opt/ta-lib/include -I/tmp/pip-build-env-uj4o5ojt/normal/lib/python3.10/site-packages/numpy/core/include -I/home/appuser/venv/include -I/usr/local/include/python3.10 -c talib/_ta_lib.c -o build/temp.linux-x86_64-cpython-310/talib/_ta_lib.o

talib/_ta_lib.c:747:10: fatal error: ta-lib/ta_defs.h: No such file or directory
747 | #include “ta-lib/ta_defs.h”
| ^~~~~~~~~~~~~~~~~~
compilation terminated.
error: command ‘/usr/bin/gcc’ failed with exit code 1
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for ta-lib
Building wheel for numpy (pyproject.toml): started

Thanks for help

Hi @thomas.chang9527

Yes, you’re right.
It turn’s out we have two issues here:

First one: the global option is not being respected, it looks like it’s an expected behavior on the newer versions
Second one: current TA Lib version (0.4.25) has some issue with deprecated numpy functions

subprocess.check_call([sys.executable, "-m", "pip", "install", "--global-option=build_ext", "--global-option=-L/home/appuser/lib/", "--global-option=-I/home/appuser/include/", "ta-lib"])

So we have to change the include and lib directory and also fix the ta-lib version to 0.4.24.
Follow below the new code, please let me know if that works well for you

import streamlit as st
import requests
import os
import sys
import subprocess

# check if the library folder already exists, to avoid building everytime you load the pahe
if not os.path.isdir("/tmp/ta-lib"):

    # Download ta-lib to disk
    with open("/tmp/ta-lib-0.4.0-src.tar.gz", "wb") as file:
        response = requests.get(
            "http://prdownloads.sourceforge.net/ta-lib/ta-lib-0.4.0-src.tar.gz"
        )
        file.write(response.content)
    # get our current dir, to configure it back again. Just house keeping
    default_cwd = os.getcwd()
    os.chdir("/tmp")
    # untar
    os.system("tar -zxvf ta-lib-0.4.0-src.tar.gz")
    os.chdir("/tmp/ta-lib")
    # build
    os.system("./configure --prefix=/home/appuser/venv/")
    os.system("make")
    # install
    os.system("mkdir -p /home/appuser/venv/")
    os.system("make install")
    os.system("ls -la /home/appuser/venv/")
    # back to the cwd
    os.chdir(default_cwd)
    sys.stdout.flush()

# add the library to our current environment
from ctypes import *

lib = CDLL("/home/appuser/venv/lib/libta_lib.so.0.0.0")
# import library
try:
    import talib
except ImportError:
    subprocess.check_call([sys.executable, "-m", "pip", "install", "--global-option=build_ext", "--global-option=-L/home/appuser/venv/lib/", "--global-option=-I/home/appuser/venv/include/", "ta-lib==0.4.24"])
finally:
    import talib

Cheers

Francisco

2 Likes

Hi @franciscoed,
Thank you very much, it works. :smiling_face_with_three_hearts: :smiling_face_with_three_hearts: :smiling_face_with_three_hearts:

I am having issues as well with TA-lib. Is it accurate that the code shown is to be added at the top of main Python app? Is that only applicable when running in Streamlit Cloud or should that work for local development as well?

In reference to the code above, do the various paths such as ‘/home/appuser/venv/’ need to be updated based on local config?

Note:
I’ve seen several posts about integration errors with TA-lib. Once a consistent solution is available, it might be helpful to consolidate the various posts so there is a helpful guide located within one thread. Just a thought…

Thanks in advance!

Have you tried the pre-built binary?

$ pip install ta-lib-bin

I’ve stopped using TA-Lib due to similar deployment issues, and since I’m not using the candle patterns in ta-lib, I started using pandas-ta instead.

Arvindra

1 Like

Thanks Arvindra,

I did and I received the following errors:
ERROR: Could not find a version that satisfies the requirement ta-lib-bin (from versions: none)
ERROR: No matching distribution found for ta-lib-bin

I don’t plan to use TA-lib for the long run, so I appreciate your suggestion on pandas-ta.

I imagine others will try TA-lib so I thought it would be worthwhile to see if there could be a relatively easy solution going forward.

:frowning: There’s also a talib-binary you could try to pip install.

I’ll have a look, thanks.

1 Like

I also have this problem. I put a call to the code franciscoed provided right after the streamlit line of my driver script. I’m getting the following error:

Collecting TA-Lib
  Downloading TA-Lib-0.4.28.tar.gz (357 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 357.1/357.1 KB 235.2 MB/s eta 0:00:00[2023-08-24 17:44:05.135903] 
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: pyngrok, validators, TA-Lib
  Building wheel for pyngrok (setup.py): started
  Building wheel for pyngrok (setup.py): finished with status 'done'
  Created wheel for pyngrok: filename=pyngrok-6.0.0-py3-none-any.whl size=19867 sha256=58cabd5d7d2e2e0db3d05b2219e0ecc715a7b11c246ff8bc19b6c0f9870c7e26
  Stored in directory: /tmp/pip-ephem-wheel-cache-ofmemp71/wheels/b6/2f/3d/4957f583fdfd1c359843d2e57981c634692c579788b2dfc088
  Building wheel for validators (setup.py): started
  Building wheel for validators (setup.py): finished with status 'done'
  Created wheel for validators: filename=validators-0.20.0-py3-none-any.whl size=19575 sha256=6555b359c039468ac4f273250b1a7ca703772cb3deea7505ab0a910c5c84a345
  Stored in directory: /tmp/pip-ephem-wheel-cache-ofmemp71/wheels/82/35/dc/f88ec71edf2a5596bd72a8fa1b697277e0fcd3cde83048b8bf
  Building wheel for TA-Lib (pyproject.toml): started
  Building wheel for TA-Lib (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Building wheel for TA-Lib (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      <string>:77: UserWarning: Cannot find ta-lib library, installation may fail.
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-311
      creating build/lib.linux-x86_64-cpython-311/talib
      copying talib/__init__.py -> build/lib.linux-x86_64-cpython-311/talib
      copying talib/stream.py -> build/lib.linux-x86_64-cpython-311/talib
      copying talib/abstract.py -> build/lib.linux-x86_64-cpython-311/talib
      copying talib/deprecated.py -> build/lib.linux-x86_64-cpython-311/talib
      running build_ext
      building 'talib._ta_lib' extension
      creating build/temp.linux-x86_64-cpython-311
      creating build/temp.linux-x86_64-cpython-311/talib
      gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include -I/usr/local/include -I/opt/include -I/opt/local/include -I/opt/homebrew/include -I/opt/homebrew/opt/ta-lib/include -I/tmp/pip-build-env-t5e634sm/normal/lib/python3.11/site-packages/numpy/core/include -I/home/adminuser/venv/include -I/usr/local/include/python3.11 -c talib/_ta_lib.c -o build/temp.linux-x86_64-cpython-311/talib/_ta_lib.o
      talib/_ta_lib.c:1082:10: fatal error: ta-lib/ta_defs.h: No such file or directory
       1082 | #include "ta-lib/ta_defs.h"
            |          ^~~~~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for TA-Lib

Blockquote

How do I work around this? Thanks.

Use ta-lib-bin instead

I have the following at the top of my requirements.txt:
ta-lib-bin
TA-Lib==0.4.26

(I also tried commenting out TA-Lib and even ta-lib-bin too)

It doesn’t matter, I still get:

      gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/include -I/usr/local/include -I/opt/include -I/opt/local/include -I/opt/homebrew/include -I/opt/homebrew/opt/ta-lib/include -I/tmp/pip-build-env-f3t5prjq/normal/lib/python3.11/site-packages/numpy/core/include -I/home/adminuser/venv/include -I/usr/local/include/python3.11 -c talib/_ta_lib.c -o build/temp.linux-x86_64-cpython-311/talib/_ta_lib.o
      talib/_ta_lib.c:747:10: fatal error: ta-lib/ta_defs.h: No such file or directory
        747 | #include "ta-lib/ta_defs.h"
            |          ^~~~~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

Remove TA-Lib and leave only ta-lib-bin
And if this does not help, show us your public github repo, otherwise we are poking around in the dark.

I tried what you suggested, and it failed. My repo is private, and I’m not comfortable making it public. I guess I have to give up.