App that had previously worked for several months stopped working: "Failed to build hdbscan ERROR: Could not build wheels for hdbscan, which is required to install pyproject.toml-based projects"

Community Cloud
1

I had deployed my app on the community server several months ago and it worked fine. It went to sleep several times and every time I was able to restart it, however last week when I wanted to reboot the app, rebooting keeps failing. The error message at the end the log is “Failed to build hdbscan
ERROR: Could not build wheels for hdbscan, which is required to install pyproject.toml-based projects”. It seems to have something to do with cython.
There are more details regarding the unsuccessful installation of hdbscan in the log – see below.
Could somebody please help? I need to get the app back up.
Thanks!

my requirements file:

streamlit == 1.20.0
streamlit-aggrid == 0.2.2.post2
pandas == 1.4.0
umap-learn == 0.5.3
numpy == 1.21.6
plotly == 5.6.0
colorcet == 3.0.0
seaborn == 0.11.2
matplotlib == 3.5.1
bokeh == 2.4.2
datashader == 0.13.0
holoviews == 1.14.8
hdbscan == 0.8.28
joblib == 1.1.0

The error:

Building wheel for hdbscan (pyproject.toml): started
Building wheel for hdbscan (pyproject.toml): finished with status ‘error’
error: subprocess-exited-with-error

× Building wheel for hdbscan (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [168 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-39
creating build/lib.linux-x86_64-cpython-39/hdbscan
copying hdbscan/plots.py → build/lib.linux-x86_64-cpython-39/hdbscan
copying hdbscan/init.py → build/lib.linux-x86_64-cpython-39/hdbscan
copying hdbscan/validity.py → build/lib.linux-x86_64-cpython-39/hdbscan
copying hdbscan/robust_single_linkage_.py → build/lib.linux-x86_64-cpython-39/hdbscan
copying hdbscan/prediction.py → build/lib.linux-x86_64-cpython-39/hdbscan
copying hdbscan/hdbscan_.py → build/lib.linux-x86_64-cpython-39/hdbscan
copying hdbscan/flat.py → build/lib.linux-x86_64-cpython-39/hdbscan
creating build/lib.linux-x86_64-cpython-39/hdbscan/tests
copying hdbscan/tests/test_rsl.py → build/lib.linux-x86_64-cpython-39/hdbscan/tests
copying hdbscan/tests/init.py → build/lib.linux-x86_64-cpython-39/hdbscan/tests
copying hdbscan/tests/test_prediction_utils.py → build/lib.linux-x86_64-cpython-39/hdbscan/tests
copying hdbscan/tests/test_hdbscan.py → build/lib.linux-x86_64-cpython-39/hdbscan/tests
copying hdbscan/tests/test_flat.py → build/lib.linux-x86_64-cpython-39/hdbscan/tests
running build_ext
/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive ‘language_level’ not set, using ‘3str’ for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-t93c1xm8/hdbscan_60471374b0774a559eeccd19145f40bd/hdbscan/_hdbscan_tree.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
Compiling hdbscan/_hdbscan_tree.pyx because it changed.
[1/1] Cythonizing hdbscan/_hdbscan_tree.pyx
building ‘hdbscan._hdbscan_tree’ extension
creating build/temp.linux-x86_64-cpython-39
creating build/temp.linux-x86_64-cpython-39/hdbscan
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/adminuser/venv/include -I/usr/local/include/python3.9 -I/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/numpy/core/include -c hdbscan/_hdbscan_tree.c -o build/temp.linux-x86_64-cpython-39/hdbscan/_hdbscan_tree.o
In file included from /tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/numpy/core/include/numpy/ndarraytypes.h:1822,
from /tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from hdbscan/_hdbscan_tree.c:1097:
/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " “#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION” [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with "
| ^~~~~~~
gcc -pthread -shared build/temp.linux-x86_64-cpython-39/hdbscan/_hdbscan_tree.o -L/usr/local/lib -o build/lib.linux-x86_64-cpython-39/hdbscan/_hdbscan_tree.cpython-39-x86_64-linux-gnu.so
/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/Cython/Compiler/Main.py:381: FutureWarning: Cython directive ‘language_level’ not set, using ‘3str’ for now (Py3). This has changed from earlier releases! File: /tmp/pip-install-t93c1xm8/hdbscan_60471374b0774a559eeccd19145f40bd/hdbscan/_hdbscan_linkage.pyx
tree = Parsing.p_module(s, pxd, full_module_name)

  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  import numpy as np
  cimport numpy as np
  
  from libc.float cimport DBL_MAX
  
  from dist_metrics cimport DistanceMetric
  ^
  ------------------------------------------------------------
  
  hdbscan/_hdbscan_linkage.pyx:12:0: 'dist_metrics.pxd' not found
  
  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  import numpy as np
  cimport numpy as np
  
  from libc.float cimport DBL_MAX
  
  from dist_metrics cimport DistanceMetric
  ^
  ------------------------------------------------------------
  
  hdbscan/_hdbscan_linkage.pyx:12:0: 'dist_metrics/DistanceMetric.pxd' not found
  
  Error compiling Cython file:
  ------------------------------------------------------------
  ...
  
  
  cpdef np.ndarray[np.double_t, ndim=2] mst_linkage_core_vector(
          np.ndarray[np.double_t, ndim=2, mode='c'] raw_data,
          np.ndarray[np.double_t, ndim=1, mode='c'] core_distances,
          DistanceMetric dist_metric,
          ^
  ------------------------------------------------------------
  
  hdbscan/_hdbscan_linkage.pyx:58:8: 'DistanceMetric' is not a type identifier
  
  Error compiling Cython file:
  ------------------------------------------------------------
  ...
                  continue
  
              right_value = current_distances[j]
              right_source = current_sources[j]
  
              left_value = dist_metric.dist(&raw_data_ptr[num_features *
                                            ^
  ------------------------------------------------------------
  
  hdbscan/_hdbscan_linkage.pyx:129:42: Cannot convert 'double_t *' to Python object
  
  Error compiling Cython file:
  ------------------------------------------------------------
  ...
              right_value = current_distances[j]
              right_source = current_sources[j]
  
              left_value = dist_metric.dist(&raw_data_ptr[num_features *
                                                          current_node],
                                            &raw_data_ptr[num_features * j],
                                            ^
  ------------------------------------------------------------
  
  hdbscan/_hdbscan_linkage.pyx:131:42: Cannot convert 'double_t *' to Python object
  Compiling hdbscan/_hdbscan_linkage.pyx because it changed.
  [1/1] Cythonizing hdbscan/_hdbscan_linkage.pyx
  Traceback (most recent call last):
    File "/home/adminuser/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/home/adminuser/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/adminuser/venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 416, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 401, in _build_with_temp_dir
      self.run_setup()
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 487, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
      exec(code, locals())
    File "<string>", line 96, in <module>
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
      return run_commands(dist)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 349, in run
      self.run_command("build")
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
      self.run_command(cmd_name)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
      self.distribution.run_command(command)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
      super().run_command(command)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
      cmd_obj.run()
    File "<string>", line 26, in run
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
      self.build_extensions()
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
      self._build_extensions_serial()
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
      self.build_extension(ext)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/Cython/Distutils/build_ext.py", line 122, in build_extension
      new_ext = cythonize(
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1134, in cythonize
      cythonize_one(*args)
    File "/tmp/pip-build-env-lntya_qh/overlay/lib/python3.9/site-packages/Cython/Build/Dependencies.py", line 1301, in cythonize_one
      raise CompileError(None, pyx_file)
  Cython.Compiler.Errors.CompileError: hdbscan/_hdbscan_linkage.pyx
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for hdbscan

PS I also have a problem with logging in and I posted this problem to SF forums because that’s where I was referred to after a failed log in. Once I myself have figured out a workaround to log in, I also posted this question. However, nobody has replied to my log in problem posted almost a week ago, and so I am also posting this question here where the chances seem to be higher. Sorry for duplicating, but I really need help urgently. Thanks.

2

Hi!,

While I do not know if this will work in your case, I had this same problem once with another library. I updated the requirements file and removed the version number (just leave it at hdbscan) and restarted the app, which worked.
Like I said, it may not work in your case, but if you have the time, can you specify another version of hdbscan (or leave it as ‘hdbscan’ in the requirements file) and restart the app? Fingers crossed?

Cheers,
Moiz

3

Thanks, I will try it but I am skeptical. I tend to think I need to install something related to cython/C++ interpreter, but I don’t know exactly what, and I don’t even know what kind of OS the server is running so it’s hard to make targeted research.

4

Well, it’s worth a shot :slight_smile: . If it helps, I think the Streamlit community cloud server runs on a Debian based OS.

5

I’ve created a minimalistic app which doesn’t really DO anything, simply imports hdbscan and creates hdbscan model object, and requirements only contain streamlit and hdbscan without any version specification, but now I can’t deploy it at all - I don’t even get to the stage of installing imports.
tmp_error
Weird. The repo is public and I don’t have multiple logins. Hmm, perhaps in my haste to test it I’ve done something dumb?

tmp_test
tmp_test627×518 43.5 KB

6

Hi again!
I think I may have found the solution. Create another file in your repo named packages.txt and add build-essential in it.

I’ve deployed an app with HDBscan which is currently working. Here’s the link to the app, and the repo:

https://testing-123.streamlit.app

Cheers,
Moiz

1 Like
7

Many thanks, this looks promising! I will report back as soon as I’ve tested it.

1 Like
8

Amazing. I have not changed anything - I finally managed to log in, and I didn’t even have to reboot the test app, it was ready to use (I mean the minimalistic app as I described it above consisting of the requirements file and the python file works now without adding the packages.txt ).
image

Unfortunately the original app still failed with the hdbscan error. I’ve now removed the version specification of hdbscan as you have suggested but for a change I’m now facing the “app in the oven forever” issue, so I may need to try later again. Meanwhile I’ve also found out that a similar issue has recently been reported in the hdbscan repo for google colab Unable to install hdbscan on colab. · Issue #600 · scikit-learn-contrib/hdbscan (github.com)

9

It’s working again! In the end I deleted the app and re-deployed it using the requirements file without hdbscan version specification to pull the most recent one which currently is 0.8.33 and it has built and started correctly. I did not have to add the packages.txt file

image
image1556×786 48.2 KB

Now I just have to make sure that the new version of hdbscan doesn’t change something fundamental.
Thanks @momos for your help.

1 Like