Resource Limit Overages

Community Cloud
1

Greetings, I have deployed a Pytorch model on Streamlit Sharing (which is absolutely amazing by the way!!!) to https://share.streamlit.io/bigjoedata/rockbot/main/src/main.py

Full repository at:

It works for a bit, but then I’m running into time-outs (i.e. “[manager] Error checking Streamlit healthz: Get “http://localhost:8501/healthz”: dial tcp 127.0.0.1:8501: connect: connection refused” in the Streamlit console) or resource limit overages. I have done some profiling on my local Linux box and it looks like it uses just over 1GB of ram with 1 user, and just over 1.3GB of ram with 2 users. CPU is greedy and looks like it will choose to max out all cores.

I’m using the GPT-2 small model so my model is the smallest it can be using off the shelf GPT-2. I’m finetuning the DistilGPT model right now to see if that will work, but in the meantime I wanted to see if there were any other tricks I could incorporate to reduce resource consumption.

My question is, are there any recommended solutions to remedy this, or is it just a current limitation of the free Streamlit Sharing option?

Here is how I’m loading & caching the model, could this be improved? I don’t believe it’s loading multiple times, but perhaps?

    @st.cache(allow_output_mutation=True, ttl=120000, max_entries=1)
    def load_aitextgen():
        return aitextgen(model="bigjoedata/rockbot")

My log is attached inline below, I don’t believe I could remove any Streamlit dependencies? Perhaps there is a lighter version of Pytorch you recommend? I believe “torch==1.7.1” in my requirements.txt is loading the CPU version, but maybe not?

The log shows a GPU error, but I believe it doesn’t affect anything since it still works after defaulting back to CPU.

I don’t know your infrastructure, but If RAM is the issue, perhaps there’s a way to enable compression in the VM? Perhaps this is already enabled? I suspect this wouldn’t help since it’s probably a CPU bound issue.

Thanks in advance!

My log (relevant parts):

    [manager] Starting up for owner: 'bigjoedata', repository: 'rockbot', branch: 'main', main module: 'src/main.py'
    [manager] Cloning repository ...
    [manager] Cloning into 'rockbot'...

    [manager] Cloned repository!
    [manager] Processing dependencies ...
    Requirement already satisfied: altair==4.1.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (4.1.0)
    Requirement already satisfied: argon2-cffi==20.1.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (20.1.0)
    Requirement already satisfied: astor==0.8.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (0.8.1)
    Requirement already satisfied: async-generator==1.10 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (1.10)
    Requirement already satisfied: attrs==20.3.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (20.3.0)
    Requirement already satisfied: backcall==0.2.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (0.2.0)
    Requirement already satisfied: base58==2.0.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (2.0.1)
    Requirement already satisfied: bleach==3.2.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (3.2.1)
    Requirement already satisfied: blinker==1.4 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (1.4)
    Requirement already satisfied: cachetools==4.2.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (4.2.0)
    Requirement already satisfied: certifi==2020.12.5 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (2020.12.5)
    Requirement already satisfied: cffi==1.14.4 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (1.14.4)
    Requirement already satisfied: chardet in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (4.0.0)
    Requirement already satisfied: click==7.1.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 18)) (7.1.2)
    Requirement already satisfied: decorator==4.4.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 19)) (4.4.2)
    Requirement already satisfied: defusedxml==0.6.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 20)) (0.6.0)
    Requirement already satisfied: entrypoints==0.3 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 21)) (0.3)
    Requirement already satisfied: gitdb==4.0.5 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 26)) (4.0.5)
    Requirement already satisfied: idna==2.10 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 31)) (2.10)
    Requirement already satisfied: ipykernel==5.4.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 32)) (5.4.2)
    Requirement already satisfied: ipython==7.19.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 33)) (7.19.0)
    Requirement already satisfied: ipython-genutils==0.2.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 34)) (0.2.0)
    Requirement already satisfied: ipywidgets==7.6.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 35)) (7.6.2)
    Requirement already satisfied: jedi==0.18.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 36)) (0.18.0)
    Requirement already satisfied: Jinja2==2.11.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 37)) (2.11.2)
    Requirement already satisfied: jsonschema==3.2.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 39)) (3.2.0)
    Requirement already satisfied: jupyter-client==6.1.7 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 40)) (6.1.7)
    Requirement already satisfied: jupyter-core==4.7.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 41)) (4.7.0)
    Requirement already satisfied: jupyterlab-pygments==0.1.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 42)) (0.1.2)
    Requirement already satisfied: jupyterlab-widgets==1.0.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 43)) (1.0.0)
    Requirement already satisfied: MarkupSafe==1.1.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 45)) (1.1.1)
    Requirement already satisfied: mistune==0.8.4 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 46)) (0.8.4)
    Requirement already satisfied: nbclient==0.5.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 48)) (0.5.1)
    Requirement already satisfied: nbconvert==6.0.7 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 49)) (6.0.7)
    Requirement already satisfied: nbformat==5.0.8 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 50)) (5.0.8)
    Requirement already satisfied: nest-asyncio==1.4.3 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 51)) (1.4.3)
    Requirement already satisfied: notebook==6.1.6 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 52)) (6.1.6)
    Requirement already satisfied: packaging==20.8 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 55)) (20.8)
    Requirement already satisfied: pandas==1.2.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 56)) (1.2.0)
    Requirement already satisfied: pandocfilters==1.4.3 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 57)) (1.4.3)
    Requirement already satisfied: parso==0.8.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 58)) (0.8.1)
    Requirement already satisfied: pexpect==4.8.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 59)) (4.8.0)
    Requirement already satisfied: pickleshare==0.7.5 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 60)) (0.7.5)
    Requirement already satisfied: prometheus-client==0.9.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 62)) (0.9.0)
    Requirement already satisfied: protobuf==3.14.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 64)) (3.14.0)
    Requirement already satisfied: ptyprocess==0.7.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 65)) (0.7.0)
    Requirement already satisfied: pyarrow==2.0.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 66)) (2.0.0)
    Requirement already satisfied: pycparser==2.20 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 69)) (2.20)
    Requirement already satisfied: pydeck==0.5.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 70)) (0.5.0)
    Requirement already satisfied: Pygments==2.7.3 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 71)) (2.7.3)
    Requirement already satisfied: pyparsing==2.4.7 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 72)) (2.4.7)
    Requirement already satisfied: pyrsistent==0.17.3 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 73)) (0.17.3)
    Requirement already satisfied: python-dateutil==2.8.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 74)) (2.8.1)
    Requirement already satisfied: pytz==2020.5 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 76)) (2020.5)
    Requirement already satisfied: pyzmq==20.0.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 78)) (20.0.0)
    Requirement already satisfied: requests==2.25.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 80)) (2.25.1)
    Requirement already satisfied: Send2Trash==1.5.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 84)) (1.5.0)
    Requirement already satisfied: six==1.15.0 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 85)) (1.15.0)
    Requirement already satisfied: smmap==3.0.4 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 86)) (3.0.4)
    Requirement already satisfied: streamlit==0.73.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 87)) (0.73.1)
    Requirement already satisfied: testpath==0.4.4 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 92)) (0.4.4)
    Requirement already satisfied: toml==0.10.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 94)) (0.10.2)
    Requirement already satisfied: toolz==0.11.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 95)) (0.11.1)
    Requirement already satisfied: tornado==6.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 97)) (6.1)
    Requirement already satisfied: traitlets==5.0.5 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 99)) (5.0.5)
    Requirement already satisfied: typing-extensions==3.7.4.3 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 101)) (3.7.4.3)
    Requirement already satisfied: tzlocal==2.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 102)) (2.1)
    Requirement already satisfied: urllib3==1.26.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 103)) (1.26.2)
    Requirement already satisfied: validators==0.18.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 104)) (0.18.2)
    Requirement already satisfied: watchdog==1.0.2 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 105)) (1.0.2)
    Requirement already satisfied: wcwidth==0.2.5 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 106)) (0.2.5)
    Requirement already satisfied: webencodings==0.5.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 107)) (0.5.1)
    Requirement already satisfied: widgetsnbextension==3.5.1 in /usr/local/lib/python3.7/site-packages (from -r requirements.txt (line 109)) (3.5.1)
    Collecting absl-py==0.11.0
      Downloading absl_py-0.11.0-py3-none-any.whl (127 kB)
    Collecting aiohttp==3.7.3
      Downloading aiohttp-3.7.3-cp37-cp37m-manylinux2014_x86_64.whl (1.3 MB)
    Collecting aitextgen==0.3.0
      Downloading aitextgen-0.3.0.tar.gz (569 kB)
    Collecting async-timeout==3.0.1
      Downloading async_timeout-3.0.1-py3-none-any.whl (8.2 kB)
    Collecting filelock==3.0.12
      Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
    Collecting fire==0.3.1
      Downloading fire-0.3.1.tar.gz (81 kB)
    Collecting fsspec==0.8.5
      Downloading fsspec-0.8.5-py3-none-any.whl (98 kB)
    Collecting future==0.18.2
      Downloading future-0.18.2.tar.gz (829 kB)
    Collecting GitPython==3.1.12
      Downloading GitPython-3.1.12-py3-none-any.whl (159 kB)
    Collecting google-auth==1.24.0
      Downloading google_auth-1.24.0-py2.py3-none-any.whl (114 kB)
    Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.7/site-packages (from google-auth==1.24.0->-r requirements.txt (line 28)) (51.0.0)
    Collecting google-auth-oauthlib==0.4.2
      Downloading google_auth_oauthlib-0.4.2-py2.py3-none-any.whl (18 kB)
    Collecting grpcio==1.34.0
      Downloading grpcio-1.34.0-cp37-cp37m-manylinux2014_x86_64.whl (3.9 MB)
    Collecting joblib==1.0.0
      Downloading joblib-1.0.0-py3-none-any.whl (302 kB)
    Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/site-packages (from jsonschema==3.2.0->-r requirements.txt (line 39)) (3.3.0)
    Collecting Markdown==3.3.3
      Downloading Markdown-3.3.3-py3-none-any.whl (96 kB)
    Collecting multidict==5.1.0
      Downloading multidict-5.1.0-cp37-cp37m-manylinux2014_x86_64.whl (142 kB)
    Collecting numpy==1.19.5
      Downloading numpy-1.19.5-cp37-cp37m-manylinux2010_x86_64.whl (14.8 MB)
    Collecting oauthlib==3.1.0
      Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
    Collecting Pillow==8.1.0
      Downloading Pillow-8.1.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
    Collecting prompt-toolkit==3.0.9
      Downloading prompt_toolkit-3.0.9-py3-none-any.whl (355 kB)
    Collecting pyasn1==0.4.8
      Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
    Collecting pyasn1-modules==0.2.8
      Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
    Collecting pytorch-lightning==1.1.3
      Downloading pytorch_lightning-1.1.3-py3-none-any.whl (680 kB)
    Collecting PyYAML==5.3.1
      Downloading PyYAML-5.3.1.tar.gz (269 kB)
    Collecting regex==2020.11.13
      Downloading regex-2020.11.13-cp37-cp37m-manylinux2014_x86_64.whl (719 kB)
    Collecting requests-oauthlib==1.3.0
      Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
    Collecting rsa==4.6
      Downloading rsa-4.6-py3-none-any.whl (47 kB)
    Collecting sacremoses==0.0.43
      Downloading sacremoses-0.0.43.tar.gz (883 kB)
    Collecting tensorboard==2.4.0
      Downloading tensorboard-2.4.0-py3-none-any.whl (10.6 MB)
    Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/site-packages (from tensorboard==2.4.0->-r requirements.txt (line 88)) (0.36.2)
    Collecting tensorboard-plugin-wit==1.7.0
      Downloading tensorboard_plugin_wit-1.7.0-py3-none-any.whl (779 kB)
    Collecting termcolor==1.1.0
      Downloading termcolor-1.1.0.tar.gz (3.9 kB)
    Collecting terminado==0.9.2
      Downloading terminado-0.9.2-py3-none-any.whl (14 kB)
    Collecting tokenizers==0.9.4
      Downloading tokenizers-0.9.4-cp37-cp37m-manylinux2010_x86_64.whl (2.9 MB)
    Collecting torch==1.7.1
      Downloading torch-1.7.1-cp37-cp37m-manylinux1_x86_64.whl (776.8 MB)
    Collecting tqdm==4.55.1
      Downloading tqdm-4.55.1-py2.py3-none-any.whl (68 kB)
    Collecting transformers==4.1.1
      Downloading transformers-4.1.1-py3-none-any.whl (1.5 MB)
    Collecting Werkzeug==1.0.1
      Downloading Werkzeug-1.0.1-py2.py3-none-any.whl (298 kB)
    Collecting yarl==1.6.3
      Downloading yarl-1.6.3-cp37-cp37m-manylinux2014_x86_64.whl (294 kB)
    Collecting chardet
      Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/site-packages (from importlib-metadata->jsonschema==3.2.0->-r requirements.txt (line 39)) (3.4.0)
    Building wheels for collected packages: aitextgen, fire, future, PyYAML, sacremoses, termcolor
      Building wheel for aitextgen (setup.py): started
      Building wheel for aitextgen (setup.py): finished with status 'done'
      Created wheel for aitextgen: filename=aitextgen-0.3.0-py3-none-any.whl size=572525 sha256=ed5b2ba7926aacd7a10e61359d73a17e354d6d048116429d025c22433ec09ef2
      Stored in directory: /home/appuser/.cache/pip/wheels/77/69/72/10f4641cea6f2d0eafe8f14057acb2d210eb77e09f35d06550
      Building wheel for fire (setup.py): started
      Building wheel for fire (setup.py): finished with status 'done'
      Created wheel for fire: filename=fire-0.3.1-py2.py3-none-any.whl size=111006 sha256=839687330fa1b3ce4771d32b0204ba0e1d1bea5188be18fab7da3e803c887449
      Stored in directory: /home/appuser/.cache/pip/wheels/95/38/e1/8b62337a8ecf5728bdc1017e828f253f7a9cf25db999861bec
      Building wheel for future (setup.py): started
      Building wheel for future (setup.py): finished with status 'done'
      Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491059 sha256=6af78e111c0be8534030a026749bae04e796b0f5960afb66fb44e1701d7ffb5c
      Stored in directory: /home/appuser/.cache/pip/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0
      Building wheel for PyYAML (setup.py): started
      Building wheel for PyYAML (setup.py): finished with status 'done'
      Created wheel for PyYAML: filename=PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl size=44620 sha256=5f8925a00096cb87c44ecda706b0324f269863c96e90160aa33a446952daaafb
      Stored in directory: /home/appuser/.cache/pip/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653
      Building wheel for sacremoses (setup.py): started
      Building wheel for sacremoses (setup.py): finished with status 'done'
      Created wheel for sacremoses: filename=sacremoses-0.0.43-py3-none-any.whl size=893258 sha256=85d8192b640a426b77f25f4501745f62a97ae849c9f4ec177abf7b4a8ecc3abe
      Stored in directory: /home/appuser/.cache/pip/wheels/69/09/d1/bf058f7d6fa0ecba2ce7c66be3b8d012beb4bf61a6e0c101c0
      Building wheel for termcolor (setup.py): started
      Building wheel for termcolor (setup.py): finished with status 'done'
      Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4829 sha256=864c39516f39002d84e55e96e747d74bcb28cb66bd76bdc722e0399e3eafcd3b
      Stored in directory: /home/appuser/.cache/pip/wheels/3f/e3/ec/8a8336ff196023622fbcb36de0c5a5c218cbb24111d1d4c7f2
    Successfully built aitextgen fire future PyYAML sacremoses termcolor
    Installing collected packages: prompt-toolkit, pyasn1, chardet, terminado, rsa, pyasn1-modules, oauthlib, multidict, yarl, requests-oauthlib, google-auth, async-timeout, Werkzeug, tqdm, tensorboard-plugin-wit, regex, numpy, Markdown, joblib, grpcio, google-auth-oauthlib, fsspec, aiohttp, absl-py, torch, tokenizers, termcolor, tensorboard, sacremoses, PyYAML, future, filelock, transformers, pytorch-lightning, Pillow, GitPython, fire, aitextgen
    Successfully installed GitPython-3.1.12 Markdown-3.3.3 Pillow-8.1.0 PyYAML-5.3.1 Werkzeug-1.0.1 absl-py-0.11.0 aiohttp-3.7.3 aitextgen-0.3.0 async-timeout-3.0.1 chardet-3.0.4 filelock-3.0.12 fire-0.3.1 fsspec-0.8.5 future-0.18.2 google-auth-1.24.0 google-auth-oauthlib-0.4.2 grpcio-1.34.0 joblib-1.0.0 multidict-5.1.0 numpy-1.19.5 oauthlib-3.1.0 prompt-toolkit-3.0.9 pyasn1-0.4.8 pyasn1-modules-0.2.8 pytorch-lightning-1.1.3 regex-2020.11.13 requests-oauthlib-1.3.0 rsa-4.6 sacremoses-0.0.43 tensorboard-2.4.0 tensorboard-plugin-wit-1.7.0 termcolor-1.1.0 terminado-0.9.2 tokenizers-0.9.4 torch-1.7.1 tqdm-4.55.1 transformers-4.1.1 yarl-1.6.3
    [manager] Processed dependencies!
    /home/appuser/.local/lib/python3.7/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:100.)
      return torch._C._cuda_getDeviceCount() > 0
    2021-01-07 18:09:56.334 Downloading bigjoedata/rockbot model to /aitextgen.

    Downloading:   0%|          | 0.00/705 [00:00<?, ?B/s][2021-01-07 18:09:57.074867] 
    Downloading: 100%|██████████| 705/705 [00:00<00:00, 652kB/s][2021-01-07 18:09:57.520476] 

    Downloading:   0%|          | 0.00/510M [00:00<?, ?B/s][2021-01-07 18:09:57.629546] 
    Downloading:   0%|          | 1.11M/510M [00:00<00:49, 10.2MB/s][2021-01-07 18:09:57.72
    ...
    Downloading: 100%|██████████| 510M/510M [00:27<00:00, 18.6MB/s][2021-01-07 18:10:32.262739] 
    2021-01-07 18:10:32.262 Using the tokenizer for bigjoedata/rockbot.
    Model name 'bigjoedata/rockbot' not found in model shortcut name list (gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2). Assuming 'bigjoedata/rockbot' is a path, a model identifier, or url to a directory containing tokenizer files.

    Downloading:   0%|          | 0.00/899k [00:00<?, ?B/s][2021-01-07 18:10:33.003224] 
    Downloading:   4%|▍         | 36.9k/899k [00:00<00:02, 314kB/s][2021-01-07 18:10:33.130799] 
    Downloading:  23%|██▎       | 205k/899k [00:00<00:01, 407kB/s] [2021-01-07 18:10:33.258380] 
    Downloading:  98%|█████████▊| 877k/899k [00:00<00:00, 563kB/s][2021-01-07 18:10:33.260068] 
    Downloading: 100%|██████████| 899k/899k [00:00<00:00, 2.40MB/s][2021-01-07 18:10:33.866440] 

    Downloading:   0%|          | 0.00/456k [00:00<?, ?B/s][2021-01-07 18:10:33.987355] 
    Downloading:   8%|▊         | 36.9k/456k [00:00<00:01, 306kB/s][2021-01-07 18:10:34.114363] 
    Downloading:  45%|████▍     | 205k/456k [00:00<00:00, 397kB/s] [2021-01-07 18:10:34.179494] 
    Downloading: 100%|██████████| 456k/456k [00:00<00:00, 1.46MB/s][2021-01-07 18:10:34.838549] 

    Downloading:   0%|          | 0.00/387 [00:00<?, ?B/s][2021-01-07 18:10:34.840010] 
    Downloading: 100%|██████████| 387/387 [00:00<00:00, 309kB/s][2021-01-07 18:10:35.269251] 

    Downloading:   0%|          | 0.00/507 [00:00<?, ?B/s][2021-01-07 18:10:35.271364] 
    Downloading: 100%|██████████| 507/507 [00:00<00:00, 300kB/s][2021-01-07 18:10:35.487555] 
    loading file https://huggingface.co/bigjoedata/rockbot/resolve/main/vocab.json from cache at aitextgen/d0691afbae042a3bd4c8865ec801754a2a2f953819015b858639dfc70894b013.a552555fdda56a1c7c9a285bccfd44ac8e4b9e26c8c9b307831b3ea3ac782b45
    loading file https://huggingface.co/bigjoedata/rockbot/resolve/main/merges.txt from cache at aitextgen/f6b118284229eb46c9d543c0472dc46fd9a0c05a30024e28f92574bcca79f36f.5d12962c5ee615a4c803841266e9c3be9a691a924f72d395d3a6c6c81157788b
    loading file https://huggingface.co/bigjoedata/rockbot/resolve/main/added_tokens.json from cache at None
    loading file https://huggingface.co/bigjoedata/rockbot/resolve/main/special_tokens_map.json from cache at aitextgen/ba6c6cec6b3f197802aa9848326c053b80bedb7e35f20dd57558ea8011d13abe.09040d55a052267d3a9e1622f29f19d3664f0f0fcb2efd0ad7cc8b7fed2b4388
    loading file https://huggingface.co/bigjoedata/rockbot/resolve/main/tokenizer_config.json from cache at aitextgen/aded29162254404b8ace0ccd0116b96472e8178dda5da644dde0842185eb5742.5917d35bb936f0eb3d4fe369e7f6db44415999d90f1eb8c8860a50f39ad8aeb6
    loading file https://huggingface.co/bigjoedata/rockbot/resolve/main/tokenizer.json from cache at None
    [manager] Error checking Streamlit healthz: Get "http://localhost:8501/healthz": dial tcp 127.0.0.1:8501: connect: connection refused
    [manager] Error checking Streamlit healthz: Get "http://localhost:8501/healthz": dial tcp 127.0.0.1:8501: connect: connection refused
    [manager] Error checking Streamlit healthz: Get "http://localhost:8501/healthz": dial tcp 127.0.0.1:8501: connect: connection refused
    [manager] Error checking Streamlit healthz: Get "http://localhost:8501/healthz": dial tcp 127.0.0.1:8501: connect: connection refused
    [manager] Error checking Streamlit healthz: Get "http://localhost:8501/healthz": dial tcp 127.0.0.1:8501: connect: connection refused
    [manager] Error checking Streamlit healthz: Get "http://localhost:8501/healthz": dial tcp 127.0.0.1:8501: connect: connection refused
    [manager] Error checking Streamlit healthz: Get "http://localhost:8501/healthz": dial tcp 127.0.0.1:8501: connect: connection refused
    [manager] Error checking Streamlit healthz: Get "http://localhost:8501/healthz": dial tcp 127.0.0.1:8501: connect: connection refused
    [manager] Error checking Streamlit healthz: Get "http://localhost:8501/healthz": dial tcp 127.0.0.1:8501: connect: connection refused
    [manager] Error checking Streamlit healthz: Get "http://localhost:8501/healthz": dial tcp 127.0.0.1:8501: connect: connection refused
    [manager] Streamlit server consistently failed status checks
    [manager] Please fix the errors, push an update to the git repo, or reboot the app.
2

Hi @bigjoedata, glad you’re finding Streamlit sharing to be a useful service!

I think your instincts are correct about pytorch being the culprit. I don’t have any firm evidence, but I feel like most of the resource issues that are being reported seem to have this use case in mind. Is it possible to do a CPU-only version of Torch? I know you are saying that the app does fallback to CPU, but I wonder if there is a CPU-only version, it could be lighter on RAM usage without loading all the GPU code, exception handling, etc.

Internally, we are deeply analyzing the performance characteristics of the currently deployed apps, so that we can figure out how to provide more resources to the community without dramatically increasing our costs. Part of that will be creating paid tiers, but hopefully we can also find a way to set users resources in such a way that apps that obviously don’t need many resources can be scaled down, apps like yours can be scaled up, and yet maintain the same relative overall costs to us.

Best,
Randy

1 Like
3

@randyzwitch Yes indeed, I really like how the sharing is set up, it’s a pretty great experience and incredibly polished! I really like the automatic update upon push and console to see what’s going on during spin-up and installation.

I did some more tinkering under the hood and found a few things:

  • Retraining with a DistilGPT-2 time cut generation times by about 1/3 but the output quality dropped notably so I stuck with the original GPT-2 124M model.

  • Changing length of text massively impacts generation time so I have set the default lower

  • Pytorch was partially a culprit, but optimizing only helps a bit. For reference for any others, I forced the smaller pytorch version by changing my requirements.txt from
    torch
    to
    --find-links https://download.pytorch.org/whl/torch_stable.html torch==1.7.1+cpu

I pushed out a new commit and it seems to run stably, though I did notice it crashed at the time of another commit; one guess is that something is off in the caching, but I’ll keep an eye on it.

Thanks again!

1 Like