Deployment issues: is inference with large models like BART possible?

☁️ Community Cloud
1

Hi,

I am having trouble deploying my news summarisation app. Originally I had requirements issues related to pytorch (required for transformers) but these appear to have been fixed according to the latest logs.

Despite receiving no errors and it being in my requirements.txt file, it looks like newsapi hasn’t installed?

Any help would be much appreciated.

2

Hi @jamie.dorri,

Welcome to our Streamlit Community! :partying_face: :partying_face: :partying_face: :partying_face: :partying_face: :tada:

Without a link to your github repo we won’t be able to help you debug this (there could be many things happening), can you link your repo here please?

Thanks!
Marisa

1 Like
3

Sorry, I fixed this (newsapi-python not newsapi is the package name), but now having with the pytorch backend (a Nvidia driver issue). To be clear, I am performing only CPU inference. Are you able to see my requirements file?

4

Sorry, yeah, it’s here:

5

Hey!

So on this one do you get an error when you’re trying to deploy? I am able to see your requirements file now thanks!

Marisa

6

Yes, I do. Here is the error:

/home/appuser/.local/lib/python3.7/site-packages/torch/cuda/init.py:52: UserWarning: CUDA initialization: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:100.)

I originally installed torch via transformers[torch] but modified the req file to work with streamlit sharing. Maybe I can fix this with the +cpu syntax, i.e., torch==1.7.0+cpu

7

My Streamlit app works locally. Don’t worry, I will get to the bottom of it eventually!

8

I’ve fixed these on my side now using the +cpu change. However, the app is crashing when I try to perform inference.

Is this a memory error? I’m using transformer’s bart-large-cnn model (1.6GB). Here is an example of an Streamlit app using large NLP models successfully, showing its possible:

https://share.streamlit.io/e-tony/story_generator/main/app.py

Thanks again!