Streamlit creates too many files and stops working

[nekitmm]

Hi, everyone!

We created a small app using Stremalit that’s hosted on an EC2 instance and after a small stress test with only a dozen of users encountered this error:

handle: <Handle BaseAsyncIOLoop._handle_events(12, 1)>
Traceback (most recent call last):
File “/root/anaconda3/envs/esg/lib/python3.8/asyncio/events.py”, line 81, in _run
File “/root/anaconda3/envs/esg/lib/python3.8/site-packages/tornado/platform/asyncio.py”, line 206, in _handle_events
File “/root/anaconda3/envs/esg/lib/python3.8/site-packages/tornado/netutil.py”, line 267, in accept_handler
File “/root/anaconda3/envs/esg/lib/python3.8/socket.py”, line 292, in accept
OSError: [Errno 24] Too many open files

I looked online and found some advice on increasing the maximum number of files per process allowed (I used ulimit -n 50000, was 1024) but that doesn’t seem clean…

I am now monitoring the number of files created by streamlit (using ls /proc/$pid/fd/ | wc -l command) and it seems like the number of files is increasing by 2 on almost every click and the number of files is mounting.

Any ideas on what we might be doing wrong? We use the latest version of streamlit (1.11.1) and Python 3.8.

Thank you very much!

[willhuang]

Hi @nekitmm , I’m not sure if I can currently help without a code snippet. However, is there a reason for opening so many files? Can you not close any?

[nekitmm]

Hi, all!

@willhuang - thanks for your response - the whole app is pretty large and it would be impossible to give a snippet, especially given that I had no idea what was causing the problem.

Turns out the problem was not related to tornado or streamlit. We use Snowpark connector to connect to Snowflake and every connection creates an ocsp response cache file that’s not deleted until you explicitly close the connection. We were accumulating these files up to the limit.

This raises a whole number of questions on how to handle this situation as opening and closing connection on every interaction with Snowflake is too slow, but we can not close the connection as we don’t know when the user will leave the app. However, this is a whole another problem.

Hope this might help someone else. Thank you!