You might be suffering from the issue described here maximum recursion depth exceeded · Issue #3094 · streamlit/streamlit (github.com). Pretty much st.experimental_rerun is not designed to support an infinite rerun loop according to @kmcgrady.
Not sure why this limitation is there, but I would love it to be lifted. Like you, I am trying to write an app that would consume some incoming data stream.
Some people have suggested using asyncio, but unfortunately input widgets do not work with this technique.
… Ah! I just noticed that @kmcgrady has written an autorefresh component Streamlit Autorefresh - 
Streamlit Components - Streamlit. Perhaps this would help us avoid the RecursionError.