FileNotFoundError: What is the location of file that is written using streamlit script hosted on streamlit cloud (subprocess using written file gives error on streamlit cloud but works on local machine)

streamlit app is working fine on local machine but getting Error on streamlit cloud.

Through this app - after the user file upload and performing Analytics from this page on it I am trying to run quarto document qmd file using subprocess to generate downloadable html report from this app page where the issue arises in generating report.

It’s hosted on streamlit cloud with streamlit link Github Repo link.

Doubt/Issue: In order to run quarto document (Report-template.qmd) that requires data processed in streamlit pages I am writing dataframe and a text file (which were processed in streamlit pages) to disk using below code:

ss.pl_df.write_csv(ss.file_name_csv)

with open(ss.ai_failed_file_name, 'w') as f:
    f.write(ss.assistant_response)

this works fine when working in local environment widows 11 machine and even working here but it I am not sure where the files get written when on streamlit cloud/hosting.

I think these files that I am writing within streamlit cloud doesn’t go to github repo so the file location mapping is failing in cloud where as it was able to pickup file correctly from root directory when running in local machine.

What should be the location of the files that I am writing it in streamlit cloud to provide the correct path to the quarto document to use these files.
(I am aware it will be available only temporarily/ till session last and that is the purpose as well.)

I am not sure if this is the right cause of issue or something else but this seems to be the most logical possibility to me.

Error while running app:

```
────────────────────── Traceback (most recent call last) ───────────────────────
```

```
  /home/adminuser/venv/lib/python3.12/site-packages/streamlit/runtime/scriptru  
```

```
  nner/exec_code.py:88 in exec_func_with_error_handling                         
```


```
  /home/adminuser/venv/lib/python3.12/site-packages/streamlit/runtime/scriptru  
```

```
  nner/script_runner.py:590 in code_to_exec                                     
```


```
  /mount/src/healthquant/pages/6_AnalyticsReport_Download.py:109 in <module>    
```


```
    106 │   # st.write(ss.cmd_str)                                              
```

```
    107 │                                                                       
```

```
    108 │   # run Quarto Document using command through subprocess              
```

```
  ❱ 109 │   subprocess.run(ss.cmd_str)                                          
```

```
    110 │                                                                       
```

```
    111 │   # result = subprocess.run(ss.cmd_str, capture_output=True)          
```

```
    112 │   # print(result)                                                     
```


```
  /usr/local/lib/python3.12/subprocess.py:548 in run                            
```


```
     545 │   │   kwargs['stdout'] = PIPE                                        
```

```
     546 │   │   kwargs['stderr'] = PIPE                                        
```

```
     547 │                                                                      
```

```
  ❱  548 │   with Popen(*popenargs, **kwargs) as process:                       
```

```
     549 │   │   try:                                                           
```

```
     550 │   │   │   stdout, stderr = process.communicate(input, timeout=timeo  
```

```
     551 │   │   except TimeoutExpired as exc:                                  
```


```
  /usr/local/lib/python3.12/subprocess.py:1026 in __init__                      
```

```
    1023 │   │   │   │   │   self.stderr = io.TextIOWrapper(self.stderr,        
```

```
    1024 │   │   │   │   │   │   │   encoding=encoding, errors=errors)          
```

```
    1025 │   │   │                                                              
```

```
  ❱ 1026 │   │   │   self._execute_child(args, executable, preexec_fn, close_f  
```

```
    1027 │   │   │   │   │   │   │   │   pass_fds, cwd, env,                    
```

```
    1028 │   │   │   │   │   │   │   │   startupinfo, creationflags, shell,     
```

```
    1029 │   │   │   │   │   │   │   │   p2cread, p2cwrite,                     
```


```
  /usr/local/lib/python3.12/subprocess.py:1955 in _execute_child                
```

```
    1952 │   │   │   │   │   if errno_num != 0:                                 
```

```
    1953 │   │   │   │   │   │   err_msg = os.strerror(errno_num)               
```

```
    1954 │   │   │   │   │   if err_filename is not None:                       
```

```
  ❱ 1955 │   │   │   │   │   │   raise child_exception_type(errno_num, err_msg  
```

```
    1956 │   │   │   │   │   else:                                              
```

```
    1957 │   │   │   │   │   │   raise child_exception_type(errno_num, err_msg  
```

```
    1958 │   │   │   │   raise child_exception_type(err_msg)                    
```

```
────────────────────────────────────────────────────────────────────────────────
```

```
FileNotFoundError: [Errno 2] No such file or directory: 'quarto render 
```

```
Report-template.qmd -P file_name:"df_65e5cd09-3f9a-4ec0-af4c-617699c674e7.csv" 

-P user_name:"Mr. /Ms. xxx" --output 

"Report-out-65e5cd09-3f9a-4ec0-af4c-617699c674e7.html"'

```

```
/mount/src/healthquant/utilities_v3.py:64: SyntaxWarning:
```

```
invalid escape sequence '\|'
```

```
/mount/src/healthquant/functions_parse.py:70: SyntaxWarning:
```

```
invalid escape sequence '\|'
```

```
/mount/src/healthquant/functions_parse_max.py:57: SyntaxWarning:
```

```
invalid escape sequence '\|'
```

Requirements.txt

streamlit==1.38.0
pandas==2.2.2
polars==1.6.0
streamlit_lottie==0.0.5
nest_asyncio==1.6.0
llama_parse==0.5.0
fuzzywuzzy==0.18.0
plotly==5.24.1
streamlit_option_menu==0.3.13
pdftotext==2.2.2
groq==0.11.0
httpx==0.27.2
pip==24.3.1
papermill==2.6.0
plotnine==0.14.3
nbclient==0.10.1
nbformat==5.10.4
uuid==1.30
quarto-cli==1.6.37

Github files:

If I do df.write_csv(‘filename.csv’) in streamlit script and then try to read it back in streamlit then what would be the path/location of the file when hosted on streamlit cloud ?

You can read it back with just pd.read_csv('filename.csv'). The absolute path in streamlit cloud would be, in your case, /mount/src/healthquant/filename.csv, but using relative paths is better. This is assuming that the working directory doesn’t change.

thanks @Goyo but seems like this Error is not related to .csv files writing as I can see them getting created.

I think it is not able to recognize quarto render command in subprocess so I tried to install quarto==1.6.39 & quarto==1.6.37 qurto_link through requirements.txt but getting error in installing quarto through pip:

ERROR: Could not find a version that satisfies the requirement quarto==1.6.39 (from versions: 0.1.0)
ERROR: No matching distribution found for quarto==1.6.39

Is there a way to install packages through conda command on streamlit cloud. I think I installed quarto package through conda install conda-forge::quarto in my local machine when it was not installing through pip in my local machine.

Or how can I include quarto from packages.txt if that could be a solution ?

Indeed, the format of the traceback is messed up, but I think you are calling subprocess.run with the wrong arguments.

There is nothing like that in pypi

Maybe. See Other Python package managers.

There is quarto-cli, that looks promising.

Thanks for the quick response and sharing the relevant links with me. And yes the error and traceback is not at all helpful in this case.

  1. I already have quarto-cli installed from the beginning but was still getting error.

  2. To cross check the cause of error I tried below command

'quarto render Report-test.qmd --output Report-test.html'

which is without the arguments (of other files) on a new quarto document Report-test.qmd that is suppose to just read csv file already present in the Github repo and print head and tried this in my local machine as well and it worked in local.

```{python}

df = pd.read_csv(r"2024-09-25_All_Combined.csv",parse_dates=True,dayfirst=True)

df = df.drop_duplicates()

```

```{python}

df.head()

```

but this too failed on streamlit cloud with the same error as earlier.

FileNotFoundError: [Errno 2] No such file or directory: 'quarto render 
Report-test.qmd --output Report-test.html'

I think there is high probability that quarto command is not getting recognized due to missing quarto package and I would need to install that using conda as that is what I did to run it in my local machine as well.

I am thinking of including environment.yml file which will run conda-forge::quarto and will also call the requirements.txt as it was shown in this SO link.

name: test-env
channels:
  - conda-forge
dependencies:
  - python>=3.12.4
  - conda-forge::quarto
  - pip
  - pip:
    - -r file:requirements.txt

I am not sure if this can work but need to try to test out.

update: it is not loading any packages … may be I need to mention all the packages in this env*.yml file only

As I said, the error message suggests that your call to subprocess is wrong. If you are doing something like this

subprocess.run('quarto render Report-test.qmd --output Report-test.html')

it won’t work on streamlit cloud. As stated in the docs:

On POSIX, if args is a string, the string is interpreted as the name or path of the program to execute. However, this can only be done if not passing arguments to the program.

See also the note right below that on how to break a shell command into a sequence of arguments. Something like this should work:

subprocess.run(
    shlex.split("quarto render Report-test.qmd --output Report-test.html")
)

Thank you so much. I had no idea about this as it was working fine without any of this in my local so never thought I would need all (shlex) of this.

So it has now worked using shlex and processed which is a big relief but unfortunately with few more errors.

so qmd is sort of a jupyter file with yaml headers and now its giving me error related to jupyter not found even when jupyter is already mentioned in requirements.txt

Starting python3 kernel...[2024-12-17 12:15:30.617759] Traceback (most recent call last):

  File "/home/adminuser/venv/lib/python3.12/site-packages/quarto_cli/share/jupyter/jupyter.py", line 21, in <module>

    from notebook import notebook_execute, RestartKernel

  File "/home/adminuser/venv/lib/python3.12/site-packages/quarto_cli/share/jupyter/notebook.py", line 15, in <module>

    from yaml import safe_load as parse_string

ModuleNotFoundError: No module named 'yaml'

Python 3 installation:
  Version: 3.12.7
  Path: /usr/local/bin/python3
  Jupyter: (None)

Jupyter is not available in this Python installation.
Install with python3 -m pip install jupyter

There is a requirements.txt file in this directory. Is this for a venv that you need to restore?

However, jupyter & yaml are already there in requirements.txt

Does it require some other ways to install packages separately for file running through subprocess or something to process jupyter ?

This is hard to grasp without having the code but it looks like quarto is running the wrong python.

Python 3 installation:
  Version: 3.12.7
  Path: /usr/local/bin/python3
  Jupyter: (None)

But your app is running in a virtual environment and you want quarto to use the python in that same environment. I have no idea how to do that, you may have to sort it out with the quarto devs or users.

Maybe try this

subprocess.run(
    shlex.split(f"{sys.executable} -m quarto render Report-test.qmd --output Report-test.html")
)

well this has worked. It is not giving me that error now but now it is giving another error [14:13:21] 📦 Processed dependencies! /home/adminuser/venv/bin/python: No module named quarto

quarto-cli is already there. Should I try installing Quarto through conda ?

I have created a Sample one page app hosted on streamlit cloud with public git repo to run quarto qmd file with same packages. It is also giving the same error.

Anyone from streamlit team can try and access the codes through this repo and can reproduce the error.

url: https://quarto-test.streamlit.app/
Github repo: quarto_test/Home.py at main · sunshineeast/quarto_test · GitHub

Apparently you cannot run quarto as a python module, so you need

subprocess.run(shlex.split("quarto render Report-test.qmd --output Report-test.html"))

and then the problem is that quarto runs python code using the system python environment instead of the virtual environment.

I tried to replicate this behavior locally without success. I created a virtual environment, activated it, installed all the needed packages and ran your application without issues. Crucially, quarto seemed to run the python code in the same virtual environment that the app is running in and was able to find packages that are installed only in the virtual environment.

However streamlit cloud seems to be doing something different, but I have no idea what the difference can be.

I was able to make this work by creating a bash script that activates the venv and then runs quarto.

source /home/adminuser/venv/bin/activate
quarto render $1 --output $2

Then I can run the script from python instead of running quarto directly.

subprocess.run(["bash", script_path, "Report-test.qmd", "Report-test.html"])

Woww… This looks great !!

I haven’t tried it yet but will run it soon and hopefully it will work for me as well.

Thanks alot for the great help and trying to figure out alternate solutions to get this to work. Couldn’t have manage to work without your help. Really Really appreciate your help.

You are amazing !!!

It worked well.

I was doing a mistake by creating shell script in windows which was giving me error \r not recognized which was due to using file editor in windows.

After taking care of this it has worked in this sample app. Thank you so much again :slight_smile:

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.