*Connection Error When Deploying Streamlit App Using Local Ollama LLM Model*

Deployment
,
1

I’m experiencing a persistent connection error when deploying my Streamlit app that relies on an Ollama LLM model running on my local machine.

Environment Details:

  • Local Machine:
    • Operating System: macOS
    • Ollama Version: 0.1.31 (also tried downgrading to 0.0.11)
  • Streamlit App:
    • Uses LangChain, Ollama LLM, and Chroma for vector storage
  • Deployment Platform:
    • Streamlit Community Cloud

Problem Description:

  • When running the app locally, it functions correctly and communicates with the Ollama server without issues.

  • Upon deploying the app on Streamlit Community Cloud, I encounter the following error:

    Connection error: HTTPConnectionPool(host='localhost', port=11434): Max retries exceeded with url: /api/generate (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f01295e4380>: Failed to establish a new connection: [Errno 111] Connection refused'))

What I’ve Tried So Far:

  1. Verified Ollama Server is Running:

    • Started the Ollama server using ollama serve.

    • Confirmed it’s listening on port 11434.

    • Checked running processes with ps aux | grep ollama and saw:

      /Applications/Ollama.app/Contents/Resources/ollama serve

  2. Checked API Endpoints:

    • Accessed http://localhost:11434 and received "ollama is running".
    • Tried curl http://localhost:11434/api/ps and got {"models":[]}.
    • Attempted curl http://localhost:11434/api/models and curl http://localhost:11434/api/generate, but received 404 Not Found errors.

  3. Verified Installed Models:

    • Ran ollama list and confirmed models like llama2, llama3, and others are installed.

      NAME                  ID              SIZE      MODIFIED
llama2:latest         78e26419b446    3.8 GB    3 hours ago
llama3:latest         365c0bd3c000    4.7 GB    13 days ago

  4. Tested API Calls:

    • Tried curl -X POST http://localhost:11434/api/generate -d '{"model": "llama2", "prompt": "Hello"}' and received a 404 Not Found error.
    • Noted that curl http://localhost:11434/api/ps returns {"models":[]}, indicating no models are running.

  5. Checked for Port Conflicts and Firewall Issues:

    • Used lsof -i :11434 to confirm no other service is occupying the port.
    • Ensured firewall settings are not blocking port 11434.

  6. Verified Ollama Version and API Changes:

    • Found that Ollama version 0.1.31 has deprecated the REST API endpoints like /api/generate.
    • Recognized that the 404 errors are due to these deprecated endpoints.

  7. Attempted to Use Correct API Endpoints:

    • Modified the base URL in the code to remove /api prefix.
    • Tested endpoints like http://localhost:11434/generate, but still received 404 Not Found errors.

  8. Attempted to Uninstall Ollama and Install an Older Version:

    • Tried brew uninstall ollama, but received Error: Cask 'ollama' is not installed.
    • Realized Ollama was installed as a standalone application in /Applications/Ollama.app.

  9. Installed Ollama Version 0.0.11 with REST API Support:

    • Downloaded the older version from Ollama Releases.
    • Installed the binary to /usr/local/bin.
    • Verified installation with ollama -v, which now shows ollama version is 0.0.11.
    • Started the server with ollama serve.
    • Successfully accessed the REST API endpoints.

  10. Deployment Challenges Remain:

    • Deployed the app on Streamlit Community Cloud.
    • Still receiving connection errors because the deployed app cannot reach the local Ollama server.

My Questions:

  1. Is it possible for a deployed Streamlit app to connect to a local Ollama server?

    • Given that the app is running on Streamlit Community Cloud and the Ollama server is on my local machine, I suspect network connectivity issues are preventing communication.

    • Considering the complexities and potential security risks, would switching to a cloud-based LLM service be more practical?

What I’m Seeking:

  • Advice on how to enable communication between the deployed Streamlit app and the Ollama server.

Thank you for your assistance!

I appreciate any guidance or suggestions on how to proceed.