How to use pyautogui in streamlit app?

🎈 Using Streamlit
1 
Traceback (most recent call last):

  File "/home/adminuser/venv/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 534, in _run_script

    exec(code, module.__dict__)

  File "/mount/src/virtual_home_decorator_app/app.py", line 4, in <module>

    import pyautogui

  File "/home/adminuser/venv/lib/python3.9/site-packages/pyautogui/__init__.py", line 246, in <module>

    import mouseinfo

  File "/home/adminuser/venv/lib/python3.9/site-packages/mouseinfo/__init__.py", line 223, in <module>

    _display = Display(os.environ['DISPLAY'])

  File "/usr/local/lib/python3.9/os.py", line 679, in __getitem__

    raise KeyError(key) from None

KeyError: 'DISPLAY'
2

You cannot use any graphic library that bypasses the browser. This will fail as soon as the streamlit app runs on any server or hosted environment.

3

I am trying to get mouse coordinates , I have tried pywin32 and then pyautogui but both are not working on streamlit cloud, so whats the alternative?

4

You have to refactor your whole approach, it will not work this way.

Of course, this does not work because there is a fundamental misunderstanding about how Streamlit works:

  1. Besides the main problem below, pywin32 wil not work at all, because it is a windows only library and streamlit cloud is a linux debian based environment.

  2. But the main problem is:
    Streamlit consists of a frontend and backend part aka Client and Server. Any interaction with streamlit must be done via the browser or browser APIs. Anything trying to bypass this, may work on your local computer, but will fail as soon as streamlit is hosted, i.e. streamlit no longer runs on the same computer as the browser. It’s just a lucky coincidence that this works on the local computer, because the client and server are identical in this case.
    But what happens on Streamlit Cloud when you try to open pyautogui (or other) gui widgets or try to access hardware? They will be opened on the computer on which the python application is running. But there is neither an UI nor a graphics driver nor a mouse device on streamlit cloud (aka Server) and even if there was, it would be the wrong computer.

You need a Streamlit component that can read mouse coordinates. I don’t know if this already exists.

5

This topic was automatically closed 2 days after the last reply. New replies are no longer allowed.