Using Playwright with Streamlit?

🎈 Using Streamlit
1

Has anyone been able to get playwright to play nicely with Streamlit? I’ve got a very simple test that’s kicking up an error message I haven’t been able to make sense of. Here’s the app, using the hello-world example from playwright’s Python documentation:

import streamlit as st
from playwright.sync_api import sync_playwright

st.write(“Starting the test…”)

with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto(http://playwright.dev)
# st.write((page.title())
browser.close()

And here’s the error message:

NotImplementedError
Traceback:
File “c:\users\aschneiderman\anaconda3\lib\site-packages\streamlit\scriptrunner\script_runner.py”, line 557, in _run_script
exec(code, module.dict)
File “app.py”, line 7, in
with sync_playwright() as p:
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright\sync_api_context_manager.py”, line 87, in enter
dispatcher_fiber.switch()
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright\sync_api_context_manager.py”, line 67, in greenlet_main
self._loop.run_until_complete(self._connection.run_as_sync()) this
File “c:\users\aschneiderman\anaconda3\lib\asyncio\base_events.py”, line 616, in run_until_complete
return future.result()
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright_impl_connection.py”, line 210, in run_as_sync
await self.run()
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright_impl_connection.py”, line 219, in run
await self._transport.connect()
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright_impl_transport.py”, line 139, in connect
raise exc
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright_impl_transport.py”, line 127, in connect
self._proc = await asyncio.create_subprocess_exec(
File “c:\users\aschneiderman\anaconda3\lib\asyncio\subprocess.py”, line 236, in create_subprocess_exec
transport, protocol = await loop.subprocess_exec(
File “c:\users\aschneiderman\anaconda3\lib\asyncio\base_events.py”, line 1630, in subprocess_exec
transport = await self._make_subprocess_transport(
File “c:\users\aschneiderman\anaconda3\lib\asyncio\base_events.py”, line 491, in _make_subprocess_transport
raise NotImplementedError
Made with Streamlit

It’s choking on calling sync_playwright, so I assume it has something to do with the fact that I’m telling it to run synchronously, but I have to admit I don’t understand the wonderful world of synchronous vs asynchronous. Any thoughts on what to try? Googling didn’t produce helpful results. Thanks!

2

I ran into the same issue, and found an answer here: python - Why am I getting NotImplementedError with async and await on Windows? - Stack Overflow

Apparantly it’s Windows’s specific. Using the “ProactorEventLoop()” instead of the normal one does the job.

loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)

A working example:

import streamlit as st
import asyncio

async def main():
    proc = await asyncio.create_subprocess_exec(
        'ping', 'www.github.com', '-n', '1',
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE)

    stdout, stderr = await proc.communicate()
    print(stdout)
    st.text(stdout.decode())

if __name__ == '__main__':
    loop = asyncio.ProactorEventLoop()
    loop.run_until_complete(main())
3

Thank you so much, Nick!

4

Nick, maybe I’m being a little slow here. But I’m having trouble seeing how I would use what you discovered. How would I rewrite my example so that it would work? Are you saying but I would need to rewrite part of the playwright library? thanks!

5

Hi, Anders
Try this:

import streamlit as st
import asyncio
from playwright.async_api import async_playwright

st.write("Starting the test…")

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("http://playwright.dev")
        title = await page.title()
        st.write(title)
        await browser.close()
        return title

if __name__ == '__main__':
    loop = asyncio.ProactorEventLoop()
    asyncio.set_event_loop(loop)
    title=loop.run_until_complete(main())
    print(title)

NickDG give the right direction to solve the problem.

In fact, there are two problems here.

First, Streamlit does not support Playwright’s sync mode, so you need to change all the syncs to async. Here are the steps:

  1. Put everything into a function and write async before it.
  2. Change ‘with sync_playwright()’ to ‘with async_playwright’.
  3. Add await before every operation related to playwright.

This completes the process of changing sync to async, but there is another issue.

Secondly, as pointed out by NickDG, Streamlit seems to be constantly looping itself. In this case, async needs to be protected using a ProactorEventLoop() in order to isolate the execution of playwright.

6

And, This is only for local windows.
For Streamlit Cloud, I am still trying…

7

Thanks so much! Really appreciate it

8

However, I have been unable to successfully deploy Playwright to Streamlit Cloud. The following solutions have been tested:

  • Recognizing the Linux system.
  • Wrapping async in various ways.
  • Separating the Playwright part into a separate Python program and calling it through subprocess.

All of these have been tested but cannot run properly. In the end, I used Selenium to achieve my program’s goals.