Using Playwright with Streamlit?

[Anders]

Has anyone been able to get playwright to play nicely with Streamlit? I’ve got a very simple test that’s kicking up an error message I haven’t been able to make sense of. Here’s the app, using the hello-world example from playwright’s Python documentation:

import streamlit as st
from playwright.sync_api import sync_playwright

st.write(“Starting the test…”)

with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto(http://playwright.dev)
# st.write((page.title())
browser.close()

And here’s the error message:

NotImplementedError
Traceback:
File “c:\users\aschneiderman\anaconda3\lib\site-packages\streamlit\scriptrunner\script_runner.py”, line 557, in _run_script
exec(code, module.dict)
File “app.py”, line 7, in
with sync_playwright() as p:
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright\sync_api_context_manager.py”, line 87, in enter
dispatcher_fiber.switch()
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright\sync_api_context_manager.py”, line 67, in greenlet_main
self._loop.run_until_complete(self._connection.run_as_sync()) this
File “c:\users\aschneiderman\anaconda3\lib\asyncio\base_events.py”, line 616, in run_until_complete
return future.result()
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright_impl_connection.py”, line 210, in run_as_sync
await self.run()
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright_impl_connection.py”, line 219, in run
await self._transport.connect()
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright_impl_transport.py”, line 139, in connect
raise exc
File “c:\users\aschneiderman\anaconda3\lib\site-packages\playwright_impl_transport.py”, line 127, in connect
self._proc = await asyncio.create_subprocess_exec(
File “c:\users\aschneiderman\anaconda3\lib\asyncio\subprocess.py”, line 236, in create_subprocess_exec
transport, protocol = await loop.subprocess_exec(
File “c:\users\aschneiderman\anaconda3\lib\asyncio\base_events.py”, line 1630, in subprocess_exec
transport = await self._make_subprocess_transport(
File “c:\users\aschneiderman\anaconda3\lib\asyncio\base_events.py”, line 491, in _make_subprocess_transport
raise NotImplementedError
Made with Streamlit

It’s choking on calling sync_playwright, so I assume it has something to do with the fact that I’m telling it to run synchronously, but I have to admit I don’t understand the wonderful world of synchronous vs asynchronous. Any thoughts on what to try? Googling didn’t produce helpful results. Thanks!

[NickDG]

I ran into the same issue, and found an answer here: python - Why am I getting NotImplementedError with async and await on Windows? - Stack Overflow

Apparantly it’s Windows’s specific. Using the “ProactorEventLoop()” instead of the normal one does the job.

loop = asyncio.ProactorEventLoop()
asyncio.set_event_loop(loop)

A working example:

import streamlit as st
import asyncio

async def main():
    proc = await asyncio.create_subprocess_exec(
        'ping', 'www.github.com', '-n', '1',
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE)

    stdout, stderr = await proc.communicate()
    print(stdout)
    st.text(stdout.decode())

if __name__ == '__main__':
    loop = asyncio.ProactorEventLoop()
    loop.run_until_complete(main())

[Anders]

Thank you so much, Nick!

[Anders]

Nick, maybe I’m being a little slow here. But I’m having trouble seeing how I would use what you discovered. How would I rewrite my example so that it would work? Are you saying but I would need to rewrite part of the playwright library? thanks!

[goldengrape]

Hi, Anders
Try this:

import streamlit as st
import asyncio
from playwright.async_api import async_playwright

st.write("Starting the test…")

async def main():
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        page = await browser.new_page()
        await page.goto("http://playwright.dev")
        title = await page.title()
        st.write(title)
        await browser.close()
        return title

if __name__ == '__main__':
    loop = asyncio.ProactorEventLoop()
    asyncio.set_event_loop(loop)
    title=loop.run_until_complete(main())
    print(title)

NickDG give the right direction to solve the problem.

In fact, there are two problems here.

First, Streamlit does not support Playwright’s sync mode, so you need to change all the syncs to async. Here are the steps:

1. Put everything into a function and write async before it.
2. Change ‘with sync_playwright()’ to ‘with async_playwright’.
3. Add await before every operation related to playwright.

This completes the process of changing sync to async, but there is another issue.

Secondly, as pointed out by NickDG, Streamlit seems to be constantly looping itself. In this case, async needs to be protected using a ProactorEventLoop() in order to isolate the execution of playwright.

[goldengrape]

And, This is only for local windows.
For Streamlit Cloud, I am still trying…

[Anders]

Thanks so much! Really appreciate it

[goldengrape]

However, I have been unable to successfully deploy Playwright to Streamlit Cloud. The following solutions have been tested:

• Recognizing the Linux system.
• Wrapping async in various ways.
• Separating the Playwright part into a separate Python program and calling it through subprocess.

All of these have been tested but cannot run properly. In the end, I used Selenium to achieve my program’s goals.

[Theo_K]

Hello, is there any solution comes out?

[contractorwolf]

I have an idea why this may not work. When using Playwright a simple “pip install playwright” is not all you need to do to use Playwright locally. You must also run the command “playwright install” to download the necessary browser code. I hit this same issue when trying to box up my streamlit app to add it to the cloud. I made sure to use a fresh conda environment and hit this same issue locally

The solve is right there in the command line “playwright install” but i dont know how one could do that in the cloud streamlit. When i hit that same error locally I did just that and executed the “playwright install” command and had to wait for the browser to download and install and then it worked. I am new to cloud streamlit and am unsure how one might do that here.

also I have custom GPT’s for both streamlit:

ChatGPT - Streamlit Developer Assistant

and playwright:

ChatGPT - Playwright in Python Coding Assistant

if anyone needs some advice you are welcome to use them, they have helper me a ton

[contractorwolf]

Sorry, I took some of the idea above and got something that is working now. First I added these to my code:

os.system('playwright install')
os.system('playwright install-deps')

I also created a packages.txt file that contained all of the following:

libnss3
libnspr4
libatk1.0-0
libatk-bridge2.0-0
libcups2
libdrm2
libxkbcommon0
libxcomposite1
libxdamage1
libxfixes3
libxrandr2
libgbm1
libpango-1.0-0
libcairo2
libasound2
libatspi2.0-0
libwayland-client0
libxcursor1
libgtk-3-0
libpangocairo-1.0-0
libcairo-gobject2
libgdk-pixbuf-2.0-0
libx11-xcb1

I uploaded it all and the app now works as expected:

streamlit

[ThunderStriker]

Yupp. That worked for me too. I replaced ProactorEventLoop() to new_event_loop() as in this code:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
Apart from that, @contractorwolf posted a great solution

[ffsxsy]

hi,i take similar steps with you ,but fail:
‘’’
import os
os.system(“playwright install --with-deps chromium”)
‘’’
and same lines in packages.txt file,
but get the following error resp: