How to use the Playwright library in jupyter notebook instead of using a .py script

Question

I want to use an automated browser and execute my steps with jupyter notebook cells instead of using .py scripts. This works fine with the browser automation library called selenium.

It does not work fine with the library called Playwright. In fact it doesn't work at all. I tried every single line of code they provided in their manual. NOTHING works in jupyter notebooks. EVERYTHING works fine on my machine as long as copy-pasting the same code in some .py file and executing it. Various examples that I'm talking about can be found here: https://playwright.dev/python/docs/intro

I really don't get why I'm unable to make it work in a jupyter notebook, especially if it works fine in literally every .py file.

Answer 1

as mentioned in https://github.com/microsoft/playwright-python/issues/480

Jupyter notebook uses asyncio event loop, so you should use async api.

from playwright.async_api import async_playwright

playwright = await async_playwright().start()
browser = await playwright.chromium.launch(headless = False)
page = await browser.new_page()

await page.goto("http://whatsmyuseragent.org/")

# await page.screenshot(path="example.png")
# await browser.close()
# await playwright.stop()

---

If you use sync API, it will throw an Error like this:

from playwright.sync_api import sync_playwright
playwright = sync_playwright().start()

'''
Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.
'''

Answer 2

If you can't use async API in jupyter notebook, you can try to create a virtual environment for playwright:

In terminal:

# create a virtual environment for playwright
python3 -m venv playwright_new
source ~/playwright_new/bin/activate
pip install playwright ipykernel requests 
playwright install

Then, create a kernel link for jupyter notebook:

source ~/playwright_new/bin/activate

# create kernel link for jupyter notebook 
python -m ipykernel install --user --name playwright_new --display-name "playwright_new"

# in mac
ls /Users/xxx/Library/Jupyter/kernels/
tree /Users/xxx/Library/Jupyter/kernels/playwright_new
    /Users/xxx/Library/Jupyter/kernels/playwright_new
    ??? kernel.json
    ??? logo-32x32.png
    ??? logo-64x64.png

# or in linux
tree /root/.local/share/jupyter/kernels

Then, run the python code again.

from playwright.async_api import async_playwright

playwright = await async_playwright().start()
browser = await playwright.chromium.launch(headless = False)
page = await browser.new_page()

await page.goto("http://whatsmyuseragent.org/")

---

Error debug

if exec python code throws an Error:

Error: Executable doesn't exist at /Users/xxxx/Library/Caches/ms-playwright/chromium-1000/chrome-mac/Chromium.app/Contents/MacOS/Chromium
???????????????????????????????????????????????????????????????????????????
? Looks like Playwright Test or Playwright was just installed or updated. ?
? Please run the following command to download new browsers:              ?
?                                                                         ?
?     playwright install                                                  ?
?                                                                         ?
? <3 Playwright Team                                                      ?
???????????????????????????????????????????????????????????????????????????

In terminal:

# you already install playwright
playwright install 

cd /Users/xxxx/Library/Caches/ms-playwright
ls
    chromium-978106/ ffmpeg-1007/     firefox-1319/    webkit-1616/

# but the folder ms-playwright/chromium-1000 NOT EXISTS
# COPY the exists chromium folder with a new name  `chromium-1000`
cp -r chromium-978106 chromium-1000