How can we use Selenium Webdriver in colab.research.google.com?

Question

I want to use Selenium Webdriver of Chrome in colab.research.google.com for fast processing. I was able to install Selenium using !pip install selenium but the webdriver of chrome needs a path to webdriverChrome.exe. How am I suppose to use it?

P.S.- colab.research.google.com is an online platform which provides GPU for fast computational problems related to deep learning. Please refrain from solutions such as webdriver.Chrome(path).

Answer 1

You can do it by installing the chromium webdriver and adjusting some options such that it does not crash in google colab:

!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
wd.get("https://www.webite-url.com")

Answer 2

this one worked in colab

!pip install selenium
!apt-get update 
!apt install chromium-chromedriver

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--disable-dev-shm-usage')
driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)

Answer 3

I made my own library to make it easy.

!pip install kora -q
from kora.selenium import wd
wd.get("https://www.website.com")

PS: I forget how I searched and experimented until it worked. But I first wrote and shared it in this gist in Dec 2018.

Answer 4

Don't have enough repu to comment. :(

However @Thomas answer still works in 06.10.2021, but with just one simple change since right of the bat you'll get DeprecationWarning: use options instead of chrome_options

Working code below:

!pip install selenium
!apt-get update # to update ubuntu to correctly run apt install
!apt install chromium-chromedriver
!cp /usr/lib/chromium-browser/chromedriver /usr/bin
import sys
sys.path.insert(0,'/usr/lib/chromium-browser/chromedriver')
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument('--disable-dev-shm-usage')
wd = webdriver.Chrome('chromedriver',options=options)
wd.get("https://stackoverflow.com/questions/51046454/how-can-we-use-selenium-webdriver-in-colab-research-google-com")
wd.title

Answer 5

to use selenium in GOOGLE COLAB do the next steps in the colab notebook

!pip install kora -q

HOW TO USE IT INSIDE COLAB :

from kora.selenium import wd
wd.get("enter any website here")

YOU CAN ALSO USE IT WITH Beautiful Soup

import bs4 as soup
wd.get("enter any website here")
html = soup.BeautifulSoup(wd.page_source)