How to get request headers in Selenium

Question

https://www.sahibinden.com/en

If you open it incognito window and check headers in Fiddler then these are the two main headers you get:

When I click the last one and check request headers this is what I get

I want to get these headers in Python. Is there any way that I can get these using Selenium? Im a bit clueless here.

Answer 1

You can use Selenium Wire. It is a Selenium extension which has been developed for this exact purpose.

https://pypi.org/project/selenium-wire/

An example after pip install:

##  Import webdriver from Selenium Wire instead of Selenium
from seleniumwire import webdriver

##  Get the URL
driver = webdriver.Chrome("my/path/to/driver", options=options)
driver.get("https://my.test.url.com")

##  Print request headers
for request in driver.requests:
  print(request.url) # <--------------- Request url
  print(request.headers) # <----------- Request headers
  print(request.response.headers) # <-- Response headers

Answer 2

You can run JS command like this;

var req = new XMLHttpRequest()
req.open('GET', document.location, false)
req.send(null)
return req.getAllResponseHeaders()

On Python;

driver.get("https://t.me/codeksiyon")
headers = driver.execute_script("var req = new XMLHttpRequest();req.open('GET', document.location, false);req.send(null);return req.getAllResponseHeaders()")

# type(headers) == str

headers = headers.splitlines()

Answer 3

The bottom line is, No, you can't retrieve the request headers using Selenium.

---

Details

It had been a long time demand from the Selenium users to add the WebDriver methods to read the HTTP status code and headers from a HTTP response. We have discussed about implementing this feature through Selenium at length within the discussion WebDriver lacks HTTP response header and status code methods.

However, Jason Leyba (Selenium contributor) in his comment straightly mentioned:

We will not be adding this feature to the WebDriver API as it falls outside of our current scope (emulating user actions).

Ashley Leyba further added, attempting to make WebDriver the ideal web testing tool will suffer in overall quality as driver.get(url) blocks until the browser has loaded the page and return the response for the final loaded page. So in case of a login redirects, status codes and headers will always end up with a 200 instead of the 302 you're looking for.

Finally, Simon M Stewart (WebDriver creator) in his comment concluded that:

This feature isn't going to happen. The recommended approach is to either extend the HtmlUnitDriver to access the information you require or to make use of an external proxy that exposes this information such as the BrowserMob Proxy

Answer 4

It's not possible to get headers using Selenium. Further information

However, you might use other libraries such as requests, BeautifulSoup to get headers.

Answer 5

Maybe you can use BrowserMob Proxy for this. Here is a example:

import settings

from browsermobproxy import Server
from selenium.webdriver import DesiredCapabilities

config = settings.Config

server = Server(config.BROWSERMOB_PATH)
server.start()
proxy = server.create_proxy()

from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % proxy.proxy)
chrome_options.add_argument('--headless')

capabilities = DesiredCapabilities.CHROME.copy()
capabilities['acceptSslCerts'] = True
capabilities['acceptInsecureCerts'] = True

driver = webdriver.Chrome(options=chrome_options,
    desired_capabilities=capabilities,
   executable_path=config.CHROME_PATH)

proxy.new_har("sahibinden", options={'captureHeaders': True})
driver.get("https://www.sahibinden.com/en")

entries = proxy.har['log']["entries"]
for entry in entries:
    if 'request' in entry.keys():
        print(entry['request']['url'])
        print(entry['request']['headers'])
        print('\n')

proxy.close()
driver.quit()

Answer 6

js_headers = '''
    const _xhr = new XMLHttpRequest();
    _xhr.open("HEAD", document.location, false);
    _xhr.send(null);

    const _headers = {};

    _xhr.getAllResponseHeaders().trim().split(/[\\r\\n]+/).map((value) => value.split(/: /)).forEach((keyValue) => {
        _headers[keyValue[0].trim()] = keyValue[1].trim();
    });

    return _headers;
'''

page_headers = driver.execute_script(js_headers)

type(page_headers) # -> dict