'problems getting ThreadedTCPServer to work with selenium and report-portal
I've encountered a behavior that I simply do not understand. Implementing a ThreadedTCPServer with selenium was working fine until I decided to attach reportportal...
For the sake of simplicity - I've tried to boil down the code to the minimum necessary to reproduce the issue:
import http.server
import socketserver
from typing import Optional
from selenium.webdriver.chrome.webdriver import WebDriver
class ThreadedTCPServer(socketserver.ThreadingMixIn, socketserver.TCPServer):
pass
def title_extractor(
headless: bool = True
):
import concurrent.futures
with concurrent.futures.ThreadPoolExecutor() as executor:
host, port = "localhost", 8050
with ThreadedTCPServer((host, port), http.server.SimpleHTTPRequestHandler) as server:
print(f"serving at port {port}")
# Start a thread for the server
print('Starting server ...')
executor.submit(server.serve_forever)
# Start a thread for selenium
print('Starting page_worker ...')
future_nonce = executor.submit(page_worker, None, headless)
print('... page_worker done')
done_set, not_done_set = concurrent.futures.wait(
[future_nonce],
timeout=300,
return_when=concurrent.futures.ALL_COMPLETED
)
for future in done_set:
if future.done():
print('encountered future.done()')
pass
if future.cancelled():
print('encountered future.cancelled()')
pass
if future.exception(timeout=3):
print(f'encountered future.exception(): {str(future.exception())}')
result = None
continue
# raise RuntimeError(str(future.exception()))
print(f'future: {str(future)}')
result = future.result(timeout=3)
print('... future result finished')
print(f"Result:\t{result}")
print('Shutting down server ...')
server.shutdown()
executor.shutdown(wait=False)
print('... server and executor shut down')
return result
def page_worker(
driver: Optional[WebDriver] = None,
headless: bool = True
) -> Optional[str]:
import selenium
from selenium.webdriver.chrome.options import Options
terminate_driver = False
options = Options()
options.add_argument("--no-sandbox")
options.add_argument("--ignore-certificate-errors")
if headless:
options.add_argument("--headless")
# Start a new driver and stop it afterwards at the end of the worker call
if driver is None:
terminate_driver = True
url = "https://google.com"
print(f"Starting chrome driver with URL={url}\n"
f"headless={headless}\toptions={str(options)}")
driver = selenium.webdriver.Chrome(options=options)
driver.get(url)
print(f"driver={driver}")
title = driver.title
if terminate_driver:
print("Stopping driver")
driver.quit()
print(f"Worker terminating")
return title
def test__page_worker_with_report_portal():
title = title_extractor(headless=True)
print(f"Worker finished. Page title: {title}")
Here's the call (without reportportal integration) that works:
python3 -m pytest --show-capture=all -s -v ./Tests/Debugging/NonceWithRP.py
Here's the result:
(venv) bartoszc@bartoszclnx:~/PycharmProjects/ryd-tests$ python3 -m pytest --show-capture=all -s -v ./Tests/Debugging/NonceWithRP.py
MARKERS: {''}
======================================================= test session starts =======================================================
platform linux -- Python 3.8.10, pytest-7.1.1, pluggy-1.0.0 -- .../venv/bin/python3
cachedir: .pytest_cache
rootdir: ..., configfile: pytest.ini
plugins: allure-pytest-2.9.45, reportportal-5.1.0, repeat-0.9.1
collected 1 item
Tests/Debugging/NonceWithRP.py::test__page_worker_with_report_portal serving at port 8050
Starting server ...
Starting page_worker ...
Starting chrome driver with URL=https://google.com
headless=True options=<selenium.webdriver.chrome.options.Options object at 0x7fa3149e8eb0>... page_worker done
---------------------------------------------------------- live log call ----------------------------------------------------------
DEBUG 17:38:10: POST http://localhost:58501/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "pageLoadStrategy": "normal", "goog:chromeOptions": {"extensions": [], "args": ["--no-sandbox", "--ignore-certificate-errors", "--headless"]}}}, "desiredCapabilities": {"browserName": "chrome", "pageLoadStrategy": "normal", "goog:chromeOptions": {"extensions": [], "args": ["--no-sandbox", "--ignore-certificate-errors", "--headless"]}}}
DEBUG 17:38:10: Starting new HTTP connection (1): localhost:58501
DEBUG 17:38:10: http://localhost:58501 "POST /session HTTP/1.1" 200 752
DEBUG 17:38:10: Finished Request
DEBUG 17:38:10: POST http://localhost:58501/session/cbf04df70cb13b87b74624492c5682ae/url {"url": "https://google.com"}
DEBUG 17:38:11: http://localhost:58501 "POST /session/cbf04df70cb13b87b74624492c5682ae/url HTTP/1.1" 200 14
DEBUG 17:38:11: Finished Request
driver=<selenium.webdriver.chrome.webdriver.WebDriver (session="cbf04df70cb13b87b74624492c5682ae")>
DEBUG 17:38:11: GET http://localhost:58501/session/cbf04df70cb13b87b74624492c5682ae/title {}
DEBUG 17:38:11: http://localhost:58501 "GET /session/cbf04df70cb13b87b74624492c5682ae/title HTTP/1.1" 200 18
DEBUG 17:38:11: Finished Request
Stopping driver
DEBUG 17:38:11: DELETE http://localhost:58501/session/cbf04df70cb13b87b74624492c5682ae {}
DEBUG 17:38:11: http://localhost:58501 "DELETE /session/cbf04df70cb13b87b74624492c5682ae HTTP/1.1" 200 14
DEBUG 17:38:11: Finished Request
Worker terminating
encountered future.done()
future: <Future at 0x7fa314b0e340 state=finished returned str>
... future result finished
Result: Google
Shutting down server ...
... server and executor shut down
Worker finished. Page title: Google
PASSED
Here's the call with reportportal integration, where it breaks:
python3 -m pytest --reportportal --show-capture=all -s -v ./Tests/Debugging/NonceWithRP.py
Here's the result:
MARKERS: {''}
------------------------------------------------------ live log sessionstart ------------------------------------------------------
DEBUG 17:41:37: ReportPortal - Init service: endpoint=...
DEBUG 17:41:37: Starting new HTTP connection (1): ...
DEBUG 17:41:37: ... "GET /api/v1/testchuck/settings HTTP/1.1" 200 None
DEBUG 17:41:37: ReportPortal - Start launch: request_body=...
DEBUG 17:41:37: ... "POST /api/v2/testchuck/launch HTTP/1.1" 201 None
DEBUG 17:41:37: start_launch - ID: ...
DEBUG 17:41:37: ReportPortal - Launch started: id=...
DEBUG 17:41:37: Starting new HTTPS connection (1): www.google-analytics.com:443
DEBUG 17:41:38: https://www.google-analytics.com:443 "POST /collect HTTP/1.1" 200 35
======================================================= test session starts =======================================================
platform linux -- Python 3.8.10, pytest-7.1.1, pluggy-1.0.0 -- .../venv/bin/python3
cachedir: .pytest_cache
rootdir: ..., configfile: pytest.ini
plugins: allure-pytest-2.9.45, reportportal-5.1.0, repeat-0.9.1
collected 1 item
------------------------------------------------------- live log collection -------------------------------------------------------
DEBUG 17:41:38: ReportPortal - Start TestItem: request_body={'attributes': [], 'name': 'Tests/Debugging/NonceWithRP.py::test__page_worker_with_report_portal', 'description': None, 'start_time': '1650555698079', 'item_type': 'STEP', 'code_ref': 'Tests/Debugging/NonceWithRP.py:test__page_worker_with_report_portal', 'parameters': None, 'parent_item_id': None, 'test_case_id': 'Tests/Debugging/NonceWithRP.py:test__page_worker_with_report_portal'}
DEBUG 17:41:38: ... "POST /api/v2/testchuck/item HTTP/1.1" 201 None
DEBUG 17:41:38: start_test_item - ID: ...
Tests/Debugging/NonceWithRP.py::test__page_worker_with_report_portal serving at port 8050
Starting server ...
Starting page_worker ...
Starting chrome driver with URL=https://google.com
headless=True options=<selenium.webdriver.chrome.options.Options object at 0x7fcac84b5370>
... page_worker done
---------------------------------------------------------- live log call ----------------------------------------------------------
DEBUG 17:41:38: POST http://localhost:56813/session {"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome", "pageLoadStrategy": "normal", "goog:chromeOptions": {"extensions": [], "args": ["--no-sandbox", "--ignore-certificate-errors", "--headless"]}}}, "desiredCapabilities": {"browserName": "chrome", "pageLoadStrategy": "normal", "goog:chromeOptions": {"extensions": [], "args": ["--no-sandbox", "--ignore-certificate-errors", "--headless"]}}}
encountered future.done()
encountered future.exception(): 'NoneType' object has no attribute 'log'
Shutting down server ...
... server and executor shut down
Worker finished. Page title: None
PASSED
--------------------------------------------------------- live log finish ---------------------------------------------------------
DEBUG 17:41:39: ReportPortal - Finish TestItem: request_body={'end_time': '1650555699144', 'status': 'PASSED', 'issue': None, 'item_id': '6647d85e-4c35-4798-acc5-17b18dcb5c48'}
DEBUG 17:41:39: ... "PUT /api/v2/testchuck/item/6647d85e-4c35-4798-acc5-17b18dcb5c48 HTTP/1.1" 200 None
DEBUG 17:41:39: finish_test_item - ID: 6647d85e-4c35-4798-acc5-17b18dcb5c48
DEBUG 17:41:39: response message: Accepted finish request for test item ID = 6647d85e-4c35-4798-acc5-17b18dcb5c48
----------------------------------------------------- live log sessionfinish ------------------------------------------------------
DEBUG 17:41:39: ReportPortal - Finish launch: request_body={'end_time': '1650555699384'}
DEBUG 17:41:39: ... "PUT /api/v2/testchuck/launch/1c4b1457-4e0a-4804-8749-391f77899385/finish HTTP/1.1" 200 None
DEBUG 17:41:39: finish_launch - ID: 1c4b1457-4e0a-4804-8749-391f77899385
DEBUG 17:41:39: response message: <reportportal_client.static.defines._PresenceSentinel object at 0x7fcac9ed9cd0>
DEBUG 17:41:39: Waiting for worker <reportportal_client.core.worker.APIWorker object at 0x7fcac8e27550> to completeprocessing batches.
DEBUG 17:41:39: [APIWorker] Received {ControlCommand.STOP} command
DEBUG 17:41:39: [APIWorker] Processing {ControlCommand.STOP} command
DEBUG 17:41:39: [APIWorker] Exiting due to {ControlCommand.STOP} command
It seems that there is a problem with the logging that seems to be invoked by the web-driver. What I don't understand is - why does the future_nonce terminate as 'done' when the log.debug messages begin - and only after the first debug message?
For the sake of what I need to do in the end - I do need to have the multi-threaded solution, where one thread serves the page, another thread operates selenium automation on the page. The code above is just a minimal implementation to reproduce the issue.
Could someone please help me in obtaining the same behavior (meaning: a complete execution of the page_worker, returning the page title), but with reportportal integration in this 'infrastructure' ?
Solution 1:[1]
This was a bug inside Report Portal client which is fixed in version 5.2.2 Please use the latest version of the client: https://pypi.org/project/reportportal-client/
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | HardNorth |