JupyterHub: api_request to the proxy failed with status code 599

Question

• AKS
• proxy-public is a Azure internal LoadBalancer

We are getting below error and hub is in a continuous crash loop. Any help to resolve this issues is very appreciated. Thanks

[W 2021-03-16 06:55:46.808 JupyterHub proxy:814] api_request to the proxy failed with status code 599, retrying... [W 2021-03-16 06:55:50.635 JupyterHub proxy:814] api_request to the proxy failed with status code 599, retrying... [E 2021-03-16 06:55:50.636 JupyterHub app:2859] Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/JupyterHub/app.py", line 2857, in launch_instance_async await self.start() File "/usr/local/lib/python3.8/dist-packages/jupyterhub/app.py", line 2632, in start await self.proxy.get_all_routes() File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 861, in get_all_routes resp = await self.api_request('', client=client) File "/usr/local/lib/python3.8/dist-packages/jupyterhub/proxy.py", line 825, in api_request result = await exponential_backoff( File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 183, in exponential_backoff raise TimeoutError(fail_message) TimeoutError: Repeated api_request to proxy path "" failed.