Why are only top-level requests proxied when configuring http clients such as postman or K6

Question

The Goal

• Run integration tests but stub sub-requests to a specific domain that occur within the API code itself
• Ideally, do this via docker-compose because I want to be able to run this in Circle CI or Jenkins

The attempt

I'm using the mockttp library as outlined here to create a proxy server that allows all traffic to pass through untouched with the exception of one specific domain where I will have a set response.

The proxy server, API, and K6 service are spun up using docker-compose, and the HTTP(S)_PROXY variables are set in the K6 service.

The problem

Direct requests are proxied but requests instigated by those requests are not. For example, if I make a request for the specific domain I am attempting to a stub, the proxy does its job. But that domain isn't called directly, it is part of the API code and the proxy has no effect at this level.

If I spin up the proxy server and configure my laptop to use the proxy server, everything works correctly. However, if I configure Postman or K6 to use the proxy server it doesn't.

docker-compose.yml

version: '3.7'
services:
  service:
    build:
      context: ./service
    container_name: service
    ports:
      - '3000:3000'
    environment:
      - NODE_TLS_REJECT_UNAUTHORIZED=0
  proxy:
    build:
      context: ./
    container_name: proxy
    ports:
      - '8002:8002'
    environment: 
      - BASE_URL=service:3000
  k6:
    image: loadimpact/k6:latest
    container_name: k6
    depends_on: 
      - service
      - proxy
    volumes: 
      - ./service/test/:/perf
    environment: 
      - HTTPS_PROXY=https://proxy:8002
      - HTTP_PROXY=http://proxy:8002
      - BASE_URL=http://service:3000/foo
      - NODE_TLS_REJECT_UNAUTHORIZED=0
      - K6_HTTP_DEBUG=true
    entrypoint: ['k6', 'run', '--insecure-skip-tls-verify', '/perf/test.js']

I'm sure I'm missing something obvious but networking is not my strong point. My understanding is that if I tell something to use a proxy, all traffic should go through the proxy. I am finding that if I set the proxy in my computer's settings, this is indeed the case. But if I try this by configuring K6 or even Postman, the behavior is different.

What am I doing wrong?

Proxy server code for reference

(async () => {
  const server = await require('mockttp').getLocal();
  server.enableDebug();
  await server.anyRequest().forHost('hostiwanttostub').always().thenReply(201);
  await server.anyRequest().always().thenForwardTo(process.env.BASE_URL);
  await server.start(8002);
  // Print out the server details:
  console.log(`Server running on port ${server.port}`);
})(); // (Run in an async wrapper so we can use top-level await everywhere)

Answer 1

First off, without digging into mockttp's code, you might try to specify the service protocol ie. http or https:

docker-compose.yml

  ...
  proxy:
    environment: 
      - BASE_URL=http://service:3000

Secondly, you might want to ensure that K6 and Postman actually use the HTTP_PROXY and HTTPS_PROXY environment variables - they probably employ their own code that might not take those into consideration and might need to be configured to use a proxy by means of some other configuration file or option.

If there are any HTTP compression options, or alternative protocols such as HTTP/2 or Websockets, you might want to disable them - if there's no mention of it being implemented in mockttp - or you might want to use a tried and tested proxy server such as squid or nginx - which are both well documented and supported.

Proxies are quite straightforward - you can trivially implement your own proxy by simply opening a socket, reading the plain text HTTP/1.1 header, matching on the "^Host: " line, and then deciding what to rewrite or add in the header and where to open a socket to to send everything you received from - and then simply relay information between the two sockets. Things only really get more complex with range requests and compression, which require some extra work.

If you are unsure what is happening, you can simply use a utility like netcat or socat to see what's going on:

 $  nc -l -p 8082

Will listen on port 8082 - you can then issue your request to it and it will echo the headers which might help. You can test it with just a simple curl http://127.0.0.1:8082.

Or you can use socat to sit in front of your proxy:

$ socat -v TCP-LISTEN:8083 TCP4:myproxy:8082

Check the socat manual page - it too can function as a proxy. -x does a HEX Dump. You can also use tcpdump, tshark, wireshark, that might shed some light - or seeing as that you're in Javascript, just inspect what's going on in your debugger.

One more thing you can check:

     await server.anyRequest().forHost('http://hostiwanttostub').always().thenReply(201);

Again, not sure how mockttp is implemented, but it might need you to specify the protocol. Its possible - although unlikely - that your initial request comes through on http and the subsequent ones on https - and that only the one is intercepted.

Mockttp's purpose in life seems to serve as an easy to set up HTTPS proxy - also check that you've installed the relevant SSL certificate AS WELL AS certificate chains for K6 or Postman - and PLEASE only set it to blindly accept self-signed certificates IF YOU ARE JUST TESTING - they might see a bogus certificate which might change what they do.

Let us know in the comments below, or with another answer, what worked for you - so that others could learn from you too.