Exclude headers from s3v4 signature calculation

Question

We are using an onPrem S3 compatible storage server in an intranet network and we want to expose this intranet url to internet so we used a ReverseProxy with a mapping to the intranet url. When we test the intranet url it works perfectly but when we test the internet url we get the 403 error:

The request signature we calculated does not match the signature you provided. Check your Secret Access Key and signing method. For more information, see REST Authentication and SOAP Authentication for details. (Service: Amazon S3; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 0a440c7f:15cc604b1e2:12d3af:24d; S3 Extended Request ID: null), S3 Extended Request ID: null

After debugging, we found that the proxy modifies the host header used to calculate the signature in order to redirect the request to the intranet url...

So my question is how to supress some headers from the V4 signature calculation using AWS SDK or Boto3 client. or is there a better architecture to expose an onPrem S3 service.

Thanks in advance. Amir.

Answer 1

There are essentially two solutions to this.

The first one is easier: sign the request for the internal URL, then just use simple string prefix replacement to rewrite the host part of the signed URL to point it to the hostname of the external proxy. When the proxy rewrites the Host header, it will end up rewriting it back to exactly what you signed.

It is, I assume, common knowledge that signed URLs are immune to tampering, for all practical purposes: you can't change anything about a signed URL without invalidating it... but that's not what this is. The change is temporary, and the proxy's net effect is to undo the change.

The alternate solution requires the proxy or another service in the chain (before the storage service) to know the signing keys and secrets, so that it can first validate the incoming request, and if valid, modify the request and then generate a new signature that the service will accept. I once wrote a service to do this so that when a request was for HEAD, the proxy would use the same key and secret (which it knew) to generate a signature for the same request, but with GET. If it matched the signature in the incoming request, the proxy would replace the existing signature with a signature for a HEAD request -- thus allowing the client to use a URL originally signed for a GET request, to make either a GET or a HEAD request -- something S3 does not natively support, since a GET and a HEAD for the same object require two different signed URLs. The concept is the same, though -- generate a signature in the proxy for what the client is requesting, to validate the incoming signature, and then re-sign the request as needed. The solution I built used HAProxy's Lua integration to examine and modify the request in flight.