C# .NET HttpClient POST returns an empty response sometimes

Question

I am using HttpClient to send a POST request to an external API and I know I'm sending the correct payload and headers and everything because it works sometimes but other times it returns an empty string response. It always returns a 200 status code. Changing it to asynchronous is not an option so my question is, how do I run this synchronously and reliably get a response back?

This is the method that I've used in a number of other places in my application for GETs and POSTs and it works perfectly fine every time:

HttpClient client = new HttpClient { BaseAddress = new Uri("url")
client.DefaultRequestHeaders.Add("X-Authorization", "Bearer " + authToken);

var input = new {json object};

var serializerSettings = new JsonSerializerSettings();
serializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();

var json = JsonConvert.SerializeObject(input, serializerSettings);
var data = new StringContent(json, Encoding.UTF8, "application/json");

var result = client.PostAsync("api/Create", data).Result;
if (result.IsSuccessStatusCode)
{
    var jsonStringResult = result.Content.ReadAsStringAsync().Result;
    var response = JsonConvert.DeserializeObject<ApiResponse>(jsonStringResult);
    // response is null....sometimes
}

Other methods that I've tried from scouring other forum posts and blogs:

var postTask = client.PostAsync("api/Create", data);
postTask.Wait();
var result = postTask.Result;
if (result.IsSuccessStatusCode)
{
    var jsonTask = result.Content.ReadAsStringAsync();
    jsonTask.Wait();
    var jsonStringResult = jsonTask.Result;
    var response = JsonConvert.DeserializeObject<ApiResponse>(jsonStringResult);
    // response is null....sometimes
}

and

var result = client.PostAsync("api/Create", data).GetAwaiter().GetResult();
if (result.IsSuccessStatusCode)
{
    var jsonStringResult = result.Content.ReadAsStringAsync().GetAwaiter().GetResult();
    var response = JsonConvert.DeserializeObject<ApiResponse>(jsonStringResult);
    // response is null....sometimes
}

and

byte[] byteArray = Encoding.UTF8.GetBytes(json);

var request = (HttpWebRequest)WebRequest.Create("url");
request.Method = "POST";
request.ContentType = "application/json; charset=UTF-8";
request.Accept = "application/json";
request.ContentLength = byteArray.Length;
request.Headers.Add("X-Authorization", "Bearer" + authToken);

Stream dataStream = request.GetRequestStream();
dataStream.Write(byteArray, 0, byteArray.Length);
dataStream.Close();

WebResponse response = request.GetResponse();

dataStream = response.GetResponseStream();
StreamReader reader = new StreamReader(dataStream);
string responseFromServer = reader.ReadToEnd();

dataStream.Close();
response.Close();

var responseObj = JsonConvert.DeserializeObject<ApiResponse>(responseFromServer);
// resonseObj is null...sometimes

All four methods work some of the time but I need this to be more reliable. Deadlocks are not a concern at this time. Any help is appreciated!

Answer 1

In a comment above, I noted that I was having the same occasional empty response issue, but I finally figured out what was happening and got my code working. There were 2 issues I had to address:

1. My user-agent header. I was using a user-agent that included a bunch of various elements, something like Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36, and when I changed it to just Mozilla/5.0 (Windows NT 10.0; Win64; x64) everything started working more consistently. (I was having the empty responses on a Windows Server environment, so you might have to adjust this to match the user-agent string to exactly your own environment.)

2. I had used Postman to test my calls and took the generated code that it produced, but when using Fiddler I discovered that there was a header Postman was sending that was not being outputted in their generated code. Specifically: Accept-Encoding: gzip, deflate, br. So naturally, it hadn't been included it in my code.

When I modified these two aspects of my http request code, it consistently worked without any empty responses. However, just to note, one more thing had to be adjusted after adding the accept-encoding header. The response was being compressed, so I had to specify that the HttpClient automatically decompress the response. My final HttpRequest initialization code looked like this:

    Dim HttpClient = New HttpClient(New HttpClientHandler With {.UseCookies = False, .AutomaticDecompression = DecompressionMethods.GZip Or DecompressionMethods.Deflate})
    Try
        HttpClient.DefaultRequestHeaders.Add("user-agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
        HttpClient.Timeout = TimeSpan.FromSeconds(10)
        HttpClient.DefaultRequestHeaders.Add("authority", "https://blahblah")
        HttpClient.DefaultRequestHeaders.Add("Accept-Encoding", "gzip, deflate, br")  
        HttpClient.DefaultRequestHeaders.Add("accept", "application/json, text/plain, */*")
        HttpClient.DefaultRequestHeaders.Add("accept-language", "en-US,en;q=0.9")
        HttpClient.DefaultRequestHeaders.Add("cookie", CookieVar)
        HttpClient.DefaultRequestHeaders.Add("origin", "https://blahblah")
        HttpClient.DefaultRequestHeaders.Add("referer", "https://blahblah")
    Catch ex As Exception
        Console.WriteLine(ex.Message)
    End Try

A final note: once I specified the AutomaticDecompression properties, using Fiddler I discovered that the Accept-Encoding header was added automatically to the call, so the actual line that specified it could be taken out. I left it in above just to show what I was talking about.

Also, to explain the .UseCookies = False line in the HttpClientHandler, see here.