Passing a large file stream to MultipartFormDataContent with HttpClient

Question

I'm experiencing a problem when trying to use MultipartFormDataContent with HttpClient with a stream of data.

Context

I'm trying to upload a large file to ASP.NET Core Web API. A client should send the file via POST request form-data to a front-end API, which in turn should forward the file to a back-end API.

Because the file can be large, I followed the Microsoft example, i.e. I don't want to use IFormFile type but instead read the Request.Body using MultipartReader. This is to avoid loading the entire file into memory on the server, or saving it in a temporary file on server's hard drive.

Problem

The back-end API controller action looks as follows (this is almost directly copied from the ASP.NET Core 5.0 sample app with just minor simplifications):

        [HttpPost]
        [DisableRequestSizeLimit]
        public async Task<IActionResult> ReceiveLargeFile()
        {
            var request = HttpContext.Request;

            if (!request.HasFormContentType
                || !MediaTypeHeaderValue.TryParse(request.ContentType, out var mediaTypeHeader)
                || string.IsNullOrEmpty(mediaTypeHeader.Boundary.Value))
            {
                return new UnsupportedMediaTypeResult();
            }

            var reader = new MultipartReader(mediaTypeHeader.Boundary.Value, request.Body);
            /* This throws an IOException: Unexpected end of Stream, the content may have already been read by another component.  */
            var section = await reader.ReadNextSectionAsync();
            
            while (section != null)
            {
                var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition,
                    out var contentDisposition);

                if (hasContentDispositionHeader
                    && contentDisposition!.DispositionType.Equals("form-data")
                    && !string.IsNullOrEmpty(contentDisposition.FileName.Value))
                {
                    /* Fake copy to nothing since it doesn't even get here */
                    await section.Body.CopyToAsync(Stream.Null);
                    return Ok();
                }

                section = await reader.ReadNextSectionAsync();
            }

            return BadRequest("No files data in the request.");
        }

I managed to reduce the problem slightly by making an integration test using Microsoft.AspNetCore.Mvc.Testing NuGet package. The following test replaces the front-end API, so instead of reading Request.Body stream in a Web API, the test just tries to add StreamContent to MultipartFormDataContent and post it via HttpClient to the back-end API:

        [Fact]
        public async Task Client_posting_to_Api_returns_Ok()
        {
            /* Arrange */
            await using var stream = new MemoryStream();
            await using var writer = new StreamWriter(stream);
            await writer.WriteLineAsync("FILE CONTENTS");
            await writer.FlushAsync();
            stream.Position = 0;

            using var client = _factory.CreateDefaultClient();

            /* Act */
            using var response =
                await client.PostAsync(
                    "Receive",
                    new MultipartFormDataContent
                    {
                        {
                            new StreamContent(stream),
                            "file",
                            "fileName"
                        }
                    });
            
            /* Assert */
            Assert.Equal(HttpStatusCode.OK, response.StatusCode);
        }

The back-end API controller then throws an IOException at await reader.ReadNextSectionAsync(), saying "Unexpected end of Stream, the content may have already been read by another component".

GitHub Repository (Complete Example)

I uploaded a complete example of the problem (including back-end API and the test) a GitHub repo.

Question

I must be doing something wrong. How can I forward a file received in a request with form-data content type in one service (front-end API) to another service (back-end API) without loading the entire file into memory or hard-drive in the front-end API, i.e. to just forward the stream of data to the back-end API?

Thanks in advance for any help.

Answer 1

I expected the same issue as you and it turned out that the MediaTypeHeaderValue.TryParse method parses the boundary value wrong as it wraps the string with '"' characters, because HttpClient sends the content type header like this:

multipart/form-data; boundary="blablabla"

So for me the solution was to add a Trim() method to boundary like this and pass that to the MultipartReader

var boundary = mediaTypeHeader.Boundary.Value.Trim('"');
var reader = new MultipartReader(boundary, request.Body);