Rust Tcpstream: reading http request sometimes lose the body

Question

I am learning how to write a http server using tcpstream with rust.

I use this function to read stream buffer.

fn handle_connection(mut stream: TcpStream) {
    let mut buffer = [0; 1024];

    stream.read(&mut buffer).unwrap();

    println!("Request: {}", String::from_utf8_lossy(&buffer[..]));

    let response = "HTTP/1.1 200 OK\r\n\r\n";

    stream.write(response.as_bytes()).unwrap();
    stream.flush().unwrap();
}

But I found that, sometimes, the buffer lost the body.

It should be:

Request: POST /echo HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: python-requests/2.26.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 13

id=1&name=Foo

But when error,

Request: POST /echo HTTP/1.1
Host: 127.0.0.1:8080
User-Agent: python-requests/2.26.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 13

It only lose the body part.

I use python3 requests to send requests.

What caused the error?

Answer 1

I'm writing my own rust reverse proxy as a means to learn the language. I came across this problem and my solution is below.

The fundemental problem is read_to_end and read_to_string expect an EOF char which doesn't get sent in a HTTP request body. So as far as I can tell you need to write your own byte buffer and then exit once there's no data left.

I'm sure this is not optimized but will get you past this problem. Excuse my use of unwrap. Obviously if this is a production system make sure you handle those more gracefully.

 #[tokio::main]
 async fn main() {
     let listener = TcpListener::bind("127.0.0.1:8080").await.unwrap(); 

     let upstream_server = env::var("UPSTREAM").unwrap();  

     loop {
         println!("initialize stream");
         let (mut stream, _) = listener.accept().await.unwrap(); 

         println!("stream readable?");
         stream.readable().await.unwrap();

         println!("read all from stream");
         let buf = read_all(&stream).await.unwrap();

         // connect to upstream
         println!("connect to upstream");
         let mut upstream_stream = TcpStream::connect(&upstream_server).await.unwrap();

         println!("upstream writable?");
         upstream_stream.writable().await.unwrap();

         // write request body from client to upstream
         println!("write stream request body to upstream");
         upstream_stream.write_all(&buf).await.unwrap();

         println!("upstream readable?");
         upstream_stream.readable().await.unwrap();

         // read upstream response
         println!("read all from upstream");
         let upstream_buf = read_all(&upstream_stream).await.unwrap(); 

         println!("shutdown upstream connection");
         upstream_stream.shutdown().await.unwrap();

         println!("upstream writable?");
         stream.writable().await.unwrap();

         println!("write stream buf upstream response");
         stream.write_all(&upstream_buf.as_bytes()).await.unwrap(); 

         println!("shutdown stream");
         stream.shutdown().await.unwrap();
     }
 }

 async fn read_all(stream: &TcpStream) -> Result<Vec<u8>, std::io::Error> {
     let mut buf: Vec<u8> = Vec::new();

     loop {
         // Creating the buffer **after** the `await` prevents it from
         // being stored in the async task.
         let mut tmp_buf = [0; 4096]; => [u8; 4096]

         // Try to read data, this may still fail with `WouldBlock`
         // if the readiness event is a false positive.
         match stream.try_read(&mut tmp_buf) {
             Ok(0) => break,
             Ok(_) => {
                 buf.extend_from_slice(&tmp_buf)
             }
             Err(ref e) if e.kind() == std::io::ErrorKind::WouldBlock => { 
                 break;
             }
             Err(e) => { => Error
                 return Err(e.into());
             }
         }

     }

     return Ok(std::str::from_utf8(&buf).unwrap().trim_matches(char::from(0)).as_bytes().to_vec())

 }

Answer 2

This a simple way to get body content. This code reads line by line of header until get a empty file, the Rust TCPStream is used here! After read the header, there is a Content-Length in reader, so i convert the char to integer and use the size to create a vector. So is filled by read_exact function.

let mut reader = BufReader::new(stream.try_clone().unwrap());
let mut name = String::new();
loop {
let r = reader.read_line(&mut name).unwrap();
    if r < 3 { //detect empty line
        break;
    }
}
let mut size = 0;
let linesplit = name.split("\n");
for l in linesplit {
    if l.starts_with("Content-Length") {
            let sizeplit = l.split(":");
            for s in sizeplit {
                if !(s.starts_with("Content-Length")) {
                    size = s.trim().parse::<usize>().unwrap(); //Get Content-Length
            }
        }
    }
}
let mut buffer = vec![0; size]; //New Vector with size of Content   
reader.read_exact(&mut buffer).unwrap(); //Get the Body Content.