How to overlay data structures?

Question

I have an 8k buffer of bytes. The data in the first part of the buffer is highly structured, with a set of variables of size u32 and u16. The data in the later part of the buffer is for small blobs of bytes. After the structured data, there is an array of offsets that point to the start of each small blob. Something like this:

struct MyBuffer {
    myvalue: u32,
    myothervalue: u16,
    offsets: [u16], // of unknown length
    bytes: [u8] // fills the rest of the buffer
}

I'm looking for an efficient way to take an 8k blob of bytes fetched from disk, and then overlay it or cast it to the MyBuffer struct. In this way I can get/set the structured values easily (let myvar = my_buffer.myvalue), and I can also access the small blobs as slices (let myslice = my_buffer[offsets[2]..offsets[3]]).

The benefit of this approach is you get efficient, zero-copy access to the data.

The fact that the number of offsets and the number of blobs of bytes is unknown makes this tricky.

In C, it's easy; you just cast a pointer to the 8k buffer to the appropriate struct and it just works. You have two different data structures pointing at the same memory.

How can I do the same thing in Rust?

Answer 1

I have discovered that there is an entire Rust ecosystem dedicated to solving this problem. Rust itself handles it poorly.

There are many Rust serialization frameworks, including those that are "zero-copy". The best list is here: https://github.com/djkoloski/rust_serialization_benchmark

The zero-copy frameworks include abomonation, capnp, flatbuffers, rkyv, and alkahest.