The Lost Art of Structure Packing (2018)

If you do any development that requires touching Win32, structure packing and memory alignment is still very real.<p>I was recently doing that in the context of improving a language’s library support for something that had to go through the OS SDK. I don’t miss the days when that stuff was the norm.

I&#x27;ve documented how GCC does bitfield packing in the TXR reference manual. Or rather, the abstract algorithm used in the FFI to replicate it.<p><a href="https:&#x2F;&#x2F;www.nongnu.org&#x2F;txr&#x2F;txr-manpage.html#N-027D075C" rel="nofollow">https:&#x2F;&#x2F;www.nongnu.org&#x2F;txr&#x2F;txr-manpage.html#N-027D075C</a><p>This is the result of empirical investigation.<p>The description also covers allocation of non-bitfields (paragraph 3) and the padding of the structure (paragraph 9) which require few words.<p>I felt that the bitfield handling is so obscure that it had to be documented in detail. If someone is to know exactly what the layout will be, the documentation can&#x27;t just be &quot;oh, it will behave like a GCC struct&quot;. Well, what will <i>that</i> do? That is not adequately documented anywhere.<p>If I have to work with bitfields in just C, I can use that as a reference to understand what the compiler will do (at least if it&#x27;s anything compatible with GCC).

Looks like what he really wants is to use Ada which has had much better support for low-level programming than C.<p>Example:<p>Word : constant := 4; -- storage element is byte, 4 bytes per word<p>type State is (A,M,W,P); type Mode is (Fix, Dec, Exp, Signif);<p>type Byte_Mask is array (0..7) of Boolean; type State_Mask is array (State) of Boolean; type Mode_Mask is array (Mode) of Boolean;<p>type Program_Status_Word is record System_Mask : Byte_Mask;<p><pre><code> Protection_Key : Integer range 0 .. 3; Machine_State : State_Mask; Interrupt_Cause : Interruption_Code; Ilc : Integer range 0 .. 3; Cc : Integer range 0 .. 3; Program_Mask : Mode_Mask; Inst_Address : Address; end record; for Program_Status_Word use record System_Mask at 0*Word range 0 .. 7; Protection_Key at 0*Word range 10 .. 11; -- bits 8,9 unused Machine_State at 0*Word range 12 .. 15; Interrupt_Cause at 0*Word range 16 .. 31; Ilc at 1*Word range 0 .. 1; -- second word Cc at 1*Word range 2 .. 3; Program_Mask at 1*Word range 4 .. 7; Inst_Address at 1*Word range 8 .. 31; end record; for Program_Status_Word&#x27;Size use 8*System.Storage_Unit; for Program_Status_Word&#x27;Alignment use 8; </code></pre> More info: <a href="https:&#x2F;&#x2F;www.adaic.org&#x2F;resources&#x2F;add_content&#x2F;standards&#x2F;05aarm&#x2F;html&#x2F;AA-13-5-1.html" rel="nofollow">https:&#x2F;&#x2F;www.adaic.org&#x2F;resources&#x2F;add_content&#x2F;standards&#x2F;05aarm...</a>

I wrote a clang plugin to look for opportunities across a 10M line codebase and there was surprisingly little to be found. Why? Because on 64-bit Linux, the current C++ ABI mandates quite large alignment, especially once you are embedding things inside other things. Packing is still sometimes useful in speeding things up, but tends to require bitfields and flattening structs inside structs into a single struct, etc.

When I worked at a prop trading firm on a greenfield market data system this kind of optimization was very much on our minds. I assume others in this field also take care to pack structs.

It would be nice if there was an annotation that just lets the compiler do all the optimization for me for the cases where I don&#x27;t care about the memory layout of the struct. Just like the Rust compiler can do without repr(C)

I&#x27;m sort of surprised there are no tools for this. I understand why having the compiler reorder things could be bad, though it seems like there should be room to tell the compiler it&#x27;s okay to repack it for minimum space, but I don&#x27;t even see any mention of a source-level tool that would just sort the items in a struct for you.<p>It seems like something like that could be useful rather than making programmers try to order their structs by hand.

I always assumed that in C, a given structure would always have the same memory layout, no matter what the compiler is (as long as the compilers target the same architecture of course).<p>I always assumed that in C++, the memory layout could change a lot between compilers (the location of the pointer to the vtable for example). Do you know if it&#x27;s true, and if the layout do change, could you give an example?

It&#x27;s probably not talked about so much now because of the canard that memory is cheap, and because HLLs disguise things a bit too much and so mislead newbies, but to suggest it&#x27;s lost is plain wrong.

Biggest surprise for me is learning that a pointer is a whopping eight bites! I have always mistakenly assumed they were small and just four bites.

TLDR: When defining a struct, put the biggest items first. This works well in almost all cases.<p>If you need to squeeze out all padding, use compiler directives like &#x27;#pragma pack&#x27;, but be aware of the performance implications.