'An invalid or unaligned stack was encountered during an unwind operation

I have a 64bit program that works with VirtualBox COM interface and implements a frontend for the virtual machine. Recently I started getting weird An invalid or unaligned stack was encountered during an unwind operation exceptions and I wanted to at least understand the causes of this. As I understand the stack needs to be 16byte aligned so, I presume unaligned stack pointer likely to cause this. But the thing is, since all my program does is implement a couple of COM interfaces using the STDMETHOD macros from ATL that should use the correct calling convention then how could I mess up the stack?

Here is an example of the call stack when the issue occurs:

ntdll.dll!00007ffe679ac0b4() Unknown
ntdll.dll!00007ffe67913356() Unknown
msvcrt.dll!__longjmp_internal() Unknown
> VBoxREM.dll!000000006fb0f3c4() Unknown

I tried to goole the __longjmp_internal symbol but did not find anything useful - does it indicate that exception unwind is in progress?

Any pointer on how to approach debugging of this issue or comments what could mess up the stack alignment are welcome, since I understand that in this case it will be impossible to give an exact solution because VirtualBox is involved.

c++com64-bitvirtualbox


Solution 1:[1]

I've faced this baffling problem recently.

I know it only started happening after I switched from the static C/C++ runtime to the DLL version, so that probably means the static version didn't do stack unwinding.

I then traced the assembly code for longjmp() and noticed one of the 1st conditional branches was on _JUMP_BUFFER.Frame.

if it's 0, then restore a bunch of registers and return.

Aha! so that must mean if _JUMP_BUFFER.Frame = 0, unwinding is disabled. I tried it and indeed, problem solved.

I then tried to observe what Frame should be when a setjmp()/longjmp() pair succeeds. I found usually, frame = stack pointer, but when the unwinding fails, frame != SP. So I tried setting Frame to SP and that also eliminates the exception.

I don't know why that works. I know in the SYSV x86-64 ABI, the frame pointer is optional. Maybe setjmp() needs a proper frame pointer and isn't getting one?

Solution 2:[2]

Not sure if this helps, but i encountered similar problems (without VMs) with longjmp in combination with x64 Windows.

Turned out that any kind of aligned stack-data (aligned with >= 32byte) in the same scope like the longjmp causes longjmp to come up with the 0xC0000028 when compiled for x64.

#include <setjmpex.h>

void doThe_0xC0000028 ( )
{
    jmp_buf jp;

    if (!setjmp (jp))
    {
        // do some stuff ... 
        // ... then "revert" with longjmp.
        longjmp (jp, 1);
    }

    // having any aligned data on stack (align > 16) in the same scope
    // causes longjmp to go: 0xC0000028
    //------------------------------------------------------------------
    __declspec(align(32)) char buffer[12];

    // just accessing buffer somehow - this is apparently needed to generate the faulty 0xC0000028
    buffer[0];          
}

I reported that as MSVC bug: https://connect.microsoft.com/VisualStudio/feedback/details/3136150/64bit-longjmp-causing-0xc0000028-with-aligned-stack-data

Solution 3:[3]

One thing to be aware of if you are calling x64 MSVCRT’s setjmp through a foreign function interface is that it expects an undocumented second parameter: the stack pointer immediately before the call. This gets stored in the Frame member of the jmp_buf. If you do not explicitly pass this argument, the Frame just becomes whatever happens to be in RDX. longjmp calls RtlUnwindEx, which checks the target Frame looks like a valid stack pointer – and if it does not, it raises the STATUS_BAD_STACK exception.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Yale Zhang
Solution 2 Roman Pfneudl
Solution 3 Brian Nixon

Related Questions