I recently observed stack overflow issue due to STA reentrancy issue. The issue occurred when a slew of COM clients called the STA object concurrently and the STA object in question made outgoing cross-apartment (or cross-process) call. When STA COM call is made, the call is sent to a hidden window in STA COM and translated to window message. When making an out-of-apartment call from an STA apartment, STA COM spins a modal message pump while waiting for the call to return. Many calls that arrived at the message loop can now be dispatched, causing reentrance. Given a lot of calls keep entering to the STA object, the STA thread reached the max limit of the thread stack. Hence the stack overflow.

Looking at the debugger, I observed that many threads showed the same pattern as shown below.

  46  Id: 2444.1a18 Suspend: 1 Teb: 000007ff`fff5e000 Unfrozen
Child-SP          RetAddr           Call Site
00000000`03c3dce8 00000000`76f3c0b0 ntdll!ZwWaitForSingleObject+0xa
00000000`03c3dcf0 000007fe`fdca86b2 kernel32!WaitForSingleObjectEx+0x9c
00000000`03c3ddb0 000007fe`fddc9d80 ole32!GetToSTA+0x8a
00000000`03c3de00 000007fe`fddc9375 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x100
00000000`03c3de50 000007fe`fdc9f436 ole32!CRpcChannelBuffer::SendReceive2+0xf1
00000000`03c3e010 000007fe`fdc9f398 ole32!CAptRpcChnl::SendReceive+0x52
00000000`03c3e0d0 000007fe`fd7d603a ole32!CCtxComChnl::SendReceive+0x6c
00000000`03c3e180 000007fe`fd7cef90 RPCRT4!NdrProxySendReceive+0x4a
00000000`03c3e1b0 000007fe`fd7d6157 RPCRT4!NdrpClientCall3+0x246
00000000`03c3e400 000007fe`fd728772 RPCRT4!ObjectStublessClient+0xa7
00000000`03c3e770 000007fe`f99a80e8 RPCRT4!ObjectStubless+0x42
00000000`03c3e7c0 00000000`ffb329cc FastProx!CWbemSvcWrapper::XWbemServices::ExecQueryAsync+0xd4
00000000`03c3e830 00000000`ffb265ba wmiprvse!CServerObject_StaThread::ExecQueryAsync+0xd4
00000000`03c3e8a0 00000000`ffb268a8 wmiprvse!CInterceptor_IWbemSyncProvider::Helper_ExecQueryAsync+0x54a
00000000`03c3e950 000007fe`fd735ec5 wmiprvse!CInterceptor_IWbemSyncProvider::ExecQueryAsync+0x138
00000000`03c3e9f0 000007fe`fd711f46 RPCRT4!Invoke+0x65
00000000`03c3ea60 000007fe`fd7d5cae RPCRT4!NdrStubCall2+0x348
00000000`03c3f040 000007fe`f998412d RPCRT4!CStdStubBuffer_Invoke+0x66
00000000`03c3f070 000007fe`fddc89b9 FastProx!CBaseStublet::Invoke+0x19
00000000`03c3f0a0 000007fe`fddc892b ole32!SyncStubInvoke+0x5d
00000000`03c3f110 000007fe`fdc9d633 ole32!StubInvoke+0xdf
00000000`03c3f1b0 000007fe`fddc87c6 ole32!CCtxComChnl::ContextInvoke+0x19f
00000000`03c3f330 000007fe`fddc855f ole32!AppInvoke+0xc2
00000000`03c3f3a0 000007fe`fddc7314 ole32!ComInvokeWithLockAndIPID+0x407
00000000`03c3f520 000007fe`fd7368d4 ole32!ThreadInvoke+0x1f0
00000000`03c3f5d0 000007fe`fd7369f0 RPCRT4!DispatchToStubInCNoAvrf+0x14
00000000`03c3f600 000007fe`fd70b042 RPCRT4!RPC_INTERFACE::DispatchToStubWorker+0x100
00000000`03c3f6f0 000007fe`fd70afbb RPCRT4!RPC_INTERFACE::DispatchToStub+0x62
00000000`03c3f730 000007fe`fd70af4a RPCRT4!RPC_INTERFACE::DispatchToStubWithObject+0x5b
00000000`03c3f7b0 000007fe`fd737080 RPCRT4!LRPC_SCALL::DispatchRequest+0x436
00000000`03c3f820 000007fe`fd7362bb RPCRT4!LRPC_SCALL::HandleRequest+0x200
00000000`03c3f940 000007fe`fd735e1a RPCRT4!LRPC_ADDRESS::ProcessIO+0x44a
00000000`03c3fa60 000007fe`fd717769 RPCRT4!LOADABLE_TRANSPORT::ProcessIOEvents+0x24a
00000000`03c3fb10 000007fe`fd717714 RPCRT4!ProcessIOEventsWrapper+0x9
00000000`03c3fb40 000007fe`fd7177a4 RPCRT4!BaseCachedThreadRoutine+0x94
00000000`03c3fb80 00000000`76f2be3d RPCRT4!ThreadStartRoutine+0x24
00000000`03c3fbb0 00000000`77136a51 kernel32!BaseThreadInitThunk+0xd
00000000`03c3fbe0 00000000`00000000 ntdll!RtlUserThreadStart+0x1d

When checking the GetToSTA, I found the COM call was made to STA thread where WMI component did reside. I learned the client created a lot of multiple async threads and sent OLE requests simultaneously. All those COM calls were entered into STA thread message loop.

Looking at the STA thread, there is repeating pattern of the below red part. That is, the incoming call is processed by the STA thread -> STA thread calls another component and wait for reply -> STA thread peeks message queue to see if there is any other incoming message. If any, the STA thread processes the message. This is STA reentrancy and it is a COM feature.

…..
00000000`013da270 000007fe`fdca80c9 ole32!ComInvoke+0x85
00000000`013da2a0 000007fe`fdca7eae ole32!ThreadDispatch+0x29
00000000`013da2d0 00000000`7705d53e ole32!ThreadWndProc+0xaa
00000000`013da350 00000000`7705d7c6 USER32!UserCallWinProcCheckWow+0x1ad
00000000`013da410 000007fe`fdd31433 USER32!DispatchMessageWorker+0x389
00000000`013da490 000007fe`fdcb11e0 ole32!CCliModalLoop::PeekRPCAndDDEMessage+0x73
00000000`013da500 000007fe`fdc7b093 ole32!CCliModalLoop::BlockFn+0x36100
00000000`013da540 000007fe`fddc7689 ole32!ModalLoop+0x6f
…..
00000000`013dc360 000007fe`f4d50263 framedyn!Provider::CreateInstanceEnum+0x34
….
00000000`013dd5d0 000007fe`fdca80c9 ole32!ComInvoke+0x85
00000000`013dd600 000007fe`fdca7eae ole32!ThreadDispatch+0x29
00000000`013dd630 00000000`7705d53e ole32!ThreadWndProc+0xaa
00000000`013dd6b0 00000000`7705d7c6 USER32!UserCallWinProcCheckWow+0x1ad
00000000`013dd770 000007fe`fdd31433 USER32!DispatchMessageWorker+0x389
00000000`013dd7f0 000007fe`fdcb11e0 ole32!CCliModalLoop::PeekRPCAndDDEMessage+0x73
00000000`013dd860 000007fe`fdc7b093 ole32!CCliModalLoop::BlockFn+0x36100
00000000`013dd8a0 000007fe`fddb4cb0 ole32!ModalLoop+0x6f
00000000`013dd8f0 000007fe`fddcb946 ole32!SwitchSTA+0x20
00000000`013dd920 000007fe`fddc9375 ole32!CRpcChannelBuffer::SwitchAptAndDispatchCall+0x24a6
00000000`013dd970 000007fe`fdc7b3be ole32!CRpcChannelBuffer::SendReceive2+0xf1
….
00000000`013de310 000007fe`f4d50d1a FastProx!CWbemSvcWrapper::XWbemServices::GetObjectW+0x95
00000000`013de370 000007fe`f4d527d5 framedyn!Provider::GetClassObjectInterface+0xda
00000000`013de420 000007fe`fd735ec5 framedyn!CWbemProviderGlue::ExecQueryAsync+0x2ad
….
00000000`013df620 000007fe`fdca80c9 ole32!ComInvoke+0x85
00000000`013df650 000007fe`fdca7eae ole32!ThreadDispatch+0x29
00000000`013df680 00000000`7705d53e ole32!ThreadWndProc+0xaa
00000000`013df700 00000000`7705d7c6 USER32!UserCallWinProcCheckWow+0x1ad
00000000`013df7c0 00000000`ffb133f3 USER32!DispatchMessageWorker+0x389
00000000`013df840 00000000`ffb12eb8 wmiprvse!WmiThread<unsigned long>::ThreadWait+0x11b
00000000`013dfac0 00000000`ffb11aa8 wmiprvse!WmiThread<unsigned long>::ThreadDispatch+0xf4
00000000`013dfb20 00000000`76f2be3d wmiprvse!WmiThread<unsigned long>::ThreadProc+0x30
00000000`013dfb50 00000000`77136a51 kernel32!BaseThreadInitThunk+0xd
00000000`013dfb80 00000000`00000000 ntdll!RtlUserThreadStart+0x1d