DWMBlurGlass heavily conflicts with Paint.NET, causing softlock when switching user contexts, among other issues

[SKCro]

Describe the bug

A clear and concise description of what the bug is:

Paint.NET 5.1.11 seems to conflict heavily with DWMBlurGlass, causing some sort of weird issue where it causes various visual bugs in other programs and, more importantly, softlocks the computer when locking your session (via Win+L, screensaver timeout, rundll32 User32.dll,LockWorkStation or otherwise), opening a UAC dialog, pressing Ctrl+Alt+Del, shutting down/restarting/logging out, etc, all of which involve switching user contexts.

Other symptoms include Firefox context menus no longer drawing, Chrome's UI failing to render at all if the program wasn't already open before, UWP apps failing to render entirely unless it has already been opened that session (eg. Win+Period emoji picker, Settings app), and the virtual desktop switcher animation (Ctrl+Win+RightArrow/Ctrl+Win+LeftArrow) failing to play and causing an instant switch (the most reliable way to test for the presence of this bug).

The system returns to normal the moment Paint.NET closes or DWMBlurGlass is unloaded. Disabling hardware acceleration in Paint.NET and disabling all plugins does not help. The bug also cannot be mitigated by loading DWMBG after opening Paint.NET - the bug just comes back, so it's definitely some kind of conflict.

My theory is that this is some kind of weird GPU surface allocation bug? I have no idea though.

To Reproduce

Steps to reproduce the behavior:

1. Install and activate DWMBlurGlass
2. Open Paint.NET
3. Observe the bugs (virtual desktop switcher and softlock when locking/Ctrl+Alt+Del are the easiest ways to test)

Screenshots

If applicable, add screenshots to help explain your problem:

N/A, since it's hard to screenshot menus that don't draw, and it'd be hard to record a softlock since recording software runs as the user.

Additional information (please complete the following information):
OS Version : Windows 10 IoT LTSC 2021, OS Build 19044.6575
Software Version : 2.3.1

What other third-party software do you use:
Paint.NET (obviously)
Windhawk (although injection into PDN has been disabled for all mods for troubleshooting)
ExplorerBlurMica
TranslucentFlyouts (Disabling either and/or both of these doesn't help so that narrows it down to just PDN, DWMBG, and possibly my GPU driver)

Additional context
I'm using a Steam Deck, and I hear the GPU drivers are a bit buggy with edge cases so it might be an issue with that? I dunno though.

[Skyyblaze]

It might be a SteamDeck GPU issue as I've tested to see if I can replicate any of the behavior you have seen with paint.NET 5.1.11. Running DWMBlurGlass 2.3.2 Beta 1 everything seems to work as it should. I'm on RTX 4070 Ti Super GPU.

[SKCro]

It might be a SteamDeck GPU issue as I've tested to see if I can replicate any of the behavior you have seen with paint.NET 5.1.11. Running DWMBlurGlass 2.3.2 Beta 1 everything seems to work as it should. I'm on RTX 4070 Ti Super GPU.

Went ahead and tried the 2.3.2 Beta, same issue there, so it's probably some sort of weird issue with Steam Deck drivers in particular. I've experienced general bugginess with those drivers (Win+Ctrl+Shift+B is a guaranteed bluescreen...) so I wouldn't be surprised if they were the culprit.

No idea how I'd be able to debug this issue, all I know is that it's probably causing some kind of GPU surface deadlock that's causing DWM to fail to create new rendering surfaces, thus causing softlocks? I dunno how it all works on the backend so yeah.

[chuck00000]

Latest beta still not working in canary build.

[Skyyblaze]

@SKCro Sadly I think it would be better to mention this issue to whatever entity maintains the Steam Deck Windows drivers than here because it seems to be more of a systemic issue with the drivers themselves that causes the issues with DWMBlurGlass.

[SKCro]

@SKCro Sadly I think it would be better to mention this issue to whatever entity maintains the Steam Deck Windows drivers than here because it seems to be more of a systemic issue with the drivers themselves that causes the issues with DWMBlurGlass.

I think it's moreso a combo of Windows updates and driver edge-case issues that's causing this, since I had this problem a few months back and it mysteriously went away, then came back again (and no, uninstalling updates somehow didn't help).
AMD infamously does the bare minimum to maintain the Deck's Windows drivers, and Steam kinda just gives users the "this stuff is provided as-is, we don't support it" on their page linking to Windows drivers, so I guess I'm kinda on my own here.

I feel like, at the very least, a way to exclude a program from being affected by DWMBG would help mitigate this issue, and I've seen some people suggest it before, although iirc there hasn't been any word from the devs about adding it. From what I can tell, it's probably not possible since DWMBG injects into DWM, not individual applications.

[Maplespe]

Actually, there's not much else DWMBlurGlass can do to verify or fix this, since it works fine on my end.
I think it's highly likely a driver issue. DWMBlurGlass doesn't actually create or modify surfaces directly, or call low-level drawing interfaces. We rely on DirectComposition, Microsoft's public visual API for modern apps. It abstracts the UI into 'Visuals,' allowing us to add Brushes and Effects, but that's all provided by the API—we can't touch the low-level stuff.
It's just like the low FPS issue we saw before, which was an internal dcomp limitation that could be fixed via the registry. But in this case, I'm not sure what registry settings would need to be changed to tweak dcomp and solve your problem.