bug(a11y): FocusTrap doesn't match browser/spec tab order in relation to Shadow DOM

[Xiphoseer]

EDIT: Removed the regression questions, it's not a regression

Description

A project I work with uses a component library based on https://lit.dev with CUSTOM_ELEMENTS_SCHEMA. They sometimes have regular, tabbable elements (<a>, <button>) in their Shadow DOM and participate in the browser tab order as expected.

However, the CDK FocusTrap machinery and the _getFirstTabbableElement / _getLastTabbableElement don't quite consider enough of the sequential focus navigation. That means that focus trapping in a dialog only works it there's at least one focusable area not in a Shadow DOM.

For performance reason, I don't think it's necessary to build up the full sequential focus navigation order. It'd probably be sufficient to use the "DOM" selection mechanism i.e. find the first/last suitable sequentially focusable area element in shadow including tree order (host - shadow dom - light dom).

Reproduction

StackBlitz link: https://stackblitz.com/edit/components-issue-starter-2e9thbl6?file=src%2Fmain.ts
Steps to reproduce:

1. Click in the preview window, preferably somewhere in the top part of the red area.
2. Tab to see the focus (outline) on the button in the red area
3. Tab to see the focus on the hidden trap element
4. Tab to see the focus (outline) on the button in the blue area
5. Keep Tabbing to see the focus actually get trapped

Expected Behavior

The focus should remain trapped in the red area

Actual Behavior

The focus moves past the red area, to an invisible item and then only gets trapped in the blue area.

Environment

• Angular: 19, 20
• CDK/Material: 19, 20
• Browser(s): Edge, Firefox
• Operating System (e.g. Windows, macOS, Ubuntu): Windows 11 Enterprise