Skip to content

Adding interaction states for headless widgets. #19238

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 16 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 11 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
121 changes: 120 additions & 1 deletion crates/bevy_picking/src/hover.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use crate::{
};

use bevy_derive::{Deref, DerefMut};
use bevy_ecs::prelude::*;
use bevy_ecs::{entity::EntityHashSet, prelude::*};
use bevy_math::FloatOrd;
use bevy_platform::collections::HashMap;
use bevy_reflect::prelude::*;
Expand Down Expand Up @@ -275,3 +275,122 @@ fn merge_interaction_states(
new_interaction_state.insert(*hovered_entity, new_interaction);
}
}

/// A component that allows users to use regular Bevy change detection to determine when the pointer
/// enters or leaves an entity. Users should insert this component on an entity to indicate interest
/// in knowing about hover state changes.
///
/// This is similar to the old Bevy [`Interaction`] component, except that it only tracks
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels a bit awkward as "old" implies Interaction either no longer exists or is deprecated. Otherwise this is just a history lesson about what came first, which isn't really something users need to care about.

/// hover state, not button presses or other interactions.
///
/// The component's boolean value will be `true` whenever the pointer is currently hovering over the
/// entity, or any of the entity's children. This is consistent with the behavior of the CSS
/// `:hover` pseudo-class, which applies to the element and all of its children.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it makes sense to say "descendants" here instead of "children" as this applies to arbitrary levels of the hierarchy. It would also be good to use "directly" here, to connect this to HoveredDirect.

Suggested change
/// The component's boolean value will be `true` whenever the pointer is currently hovering over the
/// entity, or any of the entity's children. This is consistent with the behavior of the CSS
/// `:hover` pseudo-class, which applies to the element and all of its children.
/// The component's boolean value will be `true` whenever the pointer is currently directly hovering over the
/// entity, or any of the entity's descendants (as defined by the [`ChildOf`] relationship). This is consistent with the behavior of the CSS
/// `:hover` pseudo-class, which applies to the element and all of its children.

///
/// The contained boolean value is guaranteed to only be mutated when the pointer enters or leaves
/// the entity, allowing Bevy change detection to be used efficiently. This is in contrast to the
/// [`HoverMap`] resource, which is updated every frame.
///
/// Typically, a simple hoverable entity or widget will have this component added to it. More
/// complex widgets can have this component added to each hoverable part.
///
/// The computational cost of keeping the `Hovered` components up to date is relatively cheap,
/// and linear in the number of entities that have the `Hovered` component inserted.
///
/// [`Interaction`]: https://docs.rs/bevy/0.15.0/bevy/prelude/enum.Interaction.html
#[derive(Component, Copy, Clone, Default, Eq, PartialEq, Debug, Reflect)]
#[reflect(Component, Default, PartialEq, Debug, Clone)]
pub struct Hovered(pub bool);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find it slightly confusing that the "hovered" state in Hovered is not the same as the "hovered" entity state in HoverMap. Ex: an entity not under a pointer with a child that is under a pointer is not "hovered" according to the HoverMap, but is "hovered" according to Hovered. Given that HoverMap is a public contract, I don't think we should be giving two different answers to the "what is hovered" question.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I need something that replicates the behavior of CSS :hover pseudo-class. We can rename it if that would make things clearer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider an HTML checkbox: even though the actual checkbox is a small widget, hovering or clicking on the label element is considered the same as clicking on the actual checkbox. Now, there are several ways to arrange this, for example you could have a parent element which includes both the checkbox and the label. But you could also make the label a child of the checkbox.

Similarly, one might envision a slider where the circular thumb element extends outside of the bounds of the track.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More broadly, the purpose of a visual hover indicator is to telegraph what will happen with the subsequent click - but since clicks bubble, a click on a child element, if not intercepted, will also be handled by the parent. The hover indicator needs to signal this. This is why the CSS :hover pseudo-class behaves as it does.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also missed PickingInteraction::Hovered, which mirrors the state of HoverMap to a given entity. I think we need to come up with different terminology on one side or the other, as I consider this all to be a "single API surface".

There are now currently two "hover" concepts in bevy_picking:

  1. "Cursor is directly over an unblocked Pickable::hover entity" (currently PickingInteraction::Hovered and HoverMap). This can be more than one entity, provided Pickable::should_block_lower is false for the first hovered entity highest in the stack.
  2. "The result of (1) propagated up to all ancestors" (currently Hovered). Notably this currently ignores both Pickable::is_hoverable, Pickable::should_block_lower, as well as generally ignoring whether or not an ancestor is "pickable" at all in the first place (ex: any non-UI ancestor entity can still be Hovered, provided it has the component)

I don't have a solid solution off the top of my head, but I think its pretty clear this needs some additional consideration:

  1. Do we need both concepts?
  2. If so, how do we disambiguate these concepts?
  3. Should we indiscriminately propagate Hovered up to all ancestors, ignoring whether or not they are pickable, whether or not they are "hoverable", etc?

I do think it makes sense to ignore the should_block_lower behavior in the context of Hovered.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suppose I change the name of the component to something like HoverWithin?

I think that should_block_lower is orthogonal here: it controls whether an entity is pickable, but an entity can be non-pickable and still "contain" a hovered entity, that is, the "within" predicate is independent of whether or not the entity itself is pickable. And the non-pickable entity will still get a bubbled click event from its child, so telegraphing of future clicks should be consistent.

In terms of use cases: "hover within" is the more common case, at least in my experience (YMMV). Complex widgets which are aggregates of multiple entities generally fall into two groups:

  • Widgets which are artistic composites of parts, but which are singular in interaction (e.g. a button containing a text label and an icon). For these cases, "hover within" is the right predicate.
  • Widgets which have multiple "handles" or "cells" that are independently draggable or selectable. In this case, each of the individual handle entities will likely have it's own Hover component and highlight state.

It is possible, however, to envision a widget which doesn't fall into either scheme - for example, a widget with a large number of handles or cells, where the hovering behavior is handled at the parent level via some kind of multiplexing scheme. This widget would want to know exactly which child was hovered, however this gets tricky because the child may itself have children (text content) and so you end up having to do some kind of ancestor check anyway. I'm having a hard time envisioning a concrete use case for this, but it's not impossible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's another anecdote: A classic novice error in Bevy UI is where you have a button with a text label, and you implement a hover effect when the button is hovered - only to discover that when the mouse hovers over the text the button is no longer highlighted, because now the text, rather than the button, is in the hover map. Then what happens is that you ask on discord (this has happened) and what people tell you is to make the text non-pickable. This is the wrong answer.

It's wrong because buttons shouldn't impose this constraint on their contents. In a BSN world, buttons may contain text, images, or complex hierarchies of entities, because the content of a button is a template parameter. It's onerous to make the user sanitize all of this content to ensure that none of the entities being passed into the button are pickable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then what happens is that you ask on discord (this has happened) and what people tell you is to make the text non-pickable. This is the wrong answer.

Agreed. I do think the "hovered within" approach is the better solve here.

So "answering" my questions one by one:

  1. Do we need both concepts?

I'm reasonably convinced that we need both. The "pointer hover" system feeds into "hovered within", and I can imagine use cases that care about direct hovering info.

  1. If so, how do we disambiguate these concepts?

We don't currently have events or components for each concept, but I think we should consider these all "peers" of each other:

  • pointer over: there is a pointer directly above this entity (note that it looks like the Pointer<Over> event is driven using the HoverMap not the OverMap. We might need to rename that? Also see Pointer<Enter> and Pointer<Leave> #16289)
  • hovered: there is a pointer over this entity, and no entity "above" it in the stack is blocking it
  • hovered within: either this entity is hovered, or one of its descendants is

I think "hovered" and "hovered within" work reasonably well. Although "hovered within" is less friendly (and less immediately clear) than "hovered". In general I think people will try to reach for the "hovered" name over "hovered within", then learn that what they actually want is "hovered within".

Working backwards, if we want the "hovered within" concept to be known as "hovered" (to bias people toward using that), what can we call the "pointer hover" concept to disambiguate? Perhaps "pointer hover" vs "hover" is suitably clear, provided we always specify "pointer hover" in that context (ex: PointerHoverMap, Pointer::Hover, etc).

It is worth pulling in @NthTensor and @aevyrie to see what they think here.

  1. Should we indiscriminately propagate Hovered up to all ancestors, ignoring whether or not they are pickable, whether or not they are "hoverable", etc?

Ok yeah I see the value here. Ex: this system will also be used outside of UI, and scenes will often have a simple Transform-only entity at the top, which is not directly "pickable" by a backend. That scene could easily want a top level hover event.


/// Uses [`HoverMap`] changes to update [`Hovered`] components.
pub fn update_hovering_states(
hover_map: Option<Res<HoverMap>>,
mut hovers: Query<(Entity, &mut Hovered)>,
parent_query: Query<&ChildOf>,
) {
// Don't do any work if there's no hover map.
let Some(hover_map) = hover_map else { return };

// Don't bother collecting ancestors if there are no hovers.
if hovers.is_empty() {
return;
}

// Set which contains the hovered for the current pointer entity and its ancestors. The capacity
// is based on the likely tree depth of the hierarchy, which is typically greater for UI (because
// of layout issues) than for 3D scenes. A depth of 32 is a reasonable upper bound for most use
// cases.
let mut hover_ancestors = EntityHashSet::with_capacity(32);
if let Some(map) = hover_map.get(&PointerId::Mouse) {
for hovered_entity in map.keys() {
hover_ancestors.insert(*hovered_entity);
hover_ancestors.extend(parent_query.iter_ancestors(*hovered_entity));
}
}

// For each hovered entity, it is considered "hovering" if it's in the set of hovered ancestors.
for (entity, mut hoverable) in hovers.iter_mut() {
let is_hovering = hover_ancestors.contains(&entity);
hoverable.set_if_neq(Hovered(is_hovering));
}
}

#[cfg(test)]
mod tests {
use bevy_render::camera::Camera;

use super::*;

#[test]
fn update_hovering_is_memoized() {
let mut world = World::default();
let camera = world.spawn(Camera::default()).id();

// Setup entities
let hovered_child = world.spawn_empty().id();
let hovered_entity = world.spawn(Hovered(false)).add_child(hovered_child).id();

// Setup hover map with hovered_entity hovered by mouse
let mut hover_map = HoverMap::default();
let mut entity_map = HashMap::new();
entity_map.insert(
hovered_child,
HitData {
depth: 0.0,
camera,
position: None,
normal: None,
},
);
hover_map.insert(PointerId::Mouse, entity_map);
world.insert_resource(hover_map);

// Run the system
assert!(world.run_system_cached(update_hovering_states).is_ok());

// Check to insure that the hovered entity has the Hovered component set to true
let hover = world.get_mut::<Hovered>(hovered_entity).unwrap();
assert!(hover.0);
assert!(hover.is_changed());

// Now do it again, but don't change the hover map.
world.increment_change_tick();

assert!(world.run_system_cached(update_hovering_states).is_ok());
let hover = world.get_mut::<Hovered>(hovered_entity).unwrap();
assert!(hover.0);

// Should not be changed
// NOTE: Test doesn't work - thinks it is always changed
// assert!(!hover.is_changed());

// Clear the hover map and run again.
world.insert_resource(HoverMap::default());
world.increment_change_tick();

assert!(world.run_system_cached(update_hovering_states).is_ok());
let hover = world.get_mut::<Hovered>(hovered_entity).unwrap();
assert!(!hover.0);
assert!(hover.is_changed());
}
}
10 changes: 8 additions & 2 deletions crates/bevy_picking/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -392,6 +392,7 @@ impl Plugin for PickingPlugin {
.register_type::<Self>()
.register_type::<Pickable>()
.register_type::<hover::PickingInteraction>()
.register_type::<hover::Hovered>()
.register_type::<pointer::PointerId>()
.register_type::<pointer::PointerLocation>()
.register_type::<pointer::PointerPress>()
Expand All @@ -407,7 +408,7 @@ pub struct InteractionPlugin;
impl Plugin for InteractionPlugin {
fn build(&self, app: &mut App) {
use events::*;
use hover::{generate_hovermap, update_interactions};
use hover::{generate_hovermap, update_hovering_states, update_interactions};

app.init_resource::<hover::HoverMap>()
.init_resource::<hover::PreviousHoverMap>()
Expand All @@ -429,7 +430,12 @@ impl Plugin for InteractionPlugin {
.add_event::<Pointer<Scroll>>()
.add_systems(
PreUpdate,
(generate_hovermap, update_interactions, pointer_events)
(
generate_hovermap,
update_interactions,
update_hovering_states,
pointer_events,
)
.chain()
.in_set(PickingSystems::Hover),
);
Expand Down
61 changes: 61 additions & 0 deletions crates/bevy_ui/src/interaction_states.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
/// This module contains components that are used to track the interaction state of UI widgets.
///
// Note to implementers: This uses a combination of both marker components and newtype components
// containing a bool. Markers are used for one-way binding, "write-only" components
// (like `InteractionDisabled`) that relay instructions from the user to the framework, whereas
// newtype components are used to request state updates from the framework, which mutates the
// content of those components on update.
use bevy_a11y::AccessibilityNode;
use bevy_ecs::{
component::{Component, HookContext},
world::DeferredWorld,
};

/// A marker component to indicate that a widget is disabled and should be "grayed out".
/// This is used to prevent user interaction with the widget. It should not, however, prevent
/// the widget from being updated or rendered, or from acquiring keyboard focus.
///
/// For apps which support a11y: if a widget (such as a slider) contains multiple entities,
/// the `InteractionDisabled` component should be added to the root entity of the widget - the
/// same entity that contains the `AccessibilityNode` component. This will ensure that
/// the a11y tree is updated correctly.
#[derive(Component, Debug, Clone, Copy)]
#[component(on_add = on_add_disabled, on_remove = on_remove_disabled)]
pub struct InteractionDisabled;

// Hook to set the a11y "disabled" state when the widget is disabled.
fn on_add_disabled(mut world: DeferredWorld, context: HookContext) {
let mut entt = world.entity_mut(context.entity);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: entt seems like a slightly awkward name for this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW the name "entt" (pronounced "en-tee-tee") is one of the "short local variable names" that I often use in my code, I picked up the name from some Bevy code example, don't remember where.

if let Some(mut accessibility) = entt.get_mut::<AccessibilityNode>() {
accessibility.set_disabled();
}
}

// Hook to remove the a11y "disabled" state when the widget is enabled.
fn on_remove_disabled(mut world: DeferredWorld, context: HookContext) {
let mut entt = world.entity_mut(context.entity);
if let Some(mut accessibility) = entt.get_mut::<AccessibilityNode>() {
accessibility.clear_disabled();
}
}

/// Component that indicates whether a button is currently pressed. This will be true while
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is nothing enforcing this behavior at the moment. If this is a "universal" state intended to be shared and not something specific to, say, the upcoming "bevy core widgets" (which is where this behavior is implemented in your prototype), I think it should call out explicitly that this is a contract that implementers are expected to fulfill.

That being said, maybe whether or not a button is pressed on drag should be a widget-specific behavior? (aka maybe we should delete this line, then document the drag behavior in the relevant core widget docs?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also note that we can't call this Pressed because that name is already used in the prelude.

It's an open question how universal we want this to be. Some widgets are pressable and others are not. But "pressing" has uses outside of UI - for example PickingInteraction lives in bevy_picking, not bevy_ui, and has a Pressed variant.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this does feel a little awkward as we already have a universal "pressing" system (bevy_picking::Pressed). But by adopting the "controlled" model, we are decoupling the state of the button from that. I think this is fine: it is the job of the developer to "rewire" the bevy picking state to the button state, if they so choose. These systems are compatible / harmonious with each other. I wouldn't want to call this Pressed even if that name wasn't in the prelude, as that creates a situation where something is "pressed" according to bevy_picking, but "not pressed" according to the local UI state. At least with ButtonPressed we put a little bit of space between the two concepts (although maybe using different terms like Activated would disambiguate appropriately) .

Part of me does want to embrace an "uncontrolled" model here, as that would allow us to more directly unify the bevy_picking concepts/events/components with bevy_ui. When considering both this case and the hovered case, it feels like we aren't building a cohesive framework of concepts. Users will be confused.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Buttons are neither controlled or uncontrolled because they don't have values - that is, while buttons do have state (hover and pressed), that state is meant for internal use only. Yes, the button state is accessible to allow for custom visual styling, but that state is not part of the protocol by which the button communicates with the rest of the app - it's not like a slider or checkbox which controls a value.

What this means is that controlled widgets are only "controlled" with respect to the primary value being edited. All other state variables (like drag mode and offset) are managed by the widget internally, without the need for cooperation from the app. Or to put it even simpler: classic MVC.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok yeah that makes sense. I was imagining the pressed state as similar to the "checked" state, in that a developer might want to gate whether or not it is actually "pressed" based on their own criteria. But that seems less likely.

/// a drag action is in progress.
#[derive(Component, Default, Debug)]
pub struct ButtonPressed(pub bool);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that the other "bool state" components are immutable, maybe we should make this immutable too for consistency / to enable reliable observer-driven management?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok


/// Component that indicates whether a checkbox or radio button is in a checked state.
#[derive(Component, Default, Debug)]
#[component(immutable, on_add = on_add_checked, on_replace = on_add_checked)]
pub struct Checked(pub bool);

// Hook to set the a11y "checked" state when the checkbox is added.
fn on_add_checked(mut world: DeferredWorld, context: HookContext) {
let mut entt = world.entity_mut(context.entity);
let checked = entt.get::<Checked>().unwrap().0;
let mut accessibility = entt.get_mut::<AccessibilityNode>().unwrap();
accessibility.set_toggled(match checked {
true => accesskit::Toggled::True,
false => accesskit::Toggled::False,
});
}
2 changes: 2 additions & 0 deletions crates/bevy_ui/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
//! Spawn UI elements with [`widget::Button`], [`ImageNode`], [`Text`](prelude::Text) and [`Node`]
//! This UI is laid out with the Flexbox and CSS Grid layout models (see <https://cssreference.io/flexbox/>)

pub mod interaction_states;
pub mod measurement;
pub mod ui_material;
pub mod update;
Expand All @@ -35,6 +36,7 @@ mod ui_node;

pub use focus::*;
pub use geometry::*;
pub use interaction_states::*;
pub use layout::*;
pub use measurement::*;
pub use render::*;
Expand Down
49 changes: 49 additions & 0 deletions release-content/release-notes/headless-widgets.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
---
title: Headless Widgets
authors: ["@viridia"]
pull_requests: [19238]
---

Bevy's `Button` and `Interaction` components have been around for a long time. Unfortunately
these components have a number of shortcomings, such as the fact that they don't use the new
`bevy_picking` framework, or the fact that they are really only useful for creating buttons
and not other kinds of widgets like sliders.

As an art form, games thrive on novelty: the typical game doesn't have boring, standardize controls
reminiscent of a productivity app, but instead will have beautiful, artistic widgets that are
in harmony with the game's overall visual theme. But writing new and unique widgets requires
skill and subtlety, particularly if we want first-class accessibility support. It's not a burden we
want to put on the average indie developer.

In the web development world, "headless" widget libraries, such as
[headlessui](https://headlessui.com/) and [reakit](https://reakit.io/) have become popular. These
provide standardized widgets that implement all of the correct interactions and behavioral logic,
including integration with screen readers, but which are unstyled. It's the responsibility of the
game developer to provide the visual style and animation for the widgets, which can fit the overall
style of their game.

With this release, Bevy introduces a collection of headless or "core" widgets. These are components
which can be added to any UI Node to get widget-like behavior. The core widget set includes buttons,
sliders, scrollbars, checkboxes, radio buttons, and more. This set will likely be expanded in
future releases.

## Widget Interaction States

Many of the core widgets will define supplementary ECS components that are used to store the widget's
state, similar to how the old `Interaction` component worked, but in a way that is more flexible.
These components include:

- `InteractionDisabled` - a marker component used to indicate that a component should be
"grayed out" and non-interactive. Note that these disabled widgets are still visible and can
have keyboard focus (otherwise the user would have no way to discover them).
- `Hovering` is a simple boolean component that allows detection of whether the widget is being
hovered using regular Bevy change detection.
- `Checked` is a boolean component that stores the checked state of a checkbox or radio button.
- `ButtonPressed` is used for a button-like widget, and will be true while the button is held down.

The combination of `Hovering` and `ButtonPressed` fulfills the same purpose as the old `Interaction`
component, except that now we can also represent "roll-off" behavior (the state where you click
on a button and then, while holding the mouse down, move the pointer out of the button's bounds).
It also provides additional flexibility in cases where a widget has multiple hoverable parts,
or cases where a widget is hoverable but doesn't have a pressed state (such as a tree-view expansion
toggle).