Feature Proposal: Smooth Camera Animations & Image Carousel GUI

[illusive-chase]

Hi there,

Thanks for creating and maintaining this excellent library! I've been using viser extensively in my research on multi-view reconstruction, and it has been incredibly helpful. As I've worked with visualizing camera poses, I've identified two potential enhancements that might be useful.

1. Smooth, Frontend-Based Camera Transitions

The current documentation provides a great example for programmatic camera control by updating the camera pose in a loop from the backend. While this works, I've noticed that the resulting animation can sometimes stutter, likely due to the communication latency between the Python backend and the browser frontend. For applications requiring fluid camera movements (like cycling through a set of known camera poses), a smoother, frontend-native animation might be ideal. To this end, I envision a simple, declarative API to move the camera to a target pose over a specified duration, with the animation handled entirely on the frontend. This would look something like:

client.set_camera(
    wxyz=T_world_set.rotation().wxyz,
    position=T_world_set.translation(),
    smoothness=0.9,
    duration=2000,
)

This would provide a more efficient alternative to the current backend-looping approach, as shown in the video demo.

screenshots.mp4

The current backend-looping approach

for j in range(20):
    T_world_set = T_world_current @ tf.SE3.exp(
        T_current_target.log() * j / 19.0
    )

    # We can atomically set the orientation and the position of the camera
    # together to prevent jitter that might happen if one was set before the
    # other.
    with client.atomic():
        client.camera.wxyz = T_world_set.rotation().wxyz
        client.camera.position = T_world_set.translation()

    client.flush()  # Optional!
    time.sleep(1.0 / 60.0)

2. Image Carousel GUI Component

Besides, I also wonder if there could be an image carousel GUI component (from @mantine/carousel ). In many visualization scenarios, especially those involving datasets of images (like NeRFs or multi-view stereo), it's very useful to have a GUI component to browse through the images.

For general cases, an image carousel could be used for displaying a list of images as a standalone image gallery.

image_carousel = server.gui.add_image_carousel(
    image_list,
    names=image_names, # Optional
)

image_carousel.value = 127 # Optionally programmatic control for showing 127th image

For some specific cases, like visualizing NeRF datasets, it could be used as a synchronized view selector. For example, when a user selects an image in the carousel, the 3D view automatically animates to the corresponding camera pose. This would pair perfectly with the smooth camera transition proposed above.

image_list = ...
camera_pose_list = ...

image_carousel = server.gui.add_image_carousel(
    image_list,
    names=[f'view[{i:04d}]' for i in range(len(image_list))],
)

@image_carousel.on_update
def _(event: viser.GuiEvent) -> None:
    selection = image_carousel.value
    event.client.set_camera(
        wxyz=camera_pose_list[selection].rotation().wxyz,
        position=camera_pose_list[selection].translation(),
        smoothness=0.9,
        duration=2000,
    )

I wonder whether these features would enhance the user experience for many common visualization tasks. I'd love to hear your thoughts on these proposals. If you think they align with the project's direction, I would be happy to work on a pull request to implement them.

[brentyi]

Hi @illusive-chase! Thanks for your nice words; improvements on both of these fronts would be super appreciated!

Smoother camera transitions: if you have time to explore this, one thing that would be nice is to first narrow down where the current choppy animations are coming from. We do have some logic for smoothing out network latency, which tries to match the time delay between messages processed by the client with the time delay seen by the server:

viser/src/viser/client/src/WebsocketServerWorker.ts

Lines 158 to 186 in b75b64a

	if (
	// Flush immediately for first message.
	state.lastIdealJsMs === undefined ||
	// Flush immediately if the Python delta is large, in this case we're
	// probably not sensitive to exact timing.
	pythonTimeDeltaMs > 100 ||
	// Flush if we're more than 100ms behind real-time.
	jsNowMs - state.jsTimeMinusPythonTime - currentPythonTimestampMs > 100
	) {
	// First message or no expected delta, send immediately.
	sendFn();
	state.lastIdealJsMs = jsNowMs;
	} else {
	// For messages that are being sent frequently: smooth out the sending rate.
	const idealNextSendTimeMs = state.lastIdealJsMs + pythonTimeDeltaMs;
	const timeUntilIdealJsMs = idealNextSendTimeMs - jsNowMs;
	if (timeUntilIdealJsMs > 3) {
	// We're early! This means the previous message was processed late...
	const dampingFactor = 0.95;
	setTimeout(sendFn, timeUntilIdealJsMs * dampingFactor);
	state.lastIdealJsMs =
	state.lastIdealJsMs + pythonTimeDeltaMs * dampingFactor;
	} else {
	// Message is on-time or late: send immediately.
	sendFn();
	state.lastIdealJsMs = jsNowMs;
	}
	}

If this logic can be tuned or improved to get 90% of the performance of fully client-driven animations: that'd be a much easier change to make than expanding the API.

Image carousel: this one I feel excited about (it looks beautiful!). I'm curious if it makes sense to break it down into smaller or more reusable primitives, or if there's some way to improve the existing primitives (like add_image()) so they can be used to support this use case. From your perspective, I'm wondering how you view the pain points of:

• Creating an image with server.gui.add_image()
• Creating Next / Previous buttons with server.gui.add_button_group()
• Updating the image programmatically in Python when the next/previous buttons are clicked

We could similarly replace the next/previous buttons with dropdown, slider, etc for selecting the image. (these are clearly all a worse experience than what you have implemented, but it'd be helpful for me to understand qualitatively what the things that matter are)