Pointcloud processor pipeline shows huge publishing latency with no errors

[siddharth-w]

Checklist

• I've read the contribution guidelines.
• I've searched other issues and no duplicate issues were found.
• I'm convinced that this is not my fault but a bug.

Description

The crop_box_filter_self, crop_box_filter_mirror and distortion_corrector_node shows huge latency in subsequent message publish even though the
/pointcloud_raw_ex is getting published by the nebula_ros package with default frequency.
Following are the configuration and params with the behavior given in the "Actual behavior" section:

lidar.launch.xml :

vehicle_info.param.yaml :

vehicle_mirror.param.yaml

Expected behavior

Pointcloud pipeline has minimal latency

Actual behavior

As the crop_box_filter_self node doesnt take any other input so it shouldnt be related to the anything other than pointcloud input. Below is the actual behaviour:

pipeline_latency_ms :

ros2 topic echo /sensing/lidar/top/crop_box_filter_self/debug/pipeline_latency_ms
                                                                
stamp:
  sec: 1737126653
  nanosec: 337972075
data: 518.684237
---
stamp:
  sec: 1737126655
  nanosec: 381116577
data: 562.35845
---
stamp:
  sec: 1737126658
  nanosec: 205620609
data: 587.268885
---

processing_time_ms :

ros2 topic echo /sensing/lidar/top/crop_box_filter_self/debug/processing_time_ms 

stamp:
  sec: 1737126653
  nanosec: 337906384
data: 5.598
---
stamp:
  sec: 1737126655
  nanosec: 381072219
data: 5.79
---
stamp:
  sec: 1737126658
  nanosec: 205581197
data: 6.181
---

published timestamp

ros2 topic echo /sensing/lidar/top/self_cropped/pointcloud_ex/debug/published_time 
header:
  stamp:
    sec: 1737126652
    nanosec: 819285760
  frame_id: ''
published_stamp:
  sec: 1737126653
  nanosec: 338033688
---
header:
  stamp:
    sec: 1737126654
    nanosec: 818756096
  frame_id: ''
published_stamp:
  sec: 1737126655
  nanosec: 381204125
---
header:
  stamp:
    sec: 1737126657
    nanosec: 618349824
  frame_id: ''
published_stamp:
  sec: 1737126658
  nanosec: 205685058
---

The same behavior is carried forward till localization/util/downsample/pointcloud

Please help in an elaborated way :(

Steps to reproduce

1. Connect and configure the sensors and vehicle params
2. Launch autoware.launch.xml

Versions

OS: Ubuntu 22.04.5 LTS x86_64
ROS 2: Humble
Autoware: 0.39.0
Lidar: Velodyne VLP16
GPS: Ublox F9P
IMU: Zed2 camera IMU
CPU: Intel Xeon W-1390P (16) @ 5.100GHz
GPU: NVIDIA RTX A5000
GPU: Intel RocketLake-S GT1 [UHD Graphics P750]
Memory: 7548MiB / 64026MiB

Possible causes

No response

Additional context

No response

[xmfcx]

Hi, Could you make sure DDS is configured correctly? Please follow to check:

• https://autowarefoundation.github.io/autoware-documentation/main/installation/additional-settings-for-developers/network-configuration/dds-settings/

Also can you check if the nebula driver and the crobbox filter are in the same composable node container?

For this, you can run ros2 component list to see the containers and composable nodes.

You may also share the ros2 bag, this will make it easier to debug.

Also can you set the FOV to full 360 degrees, without limiting to see if it might have caused the issue?

[siddharth-w]

Hi, Thanks for looking into it.

This is the component list, and nebula driver and the cropbox filter are indeed in the same container.

ros2 component list
/system/component_state_monitor/container
  1  /system/component_state_monitor/component
/map/map_container
  1  /map/pointcloud_map_loader
  2  /map/lanelet2_map_loader
  3  /map/lanelet2_map_visualization
  4  /map/vector_map_tf_generator
/system/mrm_comfortable_stop_operator/mrm_comfortable_stop_operator_container
  1  /system/mrm_comfortable_stop_operator
/system/mrm_emergency_stop_operator/mrm_emergency_stop_operator_container
  1  /system/mrm_emergency_stop_operator
/control/control_container
  1  /control/glog_control_container_component
  2  /control/external_cmd_converter
  3  /control/external_cmd_selector
  4  /control/trajectory_follower/controller_node_exe
  5  /control/autoware_shift_decider
  6  /control/vehicle_cmd_gate
  7  /control/autoware_operation_mode_transition_manager
/zed/zed_container
  1  /zed/zed_node
/sensing/lidar/top/pointcloud_preprocessor/pointcloud_container
  1  /sensing/lidar/top/glog_component
  2  /sensing/lidar/top/velodyne_ros_wrapper_node
  3  /sensing/lidar/top/crop_box_filter_self
  4  /sensing/lidar/top/crop_box_filter_mirror
  5  /sensing/lidar/top/distortion_corrector_node
  6  /sensing/lidar/top/ring_outlier_filter
/adapi/container
  1  /adapi/node/autoware_state
  2  /adapi/node/diagnostics
  3  /adapi/node/fail_safe
  4  /adapi/node/heartbeat
  5  /adapi/node/interface
  6  /adapi/node/localization
  7  /adapi/node/motion
  8  /adapi/node/operation_mode
  9  /adapi/node/perception
  10  /adapi/node/planning
  11  /adapi/node/routing
  12  /adapi/node/vehicle
  13  /adapi/node/vehicle_info
  14  /adapi/node/vehicle_door
/control/control_check_container
  1  /control/glog_control_check_container_component
  2  /control/trajectory_follower/lane_departure_checker_node
  3  /control/control_validator
  5  /control/collision_detector
/system/system_monitor/system_monitor/system_monitor_container
  1  /system/system_monitor/cpu_monitor
  2  /system/system_monitor/mem_monitor
  3  /system/system_monitor/net_monitor
  4  /system/system_monitor/ntp_monitor
  5  /system/system_monitor/process_monitor
  6  /system/system_monitor/hdd_monitor
  7  /system/system_monitor/gpu_monitor
  8  /system/system_monitor/voltage_monitor
/pointcloud_container
  1  /localization/util/crop_box_filter_measurement_range
  2  /sensing/lidar/concatenate_data
  3  /localization/util/voxel_grid_downsample_filter
  4  /localization/util/random_downsample_filter
/autoware_api/external/rtc_controller/container
  1  /autoware_api/external/rtc_controller/node

I might be hallucinating, but I think changing the DDS settings made it a bit faster, although its still painfully slow. I did change the FOV, cloud_min_angle=0 to cloud_max_angle=359 and the result was still the same.
I am attaching the ros2 bag as a zip file: ros2 bag

[knzo25]

@siddharth-w
The link you provided is not valid.
Does your problem occur on both the real vehicle or just with rosbags?
If the later, please paste your rosbag command
If it happens on the vehicle, remember that latency is the time on the node vs. the header stamp, which may come from the sensor. If the delay is big even for the raw pointcloud (ros2 topic delay...), then you need to synchronize the lidar. Sadly. VLPs are not as nice as newer sensors for this

[siddharth-w]

@knzo25 Hey
It occurs with the real hardware setup. The raw pointcloud works as intended, with no dealy. What I mean by latency is the gap between 2 successive messages by the pointcloud processor nodes, such as crop_box_filter_self, its takes on avg around 5 secs between 2 successive publishes and reaches as high as 20 secs.
I'll update the link.

[xmfcx]

Hi @siddharth-w !

Even when testing with real hardware, if the lidar clock is different than the system time, and the timestamps of the point clouds come from the lidar time, you will have a time synchronization issue.

This is why we are asking to test ros2 topic delay /pointcloudtopic .

mfc@bunny:~$ ros2 topic delay --help
usage: ros2 topic delay [-h] [--window WINDOW] [--spin-time SPIN_TIME] [-s] topic

Display delay of topic from timestamp in header

It displays the time difference between current ROS 2 time and the header of the message.