[OpenADKit] Issue with Service-Based Deployment Sample

[rt-2pm2]

I am trying to run the OpenADKit sample deployments reported in the new documentation.

I started with the logging scenario.

The monolithic approach, based on a single container encapsulating the full Autoware stack, albeit a bit lagging, works.
In the contrary, the Service-Based Deployment is not running as expected.
Precisely, I am not seeing the pointcloud, neither the vehicle. I could suspect that the sensing pipeline is not working, hindering also localization and perception. I see only the speed and the steering wheel angle, so the bag is playing correctly and the interface is working.

How I run the script

I followed the instructions from the documentation page, except that I was not running detached.

I run the main components by:

docker compose --env-file logging-simulation.env --env-file logging-simulation.gpu.env -f docker-compose.yaml -f docker-compose.gpu.yaml 
--profile logging-simulation --profile visualization up

Then I started the ros bag by:

docker compose --env-file logging-simulation.env -f docker-compose.yaml up rosbag

My findings

I attach the log with the output from the various container.
service-based.log

I am suspicious for the messages:

autoware-sensing-perception | [component_container_mt-1] [WARN] [1760967377.594374054] [sensing.lidar.concatenate_data]: New subscription discovered on topic '/sensing/lidar/concatenated/pointcloud', requesting incompatible QoS. No messages will be sent to it. Last incompatible policy: RELIABILITY_QOS_POLICY
autoware-sensing-perception | [component_container_mt-1] [WARN] [1760967377.643631872] [perception.obstacle_segmentation.occupancy_grid_based_outlier_filter]: New subscription discovered on topic '/perception/obstacle_segmentation/pointcloud', requesting incompatible QoS. No messages will be sent to it. Last incompatible policy: RELIABILITY_QOS_POLICY

If I got it right, the lidar data is fed to the autoware framework after concatenating all the lidars sensors. If this step fails, maybe the perception and localization pipelines get stuck?
Indeed, till the end, I see components complaining about the missing data sources:

autoware-localization | [autoware_ndt_scan_matcher_node-1] [WARN] [1760967491.206267420] [localization.pose_estimator.ndt_scan_matcher]: No InputSource. Please check the input lidar topic

Any ideas?

[oguzkaganozt]

Hi @rt-2pm2 ,

Thanks for the feedback, I tried again with service-based logging simulation and it seems to work, the vehicle takes a bit to localize after the rosbag starts, hence I increased the default rosbag replay rate from 0.2 to 0.4 to give smoother experience.

Could you try again and confirm if you still see the issue?

Best.

[rt-2pm2]

Hi @oguzkaganozt,

Thanks for the support. I tested the script after your update, but it does not solve the problem on my specific instance.
However, I made other tests and I suppose the problem is on my side. Indeed, I was testing on a VM hosted on a arm64 machine.
I tried the same script baremetal on a x86 machine and it works.
I will investigate further and provide updates.

[mitsudome-r]

It might improve if you set CYLCLONE_DDS configuration to services.

1. create cyclonedds.xml file on your host environment(reference) :

<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
  <Domain Id="any">
    <General>
      <Interfaces>
        <NetworkInterface autodetermine="false" name="lo" priority="default" multicast="default" />
      </Interfaces>
      <AllowMulticast>default</AllowMulticast>
      <MaxMessageSize>65500B</MaxMessageSize>
    </General>
    <Internal>
      <SocketReceiveBufferSize min="10MB"/>
      <Watermarks>
        <WhcHigh>500kB</WhcHigh>
      </Watermarks>
    </Internal>
  </Domain>
</CycloneDDS>

2. For each service in your docker-compose.yaml file, add the following:

volumes:
  - /path/to/cyclonedds.xml:/autoware/cyclonedds.xml
environment:
  - CYCLONEDDS_URI=/autoware/cyclonedds.xml

3. Make sure you set multicast on your host machine

sudo ip link set lo multicast on

4. Make sure configure the network settings for Autoware

# Increase the maximum receive buffer size for network packets
sudo sysctl -w net.core.rmem_max=2147483647  # 2 GiB, default is 208 KiB

# IP fragmentation settings
sudo sysctl -w net.ipv4.ipfrag_time=3  # in seconds, default is 30 s
sudo sysctl -w net.ipv4.ipfrag_high_thresh=134217728  # 128 MiB, default is 256 KiB

[rt-2pm2]

Hello @mitsudome-r,

Thank you very much for summarizing the network configuration steps.
Yes, I have that configuration for the host and the VM, except for the network interface. In my case I am not using the Loopback, but the VM ethernet intarface. The VM network is in bridged mode.

On the container, I am mounting on
In my case I was using something like:

    environment:
      - RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
      - ROS_DOMAIN_ID=${ROS_DOMAIN_ID}
      - CYCLONEDDS_URI=file://cyclonedds.xml
    volumes:
      - ${DDSCONF}:/cyclonedds.xml

Where DDSCONF variable is set to the path to the cyclonedds.xml on the VM.

I will double-check my network configuration to make sure I didn't make any mistakes.

Thanks again for your support.

[rt-2pm2]

Update: with the current set of containers it works. Probably, there was some obsoleted container which had some issue. I pulled the most recent version and the issue was gone.