Privately Owned Vehicle Work Group Meeting - 2025/11/10 - Slot 1 & 2

[pranavreddy23]

Agenda

• Milestones
• Full pipeline integration
• QNX Integration
• ZENOH integration
• SceneSeg transfer-learning off-road WG
• AutoSteer Network (EgoPath + EgoLanes)
• CARLA integration steering pipeline
• AutoSpeed
• Object Finder
• E2E Research

Milestones

1. End of October - VisionPilot stack ready for testing in simulator (on track)
2. End of November - VisionPilot stack deployed on edge hardware and benchmarked - first target hardware is the Nvidia AGX ORIN
3. End of December - VisionPilot stack deployed on actual vehicle and able to demonstrate autonomous driving

Full pipeline integration

@pranavreddy23

Pranav has begun work on integrating the full VisionPilot pipeline in a ROS2 agnostic format for deployment on edge hardware, starting with the Nvidia ORIN AGX platform as a first target. The goal is to utilize GStreamer to create a vision pipeline with Nvidia's DeepStream SDK to pipe images directly to GPU for inference with our E2E models (AutoSpeed and AutoSteer) and to visualize the outputs in an OpenCV window and use Iceoryx pub/sub for downstream tracking and control processes. GStreamer implementation is completed and initial Iceoryx implementation is showing much better performance compared to ROS2.

Now that AutoSpeed and ObjectFinder implementations are ready , the IceOryx/IceOryx2 implmentation work can be started .

Furthermore, the BrightDrive team have been able to run SceneSeg, Scene3D, DomainSeg and AutoSpeed on their test vehicle data using a Jetson ORIN and achieved 10+ FPS (max data recording FPS).

AutoSteer Network (EgoPath + EgoLanes)

@m-zain-khawaja @TranHuuNhatHuy

We are re-formulating the AutoSteer Network to have two branches - the EgoLanes branch is being trained as a segmentation network to predict the lane masks and classify the ego-left/ego-right/other lanes using TuSimple, CurveLanes and OpenLane V1 data.

The second network branch directly predicts the vehicle tyre angle given an image and will be trained using driving datasets Comma2k19, L2D (HugginFace) and ZOD (Zenesact).

For the past 2 weeks, Tran has been extensively gathering and processing various raw datasets - TuSimple, CurveLanes, OpenLane, Jiqing, Once3DLane, and BDD100K. The total number of samples is expected to be around ~500k. With this, the full, comprehensive training of first branch - EgoLanes - will be commenced later this week.

QNX Integration

The Garraio team (Mohammad Wagdy and Mahmoud) are working on QNX integration of the VisionPilot stack. PathFinder QNX implementation and testing is completed. Currently, work is being carried out AutoSpeed Integration.

ZENOH Integration

@evshary @m-zain-khawaja

CY has completed the Zenoh pipeline for SceneSeg, Scene3D and DomainSeg using the VisionPilot/common folder libs - allowing us to run the models with both ONNX runtime and TensorRT backends using ZENOH with the same shared framework as has been used for ROS2. New models (AutoSpeed/AutoSteer) and full pipeline integration is next.

SceneSeg transfer-learning off-road WG

@mhmdrz-mirdamadi

Muhammad Reza is evaluating a 'stixels' inspired approach to the SceneSeg network for off-road environments. Network design and pre-training of the semantic branch is completed.

CARLA integration steering pipeline

@JITERN @atanasko

CARLA 0.10.0 and 0.9.16 support is complete using the Python API. The Controller nodes (Lateral and Longitudinal) have also been completed as has the PathFinder ROS2 node and CARLA integration. Jit is currently working on improving the controllers with better estimates of the feedforward term (curvature for steering, and nominal speed demand for longitudinal control).

Atanasko is integrating the CARLA Scenario Runner with CARLA 0.9.16 to run NCAP scenarios in OpenScenario format for testing in simulation.

AutoSpeed

@pranavreddy23 @atanasko

AutoSpeed network first version has been trained, and ONNX version of the model as well as TensorRT FP16 version of the model has been created with its ROS2 node in addition to a standalone version.

CPU support for AutoSpeed model is ready and sent to be added for OpenAD kit integration. The PR is merged into the main branch.

Object Finder

@pranavreddy23

Pranav has started work on the C++ version of the ObjectFinder module, which tracks the distance and speed of the closest-in-path object (CIPO) as well as other moving objects using ground truth data from the OpenLane dataset.

The Calibration module, which computes the homography matrix using the Waymo Open Dataset, has been completed.

The ObjectFinder pipeline, incorporating a position-only measurement system with a Kalman filter, has also been completed.

Additionally, ORB + BF feature mapping for detecting cut-in scenarios and maintaining the main CIPO tracking has been implemented.

Code cleaning is in progress, and a pull request for the implementation will be raised soon.

E2E Research

@auo355

Augustine is doing a literature review of full monolithic E2E models and technologies which we can experiment with. Kevin has begun work to do a PoC test of Nvidia's COSMOS network with CARLA simulator to evaluate E2E simulation pipelines.

Literature Review:
https://docs.google.com/document/d/1ls-7LhYuCL1SbgiJBoCRg62UZ9FdcgLv5uA2zSYTIkY/edit?usp=sharing

Zoom Meeting Video Recording

Video Meeting Link

Please contact the work group lead (@m-zain-khawaja) to request access to a recording of this meeting.

[pranavreddy23]

@pranavreddy23 in slot 2

[MohWagdy]

Attended slot 2

[auo355]

Attended slot 2

[atanasko]

Atanasko Boris Mitrev slot 2

[norhan-ghoniem]

attended slot 2