Privately Owned Vehicle Work Group Meeting - 2025/10/19 - Slot 1 & 2

[m-zain-khawaja]

Agenda

• Milestones
• Full pipeline integration
• QNX Integration
• ZENOH integration
• SceneSeg transfer-learning off-road WG
• AutoSteer Network (EgoPath + EgoLanes)
• CARLA integration steering pipeline
• AutoSpeed
• Object Finder
• E2E Research

Milestones

1. End of October - VisionPIlot stack ready for testing in simulator (on track)
2. End of November - VisionPilot stack deployed on edge hardware and benchmarked - first target hardware is the Nvidia AGX ORIN
3. End of December - VisionPIlot stack deployed on actual vehicle and able to demonstrate autonomous driving

Full pipeline integration

@pranavreddy23

Pranav has begun work on integrating the full VisionPilot pipeline in a ROS2 agnostic format for deployment on edge hardware, starting with the Nvidia ORIN AGX platform as a first target. The goal is to utilize GStreamer to create a vision pipeline with Nvidia's DeepStream SDK to pipe images directly to GPU for inference with our E2E models (AutoSpeed and AutoSteer) and to visualize the outputs in an OpenCV window and use Iceoryx pub/sub for downstream tracking and control processes. GStreamer implementation is completed and initial Iceoryx implementation is showing much better performance compared to ROS2.

Furthermore, the BrightDrive team have been able to run SceneSeg, Scene3D, DomainSeg and AutoSpeed on their test vehicle data using a Jetson ORIN and achieved 10+ FPS (max data recording FPS).

QNX Integration

The Garraio team (Mohammad Wagdy and Mahmoud) are working on QNX integration of the VisionPilot stack. PathFinder QNX implementation and testing is completed. Currently, work is being carried out AutoSpeed Integratoin.

ZENOH Integration

@evshary @m-zain-khawaja

CY has completed the Zenoh pipeline for SceneSeg, Scene3D and DomainSeg using the VisionPilot/common folder libs - allowing us to run the models with both ONNX runtime and TensorRT backends using ZENOH with the same shared framework as has been used for ROS2. New models (AutoSpeed/AutoSteer) and full pipeline integration is next.

SceneSeg transfer-learning off-road WG

@mhmdrz-mirdamadi

Muhammad Reza is evaluating a 'stixels' inspired approach to the SceneSeg network for off-road environments. Network design and pre-training of the semantic branch is completed.

AutoSteer Network (EgoPath + EgoLanes)

@m-zain-khawaja @TranHuuNhatHuy

We are re-formulating the AutoSteer Network to have two branches - the EgoLanes branch is being trained as a segmentation network to predict the lane masks and classify the ego-left/ego-right/other lanes using TuSimple, CurveLanes and OpenLane V1 data. The second network branch directly predicts the vehicle tyre angle given an image and will be trained using driving datasets Comma2k19, L2D (HugginFace) and ZOD (Zenesact).

CARLA integration steering pipeline

@JITERN

CARLA 0.10.0 and 0.9.16 support is complete using the Python API. The Controller nodes (Lateral and Longitudinal) have also been completed as has the PathFinder ROS2 node and CARLA integration. Jit is currently working on improving the controllers with better estimates of the feedforward term (curvature for steering, and nominal speed demand for longitudinal control).

AutoSpeed

@pranavreddy23 @atanasko

AutoSpeed network first version has been trained, and ONNX version of the model as well as TensorRT FP16 version of the model has been created with its ROS2 node in addition to a standalone version. Atanasko has completed benchmarking of the model and is working on a second version of the model with a new spatially aware loss function.

Object Finder

@pranavreddy23

Pranav has begun work on the C++ version of the ObjectFinder module to track the distance and speed of the closest-in-path-object and other moving objects using the OpenLane dataset ground truth data. In order to address the camera to road calibration challenge, he is working on a calibration pipeline which can be used for generic datasets.

E2E Research

@auo355

Augustine is doing a literature review of full monolithic E2E models and technologies which we can experiment with. Kevin has begun work to do a PoC test of Nvidia's COSMOS network with CARLA simulator to evaluate E2E simulation pipelines.

Literature Review:

https://docs.google.com/document/d/1ls-7LhYuCL1SbgiJBoCRg62UZ9FdcgLv5uA2zSYTIkY/edit?usp=sharing

Zoom Meeting Video Recording

Video Meeting Link
Please contact the work group lead (@m-zain-khawaja) to request access to a recording of this meeting.

[TranHuuNhatHuy]

Tran was here in the 1st slot

[ZaynabEM]

Zaynab EL MAWAS attended Slot 1

[MohWagdy]

attended slot 2

[pranavreddy23]

In Slot 2

[mhmdrz-mirdamadi]

attended slot 2

[AmrBadawyLondon]

Slot 2

[atanasko]

attended slot2

[M-Megawer]

Mahmoud Megawer : attended slot 2

[auo355]

i attended slot 2 meeting