Skip to content

Commit 929bb34

Browse files
ankan-banankan-ban
authored
Add documentation for NV TensorRT RTX ExecutionProvider (#24829)
### Description Add documentation for NV TensorRT RTX ExecutionProvider ### Motivation and Context The EP is already online. This is for adding some basic documentation for the options / features exposed by the EP.
1 parent 40617ef commit 929bb34

File tree

2 files changed

+251
-0
lines changed

2 files changed

+251
-0
lines changed

‎docs/build/eps.md

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -235,6 +235,21 @@ These instructions are for the latest [JetPack SDK](https://developer.nvidia.com
235235

236236
* For a portion of Jetson devices like the Xavier series, higher power mode involves more cores (up to 6) to compute but it consumes more resource when building ONNX Runtime. Set `--parallel 1` in the build command if OOM happens and system is hanging.
237237

238+
## TensorRT-RTX
239+
240+
See more information on the NV TensorRT RTX Execution Provider [here](../execution-providers/TensorRTRTX-ExecutionProvider.md).
241+
242+
### Prerequisites
243+
{: .no_toc }
244+
245+
* Follow [instructions for CUDA execution provider](#cuda) to install CUDA and setup environment variables.
246+
* Intall TensorRT for RTX from nvidia.com (TODO: add link when available)
247+
248+
### Build Instructions
249+
{: .no_toc }
250+
`build.bat --config Release --parallel 32 --build_dir _build --build_shared_lib --use_nv_tensorrt_rtx --tensorrt_home "C:\dev\TensorRT-RTX-1.1.0.3" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9" --cmake_generator "Visual Studio 17 2022" --use_vcpkg`
251+
Replace the --tensorrt_home and --cuda_home with correct paths to CUDA and TensorRT-RTX installations.
252+
238253
## oneDNN
239254

240255
See more information on oneDNN (formerly DNNL) [here](../execution-providers/oneDNN-ExecutionProvider.md).

‎docs/execution-providers/TensorRTRTX-ExecutionProvider.md

Lines changed: 236 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,236 @@
1+
---
2+
title: NVIDIA - TensorRT RTX
3+
description: Instructions to execute ONNX Runtime on NVIDIA RTX GPUs with the Nvidia TensorRT RTX execution provider
4+
parent: Execution Providers
5+
nav_order: 17
6+
redirect_from: /docs/reference/execution-providers/TensorRTRTX-ExecutionProvider
7+
---
8+
9+
# Nvidia TensorRT RTX Execution Provider
10+
{: .no_toc }
11+
12+
Nvidia TensorRT RTX execution provider is the preferred execution provider for GPU acceleration on consumer hardware (RTX PCs). It is more straightforward to use than the datacenter focused legacy TensorRT Execution provider and more performant than CUDA EP.
13+
Just some of the things that make it a better fit on RTX PCs than our legacy TensorRT Execution Provider:
14+
* Much smaller footprint
15+
* Much faster model compile/load times.
16+
* Better usability in terms of use of cached models across multiple RTX GPUs.
17+
18+
The Nvidia TensorRT RTX execution provider in the ONNX Runtime makes use of NVIDIA's [TensorRT](https://developer.nvidia.com/tensorrt) RTX Deep Learning inferencing engine (TODO: correct link to TRT RTX documentation once available) to accelerate ONNX models on RTX GPUs. Microsoft and NVIDIA worked closely to integrate the TensorRT RTX execution provider with ONNX Runtime.
19+
20+
Currently TensorRT RTX supports RTX GPUs from Ampere or later architectures. Support for Turing GPUs is coming soon.
21+
22+
## Contents
23+
{: .no_toc }
24+
25+
* TOC placeholder
26+
{:toc}
27+
28+
## Install
29+
Please select the Nvidia TensorRT RTX version of Onnx Runtime: https://onnxruntime.ai/docs/install. (TODO!)
30+
31+
## Build from source
32+
See [Build instructions](../build/eps.md#TensorRT-RTX).
33+
34+
## Requirements
35+
36+
| ONNX Runtime | TensorRT-RTX | CUDA |
37+
| :----------- | :----------- | :------------- |
38+
| main | 1.0 | 12.0-12.9 |
39+
| 1.22 | 1.0 | 12.0-12.9 |
40+
41+
## Usage
42+
### C/C++
43+
```c++
44+
const auto& api = Ort::GetApi();
45+
Ort::SessionOptions session_options;
46+
api.SessionOptionsAppendExecutionProvider(session_options, "NvTensorRtRtx", nullptr, nullptr, 0);
47+
Ort::Session session(env, model_path, session_options);
48+
```
49+
50+
The C API details are [here](../get-started/with-c.md).
51+
52+
### Python
53+
To use TensorRT RTX execution provider, you must explicitly register TensorRT RTX execution provider when instantiating the `InferenceSession`.
54+
55+
```python
56+
import onnxruntime as ort
57+
sess = ort.InferenceSession('model.onnx', providers=['NvTensorRtRtxExecutionProvider'])
58+
```
59+
60+
## Configurations
61+
TensorRT RTX settings can be configured via [TensorRT Execution Provider Session Option](./TensorRTRTX-ExecutionProvider.md#execution-provider-options).
62+
63+
Here are examples and different [scenarios](./TensorRTRTX-ExecutionProvider.md#scenario) to set NV TensorRT RTX EP session options:
64+
65+
#### Click below for Python API example:
66+
67+
<details>
68+
69+
```python
70+
import onnxruntime as ort
71+
72+
model_path = '<path to model>'
73+
74+
# note: for bool type options in python API, set them as False/True
75+
provider_options = {
76+
'device_id': 0,
77+
'nv_dump_subgraphs': False,
78+
'nv_detailed_build_log': True,
79+
'user_compute_stream': stream_handle
80+
}
81+
82+
sess_opt = ort.SessionOptions()
83+
sess = ort.InferenceSession(model_path, sess_options=sess_opt, providers=[('NvTensorRTRTXExecutionProvider', provider_options)])
84+
```
85+
86+
</details>
87+
88+
#### Click below for C++ API example:
89+
90+
<details>
91+
92+
```c++
93+
Ort::SessionOptions session_options;
94+
95+
cudaStream_t cuda_stream;
96+
cudaStreamCreate(&cuda_stream);
97+
98+
// Need to put the CUDA stream handle in a string
99+
char streamHandle[32];
100+
sprintf_s(streamHandle, "%lld", (uint64_t)cuda_stream);
101+
102+
const auto& api = Ort::GetApi();
103+
std::vector<const char*> option_keys = {
104+
"device_id",
105+
"user_compute_stream", // this implicitly sets "has_user_compute_stream"
106+
};
107+
std::vector<const char*> option_values = {
108+
"1",
109+
streamHandle
110+
};
111+
112+
Ort::ThrowOnError(api.SessionOptionsAppendExecutionProvider(session_options, "NvTensorRtRtx", option_keys.data(), option_values.data(), option_keys.size()));
113+
114+
```
115+
116+
</details>
117+
118+
### Scenario
119+
120+
| Scenario | NV TensorRT RTX EP Session Option | Type |
121+
| :------------------------------------------------- | :----------------------------------------------------------------------------------------- | :----- |
122+
| Specify GPU id for execution | [device_id](./TensorRTRTX-ExecutionProvider.md#device_id) | int |
123+
| Set custom compute stream for GPU operations | [user_compute_stream](./TensorRTRTX-ExecutionProvider.md#user_compute_stream) | string |
124+
| Set TensorRT RTX EP GPU memory usage limit | [nv_max_workspace_size](./TensorRTRTX-ExecutionProvider.md#nv_max_workspace_size) | int |
125+
| Dump optimized subgraphs for debugging | [nv_dump_subgraphs](./TensorRTRTX-ExecutionProvider.md#nv_dump_subgraphs) | bool |
126+
| Capture CUDA graph for reduced launch overhead | [nv_cuda_graph_enable](./TensorRTRTX-ExecutionProvider.md#nv_cuda_graph_enable) | bool |
127+
| Enable detailed logging of build steps | [nv_detailed_build_log](./TensorRTRTX-ExecutionProvider.md#nv_detailed_build_log) | bool |
128+
| Define min shapes | [nv_profile_min_shapes](./TensorRTRTX-ExecutionProvider.md#nv_profile_min_shapes) | string |
129+
| Define max shapes | [nv_profile_max_shapes](./TensorRTRTX-ExecutionProvider.md#nv_profile_max_shapes) | string |
130+
| Define optimal shapes | [nv_profile_opt_shapes](./TensorRTRTX-ExecutionProvider.md#nv_profile_opt_shapes) | string |
131+
132+
> Note: for bool type options, assign them with **True**/**False** in python, or **1**/**0** in C++.
133+
134+
### Execution Provider Options
135+
136+
TensorRT RTX configurations can be set by execution provider options. It's useful when each model and inference session have their own configurations. All configurations should be set explicitly, otherwise default value will be taken.
137+
138+
##### device_id
139+
140+
* Description: GPU device ID.
141+
* Default value: 0
142+
143+
##### user_compute_stream
144+
145+
* Description: define the compute stream for the inference to run on. It implicitly sets the `has_user_compute_stream` option. The stream handle needs to be printed on a string as decimal number and passed down to the session options as shown in the example above.
146+
147+
* This can also be set using the python API.
148+
* i.e The cuda stream captured from pytorch can be passed into ORT-NV TensorRT RTX EP. Click below to check sample code:
149+
150+
<Details>
151+
152+
153+
```python
154+
import onnxruntime as ort
155+
import torch
156+
...
157+
sess = ort.InferenceSession('model.onnx')
158+
if torch.cuda.is_available():
159+
s = torch.cuda.Stream()
160+
provider_options = {
161+
'device_id': 0,
162+
'user_compute_stream': str(s.cuda_stream)
163+
}
164+
165+
sess = ort.InferenceSession(
166+
model_path,
167+
providers=[('NvTensorRtRtxExecutionProvider', provider_options)]
168+
)
169+
170+
options = sess.get_provider_options()
171+
assert "NvTensorRtRtxExecutionProvider" in options
172+
assert options["NvTensorRtRtxExecutionProvider"].get("user_compute_stream", "") == str(s.cuda_stream)
173+
...
174+
```
175+
176+
</Details>
177+
178+
* To take advantage of user compute stream, it is recommended to use [I/O Binding](https://onnxruntime.ai/docs/performance/device-tensor.html) to bind inputs and outputs to tensors in device.
179+
180+
##### nv_max_workspace_size
181+
182+
* Description: maximum workspace size in bytes for TensorRT RTX engine.
183+
184+
* Default value: 0 (lets TensorRT pick the optimal).
185+
186+
##### nv_dump_subgraphs
187+
188+
* Description: dumps the subgraphs if the ONNX was split across multiple execution providers.
189+
* This can help debugging subgraphs, e.g. by using `trtexec --onnx subgraph_1.onnx` and check the outputs of the parser.
190+
191+
##### nv_detailed_build_log
192+
193+
* Description: enable detailed build step logging on NV TensorRT RTX EP with timing for each engine build.
194+
195+
##### nv_cuda_graph_enable
196+
197+
* Description: this will capture a [CUDA graph](https://developer.nvidia.com/blog/cuda-graphs/) which can drastically help for a network with many small layers as it reduces launch overhead on the CPU.
198+
199+
##### nv_profile_min_shapes
200+
201+
##### nv_profile_max_shapes
202+
203+
##### nv_profile_opt_shapes
204+
205+
* Description: build with explicit dynamic shapes using a profile with the min/max/opt shapes provided.
206+
* By default TensorRT RTX engines will support dynamic shapes, for perofmance improvements it is possible to specify one or multiple explicit ranges of shapes.
207+
* The format of the profile shapes is `input_tensor_1:dim_1xdim_2x...,input_tensor_2:dim_3xdim_4x...,...`
208+
* These three flags should all be provided in order to enable explicit profile shapes feature.
209+
* Note that multiple TensorRT RTX profiles can be enabled by passing multiple shapes for the same input tensor.
210+
* Check TensorRT doc [optimization profiles](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#opt_profiles) for more details.
211+
212+
## NV TensorRT RTX EP Caches
213+
There are two major TRT RTX EP caches:
214+
* Embedded engine model / EPContext model
215+
* Internal TensorRT RTX cache
216+
217+
The internal TensorRT RTX cache is automatically managed by the EP. The user only needs to manage EPContext caching.
218+
**Caching is important to help reduce session creation time drastically.**
219+
220+
TensorRT RTX separates compilation into an ahead of time (AOT) compiled engine and a just in time (JIT) compilation. The AOT compilation can be stored as EPcontext model, this model will be compatible across multiple GPU generations.
221+
Upon loading such an EPcontext model TensorRT RTX will just in time compile the engine to fit to the used GPU. This JIT process is accelerated by TensorRT RTX's internal cache.
222+
For an example usage see:
223+
https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/test/providers/nv_tensorrt_rtx/nv_basic_test.cc
224+
225+
### More about Embedded engine model / EPContext model
226+
* TODO: decide on a plan for using weight-stripped engines by default. Fix the EP implementation to enable that. Explain the motivation and provide example on how to use the right options in this document.
227+
* EPContext models also **enable packaging an externally compiled engine** using e.g. `trtexec`. A [python script](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/tensorrt/gen_trt_engine_wrapper_onnx_model.py) that is capable of packaging such a precompiled engine into an ONNX file is included in the python tools. (TODO: document how this works with weight-stripped engines).
228+
229+
## Performance Tuning
230+
For performance tuning, please see guidance on this page: [ONNX Runtime Perf Tuning](./../performance/tune-performance/index.md)
231+
232+
When/if using [onnxruntime_perf_test](https://github.com/microsoft/onnxruntime/tree/main/onnxruntime/test/perftest#onnxruntime-performance-test), use the flag `-e nvtensorrttrx`.
233+
234+
235+
### TensorRT RTX Plugins Support
236+
TensorRT RTX doesn't support plugins.

0 commit comments

Comments
 (0)