[Doc]: recommended docker for visual_gen

[christopher5106]

📚 The doc issue

Hi,

In the documentation you write

Recommended docker: nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc4

If you just leave like that, it's complicated to understand what to expect from that image. I saw that Python 3.12 is installed.

1. Where is Visual gen: I tried from tensorrt_llm import visual_gen without success
   Should we clone it ? Do we have to install the package ?

2. What is in /app/tensorrt_llm and in /workspace/tutorials ? I see some bin/lib/include folders in /app/tensorrt_llm, it seems you compiled something.

3. I see that pytorch is installed but some deprecations:

>>> import torch
/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:63: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
  import pynvml  # type: ignore[import]

4. Are there any SageAttention, FlashAttention installed ?

5. What is the goal of this recommendation ?

Please help

Suggest a potential alternative/fix

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[christopher5106]

I found the doc here https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tensorrt-llm/containers/release?version=1.3.0rc5.post1 but to my point of view, it's not clear how it relates to visual_gen that I didn't find in the image.

[zhenhuaw-me]

@christopher5106 Thanks for raising this issue. The feat/visual_gen branch is a prototype branch which is not included in the RC release. You need to go through the install guide which installs visual_gen as a separate python package. You can reach out to @forrestl111 if any setup issue on that branch.

The "production support" in the main branch is under development with limited model and features support. You can refer to this doc: https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/models/visual-generation.md
The first weekly release that is able to do like from tensorrt_llm import VisualGen will be 1.3.0rc8 and the release is still WIP.

[zhenhuaw-me]

I am going to convert this issue to a discussion since it's not a real doc issue. Please let me know if any concerns. Thank you for trying out TensorRT-LLM VisualGen!

[christopher5106]

@forrestl111 I did pip install -e . under TensorRT-LLM/tensorrt_llm/visual_gen/ but it still says that tensorrt_llm not available in the logs... what is wrong ?

[zhenhuaw-me]

@christopher5106 That's an error when try to pip install -e . under the visual_gen dir? Are you running in a TRT-LLM NGC container or install TRT-LLM with pip install tensorrt-llm==1.2.0rc8 --extra-index-url https://pypi.nvidia.com/?

Btw, you can try out the "early version" of VisualGen with the latest weekly release with the latest release. See https://nvidia.github.io/TensorRT-LLM/1.3.0rc7/models/visual-generation.html for the early doc.

To install:

pip install tensorrt-llm==1.3.0rc7 --extra-index-url https://pypi.nvidia.com/

[christopher5106]

I had removed the entrypoint from the docker, that was the reason. Now tensorrt_llm is well recognized. Remains a few points I'll list below + the difficulty to reproduce your performance gains.

[christopher5106]

Now, most problematic is that I don't reproduce the results you got with FP4 on Flux 2 dev on B200. Usually inference takes 40 s on H200, but I got an 2x acceleration only, not the expected 5x. Any help welcome.

I'm interested in the simple scenario "FP4 + compile + cuda graphs" where 5x gains are expected.  I followed the recommnended steps here, starting from the image you recommend nvcr.io/nvidia/tensorrt-llm/release:1.2.0rc4 . I run it with the docker params you recommend.

Let me give you some insights with the logs. But the missing libs should not have anything to do with this performance issue, no?

fused_qk_norm_rope is not available
--
  | [1773816506.135146] [modal:2    :0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /usr/local/ucx//lib/libucp.so.0)
  | [1773816506.166788] [modal:2    :0]       tcp_iface.c:941  UCX  ERROR scandir(/sys/class/net) failed: No such file or directory
  | [1773816506.425891] [modal:2    :0]     ucp_context.c:2339 UCX  WARN  UCP API version is incompatible: required >= 1.20, actual 1.19.0 (loaded from /usr/local/ucx//lib/libucp.so.0)
  | [1773816506.426617] [modal:2    :0]       tcp_iface.c:941  UCX  ERROR scandir(/sys/class/net) failed: No such file or directory
  | [Rank 0]2026-03-18 06:50:27 \| WARNING \| py.warnings \| /usr/local/lib/python3.12/dist-packages/modelopt/torch/utils/import_utils.py:32: UserWarning: Failed to import transformer engine plugin due to: AttributeError("module 'transformer_engine' has no attribute 'pytorch'"). You may ignore this warning if you do not need this plugin.   warnings.warn(
  | [Rank 0]2026-03-18 06:50:27 \| WARNING \| py.warnings \| /usr/local/lib/python3.12/dist-packages/modelopt/torch/utils/import_utils.py:32: UserWarning: Failed to import transformer_engine plugin due to: ImportError("cannot import name 'pytorch' from 'transformer_engine' (/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py)"). You may ignore this warning if you do not need this plugin.   warnings.warn(
  | [TensorRT-LLM] TensorRT LLM version: 1.2.0rc4
  | [Rank 0]2026-03-18 06:50:57 \| WARNING \| visual_gen.layers.linear \| Transformer_engine is not installed
  | [Rank 0]2026-03-18 06:50:57 \| WARNING \| visual_gen.ops.attention \| FlashAttn4 is not installed.
  | [Rank 0]2026-03-18 06:50:57 \| WARNING \| visual_gen.ops.attention \| TransformerEngine is not installed.
  | [Rank 0]2026-03-18 06:50:57 \| WARNING \| visual_gen.ops.attention \| SageAttention for Blackwell not available
  | [Rank 0]2026-03-18 06:50:57 \| WARNING \| visual_gen.ops.linear \| svdquant kernels are not compiles, svdquant only supports SM 120. [Rank 0]2026-03-18 06:50:57 \| WARNING \| visual_gen.ops.linear \| Transformer_engine is not installed: cannot import name 'pytorch' from 'transformer_engine' (/usr/local/lib/python3.12/dist-packages/transformer_engine/__init__.py)
  | [Rank 0]2026-03-18 06:50:57 \| WARNING \| visual_gen.ops.linear \| Deep_gemm is not installed
  | [Rank 0]2026-03-18 06:50:58 \| INFO \| apps.tensorrt_llm_visual_gen.app \|

 
Missing in my env:

1. UCX ERROR scandir(/sys/class/net) failed: No such file or directory: hard to understand for me
2. Transformer_engine: I installed it as explained here

module 'transformer_engine' has no attribute 'pytorch

seems to be the issue

3. fused_qk_norm_rope is not available
4. "FlashAttn4 is not installed." I can't find a way to install it:

pip install flash-attn-4
ERROR: Ignored the following yanked versions: 0.0.1
ERROR: Could not find a version that satisfies the requirement flash-attn-4 (from versions: 4.0.0b3, 4.0.0b4)
ERROR: No matching distribution found for flash-attn-4

pip install flash-attn-4==4.0.0b4
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tensorrt-llm 1.2.0rc4 requires flashinfer-python<0.4.0,>=0.3.0, but you have flashinfer-python 0.5.3 which is incompatible.
tensorrt-llm 1.2.0rc4 requires nvidia-cutlass-dsl==4.3.0.dev0; python_version >= "3.10", but you have nvidia-cutlass-dsl 4.4.2 which is incompatible.
visual-gen 0.2.0 requires nvidia-cutlass-dsl==4.3.0.dev0, but you have nvidia-cutlass-dsl 4.4.2 which is incompatible.
Successfully installed flash-attn-4-4.0.0b4 nvidia-cutlass-dsl-4.4.2 nvidia-cutlass-dsl-libs-base-4.4.2 quack-kernels-0.3.3 torch-c-dlpack-ext-0.1.5

[zhenhuaw-me]

@christopher5106 The production level support on the main branch has been released. Please check the doc https://nvidia.github.io/TensorRT-LLM/models/visual-generation.html for related information. You may follow the general install instruction here: https://nvidia.github.io/TensorRT-LLM/installation/index.html

Regarding the performance, we are WIP to improve the performance.