🐛 Bug Report: ONNXRuntime 1.23 fails to load model with external data (works fine in 1.22.x)

[harisreedhar]

Describe the issue

After upgrading from onnxruntime 1.22 to 1.23.0, loading a previously working ONNX model throws a shape inference error related to external data.
This issue does not occur in version 1.22.x the same code and model run successfully there.

[ONNXRuntimeError] : 1 : FAIL : Node (/decoder/Split_33) Op (Split) [ShapeInferenceError] Cannot parse data from external tensors. Please load external data into raw data for tensor: /decoder/Constant_1066_output_0

Related: danielgatis/rembg#790

To reproduce

import onnxruntime as ort

model_path = "BiRefNet-COD-epoch_125.onnx"
session = ort.InferenceSession(
    model_path,
    providers=["CPUExecutionProvider"]
)

BiRefNet-COD-epoch_125.onnx

Urgency

No response

Platform

Linux

OS Version

Linux Mint 22 Cinnamon

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.23.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CPU

Execution Provider Library Version

No response

[snnn]

We are investigating this issue and have made some initial findings.

Analysis so far:

1. ONNX library validation passes: The model loads successfully with the standard ONNX Python library (onnx.load() and onnx.checker.check_model()) without any errors.

2. ONNXRuntime 1.23.1 fails: Confirmed the error occurs when creating an InferenceSession:

   [ONNXRuntimeError] : 1 : FAIL : Node (/decoder/Split_33) Op (Split) [ShapeInferenceError] 
   Cannot parse data from external tensors. Please load external data into raw data for tensor: /decoder/Constant_1066_output_0
   

3. No custom operators: The model uses only standard ONNX operators (opset version 17). All 21,968 nodes use the default ai.onnx domain - no custom ops were found.

4. The tensor in question is NOT external data: Inspecting the model structure reveals that /decoder/Constant_1066_output_0 is produced by a standard Constant node (/decoder/Constant_1066) with an embedded tensor attribute (shape [16], type INT64). This tensor does not have external data - it's embedded directly in the model.

5. The problematic Split node: The node /decoder/Split_33 is a standard Split operation that takes /decoder/Constant_1066_output_0 as one of its inputs (the split sizes).

Preliminary conclusion:

This appears to be a regression in ONNXRuntime 1.23.x where the shape inference code incorrectly identifies an embedded constant tensor as having external data, or fails to properly handle the tensor data during shape inference. The error message is misleading since the tensor doesn't actually use external data storage.

We will continue investigating the ONNXRuntime source code to identify the root cause and develop a fix.

[snnn]

Update: Root cause identified through debugging

Using GDB to debug the issue, I've identified the root cause of this regression in ONNXRuntime 1.23.x:

Key findings:

1. The problematic tensor: /decoder/Constant_1066_output_0 is the value attribute of a Constant node (/decoder/Constant_1066)

2. Incorrect data location marking: During shape inference, this tensor is marked with data_location = EXTERNAL but has no actual data:

   • has_raw_data() = false
   • int64_data_size() = 0
   • data_location = onnx::TensorProto_DataLocation_EXTERNAL

3. ONNX correctly rejects it: The ONNX shape inference code (in onnx::ParseData<int64_t>()) detects the external data location and throws the error:

   Cannot parse data from external tensors. Please load external data into raw data for tensor: /decoder/Constant_1066_output_0
   

   This is correct behavior from ONNX - it cannot parse external data during shape inference.

4. The tensor should have embedded data: Our analysis of the original model shows this Constant node has an embedded tensor attribute, not external data.

Conclusion:

This is a regression in ONNXRuntime 1.23.x where Constant node tensor attributes are incorrectly marked as having external data during graph transformations, when they should retain their embedded data. The issue occurs before shape inference is invoked.

Next steps: Search for changes in ONNXRuntime between v1.22 and v1.23 related to Constant node handling or external data processing during graph transformations.

[snnn]

Critical finding: The issue is related to graph optimization levels

Testing with different graph optimization levels reveals an important pattern:

Optimization Level	Result
ORT_DISABLE_ALL	❌ FAILS with the external data error
ORT_ENABLE_BASIC	✅ SUCCESS - Model loads correctly!
ORT_ENABLE_EXTENDED	❌ FAILS with the external data error
ORT_ENABLE_ALL (default)	❌ FAILS with the external data error

Key insights:

1. The issue exists in initial model loading: It fails with ORT_DISABLE_ALL, meaning the Constant node tensor is incorrectly marked as external data when the model is first loaded, before any optimizations run.

2. BASIC optimization fixes it: Some optimization pass in the BASIC level converts or replaces the problematic Constant node in a way that resolves the issue.

3. EXTENDED/ALL don't include the fix: These optimization levels either skip the fixing optimization from BASIC, or apply additional transformations that reintroduce/expose the problem.

Workaround for users:

Until this is fixed, users can work around the issue by setting the graph optimization level to BASIC:

import onnxruntime as ort

sess_options = ort.SessionOptions()
sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC

session = ort.InferenceSession(
    model_path,
    sess_options,
    providers=["CPUExecutionProvider"]
)

This suggests we need to investigate:

1. What changed in model loading between v1.22 and v1.23 that causes Constant nodes to be marked as external
2. Which BASIC optimization is fixing the issue
3. Why EXTENDED/ALL don't include this fix

[snnn]

*ROOT CAUSE IDENTIFIEDecho BEGIN___COMMAND_OUTPUT_MARKER ; PS1= ; PS2= ; EC=0 ; echo ___BEGIN___COMMAND_DONE_MARKER___0 ; }

Through debugging and code analysis, I've found the root cause of this regression in ONNXRuntime 1.23.

The Problem:

Commit 3b97d79 ("Convert Initializers to OrtValues Phase 2", PR #25320) introduced code that converts large Constant node tensors to OrtValues for memory efficiency. The logic:

1. When a Constant node's tensor is > 127 bytes (kSmallTensorExternalDataThreshold)
2. The tensor is converted to an OrtValue
3. A new TensorProto is created that points to the OrtValue's memory buffer using the "external data" mechanism
4. This sets data_location = EXTERNAL in the TensorProto

Why it fails:

ONNX's shape inference code (onnx::ParseData) cannot distinguish between:

• External data in memory (accessible by ONNXRuntime)
• External data in files (not accessible during shape inference)

When it sees data_location = EXTERNAL, it rejects the tensor with the error we're seeing.

The specific tensor:

/decoder/Constant_1066_output_0 has shape [16] with type INT64 = 128 bytes, just over the 127-byte threshold, triggering the externalization logic.

Why ORT_ENABLE_BASIC works:

Some optimization pass in the BASIC level (likely constant folding or similar) replaces or transforms these Constant nodes before shape inference runs on the problematic nodes, avoiding the issue.

Code location:

onnxruntime/core/graph/graph.cc around line 1226-1268 (in the Graph constructor)
onnxruntime/core/framework/tensorprotoutils.cc line 1498-1510 (TensorToTensorProto function)

The fix needed:

Before running shape inference, ONNXRuntime needs to:

1. Detect tensors with in-memory external data (check for kTensorProtoMemoryAddressTag)
2. Temporarily provide the actual data to ONNX shape inference
3. Or skip shape inference for nodes with inputs that have in-memory external data

I'll work on a fix next.

[snnn]

Summary of Investigation

I've completed a thorough investigation of this issue using GDB debugging and code analysis. Here's a comprehensive summary:

---

Root Cause

Commit: 3b97d79 ("Convert Initializers to OrtValues Phase 2", PR #25320)
Introduced in: ONNXRuntime v1.23.0

What Changed

The commit introduced an optimization to convert large Constant node tensors into OrtValues for better memory management. The logic in Graph::Graph() constructor:

// If tensor size > 127 bytes (kSmallTensorExternalDataThreshold):
1. Convert tensor data to OrtValue
2. Replace TensorProto with one that has external data reference to OrtValue memory
3. This sets data_location = EXTERNAL in the TensorProto

Why It Fails

ONNX's shape inference code (onnx::ParseData<T>()) explicitly rejects any TensorProto with data_location = EXTERNAL:

if (tensor_proto->has_data_location() && 
    tensor_proto->data_location() == TensorProto_DataLocation_EXTERNAL) {
  fail_shape_inference("Cannot parse data from external tensors...");
}

ONNX cannot distinguish between:

• External data in memory (accessible by ONNXRuntime, identified by kTensorProtoMemoryAddressTag)
• External data in files (not accessible during shape inference)

The Specific Failure

The problematic tensor /decoder/Constant_1066_output_0:

• Shape: [16]
• Type: INT64
• Size: 16 × 8 = 128 bytes
• Just over the 127-byte threshold, triggering the externalization

---

Testing Results

Optimization Level	Result	Explanation
ORT_DISABLE_ALL	❌ FAILS	Issue present in initial model load
ORT_ENABLE_BASIC	✅ WORKS	Some optimization transforms/removes problematic nodes
ORT_ENABLE_EXTENDED	❌ FAILS	Doesn't include the fixing optimization
ORT_ENABLE_ALL	❌ FAILS	Same as EXTENDED

---

Code Locations

1. Externalization logic:
   onnxruntime/core/graph/graph.cc lines ~1226-1268
   Function: put_data_maybe_in_memory lambda in Graph constructor

2. TensorProto creation with external data:
   onnxruntime/core/framework/tensorprotoutils.cc lines 1485-1526
   Function: TensorToTensorProto()

3. Sets data_location = EXTERNAL:
   onnxruntime/core/framework/tensor_external_data_info.cc
   Function: ExternalDataInfo::SetExternalLocationToProto()

4. ONNX rejection:
   _deps/onnx-src/onnx/defs/tensor_proto_util.cc lines ~50-55
   Macro: DEFINE_PARSE_DATA

---

Proposed Fix

ONNXRuntime needs to handle in-memory external data before shape inference. Options:

1. Provide actual data to ONNX: Temporarily materialize the tensor data from OrtValue for shape inference
2. Skip problematic shape inference: Detect in-memory external data and skip verification for those specific cases
3. Use a different marker: Don't set data_location = EXTERNAL for in-memory references; use a different mechanism

The fix should be in onnxruntime/core/graph/graph.cc in the InferenceContextImpl::getInputData() method or similar, to detect and handle in-memory external data.

---

Temporary Workaround

Users can work around this issue by using BASIC optimization:

import onnxruntime as ort

sess_options = ort.SessionOptions()
sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC

session = ort.InferenceSession(model_path, sess_options, providers=["CPUExecutionProvider"])

[snnn]

Fix Completed! 🎉

I've created a pull request with a fix for this issue: #26263

What was done:

1. Root cause identified through GDB debugging and code analysis
2. Fix implemented in onnxruntime/core/graph/graph.cc
3. Unit test added to prevent regression
4. All optimization levels tested and working

The Fix

Modified InferenceContextImpl::getInputData() to detect in-memory external data and materialize it into a temporary TensorProto that ONNX shape inference can process.

Testing Confirmed

✅ Model from the issue loads successfully with all optimization levels:

• ORT_DISABLE_ALL
• ORT_ENABLE_BASIC
• ORT_ENABLE_EXTENDED
• ORT_ENABLE_ALL (default)

✅ Unit test passes

For Users Experiencing This Issue

Until the fix is merged and released, you can use this workaround:

import onnxruntime as ort

sess_options = ort.SessionOptions()
sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_BASIC

session = ort.InferenceSession(model_path, sess_options, providers=["CPUExecutionProvider"])

The fix is now ready for review and should be included in the next ONNXRuntime release.

[henryruhs]

Thanks for the fix.