XNNPack crashes with mv3 model using repurposed dl3 android app

[abhinaykukkadapu]

🐛 Describe the bug

Repro steps:

1. Use the dl3 demo app, modify the source code to point it to rc2 aar and change the pte file name in MainActivity.java to the pte in https://www.internalfb.com/manifold/explorer/executorch/tree/abhinayk/mv3_xnnpack_crash
2. adb push /data/local/tmp/mv3_xnnpack_fp32.pte
3. Let gradle sync
4. Click run (you can ignore the image segmentation related processing after the module forward as it is irrelevant in this case and the crash occurs before the processing during, module.forward.

P1980149620 for crash logs

Versions

v1.0.0-rc2

[GregoryComer]

Adding to 1.0 release blockers to triage / repro.

[GregoryComer]

MV2 is also reported to be crashing.

[GregoryComer]

I've reproduced this locally using MV3 on the demo app on Android emulator. Next up is compiling the AAR in debug mode.

[GregoryComer]

Here's the stack trace I captured with a debug executorch.aar build (commit 563db4d):

2025-10-10 14:52:28.090  7649-7689  libc                    org.pytorch.executorchexamples.dl3   A  Fatal signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xb400007c73286000 in tid 7689 (Thread-2), pid 7649 (rchexamples.dl3)
---------------------------- PROCESS STARTED (7696) for package org.pytorch.executorchexamples.dl3 ----------------------------
2025-10-10 14:52:28.604  7694-7694  DEBUG                   crash_dump64                         A  Cmdline: org.pytorch.executorchexamples.dl3
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A  pid: 7649, tid: 7689, name: Thread-2  >>> org.pytorch.executorchexamples.dl3 <<<
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #01 pc 00000000003e76b8  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (create_fully_connected_nc+952) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #02 pc 00000000003ea478  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (xnn_create_fully_connected_nc_f32+468) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #03 pc 00000000003e620c  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (create_fully_connected_operator+732) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #04 pc 00000000003a4a38  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (create_runtime_impl+1100) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #05 pc 000000000034e78c  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (executorch::backends::xnnpack::delegate::XNNCompiler::compileModel(void const*, unsigned long, executorch::backends::xnnpack::delegate::XNNExecutor*, executorch::backends::xnnpack::delegate::XNNWeightsCache*, xnn_workspace*, executorch::runtime::NamedDataMap const*)+1324) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #06 pc 00000000003502d8  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (executorch::backends::XnnpackBackend::init(executorch::runtime::BackendInitContext&, executorch::runtime::FreeableBuffer*, executorch::runtime::ArrayRef<executorch::runtime::CompileSpec>) const+176) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #07 pc 00000000008dc980  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (executorch::runtime::BackendDelegate::Init(executorch_flatbuffer::BackendDelegate const&, executorch::runtime::Program const*, executorch::runtime::BackendInitContext&, executorch::runtime::BackendDelegate*)+760) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #08 pc 00000000008dbefc  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (executorch::runtime::Method::init(executorch_flatbuffer::ExecutionPlan*, executorch::runtime::NamedDataMap const*)+676) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #09 pc 00000000008dbb14  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (executorch::runtime::Method::load(executorch_flatbuffer::ExecutionPlan*, executorch::runtime::Program const*, executorch::runtime::MemoryManager*, executorch::runtime::EventTracer*, executorch::runtime::NamedDataMap const*)+204) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #10 pc 000000000037d688  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (executorch::extension::module::Module::load_method(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, executorch::runtime::HierarchicalAllocator*, executorch::runtime::EventTracer*)+800) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #11 pc 000000000037dc50  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (executorch::extension::module::Module::execute(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, std::__ndk1::vector<executorch::runtime::EValue, std::__ndk1::allocator<executorch::runtime::EValue>> const&)+60) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #12 pc 00000000003614ec  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (executorch::extension::ExecuTorchJni::execute_method(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>)+760) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #13 pc 00000000003610cc  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (facebook::jni::detail::MethodWrapper<facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (executorch::extension::ExecuTorchJni::*)(facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>), &executorch::extension::ExecuTorchJni::execute(facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>), executorch::extension::ExecuTorchJni, facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator>, facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>>::dispatch(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&)+124) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #14 pc 0000000000360fc8  /data/app/~~bHtXCPRoV9yKUfl2_7Nc6Q==/org.pytorch.executorchexamples.dl3-KmtBo3dWjZ9N1UHTpwSh4Q==/lib/arm64/libexecutorch.so (facebook::jni::detail::FunctionWrapper<facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&), facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*, facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator>, facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>>::call(_JNIEnv*, _jobject*, _jstring*, facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*, facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&))+88) (BuildId: e18ee8aad1169087d3b6ca37edbddc2ea0eef78e)
2025-10-10 14:52:28.605  7694-7694  DEBUG                   crash_dump64                         A        #30 pc 0000000000001944  <anonymous:7c85550000> (org.pytorch.executorchexamples.dl3.MainActivity.run+0)

[GregoryComer]

Symbolicated with ndk-stack:

********** Crash dump: **********
Build fingerprint: 'google/sdk_gphone64_arm64/emu64a:16/BE2A.250530.026.D1/13818094:user/release-keys'
#00 0x000000000006a8d4 /apex/com.android.runtime/lib64/bionic/libc.so (__memset_aarch64+212) (BuildId: 2445fc73ede03b9536eef49355aff375)
#01 0x00000000003e4578 /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (create_fully_connected_nc+952) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    create_fully_connected_nc
                                                                                                                                                    /Users/gjcomer/src/executorch/backends/xnnpack/third-party/XNNPACK/src/operators/fully-connected-nc.c:210:25
#02 0x00000000003e7338 /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (xnn_create_fully_connected_nc_f32+468) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    create_fully_connected_nc_f32
                                                                                                                                                    /Users/gjcomer/src/executorch/backends/xnnpack/third-party/XNNPACK/src/operators/fully-connected-nc.c:1575:10
                                                                                                                                                    xnn_create_fully_connected_nc_f32
                                                                                                                                                    /Users/gjcomer/src/executorch/backends/xnnpack/third-party/XNNPACK/src/operators/fully-connected-nc.c:1651:10
#03 0x00000000003e30cc /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (create_fully_connected_operator+732) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    create_fully_connected_operator
                                                                                                                                                    /Users/gjcomer/src/executorch/backends/xnnpack/third-party/XNNPACK/src/subgraph/fully-connected.c:410:16
#04 0x00000000003a18f8 /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (create_runtime_impl+1100) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    create_runtime_impl
                                                                                                                                                    /Users/gjcomer/src/executorch/backends/xnnpack/third-party/XNNPACK/src/runtime.c:641:16
#05 0x000000000034b64c /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (executorch::backends::xnnpack::delegate::XNNCompiler::compileModel(void const*, unsigned long, executorch::backends::xnnpack::delegate::XNNExecutor*, executorch::backends::xnnpack::delegate::XNNWeightsCache*, xnn_workspace*, executorch::runtime::NamedDataMap const*)+1324) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    executorch::backends::xnnpack::delegate::XNNCompiler::compileModel(void const*, unsigned long, executorch::backends::xnnpack::delegate::XNNExecutor*, executorch::backends::xnnpack::delegate::XNNWeightsCache*, xnn_workspace*, executorch::runtime::NamedDataMap const*)
                                                                                                                                                    /Users/gjcomer/src/executorch/backends/xnnpack/runtime/XNNCompiler.cpp:1902:12
#06 0x000000000034d198 /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (executorch::backends::XnnpackBackend::init(executorch::runtime::BackendInitContext&, executorch::runtime::FreeableBuffer*, executorch::runtime::ArrayRef<executorch::runtime::CompileSpec>) const+176) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    executorch::backends::XnnpackBackend::init(executorch::runtime::BackendInitContext&, executorch::runtime::FreeableBuffer*, executorch::runtime::ArrayRef<executorch::runtime::CompileSpec>) const
                                                                                                                                                    /Users/gjcomer/src/executorch/backends/xnnpack/runtime/XNNPACKBackend.cpp:105:17
#07 0x00000000008d9840 /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (executorch::runtime::BackendDelegate::Init(executorch_flatbuffer::BackendDelegate const&, executorch::runtime::Program const*, executorch::runtime::BackendInitContext&, executorch::runtime::BackendDelegate*)+760) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    executorch::runtime::BackendDelegate::Init(executorch_flatbuffer::BackendDelegate const&, executorch::runtime::Program const*, executorch::runtime::BackendInitContext&, executorch::runtime::BackendDelegate*)
                                                                                                                                                    /Users/gjcomer/src/executorch/runtime/executor/method.cpp:105:47
#08 0x00000000008d8dbc /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (executorch::runtime::Method::init(executorch_flatbuffer::ExecutionPlan*, executorch::runtime::NamedDataMap const*)+676) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    executorch::runtime::Method::init(executorch_flatbuffer::ExecutionPlan*, executorch::runtime::NamedDataMap const*)
                                                                                                                                                    /Users/gjcomer/src/executorch/runtime/executor/method.cpp:881:19
#09 0x00000000008d89d4 /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (executorch::runtime::Method::load(executorch_flatbuffer::ExecutionPlan*, executorch::runtime::Program const*, executorch::runtime::MemoryManager*, executorch::runtime::EventTracer*, executorch::runtime::NamedDataMap const*)+204) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    executorch::runtime::Method::load(executorch_flatbuffer::ExecutionPlan*, executorch::runtime::Program const*, executorch::runtime::MemoryManager*, executorch::runtime::EventTracer*, executorch::runtime::NamedDataMap const*)
                                                                                                                                                    /Users/gjcomer/src/executorch/runtime/executor/method.cpp:794:22
#10 0x000000000037a548 /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (executorch::extension::module::Module::load_method(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, executorch::runtime::HierarchicalAllocator*, executorch::runtime::EventTracer*)+800) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    executorch::extension::module::Module::load_method(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, executorch::runtime::HierarchicalAllocator*, executorch::runtime::EventTracer*)
                                                                                                                                                    /Users/gjcomer/src/executorch/extension/module/module.cpp:235:28
#11 0x000000000037ab10 /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (executorch::extension::module::Module::execute(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, std::__ndk1::vector<executorch::runtime::EValue, std::__ndk1::allocator<executorch::runtime::EValue>> const&)+60) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    executorch::extension::module::Module::execute(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>> const&, std::__ndk1::vector<executorch::runtime::EValue, std::__ndk1::allocator<executorch::runtime::EValue>> const&)
                                                                                                                                                    /Users/gjcomer/src/executorch/extension/module/module.cpp:260:3
#12 0x000000000035e3ac /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (executorch::extension::ExecuTorchJni::execute_method(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>)+760) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    executorch::extension::ExecuTorchJni::execute_method(std::__ndk1::basic_string<char, std::__ndk1::char_traits<char>, std::__ndk1::allocator<char>>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>)
                                                                                                                                                    /Users/gjcomer/src/executorch/extension/android/jni/jni_layer.cpp:369:28
#13 0x000000000035df8c /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (facebook::jni::detail::MethodWrapper<facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (executorch::extension::ExecuTorchJni::*)(facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>), &executorch::extension::ExecuTorchJni::execute(facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>), executorch::extension::ExecuTorchJni, facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator>, facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>>::dispatch(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&)+124) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    executorch::extension::ExecuTorchJni::execute(facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>)
                                                                                                                                                    /Users/gjcomer/src/executorch/extension/android/jni/jni_layer.cpp:289:12
                                                                                                                                                    facebook::jni::detail::MethodWrapper<facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (executorch::extension::ExecuTorchJni::*)(facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>), &executorch::extension::ExecuTorchJni::execute(facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>), executorch::extension::ExecuTorchJni, facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator>, facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>>::dispatch(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&)
                                                                                                                                                    /Users/gjcomer/src/executorch/cmake-out-android-arm64-v8a/extension/android/third-party/fbjni/prefab/modules/fbjni/include/fbjni/detail/Registration-inl.h:129:14
#14 0x000000000035de88 /data/app/~~Iu3N1S73Il_roBby0bAOow==/org.pytorch.executorchexamples.dl3-iUhtAyiqgt63-h9k3V0wUA==/lib/arm64/libexecutorch.so (facebook::jni::detail::FunctionWrapper<facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&), facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*, facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator>, facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>>::call(_JNIEnv*, _jobject*, _jstring*, facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*, facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&))+88) (BuildId: 9ba0b8eda25eea5d9f33ed803ffb82a690dd7704)
                                                                                                                                                    facebook::jni::detail::CallWithJniConversions<facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&), facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator>, facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*, facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>>::call(facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*, _jstring*, facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*, facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&))
                                                                                                                                                    /Users/gjcomer/src/executorch/cmake-out-android-arm64-v8a/extension/android/third-party/fbjni/prefab/modules/fbjni/include/fbjni/detail/Registration-inl.h:58:35
                                                                                                                                                    facebook::jni::detail::FunctionWrapper<facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&), facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*, facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator>, facebook::jni::alias_ref<_jstring*>, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>>::call(_JNIEnv*, _jobject*, _jstring*, facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*, facebook::jni::basic_strong_ref<facebook::jni::JArrayClass<executorch::extension::JEValue>, facebook::jni::LocalReferenceAllocator> (*)(facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::HybridClass<executorch::extension::ExecuTorchJni, facebook::jni::detail::BaseHybridClass>::JavaPart, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::alias_ref<_jstring*>&&, facebook::jni::alias_ref<facebook::jni::detail::JTypeFor<facebook::jni::JArrayClass<facebook::jni::detail::JTypeFor<executorch::extension::JEValue, facebook::jni::JObject, void>::_javaobject*>, facebook::jni::detail::JTypeArray, void>::_javaobject*>&&))
                                                                                                                                                    /Users/gjcomer/src/executorch/cmake-out-android-arm64-v8a/extension/android/third-party/fbjni/prefab/modules/fbjni/include/fbjni/detail/Registration-inl.h:95:14
#15 0x000000000033f500 /apex/com.android.art/lib64/libart.so (art_quick_generic_jni_trampoline+144) (BuildId: b229f9d1b6196afaae086f29f029f907)
#16 0x0000000000328194 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612) (BuildId: b229f9d1b6196afaae086f29f029f907)
#17 0x00000000006793d4 /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, bool, art::JValue*)+1580) (BuildId: b229f9d1b6196afaae086f29f029f907)
#18 0x00000000005c84f4 /apex/com.android.art/lib64/libart.so (void art::interpreter::ExecuteSwitchImplCpp<false>(art::interpreter::SwitchImplContext*)+868) (BuildId: b229f9d1b6196afaae086f29f029f907)
#19 0x0000000000317a88 /apex/com.android.art/lib64/libart.so (ExecuteSwitchImplAsm+8) (BuildId: b229f9d1b6196afaae086f29f029f907)
#20 0x000000000037549c <anonymous:791e6ca000> (org.pytorch.executorch.Module.execute+0)
#21 0x0000000000317488 /apex/com.android.art/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+412) (BuildId: b229f9d1b6196afaae086f29f029f907)
#22 0x00000000006795bc /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, bool, art::JValue*)+2068) (BuildId: b229f9d1b6196afaae086f29f029f907)
#23 0x00000000005c84f4 /apex/com.android.art/lib64/libart.so (void art::interpreter::ExecuteSwitchImplCpp<false>(art::interpreter::SwitchImplContext*)+868) (BuildId: b229f9d1b6196afaae086f29f029f907)
#24 0x0000000000317a88 /apex/com.android.art/lib64/libart.so (ExecuteSwitchImplAsm+8) (BuildId: b229f9d1b6196afaae086f29f029f907)
#25 0x000000000037551c <anonymous:791e6ca000> (org.pytorch.executorch.Module.forward+0)
#26 0x0000000000317488 /apex/com.android.art/lib64/libart.so (art::interpreter::ArtInterpreterToInterpreterBridge(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame*, art::JValue*)+412) (BuildId: b229f9d1b6196afaae086f29f029f907)
#27 0x00000000006795bc /apex/com.android.art/lib64/libart.so (bool art::interpreter::DoCall<false>(art::ArtMethod*, art::Thread*, art::ShadowFrame&, art::Instruction const*, unsigned short, bool, art::JValue*)+2068) (BuildId: b229f9d1b6196afaae086f29f029f907)
#28 0x00000000005c84f4 /apex/com.android.art/lib64/libart.so (void art::interpreter::ExecuteSwitchImplCpp<false>(art::interpreter::SwitchImplContext*)+868) (BuildId: b229f9d1b6196afaae086f29f029f907)
#29 0x0000000000317a88 /apex/com.android.art/lib64/libart.so (ExecuteSwitchImplAsm+8) (BuildId: b229f9d1b6196afaae086f29f029f907)
#30 0x0000000000001944 <anonymous:7c85550000> (org.pytorch.executorchexamples.dl3.MainActivity.run+0)
#31 0x00000000002d763c /apex/com.android.art/lib64/libart.so (art::interpreter::Execute(art::Thread*, art::CodeItemDataAccessor const&, art::ShadowFrame&, art::JValue, bool, bool) (.__uniq.112435418011751916792819755956732575238.llvm.11186287395527938019)+332) (BuildId: b229f9d1b6196afaae086f29f029f907)
#32 0x00000000002d6e70 /apex/com.android.art/lib64/libart.so (artQuickToInterpreterBridge+888) (BuildId: b229f9d1b6196afaae086f29f029f907)
#33 0x000000000033f638 /apex/com.android.art/lib64/libart.so (art_quick_to_interpreter_bridge+88) (BuildId: b229f9d1b6196afaae086f29f029f907)
#34 0x00000000000046a8 [anon_shmem:dalvik-jit-code-cache] (offset 0x2000000) (java.lang.Thread.run+136)
#35 0x0000000000328194 /apex/com.android.art/lib64/libart.so (art_quick_invoke_stub+612) (BuildId: b229f9d1b6196afaae086f29f029f907)
#36 0x00000000002d9348 /apex/com.android.art/lib64/libart.so (art::ArtMethod::Invoke(art::Thread*, unsigned int*, unsigned int, art::JValue*, char const*)+216) (BuildId: b229f9d1b6196afaae086f29f029f907)
#37 0x0000000000421028 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallback(void*)+932) (BuildId: b229f9d1b6196afaae086f29f029f907)
#38 0x0000000000420c74 /apex/com.android.art/lib64/libart.so (art::Thread::CreateCallbackWithUffdGc(void*)+8) (BuildId: b229f9d1b6196afaae086f29f029f907)
#39 0x0000000000080df8 /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+232) (BuildId: 2445fc73ede03b9536eef49355aff375)
#40 0x0000000000073dd8 /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+64) (BuildId: 2445fc73ede03b9536eef49355aff375)
Crash dump is completed

[GregoryComer]

If the stack trace can be trusted, this points to the weight cache: https://github.com/google/XNNPACK/blob/3131afead790c5c69a9aa12273dfc40399789ad7/src/operators/fully-connected-nc.c#L210. Going to try to reproduce on M1 with executor_runner.

Edit: No repro running with executor_runner (with ASAN). Going to continue debugging the Android repro.

[GregoryComer]

Crash seems to not repro with XNNPACK weight cache off. The repro is finicky, so this isn't definitive.

[GregoryComer]

I think I got it. The alignment calculation in XNNWeightsCache is buggy.

When the start address sits in the upper half of the address space (the cast to intptr_t is signed), it causes the alignment start to be bumped further forward than it should be, making the aligned memory region go past the end of the allocation.

Note that the aligned address is 0x70 ahead of the raw allocation (named maybe_aligned_space in code). The alignment is 0x40, so it should only be bumped by 0x10 in this case.

125414 10-11 02:44:29.897 28132 28171 I XNNPACK : reserving space in weights cache (xnn_get_pointer_to_write_weights, /Users/gjcomer/src/executorch/backends/xnnpack/third-party/XNNPACK/src/operator-utils.c:32)
125415 10-11 02:44:29.897 28132 28171 D ExecuTorch: kPackedAllocationAlignment: 64
125416 10-11 02:44:29.897 28132 28171 D ExecuTorch: Maybe aligned space: 0xb40000763d4bfa90
125417 10-11 02:44:29.897 28132 28171 D ExecuTorch: Aligned space: 0xb40000763d4bfb00
125418 10-11 02:44:29.897 28132 28171 D ExecuTorch: Pointer in data: 0xb40000763d4bfa90
125419 10-11 02:44:29.897 28132 28171 I XNNPACK : ptr: 0xb40000763d4bfb00, aligned_weights_size: 83232 (xnn_get_pointer_to_write_weights, /Users/gjcomer/src/executorch/backends/xnnpack/third-party/XNNPACK/src/operator-utils.c:38)
125420 10-11 02:44:29.897 28132 28171 F libc    : Fatal signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0xb40000763d4d4000 in tid 28171 (Thread-2), pid 28132 (rchexamples.dl3)

executorch/backends/xnnpack/runtime/XNNWeightsCache.cpp

Lines 166 to 168 in cc6cb83

	void* maybe_aligned_space = data_container.data();
	void* aligned_space = (void*)((intptr_t)maybe_aligned_space + 64 -
	(intptr_t)maybe_aligned_space % 64);

Edit: I confirmed that it does not crash after updating to use uintptr_t. I'll probably update to use std::align rather than implementing manually. I'll put out a patch tomorrow.