Run test_models.sh with strict=False flag

[mergennachin]

This is the recommended anyway, so let's gradually start migrating to strict=False

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/12368

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 10 New Failures, 2 Unrelated Failures

As of commit c16b645 with merge base a1e3d48 ():

NEW FAILURES - The following jobs have failed:

• pull / test-llama-runner-linux (fp32, xnnpack+custom+quantize_kv, linux.2xlarge, executorch-ubuntu-22.04... / linux-job (gh)
  RuntimeError: Command docker exec -t f0690dc18a065741b96ed07983b0226cf10fafe8fc21576096b6c846c2e5d8a6 /exec failed with exit code 1
• pull / test-models-linux (ic3, xnnpack-quantization-delegation, linux.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 9d0537caf3cc4f5d4014fc030caf0decca684c32504e113084a5b4c18cf3a765 /exec failed with exit code 134
• pull / test-models-linux (ic4, xnnpack-quantization-delegation, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 500f0d5251de44fcdddb011009a0a43facdfeaff9234cd0996a40da4df017925 /exec failed with exit code 134
• pull / test-models-linux (llama3_2_vision_encoder, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t f649801d72f78b08d1c47773ac30b8a7c45a0c61611a07485273e0510dad3664 /exec failed with exit code 1
• trunk / test-arm-cortex-m-size-test (bare_metal) / linux-job (gh)
  RuntimeError: Command docker exec -t 0a57d2970f46d5d4af1d3675c7bf70186973fa268eb11d9e99e4db280e5f07c9 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (ic3, xnnpack-quantization-delegation, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 2c66cea1d40d4e50c2972ed7b8d36d314dddf0c6e35a3017d32f444af39f781d /exec failed with exit code 134
• trunk / test-models-linux-aarch64 (ic4, xnnpack-quantization-delegation, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 31309720598a2bf8316324387a744ebb5c9605ef709676bc2c5b3f4ca53d536c /exec failed with exit code 134
• trunk / test-models-linux-aarch64 (llama3_2_vision_encoder, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t 65adc97abb20c496bd5e11cc9c10732986855c459624542b4b00c9c0187d32d9 /exec failed with exit code 1
• trunk / test-models-macos (ic4, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 134
• trunk / test-models-macos (llama3_2_vision_encoder, portable) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  [ FAILED ] TensorPtrMakerTest.FailedCreateTensorUsingFromBlobWithIllegalStrides
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  [ FAILED ] TensorPtrMakerTest.FailedCreateTensorUsingFromBlobWithIllegalStrides

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[digantdesai]

I remember (not sure how accurately) but there were some PTE size bloats reported with strict==false and I am not sure we have a CI to catch that. I think we should validate that and then start embracing this.

cc @GregoryComer - who I think reported this a while back for a quant model.

[mcr229]

@digantdesai yea we should probably check that model size isn't bloated. We don't have any CI here that checks for this so right now we would have to do so manually.

[digantdesai]

@digantdesai yea we should probably check that model size isn't bloated. We don't have any CI here that checks for this so right now we would have to do so manually.

Can we add a CI to make sure it doesn't double or something, just a guard against regressing.