The Operation will be canceled. The next steps may not contain expected logs.

[tarekgh]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=696281

Build leg reported

Build / browser-wasm linux Release LibraryTests / Send to Helix

Pull Request

dotnet/runtime#103009

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The Operation will be canceled. The next steps may not contain expected logs.",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=696281
Error message validated: [The Operation will be canceled. The next steps may not contain expected logs.]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/4/2024 8:08:48 PM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
2715747	dotnet-dotnet	Run Tests (azurelinuxArm64CrossContainer)	Log
2715832	dotnet-dotnet-docker	🛡 Guardian: AntiMalware Scanner (Source)	Log
2715817	dotnet-dotnet-docker	🟪 Test Images	Log
2715803	dotnet-dotnet-docker	🛡 Guardian: Publish SDL Logs to Artifacts	Log
1049857	dotnet/roslyn	Run Integration Tests	Log
2715771	dotnet-dotnet-docker	🛡 Guardian: Publish SDL Logs to Artifacts	Log
2715590	dotnet-runtime	Send TypeNameFuzzer to OneFuzz	Log
1049688	dotnet/arcade	XHarness Android Helix Testing (Linux)	Log	dotnet/arcade#15858
1049671	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115766
1049659	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115940
1049619	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115939
1049470	dotnet/runtime	Send to Helix	Log
1049582	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115885
2715612	dotnet-runtime	Run performance job script	Log
1049514	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115935
1049496	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115934
1049471	dotnet/runtime	Send to Helix	Log
1049477	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115335
1049449	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115933
1049443	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115932
1049419	dotnet/runtime	AOT compile CoreCLR tests	Log
1049420	dotnet/runtime	Send to Helix	Log
1049367	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115856
1049355	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115909
1049349	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115925
1049339	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114126
1049336	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115921
1049309	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115858
1049304	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114172
1049234	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115919
1049227	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115826
1049209	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115852
1049193	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115785
2715331	dotnet-dotnet	Build (centOSStreamContainer)	Log
1049161	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115899
1049115	dotnet/runtime	Send tests to Helix (Unix)	Log
2715327	dotnet-runtime	Send job to Helix (Unix)	Log
1049143	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115916
1049137	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115277
1049152	dotnet/performance	Run performance job script	Log
1049100	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115914
2715286	dotnet-dotnet	Build (centOSStreamContainer)	Log
1049096	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115850
1049064	dotnet/runtime	Send to Helix	Log	dotnet/runtime#113956
1049040	dotnet/roslyn	Run Integration Tests	Log
1048990	dotnet/runtime	Send to Helix	Log
1049021	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114720
1049010	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115251
1049006	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115388
1048961	dotnet/runtime	Send to Helix	Log
1048962	dotnet/runtime	AOT compile CoreCLR tests	Log
1048958	dotnet/runtime	Send to Helix	Log
1048937	dotnet/runtime	AOT compile CoreCLR tests	Log
1048939	dotnet/runtime	Send to Helix	Log
2715175	dotnet-aspnetcore	Run build.cmd	Log
2715137	dotnet-runtime	Send job to Helix (Unix)	Log
1048885	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115798
2715038	dotnet-dotnet	🛡 1ES PT: AntiMalware MDE Linux Scanner (Binary) (1ES PT)	Log
2715136	dotnet-performance	Run performance job script	Log
1048834	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115907
2715007	dotnet-dotnet	Build	Log
1048804	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115393
2715087	dotnet-dotnet	Checkout dotnet-dotnet@restore-with-http-cache to s	Log
1048775	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115902
1048770	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115901
1048766	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115900
1048759	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115899
1048782	dotnet/xharness	Run tests in Helix	Log	dotnet/xharness#1416
2714964	dotnet-dotnet	🛡 1ES PT: AntiMalware MDE Linux Scanner (Binary) (1ES PT)	Log
1048074	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115639
1048696	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115566
1048682	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115892
2714957	dotnet-dotnet	Agent Pool Validation	Log
2714808	dotnet-dotnet	🛡 Guardian: TSAUpload	Log
1048632	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115891
2714932	dotnet-runtime	Send job to Helix (Unix)	Log
1048618	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115775
1048645	dotnet/xharness	Run tests in Helix	Log	dotnet/xharness#1416
1048603	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115885
1048594	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115826
1048569	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115888
2714920	dotnet-performance	Run performance job script	Log
1048535	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115823
1048508	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115818
1048441	dotnet/runtime	Send to Helix	Log
1048453	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115883
1048417	dotnet/runtime	Send to Helix	Log	dotnet/runtime#114720
1048395	dotnet/runtime	AOT compile CoreCLR tests	Log
1048394	dotnet/runtime	Send to Helix	Log
2714788	dotnet-runtime	Run performance job script	Log
1048374	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115854
2714683	dotnet-dotnet	Build	Log
2714591	dotnet-dotnet	Build	Log
1048332	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115858
1048314	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115877
1048306	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115857
1048230	dotnet/runtime	Send to Helix	Log	dotnet/runtime#115856
2714574	dotnet-sdk	🛡 Guardian: TSAUpload	Log
2714641	dotnet-runtime	Run performance job script	Log
1048151	dotnet/roslyn	Run Integration Tests	Log	dotnet/roslyn#78578
Displaying 100 of 1051 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
47	388	1051

[dkurepa]

These appear to be timeout cancellations, and most of them are happening on the same runtime leg that uses OSX.1200.ARM64.Open, looking at grafana, they do indeed appear to be quite high. Will check with the team on what we can do about this

[dkurepa]

For pipelines experiencing timeouts on osx.1200.arm64.open we recommend using osx.14.arm64.open, as this queue has a lot more machines

[dkurepa]

There is also a number of Publish Using Darc timeouts, these are in 99% of cases AzDo Api issues. During the Publish Using Darc job, we queue up a Maestro Promotion pipeline, that downloads the artifacts from the build calling it, and publishes them where they need to go. While publishing is happening, the build pipeline keeps busy waiting for the promotion pipeline to be done, checking the status of it every 60 seconds. In cases where this times out, the build most likely did get promoted, the AzDo API just kept telling us the wrong thing

[ivanpovazan]

@dkurepa for context, the same timeouts are also caught with this tracking #1883
where I asked about potentially bumping the queues to osx.13.amd/arm64, would that suffice?
(bumping to osx.14 is not an option right now as we still want testing on osx.13 which is the minimum supported OS version)