Checkout failure: "Git fetch failed with exit code 128"

[carlossanlop]

Build

https://dev.azure.com/dnceng/public/_build/results?buildId=1706886&view=results

Build leg reported

Libraries Test Run release coreclr Linux_musl x64 Release

Pull Request

Rolling build for runtime

Action required for the engineering services team

To triage this issue (First Responder / @dotnet/dnceng):

• Open the failing build above and investigate
• Add a comment explaining your findings

If this is an issue that is causing build breaks across multiple builds and would get benefit from being listed on the build analysis check, follow the next steps:

1. Add the label "Known Build Error"
2. Edit this issue and add an error string in the Json below that can help us match this issue with future build breaks. You should use the known issues documentation

{
   "ErrorMessage" : "Git fetch failed with exit code: 128",
   "BuildRetry": true
}

Additional information about the issue reported

The Checkout dotnet/runtime@main to s step is the one that failed. Here is the end of the log output:

remote: Compressing objects:  29% (30070/103687)        
remote: Compressing objects:  30% (31107/103687)        
remote: Compressing objects:  31% (32143/103687)        
remote: Compressing objects:  31% (32322/103687)        
remote: Compressing objects:  32% (33180/103687)        
remote: Compressing objects:  33% (34217/103687)        
remote: Compressing objects:  33% (35247/103687)        
remote: Compressing objects:  34% (35254/103687)        
remote: Compressing objects:  35% (36291/103687)        
remote: Compressing objects:  36% (37328/103687)        
remote: Compressing objects:  37% (38365/103687)        
remote: Compressing objects:  37% (38926/103687)        
remote: Compressing objects:  38% (39402/103687)        
error: RPC failed; curl 18 transfer closed with outstanding read data remaining
fatal: protocol error: bad pack header
##[error]Git fetch failed with exit code: 128
Finishing: Checkout dotnet/runtime@main to s

Report

Build	Repository	Step Name	Console log	Pull Request
3008707	dotnet-dotnet-docker	Checkout dotnet-dotnet-docker@nightly to s	Log
1482440	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129766/merge to s	Log	dotnet/runtime#129766
3008217	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/62411/merge to s	Log	#62411
3008215	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/62410/merge to s	Log	#62410
3007886	dotnet-dotnet-buildtools-prereqs-docker	Checkout dotnet-dotnet-buildtools-prereqs-docker@dkurepa/ContinueOnBuildFailures to s	Log
1480597	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129814/merge to s	Log	dotnet/runtime#129814
1480365	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129826/merge to s	Log	dotnet/runtime#129826
1479843	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129810/merge to s	Log	dotnet/runtime#129810
1479916	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54964/merge to s	Log	dotnet/sdk#54964
1479919	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67399/merge to s	Log	dotnet/aspnetcore#67399
1479513	dotnet/sdk	Checkout dotnet/sdk@refs/pull/51088/merge to s	Log	dotnet/sdk#51088
1479072	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129334/merge to s	Log	dotnet/runtime#129334
1478796	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54871/merge to s	Log	dotnet/sdk#54871
1478617	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54945/merge to s	Log	dotnet/sdk#54945
1478598	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54945/merge to s	Log	dotnet/sdk#54945
1478078	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54949/merge to s	Log	dotnet/sdk#54949
1477909	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67156/merge to s	Log	dotnet/aspnetcore#67156
1477916	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67157/merge to s	Log	dotnet/aspnetcore#67157
1477913	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67157/merge to s	Log	dotnet/aspnetcore#67157
1477915	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67157/merge to s	Log	dotnet/aspnetcore#67157
1476316	dotnet/runtime	Checkout dotnet/runtime@refs/pull/126367/merge to s	Log	dotnet/runtime#126367
1475723	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129652/merge to s	Log	dotnet/runtime#129652
1474899	dotnet/maui	Checkout dotnet/maui@darc-net11.0-167dae9d-c756-48bc-a769-baa4b1f0abe2 to s	Log
1474394	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67341/merge to s	Log	dotnet/aspnetcore#67341
1473365	dotnet/sdk	Checkout dotnet/sdk@main to s	Log
3004249	dotnet-installer	Clone dotnet/installer	Log	#62258
3004194	dotnet-maui	Checkout dotnet-maui@release/10.0.1xx-sr8 to s	Log
1472925	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54892/merge to s	Log	dotnet/sdk#54892
1472559	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54890/merge to s	Log	dotnet/sdk#54890
1471366	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129563/merge to s	Log	dotnet/runtime#129563
1471288	dotnet/maui	Checkout dotnet/maui@refs/pull/35878/merge to s	Log	dotnet/maui#35878
1471235	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129587/merge to s	Log	dotnet/runtime#129587
1470679	dotnet/maui	Checkout dotnet/maui@refs/pull/35945/merge to s	Log	dotnet/maui#35945
1470376	dotnet/runtime	Checkout dotnet/runtime@main to s	Log
1469907	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129551/merge to s	Log	dotnet/runtime#129551
1469798	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67288/merge to s	Log	dotnet/aspnetcore#67288
1469371	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67279/merge to s	Log	dotnet/aspnetcore#67279
1469373	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67279/merge to s	Log	dotnet/aspnetcore#67279
1469372	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67279/merge to s	Log	dotnet/aspnetcore#67279
1469149	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54623/merge to s	Log	dotnet/sdk#54623
3002587	dotnet-sdk	Checkout dotnet-sdk@refs/pull/62190/merge to s	Log	#62190
1469207	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54723/merge to s	Log	dotnet/sdk#54723
1468036	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54761/merge to s	Log	dotnet/sdk#54761
1467884	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54729/merge to s	Log	dotnet/sdk#54729
1467015	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129260/merge to s	Log	dotnet/runtime#129260
1466217	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129454/merge to s	Log	dotnet/runtime#129454
1466332	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2612/merge to s	Log	maestro-auth-test/build-result-analysis-test#2612
1466331	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2612/merge to s	Log	maestro-auth-test/build-result-analysis-test#2612
1466164	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1610/merge to s	Log	dotnet/xharness#1610
3001110	dotnet-runtime	Checkout dotnet-runtime@main to s	Log
3000946	dotnet-sdk	Checkout dotnet-sdk@refs/pull/62118/merge to s	Log	#62118
1464983	dotnet/xharness	Checkout dotnet/xharness@refs/pull/1609/merge to s	Log	dotnet/xharness#1609
1464606	dotnet/maui	Checkout dotnet/maui@refs/pull/35898/merge to s	Log	dotnet/maui#35898
1464506	dotnet/arcade	Checkout dotnet/arcade@refs/pull/17009/merge to s	Log	#17009
1463138	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54729/merge to s	Log	dotnet/sdk#54729
1461862	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2607/merge to s	Log	maestro-auth-test/build-result-analysis-test#2607
1461861	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2607/merge to s	Log	maestro-auth-test/build-result-analysis-test#2607
1461747	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129332/merge to s	Log	dotnet/runtime#129332
1461788	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2605/merge to s	Log	maestro-auth-test/build-result-analysis-test#2605
1461787	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2605/merge to s	Log	maestro-auth-test/build-result-analysis-test#2605
1461677	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2602/merge to s	Log	maestro-auth-test/build-result-analysis-test#2602
1461676	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2602/merge to s	Log	maestro-auth-test/build-result-analysis-test#2602
1461593	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2601/merge to s	Log	maestro-auth-test/build-result-analysis-test#2601
1461592	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2601/merge to s	Log	maestro-auth-test/build-result-analysis-test#2601
1461581	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2600/merge to s	Log	maestro-auth-test/build-result-analysis-test#2600
1461580	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2600/merge to s	Log	maestro-auth-test/build-result-analysis-test#2600
1461539	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2599/merge to s	Log	maestro-auth-test/build-result-analysis-test#2599
1461538	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2599/merge to s	Log	maestro-auth-test/build-result-analysis-test#2599
1461498	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2598/merge to s	Log	maestro-auth-test/build-result-analysis-test#2598
1461497	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2598/merge to s	Log	maestro-auth-test/build-result-analysis-test#2598
1461404	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2597/merge to s	Log	maestro-auth-test/build-result-analysis-test#2597
1461405	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2597/merge to s	Log	maestro-auth-test/build-result-analysis-test#2597
1461398	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2596/merge to s	Log	maestro-auth-test/build-result-analysis-test#2596
1461397	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2596/merge to s	Log	maestro-auth-test/build-result-analysis-test#2596
1461377	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2595/merge to s	Log	maestro-auth-test/build-result-analysis-test#2595
1461378	maestro-auth-test/build-result-analysis-test	Checkout maestro-auth-test/build-result-analysis-test@refs/pull/2595/merge to s	Log	maestro-auth-test/build-result-analysis-test#2595
1461213	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54703/merge to s	Log	dotnet/sdk#54703
1461046	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129305/merge to s	Log	dotnet/runtime#129305
2998602	dotnet-maui	Checkout dotnet-maui@inflight/current to s	Log
2998123	dotnet-dotnet	Checkout dotnet-dotnet@refs/pull/62055/merge to s	Log	#62055
1459890	dotnet/aspnetcore	Checkout dotnet/aspnetcore@release/8.0 to s	Log
1459761	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54712/merge to s	Log	dotnet/sdk#54712
2997353	dotnet-sdk	Checkout dotnet-sdk@main to s	Log
1458305	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54685/merge to s	Log	dotnet/sdk#54685
1457979	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129238/merge to s	Log	dotnet/runtime#129238
2997013	dotnet-installer	Clone dotnet/installer	Log	#62013
1455559	dotnet/maui	Checkout dotnet/maui@refs/pull/34600/merge to s	Log	dotnet/maui#34600
1455486	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54647/merge to s	Log	dotnet/sdk#54647
1455413	dotnet/runtime	Checkout dotnet/runtime@refs/pull/128768/merge to s	Log	dotnet/runtime#128768
1455160	dotnet/arcade-services	Checkout dotnet/arcade-services@refs/pull/6370/merge to s	Log	dotnet/arcade-services#6370
1454373	dotnet/arcade	Checkout dotnet/arcade@refs/pull/16990/merge to s	Log	#16990
1454302	dotnet/maui	Checkout dotnet/maui@net11.0 to s	Log
1454285	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54122/merge to s	Log	dotnet/sdk#54122
1453193	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54630/merge to s	Log	dotnet/sdk#54630
1452646	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129068/merge to s	Log	dotnet/runtime#129068
1451912	dotnet/runtime	Checkout dotnet/runtime@refs/pull/129051/merge to s	Log	dotnet/runtime#129051
1451780	dotnet/maui	Checkout dotnet/maui@refs/pull/34758/merge to s	Log	dotnet/maui#34758
2993279	dotnet-maui	Checkout dotnet-maui@release/10.0.1xx-sr7 to s	Log
1451203	dotnet/aspnetcore	Checkout dotnet/aspnetcore@refs/pull/67034/merge to s	Log	dotnet/aspnetcore#67034
1450459	dotnet/sdk	Checkout dotnet/sdk@refs/pull/54600/merge to s	Log	dotnet/sdk#54600
Displaying 100 of 139 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
4	29	139

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=323960
Error message validated: Git fetch failed with exit code: 128
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/28/2023 9:15:14 PM UTC

[ChadNedzlek]

So this is... interesting. I agree this seems like essentially a random network failure that is not the fault of the code. Also... it doesn't seem like there is anything to be done other than retry the build.

@ulisesh and @AlitzelMendez. This seems like another instance of a "known issue" that would really benefit from the ability to get retried automatically. I think Carlos did the right thing by opening the known issue, but his PR (and any future PR that hits this) is still going to be blocked, since the build doesn't run at all in this case, because the checkout failed. And it would be great to help folks that hit this.

[ulisesh]

I created the issue to track how often it was happening and measure the impact of the problem. It might be hard to get traction on networking problems but having the number of impacted builds has great value in my opinion.

In the last 24 hours, we had 37 builds failed with the same error. cc @markwilkie who might find this data interesting.

I think automatic retries for known issues will be the next feature after we are done with analyzing builds retroactively, we can discuss this if people think we should change priorities.

[markwilkie]

So 37 builds were caught with the known issues feature @ulisesh ? Could I get the query please? (super exciting)

[ulisesh]

I saw 37 hits in the office but now I only see 33 🤔. There are 33 failed jobs in 2 different builds. Sorry if I misused the terms but I still think it is great that we can track these build breaks 😄

This is the query:

KnownIssues
| where IssueId == 9009 and IssueRepository == "dotnet/arcade"

[ViktorHofer]

This happened 36 times in the last 24 hours. Should we add a retry (assuming that's possible)?