Reduce artifactory bandwidth used by ci.jenkins.io

[MarkEWaite]

Service(s)

ci.jenkins.io

Summary

Analysis of the bandwidth use on repo.jenkins-ci.org by @basil has shown that ci.jenkins.io is repeatedly downloading war files for some recent Jenkins core releases and for Jenkins plugins. The duplicate downloads include

IP address	Artifact	Downloaded (GB)
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.479.1/jenkins-war-2.479.1.war	292.538732947782
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.518/jenkins-war-2.518.war	168.361135382205
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.479.3/jenkins-war-2.479.3.war	127.754617938772
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.517/jenkins-war-2.517.war	80.2548949094489
3.139.132.49	org/jenkins-ci/main/jenkins-test-harness-tools/2.2/jenkins-test-harness-tools-2.2.jar	57.4026786452159
3.139.132.49	org/jenkins-ci/plugins/bouncycastle-api/2.30.1.80-261.v00c0e2618ec3/bouncycastle-api-2.30.1.80-261.v00c0e2618ec3.hpi	44.1197699354962
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.504.3/jenkins-war-2.504.3.war	41.6389248613268
3.139.132.49	org/jenkins-ci/main/jenkins-core/2.479.1/jenkins-core-2.479.1.jar	36.1472865268588
3.139.132.49	org/jenkins-ci/plugins/durable-task/587.v84b_877235b_45/durable-task-587.v84b_877235b_45.hpi	33.4868805855513
3.139.132.49	io/jenkins/test/fips/fips-bundle-test/27.va_f689db_8c5ea_/fips-bundle-test-27.va_f689db_8c5ea_.jar	31.0293228849769
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.516.1-rc36996.5d38c1a_d0c66/jenkins-war-2.516.1-rc36996.5d38c1a_d0c66.war	25.9519814522937
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.492.3/jenkins-war-2.492.3.war	25.1239216122776
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.504.1/jenkins-war-2.504.1.war	24.5013827104121
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.518-rc37018.68171522d59f/jenkins-war-2.518-rc37018.68171522d59f.war	23.1852552713826
3.139.132.49	org/jenkins-ci/main/cli/2.518/cli-2.518.jar	21.4586931876838
3.139.132.49	org/jenkins-ci/main/jenkins-core/2.518/jenkins-core-2.518.jar	20.3767495974898
3.139.132.49	org/eclipse/jgit/org.eclipse.jgit/6.8.0.202311291450-r/org.eclipse.jgit-6.8.0.202311291450-r.jar	17.8714849250391
3.139.132.49	org/jenkins-ci/plugins/bouncycastle-api/2.30.1.80-256.vf98926042a_9b_/bouncycastle-api-2.30.1.80-256.vf98926042a_9b_.hpi	17.6525939153507
3.139.132.49	org/jenkins-ci/main/jenkins-core/2.479.3/jenkins-core-2.479.3.jar	15.6551615986973
3.139.132.49	org/jenkins-ci/plugins/durable-task/581.v299a_5609d767/durable-task-581.v299a_5609d767.hpi	14.1808188911527
3.139.132.49	org/jenkins-ci/main/jenkins-test-harness-htmlunit/208.v18783fd22fca_/jenkins-test-harness-htmlunit-208.v18783fd22fca_.jar	13.8294054418802
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.452.4/jenkins-war-2.452.4.war	13.5711509920657
3.139.132.49	org/eclipse/jgit/org.eclipse.jgit/7.3.0.202506031305-r/org.eclipse.jgit-7.3.0.202506031305-r.jar	13.3827022798359
3.139.132.49	io/jenkins/plugins/lib-durable-task/48.v72b_86b_9ca_8e1/lib-durable-task-48.v72b_86b_9ca_8e1.jar	13.1410510651767
3.139.132.49	org/jenkins-ci/plugins/jackson2-api/2.18.3-402.v74c4eb_f122b_2/jackson2-api-2.18.3-402.v74c4eb_f122b_2.hpi	13.00744024012243.139.132.49
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.518/jenkins-war-2.518.war	168.361135382205
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.479.3/jenkins-war-2.479.3.war	127.754617938772
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.517/jenkins-war-2.517.war	80.2548949094489
3.139.132.49	org/jenkins-ci/main/jenkins-test-harness-tools/2.2/jenkins-test-harness-tools-2.2.jar	57.4026786452159
3.139.132.49	org/jenkins-ci/plugins/bouncycastle-api/2.30.1.80-261.v00c0e2618ec3/bouncycastle-api-2.30.1.80-261.v00c0e2618ec3.hpi	44.1197699354962
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.504.3/jenkins-war-2.504.3.war	41.6389248613268
3.139.132.49	org/jenkins-ci/main/jenkins-core/2.479.1/jenkins-core-2.479.1.jar	36.1472865268588
3.139.132.49	org/jenkins-ci/plugins/durable-task/587.v84b_877235b_45/durable-task-587.v84b_877235b_45.hpi	33.4868805855513
3.139.132.49	io/jenkins/test/fips/fips-bundle-test/27.va_f689db_8c5ea_/fips-bundle-test-27.va_f689db_8c5ea_.jar	31.0293228849769
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.516.1-rc36996.5d38c1a_d0c66/jenkins-war-2.516.1-rc36996.5d38c1a_d0c66.war	25.9519814522937
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.492.3/jenkins-war-2.492.3.war	25.1239216122776
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.504.1/jenkins-war-2.504.1.war	24.5013827104121
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.518-rc37018.68171522d59f/jenkins-war-2.518-rc37018.68171522d59f.war	23.1852552713826
3.139.132.49	org/jenkins-ci/main/cli/2.518/cli-2.518.jar	21.4586931876838
3.139.132.49	org/jenkins-ci/main/jenkins-core/2.518/jenkins-core-2.518.jar	20.3767495974898
3.139.132.49	org/eclipse/jgit/org.eclipse.jgit/6.8.0.202311291450-r/org.eclipse.jgit-6.8.0.202311291450-r.jar	17.8714849250391
3.139.132.49	org/jenkins-ci/plugins/bouncycastle-api/2.30.1.80-256.vf98926042a_9b_/bouncycastle-api-2.30.1.80-256.vf98926042a_9b_.hpi	17.6525939153507
3.139.132.49	org/jenkins-ci/main/jenkins-core/2.479.3/jenkins-core-2.479.3.jar	15.6551615986973
3.139.132.49	org/jenkins-ci/plugins/durable-task/581.v299a_5609d767/durable-task-581.v299a_5609d767.hpi	14.1808188911527
3.139.132.49	org/jenkins-ci/main/jenkins-test-harness-htmlunit/208.v18783fd22fca_/jenkins-test-harness-htmlunit-208.v18783fd22fca_.jar	13.8294054418802
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.452.4/jenkins-war-2.452.4.war	13.5711509920657
3.139.132.49	org/eclipse/jgit/org.eclipse.jgit/7.3.0.202506031305-r/org.eclipse.jgit-7.3.0.202506031305-r.jar	13.3827022798359
3.139.132.49	io/jenkins/plugins/lib-durable-task/48.v72b_86b_9ca_8e1/lib-durable-task-48.v72b_86b_9ca_8e1.jar	13.1410510651767
3.139.132.49	org/jenkins-ci/plugins/jackson2-api/2.18.3-402.v74c4eb_f122b_2/jackson2-api-2.18.3-402.v74c4eb_f122b_2.hpi	13.0074402401224

Download count

The download count by artifact is

IP Address	Artitact	Download Count
3.139.132.49	org/eclipse/jgit/org.eclipse.jgit/6.8.0.202311291450-r/org.eclipse.jgit-6.8.0.202311291450-r.jar	6051
3.139.132.49	org/jenkins-ci/plugins/bouncycastle-api/2.30.1.80-261.v00c0e2618ec3/bouncycastle-api-2.30.1.80-261.v00c0e2618ec3.hpi	4915
3.139.132.49	org/eclipse/jgit/org.eclipse.jgit/7.3.0.202506031305-r/org.eclipse.jgit-7.3.0.202506031305-r.jar	4354
3.139.132.49	org/jenkins-ci/main/jenkins-core/2.479.1/jenkins-core-2.479.1.jar	3272
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.479.1/jenkins-war-2.479.1.war	3272
3.139.132.49	org/jenkins-ci/plugins/jackson2-api/2.18.3-402.v74c4eb_f122b_2/jackson2-api-2.18.3-402.v74c4eb_f122b_2.hpi	3219
3.139.132.49	org/jenkins-ci/plugins/durable-task/587.v84b_877235b_45/durable-task-587.v84b_877235b_45.hpi	2888
3.139.132.49	io/jenkins/test/fips/fips-bundle-test/27.va_f689db_8c5ea_/fips-bundle-test-27.va_f689db_8c5ea_.jar	2784
3.139.132.49	org/jenkins-ci/plugins/bouncycastle-api/2.30.1.80-256.vf98926042a_9b_/bouncycastle-api-2.30.1.80-256.vf98926042a_9b_.hpi	2071
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.518/jenkins-war-2.518.war	1908
3.139.132.49	org/jenkins-ci/main/jenkins-test-harness-htmlunit/208.v18783fd22fca_/jenkins-test-harness-htmlunit-208.v18783fd22fca_.jar	1903
3.139.132.49	org/jenkins-ci/main/jenkins-core/2.518/jenkins-core-2.518.jar	1848
3.139.132.49	org/jenkins-ci/main/cli/2.518/cli-2.518.jar	1846
3.139.132.49	org/jenkins-ci/main/jenkins-core/2.479.3/jenkins-core-2.479.3.jar	1417
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.479.3/jenkins-war-2.479.3.war	1417
3.139.132.49	org/jenkins-ci/plugins/durable-task/581.v299a_5609d767/durable-task-581.v299a_5609d767.hpi	1223
3.139.132.49	io/jenkins/plugins/lib-durable-task/48.v72b_86b_9ca_8e1/lib-durable-task-48.v72b_86b_9ca_8e1.jar	1138
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.517/jenkins-war-2.517.war	985
3.139.132.49	org/jenkins-ci/main/jenkins-test-harness-tools/2.2/jenkins-test-harness-tools-2.2.jar	873
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.504.3/jenkins-war-2.504.3.war	474
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.516.1-rc36996.5d38c1a_d0c66/jenkins-war-2.516.1-rc36996.5d38c1a_d0c66.war	319
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.518-rc37018.68171522d59f/jenkins-war-2.518-rc37018.68171522d59f.war	285
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.492.3/jenkins-war-2.492.3.war	279
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.504.1/jenkins-war-2.504.1.war	279
3.139.132.49	org/jenkins-ci/main/jenkins-war/2.452.4/jenkins-war-2.452.4.war	156

Reproduction steps

Not clear which job or jobs are causing the duplicate downloads

[github-actions]

Take a look at these similar issues to see if there isn't already a response to your problem:

1. 90% Reduce artifactory bandwidth used by infra.ci.jenkins.io #4745
2. 79% Reduce Artifactory storage and bandwidth use #4533

[dduportal]

After a first quick check, it appears that most of the top elements (all WAR files, but also a few JAR files) are not cached properly by the artifact caching proxy.

I'm working on a reproduction case using only a few shell instructions to run within ci.jenkins.io before going forward.

It looks like the current profile of requests in ACP did change since we added a local cache to the BOM builds: I have to reconsider a few Nginx buffering elements to adapt. These change could help in improving this caching behavior.

On long term, the "caching .m2/repository" technique used for the BOM builds could be generalized to all other builds in ci.jenkins.io to help on that matter.

[timja]

On long term, the "caching .m2/repository" technique used for the BOM builds could be generalized to all other builds in ci.jenkins.io to help on that matter.

We could also use job cacher as a simpler alternative as well? Bom has a very specific difficult usecase to handle, the rest don't.

[dduportal]

On long term, the "caching .m2/repository" technique used for the BOM builds could be generalized to all other builds in ci.jenkins.io to help on that matter.

We could also use job cacher as a simpler alternative as well? Bom has a very specific difficult usecase to handle, the rest don't.

The cache generated from the BOM is already available, easy to reuse across builds (e.g. deduplication) and safe, I would prefer focusing on it: looks easier than Job cacher for us to manage

[dduportal]

Confirmed that some requests are not cached by the caching proxy system:

curl -o /dev/null -v -sS http://artifact-caching-proxy.artifact-caching-proxy.svc.cluster.local:8080/org/jenkins-ci/main/jenkins-war/2.479.1/jenkins-war-2.479.1.war
*   Trying 172.20.80.135:8080...
* Connected to artifact-caching-proxy.artifact-caching-proxy.svc.cluster.local (172.20.80.135) port 8080 (#0)
> GET /org/jenkins-ci/main/jenkins-war/2.479.1/jenkins-war-2.479.1.war HTTP/1.1
> Host: artifact-caching-proxy.artifact-caching-proxy.svc.cluster.local:8080
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Mon, 04 Aug 2025 11:48:16 GMT
< Content-Type: application/java-archive
< Content-Length: 96008718
< Connection: keep-alive
< x-amz-id-2: <redacted>
< x-amz-request-id: CZ1BA4VA7F3Y3YHN
< x-amz-replication-status: COMPLETED
< Last-Modified: Wed, 30 Oct 2024 13:06:49 GMT
< ETag: "0f1540962ec993aac5644cc05af44a4d"
< x-amz-storage-class: INTELLIGENT_TIERING
< x-amz-server-side-encryption: AES256
< x-amz-version-id: mOkivk_ApEOApFYNLEodIWROHo1GN5JE
< Content-Disposition: attachment;filename="jenkins-war-2.479.1.war"
< Accept-Ranges: bytes
< Server: AmazonS3
< 
{ [16776 bytes data]
* Connection #0 to host artifact-caching-proxy.artifact-caching-proxy.svc.cluster.local left intact
jenkins@jnlp-maven-21-rbb77:~/agent$ 

=> is expected to have a < X-Cache-Status: HIT response header.
Confirmed in the logs as well:

10.0.147.130 - - [04/Aug/2025:11:48:17 +0000] "GET /org/jenkins-ci/main/jenkins-war/2.479.1/jenkins-war-2.479.1.war HTTP/1.1" 200 96008718 1.089 "-" "curl/7.81.0" "-" "44.209.124.32:443 : 54.231.137.1:443" "302 : 200" "0.078 : 0.998"

(we see the upstream redirect chain which is ALWAYS present).

[dduportal]

Next step: local reproduction

[dduportal]

Next step: local reproduction

🤔 Ok, so it seems all artifacts stored in a redirect location (AWS S3) in JFrog are not cached.
Local reproduction shows that all 302 -> xxx -> 200 redirection chains are never cached.

Example:

127.0.0.1 - - [04/Aug/2025:12:31:52 +0000] "GET /org/eclipse/jgit/org.eclipse.jgit/7.4.0.202507290921-m2/org.eclipse.jgit-7.4.0.202507290921-m2.jar HTTP/1.1" 200 3301620 1.873 "-" "curl/8.7.1" "-" "44.209.124.32:443 : 52.217.230.81:443" "302 : 200" "0.816 : 1.057"
127.0.0.1 - - [04/Aug/2025:12:31:56 +0000] "GET /org/eclipse/jgit/org.eclipse.jgit/7.4.0.202507290921-m2/org.eclipse.jgit-7.4.0.202507290921-m2.jar HTTP/1.1" 200 3301620 1.562 "-" "curl/8.7.1" "-" "34.225.228.28:443 : 54.231.198.145:443" "302 : 200" "0.447 : 1.114"

=> it means something changed either in Nginx or remote JFrog platform. Investigating