Implement multiple tiers caching with fallback and backfilling

[Felixoid]

This PR addresses multiple requests to have tiered caching.

Closes #30
Closes #1020
Closes #2493
Closes #2566

What is added:

• multilevel configuration for the caching chain
• config parameter for cache writing tolerance: ignore errors, fail on write to L0, fail on a write to any level
• added stats for each caching level
• documentation
• tests
• integration tests

[codecov-commenter]

Codecov Report

❌ Patch coverage is 89.60422% with 197 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.18%. Comparing base (8d6cab9) to head (f590fb8).

Files with missing lines	Patch %	Lines
src/cache/multilevel.rs	87.54%	66 Missing ⚠️
src/cache/multilevel_test.rs	93.99%	63 Missing ⚠️
src/config.rs	79.91%	46 Missing ⚠️
src/cache/cache.rs	70.73%	12 Missing ⚠️
src/cache/disk.rs	81.25%	6 Missing ⚠️
src/cache/readonly.rs	0.00%	3 Missing ⚠️
src/server.rs	90.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2581      +/-   ##
==========================================
+ Coverage   73.38%   74.18%   +0.80%     
==========================================
  Files          68       70       +2     
  Lines       37338    39206    +1868     
==========================================
+ Hits        27400    29085    +1685     
- Misses       9938    10121     +183     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Felixoid]

I've done quite a few features today, like control for write error handling, and added server stats for multilevel caching.

[sylvestre]

this is a huge patch, can you split a bit more? thanks

[Felixoid]

Probably I could take something out of cache.rs, but the rest is actually multilevel.rs, and 2/3 of 2k LoC are tests.

[Felixoid]

Rebased to the current main.

This PR now contains the multi-level cache feature exclusively. Most of the changes are the tests and documentation, which are around 70%.

[AJIOB]

In general, I like that changes.

There's awesome work, thanks, @Felixoid!

JFYI, you have a failed CUDA test. Please, recheck this point too.

[Felixoid]

Thanks a lot for the review!

@AJIOB: it looks like the CUDA tests were temporarily broken for an unknown reason. It was Segmentation fault in sudo apt-get update. Looks working fine now.

@mathstuf: as much as I like the idea, it looks like a next iteration for now. Probably, I'd like to work on it too. But it's for sure will be a design change for the configuration, and should be discussed in advance on how to proceed.

Now, the approach is simple and logical for the current configuration.

[Felixoid]

@sylvestre do you think we can proceed?

[drahnr]

Bar the shell scripts, the core implementation does look sound. Also took a look @trxcllnt which is a more minimal approach with 2 cache layers (or using nesting).
Since this PR already got reviews I suggest to move it along.

@sylvestre if you find the time, take a look, otherwise I'll do sometime next week and we should land it imho.

[Felixoid]

Thanks for the review!

I am on vacation RN, will try to find time to address the points somewhere around the following week.

[Felixoid]

@drahnr I addressed all points. Sorry that it took so long. I tried not to touch the laptop for work

[Felixoid]

In case #2644 is merged, this PR should benefit too; there are around 50 lines of boilerplate.

[Felixoid]

Any update on the process, please?

[AJIOB]

Hi @drahnr and @sylvestre ,

Is it OK for you? Can we merge these changes?

[sunnyguan]

Thanks a lot for the work here, just curious if SCCACHE_BASEDIRS is handled properly when using the multi-tier setup?

[sylvestre]

i am not ignoring you. just higher priorities lately, sorry

[Felixoid]

Thanks a lot for the work here, just curious if SCCACHE_BASEDIRS is handled properly when using the multi-tier setup?

I don't have any specific tests for it, but I don't see how it could break.

I'll try to add it too, just in case

[Felixoid]

@sunnyguan thanks a lot for the spotlight, I indeed forgot to implement the basedirs treat for the MultiLevelStorage 🤦

[Felixoid]

I believe the cmake-modules failure is unrelated; the fix of test is in #2649

The dedicated issue for cmake 4.0 failure #2650