[FEATURE] ADDING RAM OFFLOADING SUPPORT

[R-Larocque]

🚀 Feature

Adding RAM offloading support.

Motivation

Most people don't have powerful cards.

[JiwaniZakir]

RAM offloading (CPU offloading) in this context means spilling model layers or optimizer states from VRAM to system RAM when GPU memory is exhausted — useful for running or fine-tuning large models on consumer-grade cards. In Hugging Face-based stacks like this one, the typical entry point is setting device_map="auto" or a custom layer map via accelerate, combined with llm_int8_enable_fp32_cpu_offload=True in the BitsAndBytesConfig for quantized models. The likely place to wire this in would be wherever the model is loaded and the quantization/device config is assembled — in h2o-llmstudio that's generally around the load_model logic in the backbone configuration. One caveat worth noting: CPU offloading introduces significant latency on the offloaded layers due to PCIe bandwidth, so throughput during training will degrade proportionally to how much of the model is pushed off-device.