[0.9.1][Bugfix] fix oom issue in mla and enable mla_pa for deepseek mla decode

[ganyi1996ppo]

What this PR does / why we need it?

After the disaggregated PD merged, the kv cache on deepseek will become two piece of independent buffer for kv transfer or computation. However, the current kernel, namely paged_attention_mla can only accept k_cache as a single parameter, this make us have to concat these two piece of kv cache together before the attention thus incurs a memory peak inside the attention in eager mode. In this PR we introduce a torch_npu.atb.npu_multi_head_latent_attention for mla decode path, which will be used as default path for both eager mode and aclgraph after the related torch_npu is public available. Since its still a restrict package, we add VLLM_ASCEND_MLA_PA to control its usage. This flag will be removed in the future.

Does this PR introduce any user-facing change?

Yes, add a new flag named VLLM_ASCEND_MLA_PA, but it will be removed eventually after the newest torch_npu is released.

How was this patch tested?

[ganyi1996ppo]

I'll cherry-pick this PR back to the main branch after its tested and merged

[MengqingCao]

It seems the oom issue mainly caused by this combined_cache, is removing this enough to this pr? If so, maybe we could add npu_multi_head_latent_attention when the torch-npu is available officially

[ganyi1996ppo]

Remove is not enough in this case cause, as you can see, the paged_attention_mla can only receive the concatenated cache as its input parameter, withoutnpu_multi_head_latent_attention the cat seems inevitable.

[ganyi1996ppo]

dbo test seems will oom in ci

Processed prompts: 0%| | 0/41 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Process EngineCore_0:
Traceback (most recent call last):
File "/usr/local/python3.10.17/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/local/python3.10.17/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/__w/vllm-ascend/vllm-ascend/vllm-empty/vllm/v1/engine/core.py", line 519, in run_engine_core
raise e
File "/__w/vllm-ascend/vllm-ascend/vllm-empty/vllm/v1/engine/core.py", line 508, in run_engine_core
engine_core.run_busy_loop()
File "/__w/vllm-ascend/vllm-ascend/vllm-empty/vllm/v1/engine/core.py", line 535, in run_busy_loop
self._process_engine_step()
File "/__w/vllm-ascend/vllm-ascend/vllm-empty/vllm/v1/engine/core.py", line 560, in _process_engine_step
outputs, model_executed = self.step_fn()
File "/__w/vllm-ascend/vllm-ascend/vllm-empty/vllm/v1/engine/core.py", line 231, in step
model_output = self.execute_model(scheduler_output)
File "/__w/vllm-ascend/vllm-ascend/vllm-empty/vllm/v1/engine/core.py", line 217, in execute_model
raise err
File "/__w/vllm-ascend/vllm-ascend/vllm-empty/vllm/v1/engine/core.py", line 211, in execute_model
return self.model_executor.execute_model(scheduler_output)
File "/__w/vllm-ascend/vllm-ascend/vllm-empty/vllm/v1/executor/multiproc_executor.py", line 163, in execute_model
(output, ) = self.collective_rpc("execute_model",
File "/__w/vllm-ascend/vllm-ascend/vllm-empty/vllm/v1/executor/multiproc_executor.py", line 220, in collective_rpc
result = get_response(w, dequeue_timeout)
File "/__w/vllm-ascend/vllm-ascend/vllm-empty/vllm/v1/executor/multiproc_executor.py", line 207, in get_response
raise RuntimeError(
RuntimeError: Worker failed with error 'NPU out of memory. Tried to allocate 11.70 GiB (NPU 0; 29.50 GiB total capacity; 25.09 GiB already allocated; 25.09 GiB current active; 3.49 GiB free; 25.14 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.', please check the stack trace above for the root cause
FAILED

=================================== FAILURES ===================================
____________________ test_models_distributed_DeepSeekV3_dbo ____________________