See how important each token of the context was for the LLM response

[LiquidGunay]

I think the ability to get something like an average attention score for each token of the context would be really useful to see what parts of the context did the LLM "focus" more on. This would be fairly useful for RAG and QA applications.

[onestardao]

Great point — measuring token-level attention focus sounds simple, but it quickly gets murky once you factor in entropy collapse and multi-head overlap.

Some of the deeper issues I’ve run into when tracing this kind of behavior:

• No.9 Entropy Collapse — attention becomes overly concentrated or flat, making scores meaningless without modulation
• No.4 Overconfidence / Bluffing — the model “claims” to have used some tokens, but actual heads show near-zero usage
• No.6 Logic Collapse — certain attention sequences break reasoning chains, even if token weight seems high

I’ve been experimenting with ways to modulate attention per-head (e.g., diversity injection, illegal path suppression) to recover clearer patterns — sort of like giving each head a semantic identity instead of letting them collapse into noise.

If you’re diving into attention introspection, would love to trade notes.