Lc/fix xtc special tokens server

[micuentadecasa]

Bug Description

_make_sampler in server.py constructs xtc_special_tokens as a nested list [int, [int, ...]] instead of a flat
list of ints. When XTC sampling is triggered (xtc_probability > 0), apply_xtc passes this to mask[..., xtc_special_tokens] = False which crashes:

ValueError: Initialization encountered extra dimension.                                                                   

XTC sampling via the server is completely broken — any request with xtc_probability > 0 returns an HTTP 500 on the
first token.

Root Cause

server.py lines 388-391:

xtc_special_tokens=[                                                                                                      
    tokenizer.eos_token_id,    # int                                                                                      
    tokenizer.encode("\n"),     # list -> creates [int, [int, ...]]                                                       
],                                                                                                                        

tokenizer.encode() returns a list. Wrapping it in [...] alongside an int creates a nested list. Additionally,
eos_token_id (singular) only includes one EOS token, missing others on multi-EOS models.

Fix

• Align with the correct pattern already used in generate.py:2062 and chat.py:155
• xtc_special_tokens=tokenizer.encode("\n") + list(tokenizer.eos_token_ids)

This produces a flat list and includes all EOS tokens.

Validation

Before:

xtc_special_tokens: [151645, [198]]                                                                                       
apply_xtc -> ValueError: Initialization encountered extra dimension.                                                      

After:

xtc_special_tokens: [198, 151645]                                                                                         
apply_xtc -> Success, shape (1, 151669)                                                                                   

6 unit tests in tests/test_xtc_tokens.py — all passing.