Revisit tokenizer microoptimizations #22
Description
For example, there's an old branch to use SSE 4.2 string instructions in tokenization, which realized a speedup of about 10% and could probably be pushed much further.
{{ message }}
For example, there's an old branch to use SSE 4.2 string instructions in tokenization, which realized a speedup of about 10% and could probably be pushed much further.