NousResearch

/

Hermes-2-Pro-Llama-3-8B

like 446

Follow

NousResearch 3.39k

https://ai.stackexchange.com/questions/45054/why-do-llm-tokenizers-use-a-special-symbol-for-space-such-as-%C4%A0-in-bpe-or-in-sp

my understanding is that this character simply indicates the beginning of a word. I think its presence improves model's behavior around word boundaries.

I appreciate your answer a lot!