Tokenizers for embedding models are not often implemented in languages other than Python. This becomes a major obstacle when trying to use embedding models in languages like C# or Java. Developers ...

🌿 An easy-to-use Japanese Text Processing tool, which makes it possible to switch tokenizers with small changes of code. - himkt/konoha ...