avante.nvim/crates/avante-tokenizers/README.md at main

Files

Aaron Pham 0557deeab7 feat: tokenizers (#429 )

* feat: tokenizers

This reverts commit d5a4db8321.

* fix(inputs): #422

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

---------

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

2024-08-31 13:39:50 -04:00

54 B

Raw Permalink Blame History

A simple crate to unify hf/tokenizers and tiktoken-rs

54 B Raw Permalink Blame History

54 B

Raw Permalink Blame History