Commit Graph

12 Commits

Author SHA1 Message Date
Aaron Pham
0557deeab7 feat: tokenizers (#429)
* feat: tokenizers

This reverts commit d5a4db8321.

* fix(inputs): #422

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

---------

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
2024-08-31 13:39:50 -04:00
yetone
d5a4db8321 Revert tokenizers (#423)
* Revert "fix: add missing "with" in README"

This reverts commit bf1e7f1f17.

* Revert "feat: tokenizers (#407)"

This reverts commit d2095ba267.
2024-08-31 22:45:31 +08:00
Aaron Pham
d2095ba267 feat: tokenizers (#407)
* feat: autobuild tiktoken lib and schenanigans

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* chore: revert readme changes

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* fix(build): windows

Signed-off-by: Hanchin Hsieh <me@yuchanns.xyz>

* chore(plugin): early load commands and base setup

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* fix(build): make sync

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* feat: rust go vroom vroom

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* feat: scuffed afaf implementation binding go brrrr

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* chore: remove dups

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* fix(tokens): calculate whether we should do prompt_caching (fixes #416)

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* chore: ignore lockfiles

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* Update README.md

* Update crates/avante-tokenizers/README.md

* chore: remove unused

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* chore: remove auto build

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

---------

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Hanchin Hsieh <me@yuchanns.xyz>
Co-authored-by: yuchanns <me@yuchanns.xyz>
2024-08-31 07:19:59 -04:00
yetone
1f0395ccd0 fix: makefile (#319) 2024-08-28 20:57:50 +08:00
yetone
bfa203018e fix: remove copilot provider (#318) 2024-08-28 20:51:20 +08:00
Chuang Yu
eaf37668b1 fix: remove temporary build directory (#252) 2024-08-27 00:00:36 -04:00
Aaron Pham
db1f1efb27 chore(build): from source (#241)
* chore(build): from source

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

* chore: update readme

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>

---------

Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
2024-08-26 20:36:05 +08:00
rkk1995
5159aeefd1 Include ARCH in asset name (#219) 2024-08-26 03:54:42 +08:00
Aaron Pham
2a0311fbf1 chore(build): add options to build from source on linux (#98)
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
2024-08-19 15:41:18 -04:00
Hanchin Hsieh
2674872945 feat: support build tiktoken in Linux aarch64 (#15) 2024-08-16 10:01:51 +08:00
Aaron Pham
f7de743fe0 feat(tiktoken): automatic build (#9)
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
2024-08-15 20:59:01 +08:00
yetone
9edd4202af first commit 2024-08-15 01:45:38 +08:00