Atream
|
5ec33d046d
|
optimize gguf dequant, save mem, support Q2_K
use marlin for lm_head, lm_head only calc last token for prefill
extend context window to 19K for DeepSeek-V3/R1 within 24GB VRAM
|
2025-02-22 06:13:01 +00:00 |
|
Atream
|
7e1fe256c8
|
optimize GPU
|
2025-02-21 05:06:57 +00:00 |
|
Atream
|
cf4da5fd47
|
Merge pull request #382 from ceerRep/server-prefix-cache
fix server and add prefix cache for server
|
2025-02-19 12:33:36 +08:00 |
|
Azure
|
ea75849d03
|
Merge pull request #481 from Zhoneym/main
Modify and add any incorrect or missing content in the `install.md`
|
2025-02-19 11:19:11 +08:00 |
|
Zhoneym
|
c72291832d
|
Update install.md
|
2025-02-19 10:39:43 +08:00 |
|
Zhoneym
|
3015e9dfa7
|
Modify and add any incorrect or missing content in the install.md file.
|
2025-02-19 10:01:16 +08:00 |
|
ceerrep
|
584c7d5639
|
fix: object type for non-streaming response
|
2025-02-18 23:50:28 +08:00 |
|
ceerrep
|
6d45871de8
|
fix: workaround return dummy usage
|
2025-02-18 22:39:49 +08:00 |
|
ZiWei Yuan
|
09f5c5e77e
|
Merge pull request #476 from kvcache-ai/V0.2.1.post1
🔖 release v0.2.1.post1
|
2025-02-18 21:48:30 +08:00 |
|
liam
|
75d7d423b0
|
🔖 release v0.2.1.post1
|
2025-02-18 20:45:48 +08:00 |
|
Azure
|
a435e1bbcd
|
Merge pull request #406 from hoshinohikari/main
Fix cmake error caused by lack of environment variables in Windows environment
|
2025-02-18 20:09:59 +08:00 |
|
ceerrep
|
db61375f1a
|
Merge remote-tracking branch 'kvai/main' into server-prefix-cache
|
2025-02-18 17:51:22 +08:00 |
|
Azure
|
19fd24f461
|
Merge pull request #412 from haojiubujian1985/delete-unused-link
Update README_ZH.md
|
2025-02-18 14:58:10 +08:00 |
|
Atream
|
1c1769a579
|
Merge pull request #413 from kvcache-ai/fix_precision_MLA
Fix precision mla
|
2025-02-18 14:44:50 +08:00 |
|
liam
|
592e13d453
|
⚡ add mmlu_pro test
|
2025-02-18 14:43:38 +08:00 |
|
Atream
|
1054ee3ae4
|
Merge pull request #450 from kvcache-ai/update-WeChatGroup
Add files via upload
|
2025-02-18 14:34:07 +08:00 |
|
Atream
|
261d5d393f
|
Add files via upload
|
2025-02-18 14:28:29 +08:00 |
|
Azure
|
2671fc7f4b
|
Merge pull request #445 from dedfaf/main
Fix typo in DeepseekR1_V3_tutorial.md Line 174
|
2025-02-18 14:23:29 +08:00 |
|
UnicornChan
|
ccf0f1910f
|
Merge pull request #442 from Zhoneym/patch-1
Fix `GLIBCXX_3.4.30' not found in Anaconda
|
2025-02-18 13:58:44 +08:00 |
|
liam
|
07a0555016
|
⚡ fix device and add test
|
2025-02-18 12:52:17 +08:00 |
|
ceerrep
|
73d072f609
|
Merge branch 'fix_precision_MLA' of https://github.com/kvcache-ai/ktransformers into server-prefix-cache
|
2025-02-18 11:44:28 +08:00 |
|
ceerrep
|
2ffb43f974
|
fix: adapt prefix cache in forward_linux_flashinfer
|
2025-02-18 11:40:15 +08:00 |
|
Xie Weiyu
|
f029588b61
|
fix server warmup
|
2025-02-18 11:39:45 +08:00 |
|
Zhoneym
|
8b221f532a
|
Fix `GLIBCXX_3.4.30' not found in Anaconda
|
2025-02-18 11:32:42 +08:00 |
|
ceerrep
|
c70b6f4d5b
|
fix: use 'cuda:0' by default if torch_device is 'cuda'
|
2025-02-18 11:15:17 +08:00 |
|
dedfaf
|
3e5c5a2f4a
|
Fix typo in DeepseekR1_V3_tutorial.md Line 174
|
2025-02-18 10:57:12 +08:00 |
|
Xie Weiyu
|
9f1da18630
|
update install.sh
|
2025-02-17 20:50:19 +08:00 |
|
Xie Weiyu
|
c176e516b5
|
server mix mla
|
2025-02-17 20:40:28 +08:00 |
|
ceerrep
|
ee24eb8dc3
|
fix: fix server for triton kernel
|
2025-02-17 18:08:45 +08:00 |
|
ceerrep
|
bb1cadfff3
|
Merge branch 'fix_precision_MLA' of https://github.com/kvcache-ai/ktransformers into server-prefix-cache
|
2025-02-17 18:08:04 +08:00 |
|
Atream
|
038bc30888
|
fix precision bug imported by position_ids in 0.2.0
|
2025-02-17 09:23:14 +00:00 |
|
xubo
|
49c6e2fc04
|
Update README_ZH.md
|
2025-02-17 16:56:44 +08:00 |
|
hoshinohikari
|
6d9182795c
|
Fix cmake error caused by lack of environment variables in Windows environment
|
2025-02-17 15:56:56 +08:00 |
|
ceerrep
|
cd9f7f8f34
|
fix: server: drop <think> tag in chat template
|
2025-02-17 14:25:27 +08:00 |
|
ceerrep
|
ca2090d89b
|
feat: use model name in openai endpoint
|
2025-02-17 00:27:32 +08:00 |
|
ceerrep
|
5ac266085e
|
fix: use flash_attn for faster prefill
|
2025-02-17 00:11:24 +08:00 |
|
ceerrep
|
bb0ccc7b1a
|
feat: add prefix cache for server
|
2025-02-17 00:10:55 +08:00 |
|
wang jiahao
|
c515cc49a5
|
Merge pull request #355 from kvcache-ai/qiyuxinlin-patch-1
Update install.md
|
2025-02-16 15:05:43 +08:00 |
|
wang jiahao
|
24de607be4
|
Merge pull request #357 from kvcache-ai/qiyuxinlin-patch-2
Update SUMMARY.md
|
2025-02-16 15:05:19 +08:00 |
|
Atream
|
b84524622e
|
support bf16 read
|
2025-02-16 06:43:27 +00:00 |
|
wang jiahao
|
76554dd674
|
Update SUMMARY.md
|
2025-02-16 14:30:22 +08:00 |
|
wang jiahao
|
56d19d6115
|
Update install.md
|
2025-02-16 14:23:56 +08:00 |
|
Azure
|
9f9c3738f9
|
Merge pull request #354 from Azure-Tang/fix-mockTritonMLA
[fix] Mock triton mla due to precision issue
|
2025-02-16 14:06:54 +08:00 |
|
Azure
|
ff6b265e53
|
Mock triton mla due to precision issue
|
2025-02-16 06:03:12 +00:00 |
|
Atream
|
c5f036e8a4
|
Merge pull request #333 from kvcache-ai/feat_experts_gpu
toy support for experts on GPU, no CUDA Graph
|
2025-02-15 23:30:24 +08:00 |
|
Atream
|
8ed8eb2a9e
|
Update FAQ.md
|
2025-02-15 23:27:35 +08:00 |
|
Atream
|
c189d55bd1
|
toy support for experts on GPU, no CUDA Graph
|
2025-02-15 15:16:00 +00:00 |
|
wang jiahao
|
ae8da019c1
|
Merge pull request #330 from hrz6976/fix-nonetype
thanks..., I was about to submit and found that you had already modified it. Thank you for your contribution
|
2025-02-15 22:50:51 +08:00 |
|
Azure
|
56382aa869
|
Merge pull request #290 from ZhangShuaiyi/dev/check_gguf_path
ensure that gguf_path argument is a directory.
|
2025-02-15 22:48:57 +08:00 |
|
Azure
|
c8bf250104
|
Merge pull request #324 from BiFangKNT/patch-2
Update DeepseekR1_V3_tutorial.md
|
2025-02-15 22:23:09 +08:00 |
|