RYDE-WORK/ktransformers
1
0
Fork 0
mirror of https://github.com/RYDE-WORK/ktransformers.git synced 2026-01-19 21:03:18 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #317 from kvcache-ai/develop-0.2.1

Browse Source 
[feature] update  docker image and entrypoint
This commit is contained in:
UnicornChan 2025-02-15 16:19:18 +08:00 committed by GitHub
commit 65d73ea3f8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 9 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Dockerfile
View File

@ -32,4 +32,4 @@ CPU_INSTRUCT=NATIVE KTRANSFORMERS_FORCE_BUILD=TRUE TORCH_CUDA_ARCH_LIST="8.0;8.
pip cache purge
EOF
ENTRYPOINT [ "/opt/conda/bin/ktransformers" ]
ENTRYPOINT ["tail", "-f", "/dev/null"]

8
doc/en/Docker.md
View File

@ -7,7 +7,7 @@
## Images
There is a Docker image available for our project, you can pull the docker image by：
```
docker pull approachingai/ktransformers:0.1.1
docker pull approachingai/ktransformers:0.2.1
```
**Notice**: In this image, we compile the ktransformers in AVX512 instuction CPUs, if your cpu not support AVX512, it is suggested to recompile and install ktransformer in the /workspace/ktransformers directory within the container.
@ -16,14 +16,16 @@ docker pull approachingai/ktransformers:0.1.1
- finish, execute
```bash
docker build -t approachingai/ktransformers:v0.1.1 .
docker build -t approachingai/ktransformers:0.2.1 .
```
## Usage
Assuming you have the [nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-container-toolkit) that you can use the GPU in a Docker container.
```
docker run --gpus all -v /path/to/models:/models -p 10002:10002 approachingai/ktransformers:v0.1.1 --port 10002 --gguf_path /models/path/to/gguf_path --model_path /models/path/to/model_path --web True
docker run --gpus all -v /path/to/models:/models --name ktransformers -itd approachingai/ktransformers:0.2.1
docker exec -it ktransformers /bin/bash
python -m ktransformers.local_chat --gguf_path /models/path/to/gguf_path --model_path /models/path/to/model_path --cpu_infer 33
```
More operators you can see in the [readme](../../README.md)

6
ktransformers/__init__.py
View File

@ -5,7 +5,7 @@ Description :
Author : kkk1nak0
Date : 2024-08-15 07:34:46
Version : 1.0.0
LastEditors : unicornchan
LastEditTime : 2025-02-10 00:59:53
LastEditors : chenxl
LastEditTime : 2025-02-15 03:53:02
'''
__version__ = "0.2.0"
__version__ = "0.2.1"