RYDE-WORK/ktransformers
1
0
Fork 0
mirror of https://github.com/RYDE-WORK/ktransformers.git synced 2026-02-03 21:20:33 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity

Merge pull request #317 from kvcache-ai/develop-0.2.1

Browse Source 
[feature] update  docker image and entrypoint
This commit is contained in:
UnicornChan 2025-02-15 16:19:18 +08:00 committed by GitHub
commit 65d73ea3f8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
3 changed files with 9 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
Dockerfile
View File

@ -32,4 +32,4 @@ CPU_INSTRUCT=NATIVE KTRANSFORMERS_FORCE_BUILD=TRUE TORCH_CUDA_ARCH_LIST="8.0;8.
pip cache purge pip cache purge
EOF EOF
ENTRYPOINT [ "/opt/conda/bin/ktransformers" ] ENTRYPOINT ["tail", "-f", "/dev/null"]

8
doc/en/Docker.md
View File

@ -7,7 +7,7 @@
## Images ## Images
There is a Docker image available for our project, you can pull the docker image by： There is a Docker image available for our project, you can pull the docker image by：
``` ```
docker pull approachingai/ktransformers:0.1.1 docker pull approachingai/ktransformers:0.2.1
``` ```
**Notice**: In this image, we compile the ktransformers in AVX512 instuction CPUs, if your cpu not support AVX512, it is suggested to recompile and install ktransformer in the /workspace/ktransformers directory within the container. **Notice**: In this image, we compile the ktransformers in AVX512 instuction CPUs, if your cpu not support AVX512, it is suggested to recompile and install ktransformer in the /workspace/ktransformers directory within the container.
@ -16,14 +16,16 @@ docker pull approachingai/ktransformers:0.1.1
- finish, execute - finish, execute
```bash ```bash
docker build -t approachingai/ktransformers:v0.1.1 . docker build -t approachingai/ktransformers:0.2.1 .
``` ```
## Usage ## Usage
Assuming you have the [nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-container-toolkit) that you can use the GPU in a Docker container. Assuming you have the [nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-container-toolkit) that you can use the GPU in a Docker container.
``` ```
docker run --gpus all -v /path/to/models:/models -p 10002:10002 approachingai/ktransformers:v0.1.1 --port 10002 --gguf_path /models/path/to/gguf_path --model_path /models/path/to/model_path --web True docker run --gpus all -v /path/to/models:/models --name ktransformers -itd approachingai/ktransformers:0.2.1
docker exec -it ktransformers /bin/bash
python -m ktransformers.local_chat --gguf_path /models/path/to/gguf_path --model_path /models/path/to/model_path --cpu_infer 33
``` ```
More operators you can see in the [readme](../../README.md) More operators you can see in the [readme](../../README.md)

6
ktransformers/__init__.py
View File

@ -5,7 +5,7 @@ Description :
Author : kkk1nak0 Author : kkk1nak0
Date : 2024-08-15 07:34:46 Date : 2024-08-15 07:34:46
Version : 1.0.0 Version : 1.0.0
LastEditors : unicornchan LastEditors : chenxl
LastEditTime : 2025-02-10 00:59:53 LastEditTime : 2025-02-15 03:53:02
''' '''
__version__ = "0.2.0" __version__ = "0.2.1"