From 77e57b4f5271ac7efb067e0b6fa6db635ab735c0 Mon Sep 17 00:00:00 2001 From: William Date: Thu, 1 Feb 2024 13:48:01 +0800 Subject: [PATCH] Update README-en.md --- README-en.md | 80 +++++++++++++++++++++++++--------------------------- 1 file changed, 38 insertions(+), 42 deletions(-) diff --git a/README-en.md b/README-en.md index 6a69e8d..bf3d40e 100644 --- a/README-en.md +++ b/README-en.md @@ -144,54 +144,50 @@ The capital city of China is Beijing. Beijing is not only the political center o ## Deployment on mobile phones - -After INT4 quantization, MiniCPM only occupies 2GB of space, meeting the requirements of inference on edge devices. +#### Tutorial -We utilize the open-source framework [MLC-LLM](https://github.com/mlc-ai/mlc-llm) for deployment on Android and Harmony OS. For deployment on IOS, we adapt MiniCPM using [LLMFarm](https://github.com/guinmoon/LLMFarm). We select some mobile phones for testing respectively. - -### Tutorial - - #### Android - -[Compilation and installation on Android](https://github.com/OpenBMB/mlc-MiniCPM/blob/main/README.md) - - #### IOS - -[Compilation and installation on IOS](https://github.com/OpenBMB/LLMFarm) - - #### Multimodal +* After INT4 quantization, MiniCPM only occupies 2GB of space, meeting the requirements of inference on end devices. +* We have made different adaptations for different operating systems. +* **Note: The current open-source framework is still improving its support for mobile phones, and not all chips and operating system versions can successfully run MLC-LLM or LLMFarm.** +* Android, Harmony OS + * Adapt based on [MLC-LLM](https://github.com/mlc-ai/mlc-llm). + * Adapted for text model MiniCPM, and multimodel model MiniCPM-V. + * Support MiniCPM-2B-SFT-INT4、MiniCPM-2B-DPO-INT4、MiniCPM-V. + * [Compile and Installation Guide](https://github.com/OpenBMB/mlc-MiniCPM/blob/main/README.md) +* iOS + * Adapt based on [LLMFarm](https://github.com/guinmoon/LLMFarm). + * Adapted for text model MiniCPM. + * Support MiniCPM-2B-SFT-INT4、MiniCPM-2B-DPO-INT4. + * [Compile and Installation Guide](https://github.com/OpenBMB/LLMFarm) ### Performance - -Instead of conducting in-depth optimization for deployment on mobile phones, we only verify the feasibility of MiniCPM using mobile chips for inference. +* We did not conduct in-depth optimization and system testing on the mobile inference model, only verifying the feasibility of MiniCPM using mobile phone chips for inference. +* There have been no previous attempts to deploy multimodal models on mobile phones. We have verified the feasibility of deploying MiniCPM-V on mobile phones based on MLC-LLM this time, and it can input and output normally. However, there also exist a problem of long image processing time, which needs further optimization :) +* **We welcome more developers to continuously improve the inference performance of LLMs on mobile phones and update the test results below.** -**We welcome more developers to continuously improve the inference performance of LLMs on mobile phones and update the test results below.** +| Mobile Phones | OS | Processor | Memory（GB） | Inference Throughput（token/s） | +| ----------------- | ------------- | ------------------ | ------------ | ------------------------------- | +| OPPO Find N3 | Android 13 | snapdragon 8 Gen2 | 12 | 6.5 | +| Samsung S23 Ultra | Android 14 | snapdragon 8 Gen2 | 12 | 6.4 | +| Meizu M182Q | Android 11 | snapdragon 888Plus | 8 | 3.7 | +| Xiaomi 12 Pro | Android 13 | snapdragon 8 Gen1 | 8+3 | 3.7 | +| Xiaomi Redmi K40 | Android 11 | snapdragon 870 | 8 | 3.5 | +| Oneplus LE 2100 | Android 13 | snapdragon 870 | 12 | 3.5 | +| Oneplus HD1900 | Android 11 | snapdragon 865 | 8 | 3.2 | +| Oneplus HD1900 | Android 11 | snapdragon 855 | 8 | 3.0 | +| Oneplus HD1905 | Android 10 | snapdragon 855 | 8 | 3.0 | +| Oneplus HD1900 | Android 11 | snapdragon 855 | 8 | 3.0 | +| Xiaomi MI 8 | Android 9 | snapdragon 845 | 6 | 2.3 | +| Huawei Nova 11SE | Harmony 4.0.0 | snapdragon 778 | 12 | 1.9 | +| Xiaomi MIX 2 | Android 9 | snapdragon 835 | 6 | 1.3 | +| iPhone 15 Pro | iOS 17.2.1 | A16 | 8 | 18.0 | +| iPhone 15 | iOS 17.2.1 | A16 | 6 | 15.0 | +| iPhone 12 Pro | iOS 16.5.1 | A14 | 6 | 5.8 | +| iPhone 12 | iOS 17.2.1 | A14 | 4 | 5.8 | +| iPhone 11 | iOS 16.6 | A13 | 4 | 4.6 | -|Mobile Phones|OS|Processor|Memory（GB）|Inference Throughput（token/s）| -|-|-|-|-|-| -|OPPO Find N3|Android 13|snapdragon 8 Gen2|12|6.5| -|Samsung S23 Ultra|Android 14|snapdragon 8 Gen2|12|6.4| -|Meizu M182Q|Android 11|snapdragon 888Plus|8|3.7| -|Xiaomi 12 Pro|Android 13|snapdragon 8 Gen1|8+3|3.7| -|Xiaomi Redmi K40|Android 11|snapdragon 870|8|3.5| -|Oneplus LE 2100|Android 13|snapdragon 870|12|3.5| -|Oneplus HD1900|Android 11|snapdragon 865|8|3.2| -|Oneplus HD1900|Android 11|snapdragon 855|8|3.0| -|Oneplus HD1905|Android 10|snapdragon 855|8|3.0| -|Oneplus HD1900|Android 11|snapdragon 855|8|3.0| -|Xiaomi MI 8|Android 9|snapdragon 845|6|2.3| -|Huawei Nova 11SE|Harmony 4.0.0|snapdragon 778|12|1.9| -|Xiaomi MIX 2|Android 9|snapdragon 835|6|1.3| -|iPhone 15 Pro|iOS 17.2.1|A16|8|18.0| -|iPhone 15|iOS 17.2.1|A16|6|15.0| -|iPhone 12 Pro|iOS 16.5.1|A14|6|5.8| -|iPhone 12|iOS 17.2.1|A14|4|5.8| -|iPhone 11|iOS 16.6|A13|4|4.6| - -![多模态样例](https://github.com/OpenBMB/OmniLMM/blob/main/assets/Snake_cn_Mushroom_en.gif) +![multimodel demo](https://github.com/OpenBMB/OmniLMM/blob/main/assets/Snake_cn_Mushroom_en.gif) ## Demo & API