From 96d2462e3e2701ead1c58e325e6aacbe9aceb77a Mon Sep 17 00:00:00 2001 From: zh-zheng Date: Thu, 11 Apr 2024 01:53:11 +0800 Subject: [PATCH] update readme --- README-en.md | 244 ++++++++++++++++++++++++++++++++++++++++---------- README.md | 246 +++++++++++++++++++++++++++++++++++++++++---------- 2 files changed, 395 insertions(+), 95 deletions(-) diff --git a/README-en.md b/README-en.md index b7ed728..8b3488a 100644 --- a/README-en.md +++ b/README-en.md @@ -14,7 +14,7 @@ Technical Blog | Multi-modal Model OmniLMM | CPM-C 100B Model Trial | -Join our discord and wechat +Join our discord and WeChat

MiniCPM is an End-Side LLM developed by ModelBest Inc. and TsinghuaNLP, with only 2.4B parameters excluding embeddings (2.7B in total). @@ -60,7 +60,7 @@ We release all model parameters for research and limited commercial use.

## Update Log -- 2024/04/11 We release [MiniCPM-V 2.0](https://huggingface.co/openbmb/MiniCPM-V-2.0), [MiniCPM-2B-128k](https://huggingface.co/openbmb/MiniCPM-2B-128k), [MiniCPM-MoE-8x2B](https://huggingface.co/openbmb/MiniCPM-MoE-8x2B) and [MiniCPM-1B-sft-bf16](https://huggingface.co/openbmb/MiniCPM-1B-sft-bf16)! +- 2024/04/11 We release [MiniCPM-V 2.0](https://huggingface.co/openbmb/MiniCPM-V-2.0), [MiniCPM-2B-128k](https://huggingface.co/openbmb/MiniCPM-2B-128k), [MiniCPM-MoE-8x2B](https://huggingface.co/openbmb/MiniCPM-MoE-8x2B) and [MiniCPM-1B](https://huggingface.co/openbmb/MiniCPM-1B-sft-bf16)! - 2024/03/16 Intermediate checkpoints were released [here](https://huggingface.co/openbmb/MiniCPM-2B-history)! - 2024/02/13 We support llama.cpp - 2024/02/09 We have included a [Community](#community) section in the README to encourage support for MiniCPM from the open-source community. @@ -410,86 +410,236 @@ MBPP, instead of the hand-verified set. #### Multimodal evaluation -
+
- - - - - - + + + + + + + + + + + - - - - - - + + + + + + + + + + + + + + - - - - - - - + + + + + + + + + + + + - - - - - - - + + + + + + + + + + + + + + + - - + + + + + - - + + + + - - - - - - - + + + + + + + + + + + + - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Model SizeVisual TokensMMEMMB dev (en)MMB dev (zh)MMMU valCMMMU valTextVQA valDocVQA testOCRBenchOpenCompassMMEMMB dev(en)MMB dev(zh)MMMU valMathVistaLLaVA BenchObject HalBench
LLaVA-Phi3B576133559.8- Proprietary models
Gemini Pro Vision - 74.688.168063.82148.975.274.048.945.879.9 -
MobileVLM3B144128959.6- - GPT-4V - 78.088.464563.21771.575.175.053.847.893.186.4 / 92.7
Imp-v13B576143466.5- - Open-source models 6B~34B
Yi-VL-6B6.7B45.5*17.1*29049.31915.1 68.6 68.3 40.3 28.8 51.9 -
Qwen-VL-Chat 9.6B256148761.562.6488 52.1 1860.0 60.6 56.7 35.9 30.7 37.0 33.8 67.7 56.2 / 80.0
CogVLM17.4B 12251438 63.7 53.8 32.1 Yi-VL-34B34B43.4*16.9*29052.6 2050.271.171.445.130.762.3 -
MiniCPM-V(3B)3B 641452 67.3 61.9 DeepSeek-VL-7B7.3B64.7*47.0* 43555.6 1765.4 74.1 72.8 38.3 36.877.8 -
TextMonkey9.7B64.366.7 558- - - - - -- -
CogVLM-Chat17.4B70.433.3*590 52.5 1736.6 63.7 53.8 37.3 34.7 32.1 73.9 73.6 / 87.4
Open-source models 1B~3B
DeepSeek-VL-1.3B1.7B58.4*37.9*41346.0 1531.6 64.0 61.2 33.8 29.4 51.1 -
MobileVLM V23.1B57.519.4*--1440.5(P) 63.2 -----
Mini-Gemini2.2B56.234.2*--1653.0 59.8 - 31.7 -- -
MiniCPM-V2.8B 60.638.2 36647.61650.2 67.9 65.3 38.328.951.3 78.4 / 88.5
MiniCPM-V 2.02.8B 74.171.9 60555.01808.6 69.6 68.1 38.2 38.769.2 85.5 / 92.2
+* We evaluate the officially released checkpoint by ourselves. #### DPO evaluation diff --git a/README.md b/README.md index 3743c5f..7a512ca 100644 --- a/README.md +++ b/README.md @@ -15,7 +15,7 @@ MiniCPM 技术博客 | OmniLMM 多模态模型 | CPM-C 千亿模型试用 | -加入我们的 discord 和 wechat +加入我们的 discord 和 微信群

@@ -61,10 +61,10 @@ MiniCPM 是面壁智能与清华大学自然语言处理实验室共同开源的

## 更新日志 -- 2024/04/11 开源[MiniCPM-V-2.0](https://huggingface.co/openbmb/MiniCPM-V-2.0)、[MiniCPM-2B-128k](https://huggingface.co/openbmb/MiniCPM-2B-128k)、[MiniCPM-MoE-8x2B](https://huggingface.co/openbmb/MiniCPM-MoE-8x2B)和[MiniCPM-1B-sft-bf16](https://huggingface.co/openbmb/MiniCPM-1B-sft-bf16)！ +- 2024/04/11 开源[MiniCPM-V-2.0](https://huggingface.co/openbmb/MiniCPM-V-2.0)、[MiniCPM-2B-128k](https://huggingface.co/openbmb/MiniCPM-2B-128k)、[MiniCPM-MoE-8x2B](https://huggingface.co/openbmb/MiniCPM-MoE-8x2B)和[MiniCPM-1B](https://huggingface.co/openbmb/MiniCPM-1B-sft-bf16)！ - 2024/03/16 MiniCPM-2B 的30余个中间检查点开放了！[huggingface链接](https://huggingface.co/openbmb/MiniCPM-2B-history) - 2024/02/13 支持了llama.cpp -- 2024/02/09 我们在readme里加入了一个[开源社区](#community)章节，用来收集开源社区对MiniCPM的支持案例。 +- 2024/02/09 我们在README里加入了一个[开源社区](#community)章节，用来收集开源社区对MiniCPM的支持案例。 - 2024/02/08 我们更新了[llama-format的模型权重](#llamaformat)，方便大家更加快捷地使用我们的模型。 - 2024/02/01 初始发布。 @@ -437,86 +437,236 @@ print(model.response("<用户>山东省最高的山是哪座山, 它比黄山高 #### 多模态模型评测 -
+
- - - - - - + + + + + + + + + + + - - - - - - + + + + + + + + + + + + + + - - - - - - - + + + + + + + + + + + + - - - - - - - + + + + + + + + + + + + + + + - - + + + + + - - + + + + - - - - - - - + + + + + + + + + + + + - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Model SizeVisual TokensMMEMMB dev (en)MMB dev (zh)MMMU valCMMMU valTextVQA valDocVQA testOCRBenchOpenCompassMMEMMB dev(en)MMB dev(zh)MMMU valMathVistaLLaVA BenchObject HalBench
LLaVA-Phi3B576133559.8- Proprietary models
Gemini Pro Vision - 74.688.168063.82148.975.274.048.945.879.9 -
MobileVLM3B144128959.6- - GPT-4V - 78.088.464563.21771.575.175.053.847.893.186.4 / 92.7
Imp-v13B576143466.5- - Open-source models 6B~34B
Yi-VL-6B6.7B45.5*17.1*29049.31915.1 68.6 68.3 40.3 28.8 51.9 -
Qwen-VL-Chat 9.6B256148761.562.6488 52.1 1860.0 60.6 56.7 35.9 30.7 37.0 33.8 67.7 56.2 / 80.0
CogVLM17.4B 12251438 63.7 53.8 32.1 Yi-VL-34B34B43.4*16.9*29052.6 2050.271.171.445.130.762.3 -
MiniCPM-V(3B)3B 641452 67.3 61.9 DeepSeek-VL-7B7.3B64.7*47.0* 43555.6 1765.4 74.1 72.8 38.3 36.877.8 -
TextMonkey9.7B64.366.7 558- - - - - -- -
CogVLM-Chat17.4B70.433.3*590 52.5 1736.6 63.7 53.8 37.3 34.7 32.1 73.9 73.6 / 87.4
Open-source models 1B~3B
DeepSeek-VL-1.3B1.7B58.4*37.9*41346.0 1531.6 64.0 61.2 33.8 29.4 51.1 -
MobileVLM V23.1B57.519.4*--1440.5(P) 63.2 -----
Mini-Gemini2.2B56.234.2*--1653.0 59.8 - 31.7 -- -
MiniCPM-V2.8B 60.638.2 36647.61650.2 67.9 65.3 38.328.951.3 78.4 / 88.5
MiniCPM-V 2.02.8B 74.171.9 60555.01808.6 69.6 68.1 38.2 38.769.2 85.5 / 92.2
+* 我们自己评测了正式开源的模型权重。