diff --git a/README-en.md b/README-en.md index 8c05d32..68b37bc 100644 --- a/README-en.md +++ b/README-en.md @@ -1 +1 @@ -Update soon. +Update soon. \ No newline at end of file diff --git a/README.md b/README.md index f103d37..f2290c6 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,8 @@ MiniCPM 是面壁与清华大学自然语言处理实验室共同开源的系列 |[dpo-bf16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-bf16)|[dpo-bf16](https://modelscope.cn/models/OpenBMB/MiniCPM-2B-dpo-bf16/summary)|[dpo-bf16](https://wisemodel.cn/models/OpenBMB/MiniCPM-2B-dpo-bf16) |[dpo-fp16](https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp16)|[dpo-fp16](https://modelscope.cn/models/OpenBMB/MiniCPM-2B-dpo-fp16/)|[dpo-fp16](https://wisemodel.cn/models/OpenBMB/MiniCPM-2B-dpo-fp16) |[dpo-fp32](https://huggingface.co/openbmb/MiniCPM-2B-dpo-fp32)|[dpo-fp32](https://modelscope.cn/models/OpenBMB/MiniCPM-2B-dpo-fp32)|[dpo-fp32](https://wisemodel.cn/models/OpenBMB/miniCPM-dpo-fp32) + |[v-bf16](https://huggingface.co/openbmb/MiniCPM-V)|[v-bf16](https://modelscope.cn/models/OpenBMB/MiniCPM-V)|[v-bf16](https://wisemodel.cn/models/OpenBMB/MiniCPM-V) +

@@ -86,7 +88,7 @@ python inference.py --model_path --prompt_path prompts/promp ``` #### Huggingface 模型 - +##### MiniCPM-2B * 安装`transformers>=4.36.0`以及`accelerate`后,运行以下代码 ```python from transformers import AutoModelForCausalLM, AutoTokenizer @@ -108,6 +110,31 @@ print(responds) 相对于黄山(海拔1864米),泰山海拔较低,相差约319米。 ``` +##### MiniCPM-V +```python +import torch +from PIL import Image +from transformers import AutoModel, AutoTokenizer + +model_path='openbmb/MiniCPM-V' +model = AutoModel.from_pretrained(model_path, trust_remote_code=True).to(dtype=torch.bfloat16) +tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) +model.eval().cuda() + +image = Image.open('./assets/COCO_test2015_000000262144.jpg').convert('RGB') + +question = '请描述一下该图像' +res, context, _ = model.chat( + image=image, + question=question, + context=None, + tokenizer=tokenizer, + sampling=True, + temperature=0.7 +) +print(res) +``` +

## 评测结果 @@ -156,13 +183,79 @@ print(responds) #### 多模态评测 -|模型|MME(P)|MMB-dev(en)|MMB-dev(zh)|MMMU-val|CMMMU-val| -|-|-|-|-|-|-| -|LLaVA-Phi|1335.1|59.8|/|/|/| -|MobileVLM|1288.9|59.6|/|/|/| -|Imp-v1|1434.0|66.5|/|/|/| -|Qwen-VL-Chat|**1487**|60.6|56.7|**35.9**|30.7 -|**MiniCPM-V**|1446|**67.3**|**61.9**|34.7|**32.1**| +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ModelSizeMMEMMB dev (en)MMB dev (zh)MMMU valCMMMU val
LLaVA-Phi3B133559.8- - -
MobileVLM3B128959.6- - -
Imp-v13B143466.5- - -
Qwen-VL-Chat9.6B148760.6 56.7 35.9 30.7
CogVLM17.4B 1438 63.7 53.8 32.1 -
OmniLMM-3B3B 1452 67.3 61.9 34.7 32.1
+ +
#### DPO评测 diff --git a/assets/COCO_test2015_000000262144.jpg b/assets/COCO_test2015_000000262144.jpg new file mode 100644 index 0000000..012f88d Binary files /dev/null and b/assets/COCO_test2015_000000262144.jpg differ