mirror of
https://github.com/RYDE-WORK/Langchain-Chatchat.git
synced 2026-01-19 21:37:20 +08:00
3acbf4d5d1
* close #1172: 给webui_page/utils添加一些log信息，方便定位错误 * 修复：重建知识库时页面未实时显示进度 * skip model_worker running when using online model api such as chatgpt * 修改知识库管理相关内容: 1.KnowledgeFileModel增加3个字段：file_mtime(文件修改时间),file_size(文件大小),custom_docs(是否使用自定义docs)。为后面比对上传文件做准备。 2.给所有String字段加上长度，防止mysql建表错误(pr#1177) 3.统一[faiss/milvus/pgvector]_kb_service.add_doc接口，使其支持自定义docs 4.为faiss_kb_service增加一些方法，便于调用 5.为KnowledgeFile增加一些方法，便于获取文件信息，缓存file2text的结果。 * 修复/chat/fastchat无法流式输出的问题 * 新增功能： 1、KnowledgeFileModel增加"docs_count"字段，代表该文件加载到向量库中的Document数量，并在WEBUI中进行展示。 2、重建知识库`python init_database.py --recreate-vs`支持多线程。 其它： 统一代码中知识库相关函数用词：file代表一个文件名称或路径，doc代表langchain加载后的Document。部分与API接口有关或含义重叠的函数暂未修改。 --------- Co-authored-by: liunux4odoo <liunux@qq.com>, hongkong9771
45 lines
1.4 KiB
Python
45 lines
1.4 KiB
Python
from server.knowledge_base.migrate import create_tables, reset_tables, folder2db, recreate_all_vs, list_kbs_from_folder
|
|
from configs.model_config import NLTK_DATA_PATH
|
|
import nltk
|
|
nltk.data.path = [NLTK_DATA_PATH] + nltk.data.path
|
|
from startup import dump_server_info
|
|
from datetime import datetime
|
|
|
|
|
|
if __name__ == "__main__":
|
|
import argparse
|
|
|
|
parser = argparse.ArgumentParser()
|
|
parser.formatter_class = argparse.RawTextHelpFormatter
|
|
|
|
parser.add_argument(
|
|
"--recreate-vs",
|
|
action="store_true",
|
|
help=('''
|
|
recreate all vector store.
|
|
use this option if you have copied document files to the content folder, but vector store has not been populated or DEFAUL_VS_TYPE/EMBEDDING_MODEL changed.
|
|
if your vector store is ready with the configs, just skip this option to fill info to database only.
|
|
'''
|
|
)
|
|
)
|
|
args = parser.parse_args()
|
|
|
|
dump_server_info()
|
|
|
|
start_time = datetime.now()
|
|
|
|
if args.recreate_vs:
|
|
reset_tables()
|
|
print("database talbes reseted")
|
|
print("recreating all vector stores")
|
|
recreate_all_vs()
|
|
else:
|
|
create_tables()
|
|
print("database talbes created")
|
|
print("filling kb infos to database")
|
|
for kb in list_kbs_from_folder():
|
|
folder2db(kb, "fill_info_only")
|
|
|
|
end_time = datetime.now()
|
|
print(f"总计用时： {end_time-start_time}")
|