diff --git a/.cursor/rules/git-commit-zh.mdc b/.cursor/rules/git-commit-zh.mdc new file mode 100644 index 0000000..19ddfbf --- /dev/null +++ b/.cursor/rules/git-commit-zh.mdc @@ -0,0 +1,13 @@ +--- +description: 在本仓库执行 git commit 时，提交说明使用中文 +alwaysApply: true +--- + +# Git 提交说明使用中文 + +在本仓库内由助手或开发者写入的 **提交说明**（`git commit -m` 与正文）**须以中文为主**： + +- **第一行**：简短说明本次变更；推荐使用 `type(scope): 描述`（Conventional Commits），其中 **描述为中文**。 +- **正文**（若有）：动机、影响范围、注意事项等，用**完整通顺的中文句**；技术名词、包名、API 名可保留英文。 + +避免整段英文提交说明；避免一句内中英碎片堆砌。与「先对齐再改代码」类流程并存时，仍以本约定为准。 diff --git a/.env.example b/.env.example index b19a7fa..8117e02 100644 --- a/.env.example +++ b/.env.example @@ -2,32 +2,79 @@ # 唯一环境变量模板：复制为同目录 .env 后填写（.env 勿提交仓库） # copy .env.example .env （Windows） # cp .env.example .env （Linux/macOS） +# +# 阅读方式：每节以 “# --- ... ---” 开头，首行说明「干什么、和哪几节互斥」； +# 要换跑法时，按该节里的注释行取消注释，并把其它方案里会冲突的 MA_LLM* / 凭据 改回注释。 # ============================================================================= -# --- 数据工作区根目录（可选）--- -# 不填时默认为本仓库根目录（market_assistant），数据写在 ./data/JD/。 -# 若数据盘与代码分离，可设为绝对路径（须可写，运行时会自动创建 data/JD）。 +# --- 数据工作区根目录（可选；不填=本仓库根目录，数据在 ./data/JD/；数据盘与代码分离时再填绝对路径须可写）--- # LOW_GI_PROJECT_ROOT=D:\data\low-gi-workspace -# --- Django --- +# --- Django（站点密钥与调试；生产须改 SECRET 与 DEBUG）--- DJANGO_SECRET_KEY=please-change-me DJANGO_DEBUG=True DJANGO_ALLOWED_HOSTS=localhost,127.0.0.1 - # 可选：SQLite 绝对路径；不填则使用 backend/db.sqlite3 # DJANGO_SQLITE_PATH=D:\PythonProject\Low GI\market_assistant\backend\db.sqlite3 -# --- 浏览器访问前端时的 Origin（开发：Vite 5173；生产改为实际域名）--- +# --- 浏览器访问前端时的 Origin（开发：Vite 5173；生产改为实际域名；与 CORS/CSRF 成对改）--- CORS_ALLOWED_ORIGINS=http://localhost:5173,http://127.0.0.1:5173 CSRF_TRUSTED_ORIGINS=http://localhost:5173,http://127.0.0.1:5173 -# --- LLM（配料图识别、竞品报告、策略稿；OpenAI 兼容接口，与 AI_crawler 共用本文件）--- +# --- LLM ① 自建/兼容网关（与 pipeline.openai_gateway 共用；商详多模、配料/视觉走下面 OPENAI_* + OPENAI_VISION_MODEL）--- # OPENAI_API_KEY=sk-your-key-here # OPENAI_BASE_URL=https://llm.example.com/v1 # OPENAI_VISION_MODEL=Qwen/Qwen3-Omni-30B-A3B -# 纯文本优先：OPENAI_TEXT_MODEL 或 LLM_TEXT_MODEL；未设则回退到视觉模型名 +# 若 ② 选方案 A：纯文本默认也走 ① 的 Key/基址，但可只用下面两枚变量「只拆纯文本」：任一项不设则回退 ① +# OPENAI_TEXT_API_KEY=（与配料/视觉同网关不同 Key、或分账号） +# OPENAI_TEXT_BASE_URL=（可仅换路径/集群，仍与 ① 同域名） +# 若 OPENAI_BASE_URL=https://llm.rekeymed.com/v1，可用 Key 调 GET /v1/models 看 id；以下为某次查询示例（以实际返回为准）： +# Qwen/Qwen3-Omni-30B-A3B +# openai/gpt-oss-120b +# zai-org/GLM-4.5-Air +# 纯文本专用模型名（不填时回退 OPENAI_VISION_MODEL / LLM_MODEL 等；与多模可不同） # OPENAI_TEXT_MODEL= -# 别名：LLM_API_KEY、LLM_BASE_URL、LLM_MODEL +# 别名：LLM_API_KEY、LLM_BASE_URL、LLM_MODEL、LLM_TEXT_API_KEY、LLM_TEXT_BASE_URL + +# --- LLM ② 纯文本（报告/策略/关键词等 call_llm；与 ②-B～②-D 多选一，只让一条 MA_LLM_TEXT_PROVIDER 生效，换方案时先改/注释本行再启别节）--- +# 方案 A（默认）= 经 openai_gateway：纯文本读 OPENAI_TEXT_*（若有）否则 OPENAI_*；要换 B=官站、C=Kimi、D=DeepSeek 时改下行为 openai_official|kimi|deepseek 等 +MA_LLM_TEXT_PROVIDER=crawler_openai_compatible + +# --- ②-B 纯文本用 OpenAI 官站（仅当 MA=openai_official / openai_chatgpt / chatgpt；与 ②-A、②-C 互斥；凭据与 ① 可不同）--- +# MA_LLM_TEXT_PROVIDER=openai_official +# OPENAI_OFFICIAL_API_KEY=sk-... +# OPENAI_OFFICIAL_BASE_URL=https://api.openai.com/v1 +# OPENAI_OFFICIAL_TEXT_MODEL=gpt-4o-mini +# OPENAI_OFFICIAL_CONTEXT_WINDOW=128000 +# OPENAI_OFFICIAL_TIMEOUT=600 + +# --- ②-C 纯文本用 Kimi / Moonshot 兼容（仅当 MA=kimi / moonshot / kimi_moonshot；与 ②-A、②-B 互斥；配料/视觉仍只走 ① 的 OPENAI_*）--- +# MA_LLM_TEXT_PROVIDER=kimi +# KIMI_API_KEY=sk-... +# KIMI_BASE_URL=https://api.moonshot.cn/v1 +# KIMI_TEXT_MODEL=moonshot-v1-8k +# KIMI_CONTEXT_WINDOW=8192 +# 别名：MOONSHOT_API_KEY、KIMI_MODEL、MOONSHOT_MODEL +# KIMI_TIMEOUT=600 + +# --- ②-D 纯文本用 DeepSeek 官方（仅当 MA=deepseek / deep_seek；与 ②-A～②-C 互斥；配料/视觉仍只走 ① 的 OPENAI_*）--- +# MA_LLM_TEXT_PROVIDER=deepseek +# DEEPSEEK_API_KEY=sk-... +# DEEPSEEK_BASE_URL=https://api.deepseek.com/v1 +# 思考模式（官方 Thinking Mode，默认开）：关=0；力度 high|max（https://api-docs.deepseek.com/guides/thinking_mode） +# DEEPSEEK_THINKING=1 +# DEEPSEEK_REASONING_EFFORT=high +# 未填 DEEPSEEK_TEXT_MODEL 时：开思考默认 deepseek-v4-pro，关思考默认 deepseek-chat +# DEEPSEEK_TEXT_MODEL=deepseek-v4-pro +# DEEPSEEK_CONTEXT_WINDOW=64000 +# 别名：DEEPSEEK_MODEL +# DEEPSEEK_TIMEOUT=600 + +# --- 报告/策略：部分 LLM 块单独温度（默认 0.1；与上面 MA 无冲突）--- +# MA_STRATEGY_LLM_TEMPERATURE=0.1 + +# 默认会剥掉混在「模型 content」里的思考标签块（如 redacted_thinking / think 成对），内部仍可开思考；调试要原样保留时： +# MA_LLM_PRESERVE_THINKING_IN_OUTPUT=1 # --- 可选：流水线侧 LLM 开关 --- # MA_SKIP_LLM_KEYWORD_SUGGEST=1 diff --git a/.gitignore b/.gitignore index b91606d..d6a86c2 100644 --- a/.gitignore +++ b/.gitignore @@ -29,7 +29,10 @@ venv/ .idea/ .vscode/ -.cursor/ +# Cursor 本地状态忽略；项目规则见 .cursor/rules/（可提交） +.cursor/* +!.cursor/rules/ +!.cursor/rules/** frontend/node_modules/ frontend/dist/ diff --git a/README.md b/README.md index b08a492..64849ab 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ | 仓库根 `.env` | 运行时配置（勿提交 Git）；从 `.env.example` 复制 | | `.env.example` | 模板，可随仓库分发 | -Django（`backend/config/settings.py`）与 `backend/crawler_copy/jd_pc_search/AI_crawler.py` 均从**仓库根**的 `.env` 加载。 +Django（`backend/config/settings.py`）与 `python -m pipeline.openai_gateway`（在 `backend` 下执行、用于配料多模态试跑）均从**仓库根**的 `.env` 加载。 **首次编辑建议：** @@ -119,7 +119,7 @@ npm run build 2. **任务与结果**：查看任务状态；成功任务可 **库内浏览**、文件预览与下载、导出。 3. **报告生成**：配置统计规则并重新生成分析报告文件。 4. **报告查看**：在线预览、单文件下载、加载结构化摘要、**一键下载简报包**（ZIP）。 -5. **结构化摘要**：与报告同口径的规则化 JSON，供联调或其它工具使用。 +5. **结构化摘要**：与报告**同一套计数规则**的规则化 JSON，供联调或其它工具使用。 6. **市场策略制定**：选成功任务，可选填业务备注，生成策略向 Markdown（目标、战场、定位选项、支柱与行动；规则版、非大模型）。 任务**成功结束后**会自动执行入库；也可在「库内浏览」里从批次目录重新入库。 diff --git a/backend/crawler_copy/jd_pc_search/comment/jd_h5_item_comment_requests.py b/backend/crawler_copy/jd_pc_search/comment/jd_h5_item_comment_requests.py index 1923828..56e5092 100644 --- a/backend/crawler_copy/jd_pc_search/comment/jd_h5_item_comment_requests.py +++ b/backend/crawler_copy/jd_pc_search/comment/jd_h5_item_comment_requests.py @@ -12,17 +12,18 @@ 用法（本仓库默认）: 修改下方「运行配置」后 ``python jd_h5_item_comment_requests.py``（无命令行参数）。 + +**模块划分**：评价 JSON 解析见 ``jd_item_comment_parse.py``；CSV/JSONL 落盘见 ``jd_item_comment_export.py``； +本文件保留 Node 签请求、Playwright 采集与 ``main``，并 re-export 解析/导出符号以兼容 ``jd_keyword_pipeline`` 等导入。 """ from __future__ import annotations -import csv import json import random import subprocess import sys import time -from io import StringIO from pathlib import Path from types import SimpleNamespace from typing import Any @@ -32,10 +33,27 @@ from playwright.sync_api import sync_playwright _JD_PKG_ROOT = Path(__file__).resolve().parent.parent if str(_JD_PKG_ROOT) not in sys.path: sys.path.insert(0, str(_JD_PKG_ROOT)) +_JD_COMMENT_DIR = Path(__file__).resolve().parent +if str(_JD_COMMENT_DIR) not in sys.path: + sys.path.insert(0, str(_JD_COMMENT_DIR)) +_BACKEND_ROOT = Path(__file__).resolve().parents[3] +if str(_BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(_BACKEND_ROOT)) from common.jd_delay_utils import parse_request_delay_range from _low_gi_root import low_gi_project_root # noqa: E402 -_JD_COMMENT_DIR = Path(__file__).resolve().parent +from jd_item_comment_export import ( # noqa: E402 + append_comments_jsonl, + write_comments_flat_csv, +) +from jd_item_comment_parse import ( # noqa: E402 + category_and_first_guid_from_lego, + extract_comment_rows_from_jsonl_file, + extract_comment_rows_from_parsed, + jd_business_ok, + loads_jd_plain_json, + parse_list_pages_spec, +) # --------------------------------------------------------------------------- # 运行配置（按需改这里） @@ -189,40 +207,6 @@ def _sleep_between_jd_requests( time.sleep(sec) -def parse_list_pages_spec(spec: str) -> list[str]: - """ - --list-pages：``1-5`` → 1..5；``1,3,5`` → 单页序列；单数字 ``2`` → [\"2\"]。 - """ - s = (spec or "").strip() - if not s: - return ["1"] - if "," in s: - return [p.strip() for p in s.split(",") if p.strip()] - if "-" in s: - parts = s.split("-", 1) - lo, hi = int(parts[0].strip()), int(parts[1].strip()) - if lo > hi: - lo, hi = hi, lo - return [str(i) for i in range(lo, hi + 1)] - return [s] - - -def category_and_first_guid_from_lego(parsed: Any) -> tuple[str, str]: - """从 getLegoWareDetailComment 的 commentInfoList[0] 取 category（maidianInfo 前缀）与 guid。""" - if not isinstance(parsed, dict): - return "", "" - lst = parsed.get("commentInfoList") - if not isinstance(lst, list) or not lst: - return "", "" - first = lst[0] - if not isinstance(first, dict): - return "", "" - guid = str(first.get("guid") or "").strip() - maidian = str(first.get("maidianInfo") or "").strip() - category = maidian.split("_", 1)[0].strip() if maidian else "" - return category, guid - - def _read_sku_lines(path: str) -> list[str]: p = Path(path) if not p.is_file(): @@ -237,173 +221,6 @@ def _read_sku_lines(path: str) -> list[str]: return out -def _loads_jd_plain_json(text: str) -> Any: - s = (text or "").strip() - if not s: - return None - try: - return json.loads(s) - except json.JSONDecodeError: - return None - - -def _jd_business_ok(parsed: Any) -> bool: - if not isinstance(parsed, dict): - return False - if parsed.get("success") is False: - return False - c = parsed.get("code") - if c is None: - return True - return c == 0 or str(c) == "0" - - -def _clean_text(v: Any) -> str: - if v is None: - return "" - s = str(v).strip() - return " ".join(s.split()) if s else "" - - -def _large_pic_urls_from_picture_list(pil: Any) -> list[str]: - out: list[str] = [] - if not isinstance(pil, list): - return out - for p in pil: - if not isinstance(p, dict): - continue - u = p.get("largePicURL") or p.get("largePicUrl") - if u: - t = str(u).strip() - if t and t not in out: - out.append(t) - return out - - -def _is_jd_single_comment_dict(d: dict) -> bool: - """区分「一条评价」与标签/楼层等对象（getCommentListPage 里多为 commentInfo 扁平结构）。""" - cid = d.get("commentId") - if cid is None or str(cid).strip() == "": - return False - if d.get("userNickName") is None and not ( - d.get("tagCommentContent") or d.get("commentData") - ): - return False - return True - - -def _walk_collect_comment_dicts(obj: Any, acc: list[dict[str, Any]]) -> None: - """深度遍历 JSON，收集所有像单条评价的 dict（含 Lego 的 commentInfoList 项与列表页的 commentInfo）。""" - if isinstance(obj, dict): - if _is_jd_single_comment_dict(obj): - acc.append(obj) - for v in obj.values(): - _walk_collect_comment_dicts(v, acc) - elif isinstance(obj, list): - for x in obj: - _walk_collect_comment_dicts(x, acc) - - -def _row_from_comment_dict(sku: str, item: dict[str, Any]) -> dict[str, Any]: - text = _clean_text( - item.get("tagCommentContent") or item.get("commentData") - ) - buy = _clean_text( - item.get("buyCountText") or item.get("repurchaseInfo") - ) - date = _clean_text( - item.get("commentDate") or item.get("newCommentDate") - ) - return { - "sku": str(sku).strip(), - "commentId": str(item.get("commentId") or "").strip(), - "userNickName": _clean_text(item.get("userNickName")), - "tagCommentContent": text, - "commentDate": date, - "buyCountText": buy, - "largePicURLs": _large_pic_urls_from_picture_list( - item.get("pictureInfoList") - ), - "commentScore":str(item.get("commentScore") or "").strip(), - } - - -def extract_comment_rows_from_parsed(sku: str, parsed: Any) -> list[dict[str, Any]]: - """ - 从整段 parsed 深度遍历抽取评价： - - getLegoWareDetailComment：commentInfoList / lastCommentInfoList - - getCommentListPage：result.floors → data 里 { commentInfo: {...} } 已拍平为内层字段，同上 - """ - if not isinstance(parsed, dict): - return [] - acc: list[dict[str, Any]] = [] - _walk_collect_comment_dicts(parsed, acc) - seen: set[str] = set() - rows: list[dict[str, Any]] = [] - for item in acc: - cid = str(item.get("commentId") or "").strip() - dedup_key = f"{sku}:{cid}" if cid else f"{sku}:{id(item)}" - if dedup_key in seen: - continue - seen.add(dedup_key) - rows.append(_row_from_comment_dict(sku, item)) - return rows - - -def _comment_flat_fieldnames() -> list[str]: - return [ - "sku", - "commentId", - "userNickName", - "tagCommentContent", - "commentDate", - "buyCountText", - "largePicURLs", - "commentScore", - ] - - -def _write_comments_csv(path: Path, rows: list[dict[str, Any]]) -> None: - path.parent.mkdir(parents=True, exist_ok=True) - buf = StringIO() - fn = _comment_flat_fieldnames() - w = csv.DictWriter(buf, fieldnames=fn, extrasaction="ignore") - w.writeheader() - for r in rows: - line = {k: r.get(k, "") for k in fn} - line["largePicURLs"] = json.dumps( - r.get("largePicURLs") or [], ensure_ascii=False - ) - w.writerow(line) - path.write_text("\ufeff" + buf.getvalue(), encoding="utf-8") - - -def write_comments_flat_csv(path: Path | str, rows: list[dict[str, Any]]) -> None: - """与 ``COMMENTS_OUT`` 为 ``.csv`` 时相同格式（UTF-8 BOM），供流水线等复用。""" - _write_comments_csv(Path(path), rows) - - -def _append_comments_jsonl(f, rows: list[dict[str, Any]]) -> None: - for r in rows: - f.write(json.dumps(r, ensure_ascii=False) + "\n") - - -def _extract_rows_from_jsonl_file(path: Path) -> list[dict[str, Any]]: - all_rows: list[dict[str, Any]] = [] - for line in path.read_text(encoding="utf-8").splitlines(): - s = line.strip() - if not s or s.startswith("#"): - continue - try: - rec = json.loads(s) - except json.JSONDecodeError: - continue - sku = str(rec.get("sku") or "").strip() - parsed = rec.get("parsed") - all_rows.extend(extract_comment_rows_from_parsed(sku, parsed)) - return all_rows - - def main() -> None: args = SimpleNamespace( sku=(SKU or "").strip(), @@ -436,14 +253,14 @@ def main() -> None: print("离线模式：请配置 FROM_JSONL 与 COMMENTS_OUT", file=sys.stderr) sys.exit(2) co_path = Path(comments_out) - rows = _extract_rows_from_jsonl_file(Path(from_jsonl)) + rows = extract_comment_rows_from_jsonl_file(Path(from_jsonl)) suf = co_path.suffix.lower() if suf == ".csv": - _write_comments_csv(co_path, rows) + write_comments_flat_csv(co_path, rows) elif suf == ".jsonl": co_path.parent.mkdir(parents=True, exist_ok=True) with co_path.open("w", encoding="utf-8") as cf: - _append_comments_jsonl(cf, rows) + append_comments_jsonl(cf, rows) else: print("COMMENTS_OUT 请使用 .csv 或 .jsonl 扩展名", file=sys.stderr) sys.exit(2) @@ -526,13 +343,13 @@ def main() -> None: print(f"[京东] HTTP {status}，--raise-http 已启用", file=sys.stderr) sys.exit(1) - parsed = _loads_jd_plain_json(text) + parsed = loads_jd_plain_json(text) http_ok = 200 <= status < 300 row = { "sku": sku, "http_status": status, "http_ok": http_ok, - "ok": http_ok and _jd_business_ok(parsed), + "ok": http_ok and jd_business_ok(parsed), "parsed": parsed, "raw": text if parsed is None else None, } @@ -552,7 +369,7 @@ def main() -> None: flat = extract_comment_rows_from_parsed(sku, parsed) if comments_path is not None: if comments_jsonl_f is not None: - _append_comments_jsonl(comments_jsonl_f, flat) + append_comments_jsonl(comments_jsonl_f, flat) else: comments_csv_rows.extend(flat) @@ -621,7 +438,7 @@ def main() -> None: file=sys.stderr, ) sys.exit(1) - parsed_p = _loads_jd_plain_json(text_p) + parsed_p = loads_jd_plain_json(text_p) http_ok_p = 200 <= st_p < 300 row_p = { "sku": sku, @@ -630,7 +447,7 @@ def main() -> None: "http_status": st_p, "http_ok": http_ok_p, "ok": http_ok_p - and _jd_business_ok(parsed_p), + and jd_business_ok(parsed_p), "parsed": parsed_p, "raw": text_p if parsed_p is None else None, } @@ -645,7 +462,7 @@ def main() -> None: ) if comments_path is not None: if comments_jsonl_f is not None: - _append_comments_jsonl( + append_comments_jsonl( comments_jsonl_f, flat_p ) else: @@ -659,7 +476,7 @@ def main() -> None: if comments_jsonl_f: comments_jsonl_f.close() if comments_path is not None and comments_path.suffix.lower() == ".csv": - _write_comments_csv(comments_path, comments_csv_rows) + write_comments_flat_csv(comments_path, comments_csv_rows) if __name__ == "__main__": diff --git a/backend/crawler_copy/jd_pc_search/comment/jd_item_comment_export.py b/backend/crawler_copy/jd_pc_search/comment/jd_item_comment_export.py new file mode 100644 index 0000000..f52e31a --- /dev/null +++ b/backend/crawler_copy/jd_pc_search/comment/jd_item_comment_export.py @@ -0,0 +1,46 @@ +# -*- coding: utf-8 -*- +""" +评价扁平结果**落盘**（UTF-8 BOM CSV / JSONL 行追加）。 + +解析见 ``jd_item_comment_parse``。 +""" +from __future__ import annotations + +import csv +import json +import sys +from io import StringIO +from pathlib import Path +from typing import Any + +_BACKEND_ROOT = Path(__file__).resolve().parents[3] +if str(_BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(_BACKEND_ROOT)) +from pipeline.csv.schema import COMMENT_CSV_COLUMNS, COMMENT_ROW_DICT_KEYS # noqa: E402 + + +def write_comments_flat_csv(path: Path | str, rows: list[dict[str, Any]]) -> None: + """与 COMMENTS_OUT 为 ``.csv`` 时相同格式（UTF-8 BOM），供流水线等复用。""" + _write_comments_csv(Path(path), rows) + + +def _write_comments_csv(path: Path, rows: list[dict[str, Any]]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + buf = StringIO() + fn = list(COMMENT_CSV_COLUMNS) + w = csv.DictWriter(buf, fieldnames=fn, extrasaction="ignore") + w.writeheader() + for r in rows: + line: dict[str, Any] = {} + for h, api_k in zip(COMMENT_CSV_COLUMNS, COMMENT_ROW_DICT_KEYS): + if api_k == "largePicURLs": + line[h] = json.dumps(r.get("largePicURLs") or [], ensure_ascii=False) + else: + line[h] = r.get(api_k, "") + w.writerow(line) + path.write_text("\ufeff" + buf.getvalue(), encoding="utf-8") + + +def append_comments_jsonl(f: Any, rows: list[dict[str, Any]]) -> None: + for r in rows: + f.write(json.dumps(r, ensure_ascii=False) + "\n") diff --git a/backend/crawler_copy/jd_pc_search/comment/jd_item_comment_parse.py b/backend/crawler_copy/jd_pc_search/comment/jd_item_comment_parse.py new file mode 100644 index 0000000..f72fec1 --- /dev/null +++ b/backend/crawler_copy/jd_pc_search/comment/jd_item_comment_parse.py @@ -0,0 +1,174 @@ +# -*- coding: utf-8 -*- +""" +京东商品评价 JSON 的**纯解析**：从 Lego / 列表页响应中抽取扁平评价行。 + +请求签名与 Playwright 见 ``jd_h5_item_comment_requests``。 +""" +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +def parse_list_pages_spec(spec: str) -> list[str]: + """ + --list-pages：``1-5`` → 1..5；``1,3,5`` → 单页序列；单数字 ``2`` → [\"2\"]。 + """ + s = (spec or "").strip() + if not s: + return ["1"] + if "," in s: + return [p.strip() for p in s.split(",") if p.strip()] + if "-" in s: + parts = s.split("-", 1) + lo, hi = int(parts[0].strip()), int(parts[1].strip()) + if lo > hi: + lo, hi = hi, lo + return [str(i) for i in range(lo, hi + 1)] + return [s] + + +def category_and_first_guid_from_lego(parsed: Any) -> tuple[str, str]: + """从 getLegoWareDetailComment 的 commentInfoList[0] 取 category（maidianInfo 前缀）与 guid。""" + if not isinstance(parsed, dict): + return "", "" + lst = parsed.get("commentInfoList") + if not isinstance(lst, list) or not lst: + return "", "" + first = lst[0] + if not isinstance(first, dict): + return "", "" + guid = str(first.get("guid") or "").strip() + maidian = str(first.get("maidianInfo") or "").strip() + category = maidian.split("_", 1)[0].strip() if maidian else "" + return category, guid + + +def loads_jd_plain_json(text: str) -> Any: + s = (text or "").strip() + if not s: + return None + try: + return json.loads(s) + except json.JSONDecodeError: + return None + + +def jd_business_ok(parsed: Any) -> bool: + if not isinstance(parsed, dict): + return False + if parsed.get("success") is False: + return False + c = parsed.get("code") + if c is None: + return True + return c == 0 or str(c) == "0" + + +def _clean_text(v: Any) -> str: + if v is None: + return "" + s = str(v).strip() + return " ".join(s.split()) if s else "" + + +def _large_pic_urls_from_picture_list(pil: Any) -> list[str]: + out: list[str] = [] + if not isinstance(pil, list): + return out + for p in pil: + if not isinstance(p, dict): + continue + u = p.get("largePicURL") or p.get("largePicUrl") + if u: + t = str(u).strip() + if t and t not in out: + out.append(t) + return out + + +def _is_jd_single_comment_dict(d: dict) -> bool: + """区分「一条评价」与标签/楼层等对象（getCommentListPage 里多为 commentInfo 扁平结构）。""" + cid = d.get("commentId") + if cid is None or str(cid).strip() == "": + return False + if d.get("userNickName") is None and not ( + d.get("tagCommentContent") or d.get("commentData") + ): + return False + return True + + +def _walk_collect_comment_dicts(obj: Any, acc: list[dict[str, Any]]) -> None: + """深度遍历 JSON，收集所有像单条评价的 dict（含 Lego 的 commentInfoList 项与列表页的 commentInfo）。""" + if isinstance(obj, dict): + if _is_jd_single_comment_dict(obj): + acc.append(obj) + for v in obj.values(): + _walk_collect_comment_dicts(v, acc) + elif isinstance(obj, list): + for x in obj: + _walk_collect_comment_dicts(x, acc) + + +def _row_from_comment_dict(sku: str, item: dict[str, Any]) -> dict[str, Any]: + text = _clean_text( + item.get("tagCommentContent") or item.get("commentData") + ) + buy = _clean_text( + item.get("buyCountText") or item.get("repurchaseInfo") + ) + date = _clean_text( + item.get("commentDate") or item.get("newCommentDate") + ) + return { + "sku": str(sku).strip(), + "commentId": str(item.get("commentId") or "").strip(), + "userNickName": _clean_text(item.get("userNickName")), + "tagCommentContent": text, + "commentDate": date, + "buyCountText": buy, + "largePicURLs": _large_pic_urls_from_picture_list( + item.get("pictureInfoList") + ), + "commentScore": str(item.get("commentScore") or "").strip(), + } + + +def extract_comment_rows_from_parsed(sku: str, parsed: Any) -> list[dict[str, Any]]: + """ + 从整段 parsed 深度遍历抽取评价： + - getLegoWareDetailComment：commentInfoList / lastCommentInfoList + - getCommentListPage：result.floors → data 里 { commentInfo: {...} } 已拍平为内层字段，同上 + """ + if not isinstance(parsed, dict): + return [] + acc: list[dict[str, Any]] = [] + _walk_collect_comment_dicts(parsed, acc) + seen: set[str] = set() + rows: list[dict[str, Any]] = [] + for item in acc: + cid = str(item.get("commentId") or "").strip() + dedup_key = f"{sku}:{cid}" if cid else f"{sku}:{id(item)}" + if dedup_key in seen: + continue + seen.add(dedup_key) + rows.append(_row_from_comment_dict(sku, item)) + return rows + + +def extract_comment_rows_from_jsonl_file(path: Path) -> list[dict[str, Any]]: + """离线：从采集 JSONL 每行 { sku, parsed } 抽取扁平评价行。""" + all_rows: list[dict[str, Any]] = [] + for line in path.read_text(encoding="utf-8").splitlines(): + s = line.strip() + if not s or s.startswith("#"): + continue + try: + rec = json.loads(s) + except json.JSONDecodeError: + continue + sku = str(rec.get("sku") or "").strip() + parsed = rec.get("parsed") + all_rows.extend(extract_comment_rows_from_parsed(sku, parsed)) + return all_rows diff --git a/backend/crawler_copy/jd_pc_search/detail/jd_detail_buyer_extraction.py b/backend/crawler_copy/jd_pc_search/detail/jd_detail_buyer_extraction.py new file mode 100644 index 0000000..b1a294b --- /dev/null +++ b/backend/crawler_copy/jd_pc_search/detail/jd_detail_buyer_extraction.py @@ -0,0 +1,606 @@ +# -*- coding: utf-8 -*- +""" +商详 JSON → **购买者可理解的优惠与权益摘要**（用于促销策略分析，而非字段堆砌）。 + +设计原则： +- **先回答「我买这件能怎样」**：到手价相对标价、是否有券/补贴提示、硬约束（如不可用东券）。 +- **再列可感知的物流/售后权益**：价保、退换、送达等，用短标签 + 一句说明。 +- **原始杂乱节点**（如 abData、埋点）不进入摘要。 +- **优惠拆解**（若存在 ``preferenceVO.preferencePopUp.expression``）：购买立减、红包抵扣金额、券/促销/国补占位等，与腰带价、到手价**对照阅读**。 + +输入为 ``pc_detailpage_wareBusiness`` 类接口的 **JSON 根对象**（与 ``flatten_ware_business`` 同源）； +若你保存的是完整响应，根级字段与之一致即可。 +""" +from __future__ import annotations + +import json +import re +from typing import Any + +from pipeline.csv.schema import strip_buyer_ranking_line_prefix + + +def _s(x: Any) -> str: + if x is None: + return "" + return str(x).strip() + + +def _strip_html(text: str, *, max_len: int = 800) -> str: + if not text: + return "" + t = re.sub(r"<[^>]+>", " ", text) + t = " ".join(t.split()).strip() + return t[:max_len] if max_len > 0 else t + + +def _parse_float_maybe(s: str) -> float | None: + t = (s or "").strip().replace(",", "") + if not t: + return None + m = re.search(r"(\d+(?:\.\d+)?)", t) + if not m: + return None + try: + return float(m.group(1)) + except ValueError: + return None + + +def _price_from_gather_vo(obj: dict[str, Any]) -> dict[str, Any]: + """warePriceGatherVO.priceItemList → 到手价 / 京东价等。""" + out: dict[str, Any] = { + "hand_price": "", + "hand_label": "", + "jd_price": "", + "jd_price_hit_line": False, + "raw_items": [], + } + wpg = obj.get("warePriceGatherVO") + if not isinstance(wpg, dict): + return out + pil = wpg.get("priceItemList") + if not isinstance(pil, list): + return out + for it in pil: + if not isinstance(it, dict): + continue + ptype = _s(it.get("priceType")) + price = _s(it.get("price")) + hit_line = bool(it.get("hitLine")) + labels: list[str] = [] + for lb in it.get("priceLabelList") or []: + if isinstance(lb, dict) and _s(lb.get("labelTxt")): + labels.append(_s(lb.get("labelTxt"))) + out["raw_items"].append( + {"priceType": ptype, "price": price, "hitLine": hit_line, "labels": labels} + ) + if ptype == "finalPrice" and price: + out["hand_price"] = price + out["hand_label"] = labels[0] if labels else "到手价" + if ptype == "jdPrice" and price: + out["jd_price"] = price + out["jd_price_hit_line"] = hit_line + return out + + +def _price_from_classic_price_block(obj: dict[str, Any]) -> dict[str, Any]: + """兼容仅有 price.finalPrice / price.p 的旧结构。""" + price = obj.get("price") + if not isinstance(price, dict): + return {} + fp = price.get("finalPrice") + fp = fp if isinstance(fp, dict) else {} + hand = _s(fp.get("price")) or _s(price.get("p")) + jd = _s(price.get("op")) or _s(price.get("p")) + return { + "hand_price": hand, + "hand_label": "到手价", + "jd_price": jd if jd != hand else "", + "jd_price_hit_line": bool(_s(price.get("op"))), + } + + +def _preference_bundle(obj: dict[str, Any]) -> dict[str, Any]: + """ + 详情页「优惠弹层」同源：立减、红包、券、促销、国补占位；以及包邮/返豆等短标签。 + 对应前端 preferenceVO（与 warePriceGatherVO 互补）。 + """ + out: dict[str, Any] = { + "expression": {}, + "subtrahends": [], + "shared_labels": [], + "popup_preferences": [], + } + pv = obj.get("preferenceVO") + if not isinstance(pv, dict): + return out + for lb in pv.get("againSharedLabel") or []: + if isinstance(lb, dict): + name = _s(lb.get("labelName")) + if name: + out["shared_labels"].append(name[:120]) + ppop = pv.get("preferencePopUp") + if not isinstance(ppop, dict): + return out + for it in ppop.get("againSharedPreference") or []: + if not isinstance(it, dict): + continue + line = _s(it.get("text")) + val = _s(it.get("value")) + st = _s(it.get("shortText")) + if st and val: + out["popup_preferences"].append(f"{st}：{val}"[:300]) + elif line and val: + out["popup_preferences"].append(f"{line}：{val}"[:300]) + elif val: + out["popup_preferences"].append(val[:300]) + ex = ppop.get("expression") + if not isinstance(ex, dict): + return out + out["expression"] = { + "base_price": _s(ex.get("basePrice"))[:32], + "discount_desc": _s(ex.get("discountDesc"))[:32], + "discount_amount": _s(ex.get("discountAmount"))[:32], + "red_amount": _s(ex.get("redAmount"))[:32], + "coupon_amount": _s(ex.get("couponAmount"))[:32], + "promotion_amount": _s(ex.get("promotionAmount"))[:32], + "gov_amount": _s(ex.get("govAmount"))[:32], + } + for sub in ex.get("subtrahends") or []: + if not isinstance(sub, dict): + continue + out["subtrahends"].append( + { + "category": _s(sub.get("topDesc"))[:32], + "description": _s(sub.get("preferenceDesc"))[:200], + "amount": _s(sub.get("preferenceAmount"))[:32], + "preference_type": _s(sub.get("preferenceType"))[:16], + } + ) + return out + + +def _gov_support_surface(obj: dict[str, Any]) -> dict[str, Any]: + """政府补贴/国补腰带等（页面开关与展示，非到手计算结果）。""" + g = obj.get("govSupportInfo") + if not isinstance(g, dict): + return {} + return { + "gov_subsidy_flag": bool(g.get("govSubsidy")), + "gov_support_flag": bool(g.get("govSupport")), + "subsidy_type": _s(g.get("subsidyType"))[:64], + "subsidy_scene": _s(g.get("subsidyScene"))[:32], + "right_text": _s(g.get("rightText"))[:200], + "belt_banner_url": _s(g.get("beltBanner"))[:300], + } + + +def _best_promotion_summary(obj: dict[str, Any]) -> dict[str, Any]: + bp = obj.get("bestPromotion") + if not isinstance(bp, dict): + return {"purchase_price": "", "can_get_coupon": []} + cgc = bp.get("canGetCoupon") + coupons: list[str] = [] + if isinstance(cgc, list): + for c in cgc[:12]: + if isinstance(c, dict): + t = _s(c.get("name") or c.get("desc") or c.get("couponTitle")) + if t: + coupons.append(t[:120]) + elif c: + coupons.append(_s(str(c))[:120]) + return { + "purchase_price": _s(bp.get("purchasePrice"))[:32], + "can_get_coupon": coupons, + } + + +def _warm_tips(obj: dict[str, Any]) -> list[str]: + tips: list[str] = [] + wv = obj.get("warmTipVO") + if isinstance(wv, dict): + for t in wv.get("tips") or []: + if isinstance(t, dict): + txt = _s(t.get("tipTxt")) + if txt: + tips.append(txt[:300]) + prom = obj.get("promotion") + if isinstance(prom, dict): + pr = _s(prom.get("prompt")) + if pr and pr not in tips: + tips.append(pr[:300]) + return tips + + +def _rankings(obj: dict[str, Any]) -> list[str]: + out: list[str] = [] + rl = obj.get("rankInfoList") + if isinstance(rl, list): + for it in rl[:8]: + if isinstance(it, dict): + n = _s(it.get("rankName")) + if n: + out.append(n[:200]) + return out + + +def _service_tag_labels(obj: dict[str, Any]) -> list[str]: + """主图区服务标：短标签，去重。""" + seen: set[str] = set() + labels: list[str] = [] + st = obj.get("serviceTagsVO") + if isinstance(st, dict): + for key in ("basicNewIcons", "basicIcons"): + for it in st.get(key) or []: + if isinstance(it, dict): + tx = _s(it.get("text")) + if tx and tx not in seen: + seen.add(tx) + labels.append(tx[:80]) + return labels[:16] + + +def _service_tag_details(obj: dict[str, Any], *, limit: int = 6) -> list[dict[str, str]]: + """每条：标题 + 一句「你能获得什么」说明（来自 tip，已去 HTML）。""" + out: list[dict[str, str]] = [] + st = obj.get("serviceTagsVO") + if not isinstance(st, dict): + return out + for key in ("basicNewIcons", "basicIcons"): + for it in st.get(key) or []: + if not isinstance(it, dict): + continue + title = _s(it.get("text")) + if not title: + continue + tip = _strip_html(_s(it.get("tip")), max_len=400) + out.append({"title": title[:80], "what_you_get": tip}) + if len(out) >= limit: + return out + return out + + +def _delivery_one_liner(obj: dict[str, Any]) -> str: + si = obj.get("stockInfo") + if isinstance(si, dict): + pr = _s(si.get("promiseResult") or si.get("promiseInfoText")) + if pr: + return _strip_html(pr, max_len=500) + sv = obj.get("stockVO") + if isinstance(sv, dict): + pr = _s(sv.get("promiseInfoText")) + if pr: + return _strip_html(pr, max_len=500) + return "" + + +def _logistics_icons(obj: dict[str, Any]) -> list[str]: + texts: list[str] = [] + sv = obj.get("stockVO") + if isinstance(sv, dict): + pil = sv.get("promiseIconList") + if isinstance(pil, list): + for it in pil[:10]: + if isinstance(it, dict) and _s(it.get("text")): + texts.append(_s(it.get("text"))[:40]) + return texts + + +def _bottom_cta_hint(obj: dict[str, Any]) -> str: + bb = obj.get("bottomBtnVO") + if not isinstance(bb, dict): + return "" + items = bb.get("bottomBtnItems") + if not isinstance(items, list) or not items: + return "" + it0 = items[0] + if not isinstance(it0, dict): + return "" + bs = it0.get("buttonStyle") + if not isinstance(bs, dict): + return "" + tf = bs.get("textFormat") + if not isinstance(tf, dict): + return "" + return _strip_html(_s(tf.get("text")), max_len=200) + + +def _belt_surface(obj: dict[str, Any]) -> str: + bi = obj.get("beltBannerInfo") + if isinstance(bi, dict): + r = _s(bi.get("bannerRightText") or bi.get("bannerRightTextOfficialDiscount")) + if r: + return r[:120] + return "" + + +def _enterprise_hint(obj: dict[str, Any]) -> str: + ch = obj.get("corpHighInfo") + if isinstance(ch, dict): + return _s(ch.get("tip"))[:300] + return "" + + +def _user_segment_flags(obj: dict[str, Any]) -> dict[str, bool]: + ui = obj.get("userInfo") + new_people = bool(ui.get("newPeople")) if isinstance(ui, dict) else False + return {"new_people": new_people} + + +def _build_summary_lines( + *, + gather: dict[str, Any], + classic: dict[str, Any], + bp_sum: dict[str, Any], + pref: dict[str, Any], + gov_surf: dict[str, Any], + warm: list[str], + delivery: str, + cta: str, + flags: dict[str, bool], +) -> list[str]: + """3～6 条中文短句，面向「我买能怎样」。""" + lines: list[str] = [] + + hand = gather.get("hand_price") or classic.get("hand_price") or bp_sum.get("purchase_price") + jd_p = gather.get("jd_price") or classic.get("jd_price") + hl = gather.get("hand_label") or classic.get("hand_label") or "到手价" + + if hand: + if jd_p and _parse_float_maybe(jd_p) and _parse_float_maybe(hand): + a, b = _parse_float_maybe(jd_p), _parse_float_maybe(hand) + if a is not None and b is not None and a > b: + diff = round(a - b, 2) + lines.append( + f"当前展示「{hl}」约 {hand} 元，相对页面标价 {jd_p} 元约低 {diff} 元（以结算页为准）。" + ) + else: + lines.append( + f"当前展示「{hl}」约 {hand} 元（页面标价 {jd_p} 元，以结算页为准）。" + ) + else: + lines.append(f"当前展示「{hl}」约 {hand} 元（以结算页为准）。") + + ex = pref.get("expression") or {} + dd = _s(ex.get("discount_desc")) + da = _s(ex.get("discount_amount")) + ra = _s(ex.get("red_amount")) + subs = pref.get("subtrahends") or [] + if dd or da or ra or subs: + parts: list[str] = [] + if dd and da: + parts.append(f"{dd}约 {da} 元") + elif dd: + parts.append(dd) + for s in subs: + if not isinstance(s, dict): + continue + cat = _s(s.get("category")) + desc = _s(s.get("description")) + if cat and desc: + parts.append(f"{cat}：{desc}") + elif desc: + parts.append(desc) + if ra and not subs: + parts.append(f"红包类约 {ra} 元") + elif ra and subs and not any("红包" in _s(p) for p in parts): + parts.append(f"红包类约 {ra} 元") + if parts: + lines.append( + "详情页优惠拆解（与腰带/到手价对照）：" + + "；".join(parts[:5]) + + "（以结算页为准）。" + ) + + if pref.get("shared_labels"): + lines.append( + "其他权益标签：" + + "、".join(pref["shared_labels"][:4]) + + "。" + ) + if pref.get("popup_preferences"): + lines.append( + "其他权益：" + + "；".join(pref["popup_preferences"][:4]) + + "。" + ) + + if gov_surf.get("gov_subsidy_flag") or gov_surf.get("gov_support_flag"): + rt = _s(gov_surf.get("right_text")) + if rt: + lines.append(f"国补/政府补贴相关展示：{_strip_html(rt, max_len=160)}（以活动规则为准）。") + else: + lines.append("页面含政府补贴/国补相关入口（以活动规则与结算为准）。") + + if flags.get("new_people") and cta: + lines.append( + f"新人相关文案：{_strip_html(cta)}（若你不是新人，价格与活动可能不同）。" + ) + elif cta: + lines.append(f"主按钮文案：{_strip_html(cta)}。") + + if warm: + lines.append("购买限制与提示：" + "；".join(warm[:4]) + "。") + + # 榜单仅走 visibility.rankings → buyer_ranking_line_from_profile，避免 buyer_promo 重复 + + if delivery: + lines.append("送达：" + delivery + "。") + + if bp_sum.get("can_get_coupon"): + lines.append( + "可领券/活动入口（摘要）：" + + "；".join(bp_sum["can_get_coupon"][:4]) + + "。" + ) + + return lines[:8] + + +def extract_buyer_offer_profile(obj: Any) -> dict[str, Any]: + """ + 从商详 JSON 根对象生成结构化摘要。 + + 返回 dict 可直接 ``json.dumps(..., ensure_ascii=False)``；无有效输入时仍返回带 ``schema_version`` 的空壳。 + """ + empty: dict[str, Any] = { + "schema_version": 1, + "sku_id_hint": "", + "price_snapshot": {}, + "discount_mechanism": {}, + "gov_support_surface": {}, + "best_promotion": {}, + "purchase_constraints": {"warm_tips": [], "ware_flags": {}}, + "marketing_surface": {}, + "visibility": {"rankings": []}, + "after_sales_short_labels": [], + "after_sales_details": [], + "delivery": {"one_liner": "", "logistics_icons": []}, + "buyer_summary_lines": [], + "notes": "摘要由规则生成，结算价与活动以京东下单页为准。", + } + if not isinstance(obj, dict): + return empty + + pc = obj.get("pageConfigVO") + sku_hint = "" + if isinstance(pc, dict): + sku_hint = _s(pc.get("skuid")) + + gather = _price_from_gather_vo(obj) + classic = _price_from_classic_price_block(obj) + if not gather.get("hand_price") and classic.get("hand_price"): + gather["hand_price"] = classic["hand_price"] + gather["hand_label"] = classic.get("hand_label") or gather.get("hand_label") + if not gather.get("jd_price") and classic.get("jd_price"): + gather["jd_price"] = classic["jd_price"] + + bp_sum = _best_promotion_summary(obj) + pref_bundle = _preference_bundle(obj) + gov_surf = _gov_support_surface(obj) + warm = _warm_tips(obj) + rankings = _rankings(obj) + short_labels = _service_tag_labels(obj) + details = _service_tag_details(obj) + delivery = _delivery_one_liner(obj) + log_icons = _logistics_icons(obj) + cta = _bottom_cta_hint(obj) + belt = _belt_surface(obj) + ent = _enterprise_hint(obj) + flags = _user_segment_flags(obj) + + wim = obj.get("wareInfoReadMap") + ware_flags: dict[str, str] = {} + if isinstance(wim, dict): + for k in ("isCanUseDQ", "msbybt", "productBybt"): + if k in wim: + ware_flags[k] = _s(wim.get(k)) + + bybt = obj.get("bybtInfo") + if isinstance(bybt, dict) and bybt.get("productBybt"): + ware_flags["productBybt"] = "1" + + lines = _build_summary_lines( + gather=gather, + classic=classic, + bp_sum=bp_sum, + pref=pref_bundle, + gov_surf=gov_surf, + warm=warm, + delivery=delivery, + cta=cta, + flags=flags, + ) + if ent: + et = ent[:200].strip() + if et and et[-1] not in "。！？…": + et += "。" + lines.append("企业采购提示：" + et) + + out = { + "schema_version": 1, + "sku_id_hint": sku_hint, + "price_snapshot": { + "hand_price": gather.get("hand_price") or bp_sum.get("purchase_price"), + "hand_label": gather.get("hand_label") or "到手价", + "jd_list_price": gather.get("jd_price"), + "jd_list_hit_line": gather.get("jd_price_hit_line"), + "price_items": gather.get("raw_items"), + }, + "discount_mechanism": pref_bundle, + "gov_support_surface": gov_surf, + "best_promotion": bp_sum, + "purchase_constraints": { + "warm_tips": warm, + "ware_flags": ware_flags, + }, + "marketing_surface": { + "belt_right_text": belt, + "bottom_button_hint": _strip_html(cta, max_len=200), + }, + "visibility": {"rankings": rankings}, + "after_sales_short_labels": short_labels, + "after_sales_details": details, + "delivery": {"one_liner": delivery, "logistics_icons": log_icons}, + "enterprise_channel": ent, + "user_segment": flags, + "buyer_summary_lines": lines, + "notes": "摘要由规则生成；价格、券、补贴以结算页为准，此处不罗列埋点/实验字段。", + } + return out + + +# --- 与 ``pipeline.jd.buyer_offer_export_csv`` / 流水线 detail_ware 列对齐的扁平字段 --- + +_PROMO_EXCLUDE_PREFIXES_FOR_FLAT = ("榜单/曝光", "送达：", "企业采购提示：") +_DEFAULT_BUYER_PROMO_SEP = " | " + + +def buyer_ranking_line_from_profile(prof: dict[str, Any]) -> str: + """榜单名单列（如 ``粗粮饼干热卖榜·第5名。``），无 ``榜单/曝光：`` 前缀；无榜单时为空串。""" + vis = prof.get("visibility") + if not isinstance(vis, dict): + return "" + rk = vis.get("rankings") + if not isinstance(rk, list) or not rk: + return "" + first = strip_buyer_ranking_line_prefix(str(rk[0]).strip()) + if not first: + return "" + body = first if first.endswith("。") else first + "。" + return body + + +def buyer_promo_text_from_profile( + prof: dict[str, Any], + *, + sep: str = _DEFAULT_BUYER_PROMO_SEP, +) -> str: + """ + 从 ``buyer_summary_lines`` 取句，去掉榜单/送达/企业采购句，用 ``sep`` 拼接。 + """ + raw = prof.get("buyer_summary_lines") + if not isinstance(raw, list): + return "" + parts: list[str] = [] + for line in raw: + s = str(line).strip() + if not s: + continue + if any(s.startswith(p) for p in _PROMO_EXCLUDE_PREFIXES_FOR_FLAT): + continue + parts.append(s) + return sep.join(parts) + + +def extract_buyer_offer_profile_from_json_text(text: str) -> dict[str, Any]: + """解析响应体字符串，失败时返回空壳摘要。""" + raw = (text or "").strip() + if not raw: + return extract_buyer_offer_profile({}) + try: + obj = json.loads(raw) + except json.JSONDecodeError: + return extract_buyer_offer_profile({}) + return extract_buyer_offer_profile(obj) diff --git a/backend/crawler_copy/jd_pc_search/detail/jd_detail_ware_business_requests.py b/backend/crawler_copy/jd_pc_search/detail/jd_detail_ware_business_requests.py index c874cb6..6ff2838 100644 --- a/backend/crawler_copy/jd_pc_search/detail/jd_detail_ware_business_requests.py +++ b/backend/crawler_copy/jd_pc_search/detail/jd_detail_ware_business_requests.py @@ -21,25 +21,26 @@ - 若 ``OUTPUT_SKU_AND_BODY_IMAGES_ONLY=False``：**原始接口 JSON** 单 SKU ``OUT``、批量 ``OUT_DIR`` / ``ware_{sku}.json``；解析扁平含全部 ``detail_*``；汇总表 ``OUT_PARSED_CSV``。 **解析 API**：``flatten_ware_business`` / ``parse_ware_business_response_text`` / ``ware_parsed_row``， -列 ``detail_body_ingredients`` 为配料表文本（由 ``#detail-main`` 长图经 ``AI_crawler`` 自后向前多模态识别）；列 ``detail_body_ingredients_source_url`` 为**实际用于识别**的那张长图 URL（命中即停）。未配置 API 或识别失败时配料列为原因说明、图源列为空。内部 ``meta["detail_body_image_urls"]`` 仍为全部长图 URL 串，仅供解析用。 +列 ``detail_body_ingredients`` 为配料表文本（由 ``#detail-main`` 长图经 ``pipeline.openai_gateway`` 自后向前多模态识别）；列 ``detail_body_ingredients_source_url`` 为**实际用于识别**的那张长图 URL（命中即停）。未配置 API 或识别失败时配料列为原因说明、图源列为空。内部 ``meta["detail_body_image_urls"]`` 仍为全部长图 URL 串，仅供解析用。 Cookie：``../common/jd_cookie.txt``（或配置项 ``COOKIE_FILE`` / ``COOKIE_OVERRIDE``），经 ``add_cookies`` 注入。 依赖: pip install playwright && playwright install chromium（``USE_CHROME=True`` 时用本机 Chrome） 用法: 改下方「运行配置」后执行 ``python jd_detail_ware_business_requests.py``（无命令行参数）。 + +**模块划分**：响应 JSON 的扁平化与落盘格式化在 ``jd_detail_ware_parse.py``； +Playwright 打开商品页、拦截接口、``#detail-main`` 抽图在 ``jd_detail_ware_fetch.py``； +本文件保留运行配置、CLI ``main``、配料视觉桥接与解析结果写盘辅助，并对外 re-export 解析符号以兼容旧导入路径。 """ from __future__ import annotations import csv import json -import re import sys -import time from pathlib import Path from typing import Any, Callable -from urllib.parse import parse_qs, urlparse from playwright.sync_api import sync_playwright @@ -50,6 +51,10 @@ from playwright.sync_api import sync_playwright _JD_PC_SEARCH = Path(__file__).resolve().parents[1] if str(_JD_PC_SEARCH) not in sys.path: sys.path.insert(0, str(_JD_PC_SEARCH)) +# ``import pipeline.openai_gateway`` 需将 backend 根目录加入 path（Django 启动时已有） +_backend_root = Path(__file__).resolve().parents[3] +if str(_backend_root) not in sys.path: + sys.path.insert(0, str(_backend_root)) from _low_gi_root import low_gi_project_root # noqa: E402 _PROJECT_ROOT = low_gi_project_root() @@ -91,7 +96,7 @@ LOG_API_M_JD_TRACE = False # COLLECT_DETAIL_MAIN_IMAGE_URLS：True 时在点击商品详情 tab 并等待后，从 #detail-main 抽取图文 URL（style 中 background-image、img src、zbViewWeChatMiniImages）， # 补全为 https 后写入 meta（见 DETAIL_BODY_IMAGE_URL_SEPARATOR）；再经多模态写入列 detail_body_ingredients（配料文本）与 detail_body_ingredients_source_url（命中图源） COLLECT_DETAIL_MAIN_IMAGE_URLS = True -# EXTRACT_INGREDIENTS_FROM_DETAIL_BODY_IMAGES：True 时 detail_body_ingredients 为配料表文本、detail_body_ingredients_source_url 为命中图源；False 时配料列恒为空、图源列恒为空。需 .env 中 OPENAI_* / LLM_*（见上级目录 AI_crawler.py） +# EXTRACT_INGREDIENTS_FROM_DETAIL_BODY_IMAGES：True 时 detail_body_ingredients 为配料表文本、detail_body_ingredients_source_url 为命中图源；False 时配料列恒为空、图源列恒为空。需 .env 中 OPENAI_* / LLM_*（见 pipeline.openai_gateway） EXTRACT_INGREDIENTS_FROM_DETAIL_BODY_IMAGES = True # DETAIL_BODY_IMAGE_URL_SEPARATOR：写入 CSV/JSON 单单元格时的分隔符 DETAIL_BODY_IMAGE_URL_SEPARATOR = "; " @@ -139,805 +144,38 @@ _JD_DETAIL_CONTEXT_EXTRA_HEADERS: dict[str, str] = { "sec-ch-ua-platform": '"Windows"', } -# PC 详情页底部锚点：商品详情（与页面 id 一致，改版时需对照 DOM） -_JD_TAB_PRODUCT_DETAIL_SELECTOR = "#SPXQ-tab-column" - - -def _jd_function_id_from_api_url(url: str) -> str: - """从 ``api.m.jd.com?...`` 的 query 取 ``functionId``；无则空串。""" - try: - q = parse_qs(urlparse(url).query) - v = q.get("functionId") or q.get("functionid") - if not v: - return "" - return str(v[0]).strip() - except Exception: - return "" - - -def _print_api_m_jd_trace(rows: list[dict[str, Any]], *, sku_id: str) -> None: - if not rows: - print( - f"[京东] api.m.jd.com 轨迹 sku={sku_id}：未观察到该域名任何响应（Cookie/拦截或接口已迁域）。", - file=sys.stderr, - ) - return - print( - f"[京东] api.m.jd.com 轨迹 sku={sku_id} 共 {len(rows)} 条（按时间顺序；" - "after_tab 内无新行则点击 tab 未触发该域名 XHR）", - file=sys.stderr, - ) - for i, r in enumerate(rows, 1): - fid = r.get("functionId") or "(无 query functionId)" - print( - f" {i}. [{r.get('phase')}] HTTP {r.get('status')} {fid}", - file=sys.stderr, - ) - u = (r.get("url") or "")[:900] - print(f" {u}", file=sys.stderr) - print( - "[京东] 提示：若仍对不上 DevTools，请看 POST 请求的 form/body 里的 functionId；" - "图文详情也可能在首屏 HTML、iframe 或非 api.m.jd.com 的静态资源。", - file=sys.stderr, - ) - - -def _click_jd_product_detail_tab(page: Any, *, timeout_ms: int) -> None: - """点击「商品详情」锚点 tab；失败仅打日志，不中断（部分模板无此节点）。""" - cap = max(2_000, min(12_000, int(timeout_ms))) - try: - loc = page.locator(_JD_TAB_PRODUCT_DETAIL_SELECTOR) - loc.wait_for(state="visible", timeout=cap) - loc.click(timeout=cap) - except Exception as e: - print( - f"[京东] 未点击商品详情 tab（{_JD_TAB_PRODUCT_DETAIL_SELECTOR}）: {e}", - file=sys.stderr, - ) - - -# #detail-main 内 style 块中的 background-image:url(...) -_CSS_BG_URL_RE = re.compile(r"url\s*\(\s*([^)]+)\s*\)", re.I) -# zbViewWeChatMiniImages 的 value="a.jpg,b.jpg,..." -_ZB_MINI_VALUE_RE = re.compile( - r"zbViewWeChatMiniImages[^>]*\bvalue\s*=\s*[\"']([^\"']+)[\"']", - re.I | re.DOTALL, +# --------------------------------------------------------------------------- +# 解析（JSON 扁平）与采集（Playwright 拦截）分模块，见 jd_detail_ware_parse / jd_detail_ware_fetch +# --------------------------------------------------------------------------- +from jd_detail_ware_fetch import ( # noqa: E402 + WareFetchRuntime, + _print_http_verbose, + fetch_ware_business as _fetch_ware_business_impl, +) +from jd_detail_ware_parse import ( # noqa: E402 + DETAIL_WARE_LEAN_CSV_FIELDNAMES, + SKU_BODY_IMAGES_ONLY_FIELDNAMES, + WARE_BUSINESS_MERGE_FIELDNAMES, + WARE_PARSED_CSV_FIELDNAMES, + detail_ware_lean_csv_row, + flatten_ware_business, + format_ware_response_for_save, + format_ware_response_text, + minimal_sku_body_images_row, + parse_ware_business_response_text, + ware_parsed_row, ) - -def _normalize_jd_detail_asset_url(raw: str) -> str: - """将 //、/sku/jfs、/cms/jfs 等补全为可访问的 https URL。""" - s = (raw or "").strip() - if len(s) >= 2 and s[0] in "\"'" and s[-1] == s[0]: - s = s[1:-1].strip() - if not s or s.lower().startswith("data:"): - return "" - if s.startswith("//"): - return ("https:" + s)[:900] - if s.startswith("http://"): - return ("https://" + s[7:])[:900] - if s.startswith("https://"): - return s[:900] - if s.startswith("/sku/jfs/"): - return ("https://img30.360buyimg.com" + s)[:900] - if s.startswith("/cms/jfs/"): - return ("https://img12.360buyimg.com" + s)[:900] - if s.startswith("/jfs/"): - return ("https://img30.360buyimg.com/sku/jfs" + s[4:])[:900] - if s.startswith("jfs/"): - return ("https://img30.360buyimg.com/sku/" + s)[:900] - return s[:900] - - -def _dedupe_urls_preserve_order(urls: list[str]) -> list[str]: - seen: set[str] = set() - out: list[str] = [] - for u in urls: - u = (u or "").strip() - if not u or u in seen: - continue - seen.add(u) - out.append(u) - return out - - -def _urls_from_detail_main_inner_html(html: str) -> list[str]: - raw: list[str] = [] - if not (html or "").strip(): - return raw - for m in _CSS_BG_URL_RE.finditer(html): - inner = (m.group(1) or "").strip().strip("\"'") - if inner: - raw.append(inner) - zm = _ZB_MINI_VALUE_RE.search(html) - if zm: - for part in (zm.group(1) or "").split(","): - p = part.strip() - if p: - raw.append(p) - for m in re.finditer( - r"""]*\bsrc\s*=\s*(['"])(?P.*?)\1""", - html, - re.I | re.DOTALL, - ): - u = (m.group("u") or "").strip() - if u: - raw.append(u) - for m in re.finditer(r"""]*\bsrc\s*=\s*([^\s>'"]+)""", html, re.I): - u = (m.group(1) or "").strip() - if u: - raw.append(u) - return raw - - -def scrape_detail_main_body_urls_joined( - page: Any, - *, - wait_ms: int = 8_000, - separator: str | None = None, - max_chars: int | None = None, -) -> str: - """ - 从当前页 ``#detail-main`` 收集图文资源 URL（SSD 背景图、img、zbViewWeChatMiniImages）， - 补全为 https，去重保序后用 ``separator`` 拼成一段字符串（供 CSV 单格）。 - 页面须已展示商品详情区（通常需先点 ``#SPXQ-tab-column``）。 - """ - if not COLLECT_DETAIL_MAIN_IMAGE_URLS: - return "" - sep = ( - separator - if separator is not None - else (DETAIL_BODY_IMAGE_URL_SEPARATOR or "; ") - ) - cap = max(2000, min(15_000, int(wait_ms))) - loc = page.locator("#detail-main") - try: - loc.wait_for(state="attached", timeout=cap) - except Exception: - return "" - try: - html = loc.inner_html(timeout=min(5_000, cap)) - except Exception: - html = "" - dom_srcs: list[str] = [] - try: - dom_srcs = page.evaluate( - """() => { - const r = document.querySelector('#detail-main'); - if (!r) return []; - return Array.from(r.querySelectorAll('img')) - .map(i => (i.currentSrc || i.src || '').trim()) - .filter(Boolean); - }""" - ) - except Exception: - dom_srcs = [] - if not isinstance(dom_srcs, list): - dom_srcs = [] - - candidates = _urls_from_detail_main_inner_html(html) + [str(x) for x in dom_srcs] - normalized: list[str] = [] - for c in candidates: - n = _normalize_jd_detail_asset_url(c) - if not n: - continue - low = n.lower() - if "sku-market-gw.jd.com" in low and low.endswith(".css"): - continue - if "list.jd.com" in low or "item.jd.com" in low or "mall.jd.com" in low: - if not any( - low.endswith(ext) - for ext in (".jpg", ".jpeg", ".png", ".webp", ".avif", ".gif", ".dpg") - ): - continue - normalized.append(n) - - uniq = _dedupe_urls_preserve_order(normalized) - joined = sep.join(uniq) - mxc = ( - int(max_chars) - if max_chars is not None - else int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS) - ) - if mxc > 0 and len(joined) > mxc: - joined = joined[: mxc - 3] + "..." - return joined - - -def _normalize_ware_json_tree(obj: Any, *, sort_keys: bool) -> Any: - """递归规整：字典可选按键名排序，列表保持元素顺序。""" - if isinstance(obj, dict): - pairs = [ - (k, _normalize_ware_json_tree(v, sort_keys=sort_keys)) - for k, v in obj.items() - ] - if sort_keys: - pairs.sort(key=lambda kv: kv[0]) - return dict(pairs) - if isinstance(obj, list): - return [_normalize_ware_json_tree(x, sort_keys=sort_keys) for x in obj] - return obj - - -def _format_ware_response_text( - text: str, - *, - normalize: bool, - sort_keys: bool, - indent: int, -) -> tuple[str, bool]: - """ - 尝试将接口 body 规整为可读 JSON。 - 返回 (输出文本, 是否已成功按 JSON 处理)。 - """ - if not normalize or not (text or "").strip(): - return text, False - try: - obj = json.loads(text) - except json.JSONDecodeError: - return text, False - obj = _normalize_ware_json_tree(obj, sort_keys=sort_keys) - if indent > 0: - out = json.dumps(obj, ensure_ascii=False, indent=indent) + "\n" - else: - out = json.dumps(obj, ensure_ascii=False, separators=(",", ":")) + "\n" - return out, True - - -# 与 pc_detailpage_wareBusiness 响应对应的扁平字段（字符串，缺失为空），顺序即 CSV 建议列序 -WARE_BUSINESS_MERGE_FIELDNAMES: tuple[str, ...] = ( - "detail_sku_title", - "detail_price_final", - "detail_price_original", - "detail_purchase_price", - "detail_shop_name", - "detail_shop_id", - "detail_shop_url", - "detail_vender_id", - "detail_stock_text", - "detail_delivery_promise", - "detail_sku_name", - "detail_product_id", - "detail_main_sku_id", - "detail_page_sku_id", - "detail_brand", - "detail_category_path", - "detail_main_image", - "detail_product_attributes", - "detail_belt_banner", - "detail_csfh_text", - # 来自 DOM #detail-main（非 wareBusiness JSON）；列语义为「详情长图衍生信息」，常为 URL 串，流水线可替换为配料表文本 - "detail_body_ingredients", - # 视觉识别命中时：实际用于解析配料的那张详情长图 URL（自后向前首次通过校验） - "detail_body_ingredients_source_url", +_WARE_FETCH_RUNTIME = WareFetchRuntime( + log_api_m_jd_trace=bool(LOG_API_M_JD_TRACE), + goto_wait_until=str(GOTO_WAIT_UNTIL or "domcontentloaded").strip(), + click_product_detail_tab=bool(CLICK_PRODUCT_DETAIL_TAB), + collect_detail_main_image_urls=bool(COLLECT_DETAIL_MAIN_IMAGE_URLS), + detail_body_image_url_separator=str(DETAIL_BODY_IMAGE_URL_SEPARATOR or "; "), + detail_body_image_urls_max_chars=int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS), ) -def _empty_ware_flat() -> dict[str, str]: - return {k: "" for k in WARE_BUSINESS_MERGE_FIELDNAMES} - - -def _strip_htmlish(s: str, *, max_len: int) -> str: - if not s: - return "" - t = re.sub(r"<[^>]+>", " ", s) - t = " ".join(t.split()).strip() - return t[:max_len] if max_len > 0 else t - - -def _s(obj: Any) -> str: - if obj is None: - return "" - return str(obj).strip() - - -def _join_url(u: str) -> str: - u = u.strip() - if not u: - return "" - if u.startswith("//"): - return "https:" + u - return u - - -def _jd_product_image_url(path: str) -> str: - """与搜索侧一致：jfs/ 相对路径补全为可访问 URL。""" - p = (path or "").strip() - if not p: - return "" - if p.startswith("//"): - return "https:" + p - if p.startswith("http://"): - return "https://" + p[7:] - if p.startswith("https://"): - return p[:800] - if p.startswith("jfs/"): - return "https://img13.360buyimg.com/n2/s480x480_" + p[:700] - return p[:800] - - -def flatten_ware_business(obj: Any) -> dict[str, str]: - """ - 将 ``pc_detailpage_wareBusiness`` 的 JSON 根对象压成扁平字符串字典。 - 非 dict 或字段缺失时对应值为空串。 - """ - out = _empty_ware_flat() - if not isinstance(obj, dict): - return out - - sh = obj.get("skuHeadVO") - sh = sh if isinstance(sh, dict) else {} - out["detail_sku_title"] = _s(sh.get("skuTitle"))[:2000] - - price = obj.get("price") - price = price if isinstance(price, dict) else {} - fp = price.get("finalPrice") - fp = fp if isinstance(fp, dict) else {} - out["detail_price_final"] = _s(fp.get("price")) or _s(price.get("p"))[:64] - out["detail_price_original"] = _s(price.get("op")) or _s(price.get("p"))[:64] - - bp = obj.get("bestPromotion") - bp = bp if isinstance(bp, dict) else {} - out["detail_purchase_price"] = _s(bp.get("purchasePrice"))[:64] - - ishop = obj.get("itemShopInfo") - ishop = ishop if isinstance(ishop, dict) else {} - out["detail_shop_name"] = _s(ishop.get("shopName"))[:500] - out["detail_shop_id"] = _s(ishop.get("shopId"))[:32] - out["detail_shop_url"] = _join_url(_s(ishop.get("shopUrl")))[:500] - - pc = obj.get("pageConfigVO") - pc = pc if isinstance(pc, dict) else {} - out["detail_vender_id"] = _s(pc.get("venderId"))[:32] - sk = pc.get("skuid") - out["detail_page_sku_id"] = _s(sk)[:32] - cats = pc.get("catName") - if isinstance(cats, list): - parts = [_s(x) for x in cats if _s(x)] - out["detail_category_path"] = " > ".join(parts)[:500] - src = _s(pc.get("src")) - if src: - out["detail_main_image"] = _jd_product_image_url(src) - - mi = obj.get("mainImageVO") - if isinstance(mi, dict) and not out["detail_main_image"]: - mia = mi.get("mainImageArea") - if isinstance(mia, dict): - iu = _s(mia.get("imageUrl")) - if iu: - out["detail_main_image"] = _jd_product_image_url(iu) - - si = obj.get("stockInfo") - si = si if isinstance(si, dict) else {} - out["detail_stock_text"] = _strip_htmlish(_s(si.get("stockDesc")), max_len=500) - if not out["detail_stock_text"]: - out["detail_stock_text"] = _strip_htmlish(_s(si.get("promiseResult")), max_len=500) - out["detail_delivery_promise"] = _strip_htmlish( - _s(si.get("promiseResult") or si.get("promiseInfoText")), max_len=800 - ) - - wim = obj.get("wareInfoReadMap") - wim = wim if isinstance(wim, dict) else {} - out["detail_sku_name"] = _s(wim.get("sku_name"))[:2000] - out["detail_product_id"] = _s(wim.get("product_id"))[:32] - out["detail_main_sku_id"] = _s(wim.get("main_sku_id"))[:32] - out["detail_brand"] = _s(wim.get("cn_brand"))[:200] - if not out["detail_brand"]: - pav = obj.get("productAttributeVO") - if isinstance(pav, dict): - for it in pav.get("attributes") or []: - if not isinstance(it, dict): - continue - if _s(it.get("labelName")) == "品牌": - out["detail_brand"] = _s(it.get("labelValue"))[:200] - break - - pav = obj.get("productAttributeVO") - attrs: list[str] = [] - if isinstance(pav, dict): - for it in pav.get("attributes") or []: - if not isinstance(it, dict): - continue - ln, lv = _s(it.get("labelName")), _s(it.get("labelValue")) - if ln and lv: - attrs.append(f"{ln}:{lv}") - out["detail_product_attributes"] = "; ".join(attrs)[:4000] - - out["detail_belt_banner"] = _join_url(_s(obj.get("beltBanner")))[:800] - out["detail_csfh_text"] = _s(obj.get("csfhText"))[:200] - - return out - - -# 与 OUT_PARSED_CSV / 流水线 detail CSV 列一致 -WARE_PARSED_CSV_FIELDNAMES: tuple[str, ...] = ( - "skuId", - "http_status", - *WARE_BUSINESS_MERGE_FIELDNAMES, -) - -# 仅 sku + 配料（本脚本 OUTPUT_SKU_AND_BODY_IMAGES_ONLY 时 main 写 CSV/解析 JSON 用） -SKU_BODY_IMAGES_ONLY_FIELDNAMES: tuple[str, ...] = ( - "skuId", - "detail_body_ingredients", -) - -# 与合并表 lean 商详块一致 + skuId；keyword_pipeline DETAIL_WARE_CSV_MODE=lean 写 detail_ware_export.csv -DETAIL_WARE_LEAN_CSV_FIELDNAMES: tuple[str, ...] = ( - "skuId", - "detail_brand", - "detail_price_final", - "detail_shop_name", - "detail_category_path", - "detail_product_attributes", - "detail_body_ingredients", -) - - -def detail_ware_lean_csv_row( - sku: str, - http_status: int, - response_text: str, - *, - detail_body_ingredients: str = "", - detail_body_ingredients_source_url: str = "", -) -> dict[str, str]: - """lean 详情汇总表一行（无 http_status）；字段来自 ``ware_parsed_row`` 子集。""" - full = ware_parsed_row( - sku, - http_status, - response_text, - detail_body_ingredients=detail_body_ingredients, - detail_body_ingredients_source_url=detail_body_ingredients_source_url, - ) - return {k: str(full.get(k) or "") for k in DETAIL_WARE_LEAN_CSV_FIELDNAMES} - - -def minimal_sku_body_images_row( - sku: str, - detail_body_ingredients: str, - *, - detail_body_ingredients_source_url: str = "", -) -> dict[str, str]: - """``skuId``、配料文本（图源仅内部流程使用，不写入极简 CSV）。""" - _ = detail_body_ingredients_source_url # 保留参数供调用方兼容 - u = (detail_body_ingredients or "").strip() - mxc = max(0, int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS)) - if mxc and len(u) > mxc: - u = u[:mxc] - return { - "skuId": str(sku).strip(), - "detail_body_ingredients": u, - } - - -def _write_minimal_body_images_json( - path: Path, - sku: str, - detail_body_ingredients: str, - *, - detail_body_ingredients_source_url: str = "", -) -> None: - row = minimal_sku_body_images_row( - sku, - detail_body_ingredients, - detail_body_ingredients_source_url=detail_body_ingredients_source_url, - ) - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(row, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") - print(f"[京东] 已写图文 URL JSON：{path}", file=sys.stderr) - - -def parse_ware_business_response_text(text: str) -> tuple[dict[str, str], bool]: - """ - 解析响应体字符串。 - 返回 ``(扁平字典, 是否成功解析为 JSON 对象)``；失败时扁平字典各键均为 ``""``。 - """ - raw = (text or "").strip() - if not raw: - return _empty_ware_flat(), False - try: - obj = json.loads(raw) - except json.JSONDecodeError: - return _empty_ware_flat(), False - if not isinstance(obj, dict): - return _empty_ware_flat(), False - return flatten_ware_business(obj), True - - -def ware_parsed_row( - sku: str, - http_status: int, - response_text: str, - *, - detail_body_ingredients: str = "", - detail_body_ingredients_source_url: str = "", -) -> dict[str, str]: - """单行扁平结果：``skuId``、``http_status`` 与 ``WARE_BUSINESS_MERGE_FIELDNAMES``（含 DOM 长图 URL 或配料文本）。""" - flat, _ = parse_ware_business_response_text( - response_text if http_status == 200 else "" - ) - row: dict[str, str] = { - "skuId": str(sku).strip(), - "http_status": str(http_status), - **flat, - } - u = (detail_body_ingredients or "").strip() - if u: - mxc = max(0, int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS)) - row["detail_body_ingredients"] = u[:mxc] if mxc else u - src = (detail_body_ingredients_source_url or "").strip() - if src: - mxc = max(0, int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS)) - row["detail_body_ingredients_source_url"] = src[:mxc] if mxc else src - return row - - -def ware_fetch_should_retry(http_status: int, response_text: str) -> bool: - """ - True 表示应重试：未命中接口、HTTP 非 200、正文空、非 JSON、或解析后业务字段全空。 - """ - if int(http_status) != 200: - return True - raw = (response_text or "").strip() - if not raw: - return True - flat, ok = parse_ware_business_response_text(raw) - if not ok: - return True - return not any((v or "").strip() for v in flat.values()) - - -def format_ware_response_for_save( - text: str, - *, - normalize: bool = True, - sort_keys: bool = True, - indent: int = 2, -) -> str: - """流水线等落盘用：尽量输出缩进 + 可选键排序的 JSON 文本（失败则保留原文）。""" - body, _ok = _format_ware_response_text( - text or "", - normalize=normalize, - sort_keys=sort_keys, - indent=max(0, int(indent)), - ) - return body - - -def _write_ware_parsed_json( - path: Path, - sku: str, - http_status: int, - response_text: str, - *, - detail_body_ingredients: str = "", - detail_body_ingredients_source_url: str = "", -) -> None: - row = ware_parsed_row( - sku, - http_status, - response_text, - detail_body_ingredients=detail_body_ingredients, - detail_body_ingredients_source_url=detail_body_ingredients_source_url, - ) - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(row, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") - print(f"[京东] 已写解析 JSON：{path}", file=sys.stderr) - - -def _read_jd_cookie_file_raw(cookie_file: str | None) -> str: - """多行合并为 ``; `` 分隔（与 Node ``readCookieFile`` 一致）。""" - path = Path(cookie_file) if (cookie_file or "").strip() else _DEFAULT_COOKIE_PATH - path = path.resolve() - if not path.is_file(): - return "" - chunks: list[str] = [] - for line in path.read_text(encoding="utf-8").splitlines(): - t = line.strip() - if t and not t.startswith("#"): - chunks.append(t) - return "; ".join(chunks).strip() - - -def _cookie_header_to_playwright(cookie_header: str) -> list[dict[str, Any]]: - rows: list[dict[str, Any]] = [] - for part in cookie_header.split(";"): - part = part.strip() - if not part or "=" not in part: - continue - name, _, value = part.partition("=") - name, value = name.strip(), value.strip() - if not name: - continue - rows.append( - { - "name": name, - "value": value, - "domain": ".jd.com", - "path": "/", - "secure": True, - } - ) - return rows - - -def _headers_for_verbose(h: dict[str, str], *, cookie_preview: int = 96) -> dict[str, str]: - out: dict[str, str] = {} - for k, v in h.items(): - if k.lower() == "cookie" and len(v) > cookie_preview: - out[k] = ( - v[:cookie_preview] - + f"...（共 {len(v)} 字符，完整值请用 --http-log）" - ) - else: - out[k] = v - return out - - -def _print_http_verbose(meta: dict[str, Any], *, body_max: int) -> None: - req = meta["request"] - res = meta["response"] - body = res.get("body") or "" - if len(body) > body_max: - body_show = ( - body[:body_max] - + f"\n...（stderr 已截断，响应共 {len(body)} 字符；完整见 --http-log）" - ) - else: - body_show = body - req_block: dict[str, Any] = {"method": req.get("method", "GET")} - if req.get("url"): - req_block["url"] = req["url"] - for k in ("via", "item_page", "referrer_document", "note"): - if k in req: - req_block[k] = req[k] - hdrs = req.get("headers") - if hdrs: - req_block["headers"] = _headers_for_verbose(hdrs) - block = { - "skuId": meta.get("skuId"), - "request": req_block, - "response": { - "url": res.get("url"), - "status": res.get("status"), - "status_text": res.get("status_text"), - "headers": res.get("headers"), - "body": body_show, - }, - } - sys.stderr.write( - "[京东] HTTP 详情（stderr）:\n" - + json.dumps(block, ensure_ascii=False, indent=2) - + "\n" - ) - - -def _fetch_ware_business_once( - context: Any, - page: Any, - sku_id: str, - *, - cookie_file: str | None = None, - timeout_ms: int = 30_000, - cookie_override: str = "", -) -> tuple[int, str, dict[str, Any]]: - """单次打开商品页并拦截 ``pc_detailpage_wareBusiness`` 响应（无重试）。""" - raw_cookie = (cookie_override or "").strip() or _read_jd_cookie_file_raw(cookie_file) - if not raw_cookie: - print( - "[京东] 需要 Cookie：--cookie 或 jd_cookie.txt（--cookie-file）", - file=sys.stderr, - ) - sys.exit(2) - context.clear_cookies() - try: - context.add_cookies(_cookie_header_to_playwright(raw_cookie)) - except Exception as e: - print(f"[京东] add_cookies 警告: {e}", file=sys.stderr) - item_url = f"https://item.jd.com/{str(sku_id).strip()}.html" - matches: list[tuple[int, str, str, dict[str, str]]] = [] - trace_rows: list[dict[str, Any]] = [] - trace_phase = "opening" - - def _on_response(response: Any) -> None: - try: - u = response.url - if LOG_API_M_JD_TRACE and "api.m.jd.com" in u: - ct = "" - try: - ct = (response.headers.get("content-type") or "").strip() - except Exception: - pass - trace_rows.append( - { - "phase": trace_phase, - "status": response.status, - "functionId": _jd_function_id_from_api_url(u), - "content_type": ct[:160], - "url": u[:2000], - } - ) - if ( - "api.m.jd.com" in u - and "pc_detailpage_wareBusiness" in u - and "functionId=" in u - ): - st = response.status - body = response.text() - matches.append((st, body, u, dict(response.headers))) - except Exception: - pass - - detail_joined = "" - page.on("response", _on_response) - try: - _wu = (GOTO_WAIT_UNTIL or "domcontentloaded").strip() - page.goto(item_url, wait_until=_wu, timeout=timeout_ms) - trace_phase = "loaded" - if CLICK_PRODUCT_DETAIL_TAB: - _click_jd_product_detail_tab(page, timeout_ms=timeout_ms) - trace_phase = "after_tab" - extra = min(12_000, max(3_000, timeout_ms // 2)) - page.wait_for_timeout(extra) - if COLLECT_DETAIL_MAIN_IMAGE_URLS: - try: - detail_joined = scrape_detail_main_body_urls_joined( - page, - wait_ms=min(timeout_ms, 12_000), - ) - except Exception as e: - print( - f"[京东] 抽取 #detail-main 图文 URL 失败: {e}", - file=sys.stderr, - ) - finally: - try: - page.remove_listener("response", _on_response) - except Exception: - pass - if LOG_API_M_JD_TRACE: - _print_api_m_jd_trace(trace_rows, sku_id=str(sku_id).strip()) - if not matches: - meta: dict[str, Any] = { - "skuId": str(sku_id).strip(), - "request": { - "method": "GET", - "via": "capture_page_navigation", - "item_page": item_url, - "note": "未捕获到 pc_detailpage_wareBusiness（页面可能改版或 Cookie 未登录）", - }, - "response": {"status": 0, "status_text": "", "headers": {}, "body": ""}, - } - if LOG_API_M_JD_TRACE: - meta["api_m_jd_trace"] = trace_rows - meta["detail_body_image_urls"] = detail_joined - return 0, "", meta - ok_rows = [m for m in matches if m[0] == 200] - st, body, u, rh = ok_rows[-1] if ok_rows else matches[-1] - meta = { - "skuId": str(sku_id).strip(), - "request": { - "method": "GET", - "url": u, - "via": "capture_page_navigation", - "item_page": item_url, - "captures_count": len(matches), - }, - "response": { - "url": u, - "status": st, - "status_text": "", - "headers": rh, - "body": body, - }, - } - if LOG_API_M_JD_TRACE: - meta["api_m_jd_trace"] = trace_rows - meta["detail_body_image_urls"] = detail_joined - return st, body, meta - - def fetch_ware_business( context: Any, page: Any, @@ -950,53 +188,20 @@ def fetch_ware_business( retry_delay_sec: float = 2.0, cancel_check: Callable[[], bool] | None = None, ) -> tuple[int, str, dict[str, Any]]: - """ - 打开商品页并拦截 ``pc_detailpage_wareBusiness``。 - ``max_attempts``>1 时，在结果为空或失败时按 ``retry_delay_sec`` 间隔重试。 - """ - sid = str(sku_id).strip() - n = max(1, int(max_attempts)) - last: tuple[int, str, dict[str, Any]] = (0, "", {}) - for i in range(n): - if cancel_check is not None and cancel_check(): - return last - if i > 0: - delay = max(0.0, float(retry_delay_sec)) - if delay > 0: - time.sleep(delay) - if cancel_check is not None and cancel_check(): - return last - code, text, meta = _fetch_ware_business_once( - context, - page, - sid, - cookie_file=cookie_file, - timeout_ms=timeout_ms, - cookie_override=cookie_override, - ) - last = (code, text, meta) - if OUTPUT_SKU_AND_BODY_IMAGES_ONLY and ( - meta.get("detail_body_image_urls") or "" - ).strip(): - if i > 0: - print( - f"[京东] sku={sid} 第 {i + 1} 次尝试已成功（已抽到 #detail-main 图文 URL）", - file=sys.stderr, - ) - break - if not ware_fetch_should_retry(code, text): - if i > 0: - print( - f"[京东] sku={sid} 第 {i + 1} 次尝试已成功", - file=sys.stderr, - ) - break - print( - f"[京东] sku={sid} 详情结果为空或无效 (HTTP {code})，" - f"重试 {i + 1}/{n}…", - file=sys.stderr, - ) - return last + '''打开商品页并拦截 pc_detailpage_wareBusiness（与流水线兼容的薄封装）。''' + return _fetch_ware_business_impl( + context, + page, + sku_id, + cookie_file=cookie_file, + timeout_ms=timeout_ms, + cookie_override=cookie_override, + max_attempts=max_attempts, + retry_delay_sec=retry_delay_sec, + cancel_check=cancel_check, + output_sku_and_body_images_only=bool(OUTPUT_SKU_AND_BODY_IMAGES_ONLY), + runtime=_WARE_FETCH_RUNTIME, + ) def _detail_body_ingredients_column_value( @@ -1035,6 +240,46 @@ def _detail_body_ingredients_column_value( return f"【未识别到配料】识别异常：{e}"[:800], "" +def _write_minimal_body_images_json( + path: Path, + sku: str, + detail_body_ingredients: str, + *, + detail_body_ingredients_source_url: str = "", +) -> None: + row = minimal_sku_body_images_row( + sku, + detail_body_ingredients, + detail_body_ingredients_source_url=detail_body_ingredients_source_url, + max_cell_chars=int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS), + ) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(row, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print(f"[京东] 已写图文 URL JSON：{path}", file=sys.stderr) + + +def _write_ware_parsed_json( + path: Path, + sku: str, + http_status: int, + response_text: str, + *, + detail_body_ingredients: str = "", + detail_body_ingredients_source_url: str = "", +) -> None: + row = ware_parsed_row( + sku, + http_status, + response_text, + detail_body_ingredients=detail_body_ingredients, + detail_body_ingredients_source_url=detail_body_ingredients_source_url, + max_cell_chars=int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS), + ) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(row, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + print(f"[京东] 已写解析 JSON：{path}", file=sys.stderr) + + def main() -> None: try: if hasattr(sys.stdout, "reconfigure"): @@ -1107,9 +352,9 @@ def main() -> None: vision_ok = False if EXTRACT_INGREDIENTS_FROM_DETAIL_BODY_IMAGES: try: - import AI_crawler as vision_mod # noqa: WPS433 + import pipeline.openai_gateway as vision_mod # noqa: WPS433 - vision_mod._resolve_credentials(None, None, None) + vision_mod.resolve_credentials(None, None, None) vision_ok = True except Exception as e: print( @@ -1142,6 +387,7 @@ def main() -> None: s, detail_body_ingredients, detail_body_ingredients_source_url=detail_body_ingredients_source_url, + max_cell_chars=int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS), ) if out_dir: if out_parsed_dir: @@ -1187,7 +433,7 @@ def main() -> None: if out_dir: out_p = Path(out_dir).resolve() / f"ware_{s}.json" out_p.parent.mkdir(parents=True, exist_ok=True) - body, _ok = _format_ware_response_text( + body, _ok = format_ware_response_text( text, normalize=normalize_json, sort_keys=sort_keys, @@ -1213,13 +459,14 @@ def main() -> None: text, detail_body_ingredients=detail_body_ingredients, detail_body_ingredients_source_url=detail_body_ingredients_source_url, + max_cell_chars=int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS), ) ) return if out_path: Path(out_path).parent.mkdir(parents=True, exist_ok=True) if normalize_json: - body, _ok = _format_ware_response_text( + body, _ok = format_ware_response_text( text, normalize=True, sort_keys=sort_keys, @@ -1255,6 +502,7 @@ def main() -> None: text, detail_body_ingredients=detail_body_ingredients, detail_body_ingredients_source_url=detail_body_ingredients_source_url, + max_cell_chars=int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS), ) ) return @@ -1268,6 +516,7 @@ def main() -> None: text, detail_body_ingredients=detail_body_ingredients, detail_body_ingredients_source_url=detail_body_ingredients_source_url, + max_cell_chars=int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS), ) ) return @@ -1281,7 +530,7 @@ def main() -> None: detail_body_ingredients_source_url=detail_body_ingredients_source_url, ) if normalize_json: - body, ok = _format_ware_response_text( + body, ok = format_ware_response_text( text, normalize=True, sort_keys=sort_keys, @@ -1308,6 +557,7 @@ def main() -> None: text, detail_body_ingredients=detail_body_ingredients, detail_body_ingredients_source_url=detail_body_ingredients_source_url, + max_cell_chars=int(DETAIL_BODY_IMAGE_URLS_MAX_CHARS), ) ) diff --git a/backend/crawler_copy/jd_pc_search/detail/jd_detail_ware_fetch.py b/backend/crawler_copy/jd_pc_search/detail/jd_detail_ware_fetch.py new file mode 100644 index 0000000..4dd126a --- /dev/null +++ b/backend/crawler_copy/jd_pc_search/detail/jd_detail_ware_fetch.py @@ -0,0 +1,535 @@ +# -*- coding: utf-8 -*- +""" +京东 PC 详情 **采集层**：打开商品页、拦截 ``pc_detailpage_wareBusiness``、从 ``#detail-main`` 抽图文 URL。 + +解析 JSON 扁平字段见 ``jd_detail_ware_parse``。 +""" +from __future__ import annotations + +import json +import re +import sys +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any, Callable +from urllib.parse import parse_qs, urlparse + +_JD_DIR = Path(__file__).resolve().parent +if str(_JD_DIR) not in sys.path: + sys.path.insert(0, str(_JD_DIR)) +_JD_PC_SEARCH_DIR = _JD_DIR.parent +if str(_JD_PC_SEARCH_DIR) not in sys.path: + sys.path.insert(0, str(_JD_PC_SEARCH_DIR)) + +from jd_detail_ware_parse import ware_fetch_should_retry + +_DEFAULT_COOKIE_PATH = (_JD_DIR.parent / "common" / "jd_cookie.txt").resolve() + + +@dataclass(frozen=True) +class WareFetchRuntime: + """单次打开详情页时的行为开关（与 ``jd_detail_ware_business_requests`` 顶部配置对应）。""" + + log_api_m_jd_trace: bool = False + goto_wait_until: str = "domcontentloaded" + click_product_detail_tab: bool = True + collect_detail_main_image_urls: bool = True + detail_body_image_url_separator: str = "; " + detail_body_image_urls_max_chars: int = 31000 + + +def _jd_function_id_from_api_url(url: str) -> str: + """从 ``api.m.jd.com?...`` 的 query 取 ``functionId``；无则空串。""" + try: + q = parse_qs(urlparse(url).query) + v = q.get("functionId") or q.get("functionid") + if not v: + return "" + return str(v[0]).strip() + except Exception: + return "" + + +def _print_api_m_jd_trace(rows: list[dict[str, Any]], *, sku_id: str) -> None: + if not rows: + print( + f"[京东] api.m.jd.com 轨迹 sku={sku_id}：未观察到该域名任何响应（Cookie/拦截或接口已迁域）。", + file=sys.stderr, + ) + return + print( + f"[京东] api.m.jd.com 轨迹 sku={sku_id} 共 {len(rows)} 条（按时间顺序；" + "after_tab 内无新行则点击 tab 未触发该域名 XHR）", + file=sys.stderr, + ) + for i, r in enumerate(rows, 1): + fid = r.get("functionId") or "(无 query functionId)" + print( + f" {i}. [{r.get('phase')}] HTTP {r.get('status')} {fid}", + file=sys.stderr, + ) + u = (r.get("url") or "")[:900] + print(f" {u}", file=sys.stderr) + print( + "[京东] 提示：若仍对不上 DevTools，请看 POST 请求的 form/body 里的 functionId；" + "图文详情也可能在首屏 HTML、iframe 或非 api.m.jd.com 的静态资源。", + file=sys.stderr, + ) + + +# PC 详情页底部锚点：商品详情（与页面 id 一致，改版时需对照 DOM） +_JD_TAB_PRODUCT_DETAIL_SELECTOR = "#SPXQ-tab-column" + + +def _click_jd_product_detail_tab(page: Any, *, timeout_ms: int) -> None: + """点击「商品详情」锚点 tab；失败仅打日志，不中断（部分模板无此节点）。""" + cap = max(2_000, min(12_000, int(timeout_ms))) + try: + loc = page.locator(_JD_TAB_PRODUCT_DETAIL_SELECTOR) + loc.wait_for(state="visible", timeout=cap) + loc.click(timeout=cap) + except Exception as e: + print( + f"[京东] 未点击商品详情 tab（{_JD_TAB_PRODUCT_DETAIL_SELECTOR}）: {e}", + file=sys.stderr, + ) + + +# #detail-main 内 style 块中的 background-image:url(...) +_CSS_BG_URL_RE = re.compile(r"url\s*\(\s*([^)]+)\s*\)", re.I) +# zbViewWeChatMiniImages 的 value="a.jpg,b.jpg,..." +_ZB_MINI_VALUE_RE = re.compile( + r"zbViewWeChatMiniImages[^>]*\bvalue\s*=\s*[\"']([^\"']+)[\"']", + re.I | re.DOTALL, +) + + +def _normalize_jd_detail_asset_url(raw: str) -> str: + """将 //、/sku/jfs、/cms/jfs 等补全为可访问的 https URL。""" + s = (raw or "").strip() + if len(s) >= 2 and s[0] in "\"'" and s[-1] == s[0]: + s = s[1:-1].strip() + if not s or s.lower().startswith("data:"): + return "" + if s.startswith("//"): + return ("https:" + s)[:900] + if s.startswith("http://"): + return ("https://" + s[7:])[:900] + if s.startswith("https://"): + return s[:900] + if s.startswith("/sku/jfs/"): + return ("https://img30.360buyimg.com" + s)[:900] + if s.startswith("/cms/jfs/"): + return ("https://img12.360buyimg.com" + s)[:900] + if s.startswith("/jfs/"): + return ("https://img30.360buyimg.com/sku/jfs" + s[4:])[:900] + if s.startswith("jfs/"): + return ("https://img30.360buyimg.com/sku/" + s)[:900] + return s[:900] + + +def _dedupe_urls_preserve_order(urls: list[str]) -> list[str]: + seen: set[str] = set() + out: list[str] = [] + for u in urls: + u = (u or "").strip() + if not u or u in seen: + continue + seen.add(u) + out.append(u) + return out + + +def _urls_from_detail_main_inner_html(html: str) -> list[str]: + raw: list[str] = [] + if not (html or "").strip(): + return raw + for m in _CSS_BG_URL_RE.finditer(html): + inner = (m.group(1) or "").strip().strip("\"'") + if inner: + raw.append(inner) + zm = _ZB_MINI_VALUE_RE.search(html) + if zm: + for part in (zm.group(1) or "").split(","): + p = part.strip() + if p: + raw.append(p) + for m in re.finditer( + r"""]*\bsrc\s*=\s*(['"])(?P.*?)\1""", + html, + re.I | re.DOTALL, + ): + u = (m.group("u") or "").strip() + if u: + raw.append(u) + for m in re.finditer(r"""]*\bsrc\s*=\s*([^\s>'"]+)""", html, re.I): + u = (m.group(1) or "").strip() + if u: + raw.append(u) + return raw + + +def scrape_detail_main_body_urls_joined( + page: Any, + *, + wait_ms: int = 8_000, + separator: str | None = None, + max_chars: int | None = None, + runtime: WareFetchRuntime | None = None, +) -> str: + """ + 从当前页 ``#detail-main`` 收集图文资源 URL（SSD 背景图、img、zbViewWeChatMiniImages）， + 补全为 https，去重保序后用 ``separator`` 拼成一段字符串（供 CSV 单格）。 + 页面须已展示商品详情区（通常需先点 ``#SPXQ-tab-column``）。 + """ + rt = runtime or WareFetchRuntime() + if not rt.collect_detail_main_image_urls: + return "" + sep = ( + separator + if separator is not None + else (rt.detail_body_image_url_separator or "; ") + ) + cap = max(2000, min(15_000, int(wait_ms))) + loc = page.locator("#detail-main") + try: + loc.wait_for(state="attached", timeout=cap) + except Exception: + return "" + try: + html = loc.inner_html(timeout=min(5_000, cap)) + except Exception: + html = "" + dom_srcs: list[str] = [] + try: + dom_srcs = page.evaluate( + """() => { + const r = document.querySelector('#detail-main'); + if (!r) return []; + return Array.from(r.querySelectorAll('img')) + .map(i => (i.currentSrc || i.src || '').trim()) + .filter(Boolean); + }""" + ) + except Exception: + dom_srcs = [] + if not isinstance(dom_srcs, list): + dom_srcs = [] + + candidates = _urls_from_detail_main_inner_html(html) + [str(x) for x in dom_srcs] + normalized: list[str] = [] + for c in candidates: + n = _normalize_jd_detail_asset_url(c) + if not n: + continue + low = n.lower() + if "sku-market-gw.jd.com" in low and low.endswith(".css"): + continue + if "list.jd.com" in low or "item.jd.com" in low or "mall.jd.com" in low: + if not any( + low.endswith(ext) + for ext in (".jpg", ".jpeg", ".png", ".webp", ".avif", ".gif", ".dpg") + ): + continue + normalized.append(n) + + uniq = _dedupe_urls_preserve_order(normalized) + joined = sep.join(uniq) + mxc = ( + int(max_chars) + if max_chars is not None + else int(rt.detail_body_image_urls_max_chars) + ) + if mxc > 0 and len(joined) > mxc: + joined = joined[: mxc - 3] + "..." + return joined + + +def _read_jd_cookie_file_raw(cookie_file: str | None) -> str: + """多行合并为 ``; `` 分隔（与 Node ``readCookieFile`` 一致）。""" + path = Path(cookie_file) if (cookie_file or "").strip() else _DEFAULT_COOKIE_PATH + path = path.resolve() + if not path.is_file(): + return "" + chunks: list[str] = [] + for line in path.read_text(encoding="utf-8").splitlines(): + t = line.strip() + if t and not t.startswith("#"): + chunks.append(t) + return "; ".join(chunks).strip() + + +def _cookie_header_to_playwright(cookie_header: str) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for part in cookie_header.split(";"): + part = part.strip() + if not part or "=" not in part: + continue + name, _, value = part.partition("=") + name, value = name.strip(), value.strip() + if not name: + continue + rows.append( + { + "name": name, + "value": value, + "domain": ".jd.com", + "path": "/", + "secure": True, + } + ) + return rows + + +def _headers_for_verbose(h: dict[str, str], *, cookie_preview: int = 96) -> dict[str, str]: + out: dict[str, str] = {} + for k, v in h.items(): + if k.lower() == "cookie" and len(v) > cookie_preview: + out[k] = ( + v[:cookie_preview] + + f"...（共 {len(v)} 字符，完整值请用 --http-log）" + ) + else: + out[k] = v + return out + + +def _print_http_verbose(meta: dict[str, Any], *, body_max: int) -> None: + req = meta["request"] + res = meta["response"] + body = res.get("body") or "" + if len(body) > body_max: + body_show = ( + body[:body_max] + + f"\n...（stderr 已截断，响应共 {len(body)} 字符；完整见 --http-log）" + ) + else: + body_show = body + req_block: dict[str, Any] = {"method": req.get("method", "GET")} + if req.get("url"): + req_block["url"] = req["url"] + for k in ("via", "item_page", "referrer_document", "note"): + if k in req: + req_block[k] = req[k] + hdrs = req.get("headers") + if hdrs: + req_block["headers"] = _headers_for_verbose(hdrs) + block = { + "skuId": meta.get("skuId"), + "request": req_block, + "response": { + "url": res.get("url"), + "status": res.get("status"), + "status_text": res.get("status_text"), + "headers": res.get("headers"), + "body": body_show, + }, + } + sys.stderr.write( + "[京东] HTTP 详情（stderr）:\n" + + json.dumps(block, ensure_ascii=False, indent=2) + + "\n" + ) + + +def _fetch_ware_business_once( + context: Any, + page: Any, + sku_id: str, + *, + cookie_file: str | None = None, + timeout_ms: int = 30_000, + cookie_override: str = "", + runtime: WareFetchRuntime | None = None, +) -> tuple[int, str, dict[str, Any]]: + """单次打开商品页并拦截 ``pc_detailpage_wareBusiness`` 响应（无重试）。""" + rt = runtime or WareFetchRuntime() + raw_cookie = (cookie_override or "").strip() or _read_jd_cookie_file_raw(cookie_file) + if not raw_cookie: + print( + "[京东] 需要 Cookie：--cookie 或 jd_cookie.txt（--cookie-file）", + file=sys.stderr, + ) + sys.exit(2) + context.clear_cookies() + try: + context.add_cookies(_cookie_header_to_playwright(raw_cookie)) + except Exception as e: + print(f"[京东] add_cookies 警告: {e}", file=sys.stderr) + item_url = f"https://item.jd.com/{str(sku_id).strip()}.html" + matches: list[tuple[int, str, str, dict[str, str]]] = [] + trace_rows: list[dict[str, Any]] = [] + trace_phase = "opening" + + def _on_response(response: Any) -> None: + try: + u = response.url + if rt.log_api_m_jd_trace and "api.m.jd.com" in u: + ct = "" + try: + ct = (response.headers.get("content-type") or "").strip() + except Exception: + pass + trace_rows.append( + { + "phase": trace_phase, + "status": response.status, + "functionId": _jd_function_id_from_api_url(u), + "content_type": ct[:160], + "url": u[:2000], + } + ) + if ( + "api.m.jd.com" in u + and "pc_detailpage_wareBusiness" in u + and "functionId=" in u + ): + st = response.status + body = response.text() + matches.append((st, body, u, dict(response.headers))) + except Exception: + pass + + detail_joined = "" + page.on("response", _on_response) + try: + _wu = (rt.goto_wait_until or "domcontentloaded").strip() + page.goto(item_url, wait_until=_wu, timeout=timeout_ms) + trace_phase = "loaded" + if rt.click_product_detail_tab: + _click_jd_product_detail_tab(page, timeout_ms=timeout_ms) + trace_phase = "after_tab" + extra = min(12_000, max(3_000, timeout_ms // 2)) + page.wait_for_timeout(extra) + if rt.collect_detail_main_image_urls: + try: + detail_joined = scrape_detail_main_body_urls_joined( + page, + wait_ms=min(timeout_ms, 12_000), + runtime=rt, + ) + except Exception as e: + print( + f"[京东] 抽取 #detail-main 图文 URL 失败: {e}", + file=sys.stderr, + ) + finally: + try: + page.remove_listener("response", _on_response) + except Exception: + pass + if rt.log_api_m_jd_trace: + _print_api_m_jd_trace(trace_rows, sku_id=str(sku_id).strip()) + if not matches: + meta: dict[str, Any] = { + "skuId": str(sku_id).strip(), + "request": { + "method": "GET", + "via": "capture_page_navigation", + "item_page": item_url, + "note": "未捕获到 pc_detailpage_wareBusiness（页面可能改版或 Cookie 未登录）", + }, + "response": {"status": 0, "status_text": "", "headers": {}, "body": ""}, + } + if rt.log_api_m_jd_trace: + meta["api_m_jd_trace"] = trace_rows + meta["detail_body_image_urls"] = detail_joined + return 0, "", meta + ok_rows = [m for m in matches if m[0] == 200] + st, body, u, rh = ok_rows[-1] if ok_rows else matches[-1] + meta = { + "skuId": str(sku_id).strip(), + "request": { + "method": "GET", + "url": u, + "via": "capture_page_navigation", + "item_page": item_url, + "captures_count": len(matches), + }, + "response": { + "url": u, + "status": st, + "status_text": "", + "headers": rh, + "body": body, + }, + } + if rt.log_api_m_jd_trace: + meta["api_m_jd_trace"] = trace_rows + meta["detail_body_image_urls"] = detail_joined + return st, body, meta + + +def fetch_ware_business( + context: Any, + page: Any, + sku_id: str, + *, + cookie_file: str | None = None, + timeout_ms: int = 30_000, + cookie_override: str = "", + max_attempts: int = 1, + retry_delay_sec: float = 2.0, + cancel_check: Callable[[], bool] | None = None, + output_sku_and_body_images_only: bool = False, + runtime: WareFetchRuntime | None = None, +) -> tuple[int, str, dict[str, Any]]: + """ + 打开商品页并拦截 ``pc_detailpage_wareBusiness``。 + ``max_attempts``>1 时，在结果为空或失败时按 ``retry_delay_sec`` 间隔重试。 + """ + rt = runtime or WareFetchRuntime() + sid = str(sku_id).strip() + n = max(1, int(max_attempts)) + last: tuple[int, str, dict[str, Any]] = (0, "", {}) + for i in range(n): + if cancel_check is not None and cancel_check(): + return last + if i > 0: + delay = max(0.0, float(retry_delay_sec)) + if delay > 0: + time.sleep(delay) + if cancel_check is not None and cancel_check(): + return last + code, text, meta = _fetch_ware_business_once( + context, + page, + sid, + cookie_file=cookie_file, + timeout_ms=timeout_ms, + cookie_override=cookie_override, + runtime=rt, + ) + last = (code, text, meta) + if output_sku_and_body_images_only and ( + meta.get("detail_body_image_urls") or "" + ).strip(): + if i > 0: + print( + f"[京东] sku={sid} 第 {i + 1} 次尝试已成功（已抽到 #detail-main 图文 URL）", + file=sys.stderr, + ) + break + if not ware_fetch_should_retry(code, text): + if i > 0: + print( + f"[京东] sku={sid} 第 {i + 1} 次尝试已成功", + file=sys.stderr, + ) + break + print( + f"[京东] sku={sid} 详情结果为空或无效 (HTTP {code})，" + f"重试 {i + 1}/{n}…", + file=sys.stderr, + ) + return last + + +__all__ = [ + "WareFetchRuntime", + "_print_http_verbose", + "_read_jd_cookie_file_raw", + "fetch_ware_business", + "scrape_detail_main_body_urls_joined", +] diff --git a/backend/crawler_copy/jd_pc_search/detail/jd_detail_ware_parse.py b/backend/crawler_copy/jd_pc_search/detail/jd_detail_ware_parse.py new file mode 100644 index 0000000..7b65b3e --- /dev/null +++ b/backend/crawler_copy/jd_pc_search/detail/jd_detail_ware_parse.py @@ -0,0 +1,391 @@ +# -*- coding: utf-8 -*- +""" +``pc_detailpage_wareBusiness`` 响应的**纯解析**：JSON → 扁平字段、落盘前 JSON 规整、是否应重试。 + +与 Playwright 拦截、DOM 抽图（``jd_detail_ware_fetch``）分离，便于单测与阅读。 +""" +from __future__ import annotations + +import json +import re +from typing import Any + +# 与 Excel 单元格上限兼顾的默认截断（可被调用方覆盖） +DEFAULT_DETAIL_BODY_MAX_CHARS = 31000 + + +def _normalize_ware_json_tree(obj: Any, *, sort_keys: bool) -> Any: + """递归规整：字典可选按键名排序，列表保持元素顺序。""" + if isinstance(obj, dict): + pairs = [ + (k, _normalize_ware_json_tree(v, sort_keys=sort_keys)) + for k, v in obj.items() + ] + if sort_keys: + pairs.sort(key=lambda kv: kv[0]) + return dict(pairs) + if isinstance(obj, list): + return [_normalize_ware_json_tree(x, sort_keys=sort_keys) for x in obj] + return obj + + +def format_ware_response_text( + text: str, + *, + normalize: bool, + sort_keys: bool, + indent: int, +) -> tuple[str, bool]: + """ + 尝试将接口 body 规整为可读 JSON。 + 返回 (输出文本, 是否已成功按 JSON 处理)。 + """ + if not normalize or not (text or "").strip(): + return text, False + try: + obj = json.loads(text) + except json.JSONDecodeError: + return text, False + obj = _normalize_ware_json_tree(obj, sort_keys=sort_keys) + if indent > 0: + out = json.dumps(obj, ensure_ascii=False, indent=indent) + "\n" + else: + out = json.dumps(obj, ensure_ascii=False, separators=(",", ":")) + "\n" + return out, True + + +def format_ware_response_for_save( + text: str, + *, + normalize: bool = True, + sort_keys: bool = True, + indent: int = 2, +) -> str: + """流水线等落盘用：尽量输出缩进 + 可选键排序的 JSON 文本（失败则保留原文）。""" + body, _ok = format_ware_response_text( + text or "", + normalize=normalize, + sort_keys=sort_keys, + indent=max(0, int(indent)), + ) + return body + + +# 与 pc_detailpage_wareBusiness 响应对应的扁平字段（字符串，缺失为空），顺序即 CSV 建议列序 +WARE_BUSINESS_MERGE_FIELDNAMES: tuple[str, ...] = ( + "detail_sku_title", + "detail_price_final", + "detail_price_original", + "detail_purchase_price", + "detail_shop_name", + "detail_shop_id", + "detail_shop_url", + "detail_vender_id", + "detail_stock_text", + "detail_delivery_promise", + "detail_sku_name", + "detail_product_id", + "detail_main_sku_id", + "detail_page_sku_id", + "detail_brand", + "detail_category_path", + "detail_main_image", + "detail_product_attributes", + "detail_belt_banner", + "detail_csfh_text", + # 来自 DOM #detail-main（非 wareBusiness JSON）；列语义为「详情长图衍生信息」，常为 URL 串，流水线可替换为配料表文本 + "detail_body_ingredients", + # 视觉识别命中时：实际用于解析配料的那张详情长图 URL（自后向前首次通过校验） + "detail_body_ingredients_source_url", +) + + +def _empty_ware_flat() -> dict[str, str]: + return {k: "" for k in WARE_BUSINESS_MERGE_FIELDNAMES} + + +def _strip_htmlish(s: str, *, max_len: int) -> str: + if not s: + return "" + t = re.sub(r"<[^>]+>", " ", s) + t = " ".join(t.split()).strip() + return t[:max_len] if max_len > 0 else t + + +def _s(obj: Any) -> str: + if obj is None: + return "" + return str(obj).strip() + + +def _join_url(u: str) -> str: + u = u.strip() + if not u: + return "" + if u.startswith("//"): + return "https:" + u + return u + + +def _jd_product_image_url(path: str) -> str: + """与搜索侧一致：jfs/ 相对路径补全为可访问 URL。""" + p = (path or "").strip() + if not p: + return "" + if p.startswith("//"): + return "https:" + p + if p.startswith("http://"): + return "https://" + p[7:] + if p.startswith("https://"): + return p[:800] + if p.startswith("jfs/"): + return "https://img13.360buyimg.com/n2/s480x480_" + p[:700] + return p[:800] + + +def flatten_ware_business(obj: Any) -> dict[str, str]: + """ + 将 ``pc_detailpage_wareBusiness`` 的 JSON 根对象压成扁平字符串字典。 + 非 dict 或字段缺失时对应值为空串。 + """ + out = _empty_ware_flat() + if not isinstance(obj, dict): + return out + + sh = obj.get("skuHeadVO") + sh = sh if isinstance(sh, dict) else {} + out["detail_sku_title"] = _s(sh.get("skuTitle"))[:2000] + + price = obj.get("price") + price = price if isinstance(price, dict) else {} + fp = price.get("finalPrice") + fp = fp if isinstance(fp, dict) else {} + out["detail_price_final"] = _s(fp.get("price")) or _s(price.get("p"))[:64] + out["detail_price_original"] = _s(price.get("op")) or _s(price.get("p"))[:64] + + bp = obj.get("bestPromotion") + bp = bp if isinstance(bp, dict) else {} + out["detail_purchase_price"] = _s(bp.get("purchasePrice"))[:64] + + ishop = obj.get("itemShopInfo") + ishop = ishop if isinstance(ishop, dict) else {} + out["detail_shop_name"] = _s(ishop.get("shopName"))[:500] + out["detail_shop_id"] = _s(ishop.get("shopId"))[:32] + out["detail_shop_url"] = _join_url(_s(ishop.get("shopUrl")))[:500] + + pc = obj.get("pageConfigVO") + pc = pc if isinstance(pc, dict) else {} + out["detail_vender_id"] = _s(pc.get("venderId"))[:32] + sk = pc.get("skuid") + out["detail_page_sku_id"] = _s(sk)[:32] + cats = pc.get("catName") + if isinstance(cats, list): + parts = [_s(x) for x in cats if _s(x)] + out["detail_category_path"] = " > ".join(parts)[:500] + src = _s(pc.get("src")) + if src: + out["detail_main_image"] = _jd_product_image_url(src) + + mi = obj.get("mainImageVO") + if isinstance(mi, dict) and not out["detail_main_image"]: + mia = mi.get("mainImageArea") + if isinstance(mia, dict): + iu = _s(mia.get("imageUrl")) + if iu: + out["detail_main_image"] = _jd_product_image_url(iu) + + si = obj.get("stockInfo") + si = si if isinstance(si, dict) else {} + out["detail_stock_text"] = _strip_htmlish(_s(si.get("stockDesc")), max_len=500) + if not out["detail_stock_text"]: + out["detail_stock_text"] = _strip_htmlish(_s(si.get("promiseResult")), max_len=500) + out["detail_delivery_promise"] = _strip_htmlish( + _s(si.get("promiseResult") or si.get("promiseInfoText")), max_len=800 + ) + + wim = obj.get("wareInfoReadMap") + wim = wim if isinstance(wim, dict) else {} + out["detail_sku_name"] = _s(wim.get("sku_name"))[:2000] + out["detail_product_id"] = _s(wim.get("product_id"))[:32] + out["detail_main_sku_id"] = _s(wim.get("main_sku_id"))[:32] + out["detail_brand"] = _s(wim.get("cn_brand"))[:200] + if not out["detail_brand"]: + pav = obj.get("productAttributeVO") + if isinstance(pav, dict): + for it in pav.get("attributes") or []: + if not isinstance(it, dict): + continue + if _s(it.get("labelName")) == "品牌": + out["detail_brand"] = _s(it.get("labelValue"))[:200] + break + + pav = obj.get("productAttributeVO") + attrs: list[str] = [] + if isinstance(pav, dict): + for it in pav.get("attributes") or []: + if not isinstance(it, dict): + continue + ln, lv = _s(it.get("labelName")), _s(it.get("labelValue")) + if ln and lv: + attrs.append(f"{ln}:{lv}") + out["detail_product_attributes"] = "; ".join(attrs)[:4000] + + out["detail_belt_banner"] = _join_url(_s(obj.get("beltBanner")))[:800] + out["detail_csfh_text"] = _s(obj.get("csfhText"))[:200] + + return out + + +# 与 OUT_PARSED_CSV / 流水线 detail CSV 列一致 +WARE_PARSED_CSV_FIELDNAMES: tuple[str, ...] = ( + "skuId", + "http_status", + *WARE_BUSINESS_MERGE_FIELDNAMES, +) + +# 仅 sku + 配料（本脚本 OUTPUT_SKU_AND_BODY_IMAGES_ONLY 时 main 写 CSV/解析 JSON 用） +SKU_BODY_IMAGES_ONLY_FIELDNAMES: tuple[str, ...] = ( + "skuId", + "detail_body_ingredients", +) + +# 与合并表 lean 商详块一致 + SKU；keyword_pipeline DETAIL_WARE_CSV_MODE=lean 写 detail_ware_export.csv（纯中文表头） +DETAIL_WARE_LEAN_CSV_FIELDNAMES: tuple[str, ...] = ( + "SKU", + "品牌", + "到手价", + "店铺名称", + "类目路径", + "商品参数", + "配料表", + "榜单排名", + "促销摘要", +) + +# ``ware_parsed_row`` 可提供的 lean 列（购买者摘要由独立抽取补充） +_DETAIL_WARE_LEAN_FROM_RESPONSE_KEYS: tuple[str, ...] = ( + "skuId", + "detail_brand", + "detail_price_final", + "detail_shop_name", + "detail_category_path", + "detail_product_attributes", + "detail_body_ingredients", +) + + +def detail_ware_lean_csv_row( + sku: str, + http_status: int, + response_text: str, + *, + detail_body_ingredients: str = "", + detail_body_ingredients_source_url: str = "", + buyer_ranking_line: str = "", + buyer_promo_text: str = "", + max_cell_chars: int | None = None, +) -> dict[str, str]: + """lean 详情汇总表一行（无 http_status）；字段来自 ``ware_parsed_row`` 子集 + 购买者摘要列。""" + full = ware_parsed_row( + sku, + http_status, + response_text, + detail_body_ingredients=detail_body_ingredients, + detail_body_ingredients_source_url=detail_body_ingredients_source_url, + max_cell_chars=max_cell_chars, + ) + out = {k: str(full.get(k) or "") for k in _DETAIL_WARE_LEAN_FROM_RESPONSE_KEYS} + out["buyer_ranking_line"] = (buyer_ranking_line or "").strip() + out["buyer_promo_text"] = (buyer_promo_text or "").strip() + _cn = DETAIL_WARE_LEAN_CSV_FIELDNAMES + _en = ( + "skuId", + "detail_brand", + "detail_price_final", + "detail_shop_name", + "detail_category_path", + "detail_product_attributes", + "detail_body_ingredients", + "buyer_ranking_line", + "buyer_promo_text", + ) + return {_cn[i]: str(out.get(_en[i]) or "") for i in range(len(_cn))} + + +def minimal_sku_body_images_row( + sku: str, + detail_body_ingredients: str, + *, + detail_body_ingredients_source_url: str = "", + max_cell_chars: int | None = None, +) -> dict[str, str]: + """``skuId``、配料文本（图源仅内部流程使用，不写入极简 CSV）。""" + _ = detail_body_ingredients_source_url # 保留参数供调用方兼容 + u = (detail_body_ingredients or "").strip() + mxc = max(0, int(max_cell_chars if max_cell_chars is not None else DEFAULT_DETAIL_BODY_MAX_CHARS)) + if mxc and len(u) > mxc: + u = u[:mxc] + return { + "skuId": str(sku).strip(), + "detail_body_ingredients": u, + } + + +def parse_ware_business_response_text(text: str) -> tuple[dict[str, str], bool]: + """ + 解析响应体字符串。 + 返回 ``(扁平字典, 是否成功解析为 JSON 对象)``；失败时扁平字典各键均为 ``""``。 + """ + raw = (text or "").strip() + if not raw: + return _empty_ware_flat(), False + try: + obj = json.loads(raw) + except json.JSONDecodeError: + return _empty_ware_flat(), False + if not isinstance(obj, dict): + return _empty_ware_flat(), False + return flatten_ware_business(obj), True + + +def ware_parsed_row( + sku: str, + http_status: int, + response_text: str, + *, + detail_body_ingredients: str = "", + detail_body_ingredients_source_url: str = "", + max_cell_chars: int | None = None, +) -> dict[str, str]: + """单行扁平结果：``skuId``、``http_status`` 与 ``WARE_BUSINESS_MERGE_FIELDNAMES``（含 DOM 长图 URL 或配料文本）。""" + flat, _ = parse_ware_business_response_text( + response_text if http_status == 200 else "" + ) + row: dict[str, str] = { + "skuId": str(sku).strip(), + "http_status": str(http_status), + **flat, + } + mxc = max(0, int(max_cell_chars if max_cell_chars is not None else DEFAULT_DETAIL_BODY_MAX_CHARS)) + u = (detail_body_ingredients or "").strip() + if u: + row["detail_body_ingredients"] = u[:mxc] if mxc else u + src = (detail_body_ingredients_source_url or "").strip() + if src: + row["detail_body_ingredients_source_url"] = src[:mxc] if mxc else src + return row + + +def ware_fetch_should_retry(http_status: int, response_text: str) -> bool: + """ + True 表示应重试：未命中接口、HTTP 非 200、正文空、非 JSON、或解析后业务字段全空。 + """ + if int(http_status) != 200: + return True + raw = (response_text or "").strip() + if not raw: + return True + flat, ok = parse_ware_business_response_text(raw) + if not ok: + return True + return not any((v or "").strip() for v in flat.values()) diff --git a/backend/crawler_copy/jd_pc_search/jd_competitor_report.py b/backend/crawler_copy/jd_pc_search/jd_competitor_report.py index ee1f9ba..f457981 100644 --- a/backend/crawler_copy/jd_pc_search/jd_competitor_report.py +++ b/backend/crawler_copy/jd_pc_search/jd_competitor_report.py @@ -1,2312 +1,22 @@ # -*- coding: utf-8 -*- """ -关键词 → 调用 ``jd_keyword_pipeline`` 全链路采集 → 生成 **标准化竞品分析报告**（Markdown）。 +在爬虫目录下直接执行时的入口：将 ``backend`` 加入 ``sys.path`` 后调用 +``pipeline.competitor_report.jd_report``（与 ``python -m pipeline.competitor_report.jd_report`` 一致）。 -报告结构对齐常见竞品分析框架：研究范围与方法、执行摘要、**整体市场观察（列表可见度 proxy）**、 -市场与竞争结构、**按细分类目分组的竞品对比矩阵**、价格分析、产品与宣称、**按细分类目的消费者反馈与用户画像**、策略提示与附录；并明确数据边界。 -若运行配置中提供了外部市场规模摘录（``EXTERNAL_MARKET_TABLE_ROWS``），则追加对应表格小节；否则不输出占位行。 +推荐在 ``backend`` 目录执行:: -依赖：全量抓取时与 ``jd_keyword_pipeline.py`` 相同（Node、h5st、Playwright、``common/jd_cookie.txt``）。 -**仅复用已有目录生成报告时**不需要跑浏览器，只需该目录下已有 CSV / ``run_meta.json``。 - -用法： - -- **重新抓取并出报告**：``EXISTING_RUN_DIR = None``，配置 ``KEYWORD``（及可选 ``OVERRIDE_*``），执行 ``python jd_competitor_report.py``。 -- **只分析已有批次**：将 ``EXISTING_RUN_DIR`` 设为 ``pipeline_runs/<时间戳>_<关键词>/`` 的绝对或相对路径（相对当前工作目录）， - 再执行同一命令；**不重新抓取**。关键词优先用本文件 ``KEYWORD``，否则读 ``run_meta.json`` 的 ``keyword``，再否则从目录名 - ``YYYYMMDD_HHMMSS_<词>`` 推断。 - -流水线其余参数（评论分页、延迟等）仍在 ``jd_keyword_pipeline.py`` 顶部配置。 - -输出：在对应运行目录下覆盖写入 ``competitor_analysis.md``。 + python -m pipeline.competitor_report.jd_report """ - from __future__ import annotations -import csv -import json -import math -import re -import statistics import sys -from collections import Counter from pathlib import Path -from typing import Any -_ROOT = Path(__file__).resolve().parent -if str(_ROOT) not in sys.path: - sys.path.insert(0, str(_ROOT)) - -import jd_keyword_pipeline as kpl # noqa: E402 - -# --------------------------------------------------------------------------- -# 运行配置（按需改这里） -# --------------------------------------------------------------------------- -# KEYWORD：京东 PC 搜索词；全量抓取时必填。「仅已有目录」模式下可留空，改从 run_meta / 目录名推断。 -KEYWORD = "低GI" -# 已有流水线目录（含 keyword_pipeline_merged.csv 等）时设为路径则**不重新抓取**，只生成 competitor_analysis.md。 -EXISTING_RUN_DIR = None -# EXISTING_RUN_DIR = r"data\JD\pipeline_runs\20260408_144606_低GI" # 相对数据根或绝对路径 -# 以下非 None 时仅本次运行临时覆盖 jd_keyword_pipeline 中同名变量（不改 pipeline 文件） -OVERRIDE_MAX_SKUS: int | None = None -OVERRIDE_PAGE_START: int | None = None -OVERRIDE_PAGE_TO: int | None = None - -# 评价/预览文本中可统计的「低 GI / 控糖」语境词（命中次数供侧写，非严谨 NLP） -# 可选：第三方市场规模 / 行业增速等（每行四列：指标 | 数值与口径 | 来源 | 年份）。留空则不生成该小节。 -EXTERNAL_MARKET_TABLE_ROWS: tuple[tuple[str, str, str, str], ...] = () - -COMMENT_FOCUS_WORDS: tuple[str, ...] = ( - "口感", - "甜", - "糖", - "血糖", - "控糖", - "低糖", - "无糖", - "饱腹", - "升糖", - "GI", - "gi", - "孕妇", - "老人", - "糖尿病", - "价格", - "贵", - "便宜", - "回购", - "包装", - "物流", -) - -# 用途/场景：每组 (展示名, 触发子串…)。每条评价若命中组内任一子串则该组 +1；同一条可属多组。 -COMMENT_SCENARIO_GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = ( - ("早餐/代餐", ("早餐", "代餐", "早饭", "当早餐", "当早饭", "早上吃", "晨起")), - ("零食/加餐/解馋", ("零食", "加餐", "嘴馋", "小零食", "解馋", "垫肚子", "饿了", "肚子饿", "两餐之间", "间食")), - ("控糖/血糖相关", ("控糖", "血糖高", "升糖", "糖友", "糖尿病", "孕期控糖", "妊娠糖", "血糖")), - ("孕期/育儿", ("孕期", "孕妇", "怀孕", "产妇", "坐月子", "哺乳", "给宝宝", "给娃", "孩子吃", "小孩吃", "宝宝吃")), - ("健身/减脂", ("减肥", "减脂", "瘦身", "健身", "卡路里", "热量低", "低脂")), - ("长辈/家庭", ("老人", "爸妈", "父母", "长辈", "爷爷奶奶", "给家里")), - ("办公/外出", ("办公室", "上班吃", "出门", "外出", "随身带", "包里", "便携")), - ("送礼/囤货", ("送礼", "送人", "囤货", "年货")), - ("夜宵/熬夜", ("夜宵", "熬夜", "晚上饿")), -) - - -def _normalize_focus_words(raw: Any) -> tuple[str, ...]: - if not isinstance(raw, list) or not raw: - return COMMENT_FOCUS_WORDS - out: list[str] = [] - for x in raw[:120]: - s = str(x).strip() - if len(s) > 48: - s = s[:48] - if s: - out.append(s) - return tuple(out) if out else COMMENT_FOCUS_WORDS - - -def _normalize_scenario_groups( - raw: Any, -) -> tuple[tuple[str, tuple[str, ...]], ...]: - if not isinstance(raw, list) or not raw: - return COMMENT_SCENARIO_GROUPS - parsed: list[tuple[str, tuple[str, ...]]] = [] - for item in raw[:40]: - label = "" - triggers: list[str] = [] - if isinstance(item, dict): - label = str(item.get("label") or "").strip()[:80] - tr = item.get("triggers") - if isinstance(tr, list): - for t in tr[:48]: - s = str(t).strip() - if len(s) > 48: - s = s[:48] - if s: - triggers.append(s) - elif isinstance(item, (list, tuple)) and len(item) >= 2: - label = str(item[0]).strip()[:80] - tr = item[1] - if isinstance(tr, (list, tuple)): - for t in tr[:48]: - s = str(t).strip() - if len(s) > 48: - s = s[:48] - if s: - triggers.append(s) - if label and triggers: - parsed.append((label, tuple(triggers))) - return tuple(parsed) if parsed else COMMENT_SCENARIO_GROUPS - - -def _normalize_external_market_rows( - raw: Any, -) -> tuple[tuple[str, str, str, str], ...]: - if not isinstance(raw, list) or not raw: - return EXTERNAL_MARKET_TABLE_ROWS - rows: list[tuple[str, str, str, str]] = [] - - def _four_cells(x: Any) -> tuple[str, str, str, str] | None: - if isinstance(x, (list, tuple)) and len(x) >= 4: - return tuple(str(c)[:500] for c in x[:4]) - if isinstance(x, dict): - a = str(x.get("indicator") or x.get("a") or "").strip()[:500] - b = str(x.get("value_and_scope") or x.get("b") or "").strip()[:500] - c = str(x.get("source") or x.get("c") or "").strip()[:500] - d = str(x.get("year") or x.get("d") or "").strip()[:500] - if any((a, b, c, d)): - return (a, b, c, d) - return None - - for item in raw[:24]: - r = _four_cells(item) - if r: - rows.append(r) - return tuple(rows) if rows else EXTERNAL_MARKET_TABLE_ROWS - - -def resolve_report_tuning( - report_config: dict[str, Any] | None, -) -> tuple[ - tuple[str, ...], - tuple[tuple[str, tuple[str, ...]], ...], - tuple[tuple[str, str, str, str], ...], -]: - if not report_config: - return COMMENT_FOCUS_WORDS, COMMENT_SCENARIO_GROUPS, EXTERNAL_MARKET_TABLE_ROWS - return ( - _normalize_focus_words(report_config.get("comment_focus_words")), - _normalize_scenario_groups(report_config.get("comment_scenario_groups")), - _normalize_external_market_rows( - report_config.get("external_market_table_rows") - ), - ) - - -def _cell(row: dict[str, str], *keys: str) -> str: - for k in keys: - v = str(row.get(k) or "").strip() - if v: - return v - return "" - - -# 合并表列名：lean 仅有搜索侧「类目(...)」；full 或历史文件可能含商详「detail_category_path」。勿用内部键 leaf_category（CSV 中不存在）。 -_MERGED_CATEGORY_KEYS: tuple[str, ...] = ( - "detail_category_path", - "leaf_category", - "类目(leafCategory,cid3Name,catid)", -) -_K_CAT_COL = "类目(leafCategory,cid3Name,catid)" -_K_PROP_COL = "规格属性(propertyList,color,catid,shortName)" - - -def _shortname_from_prop(prop: str) -> str: - m = re.search(r"简称[:：]\s*([^|]+)", prop or "") - return m.group(1).strip()[:120] if m else "" - - -def _category_cell(row: dict[str, str]) -> str: - c = _cell(row, *_MERGED_CATEGORY_KEYS) - if c: - return c - prop = _cell(row, _K_PROP_COL) - sn = _shortname_from_prop(prop) - if re.search(r"类目[:：]\s*\d+", prop): - if sn: - return sn - return "" - return "" - - -def _search_export_catid_to_shortname_map(rows: list[dict[str, str]]) -> dict[str, str]: - """列表导出中叶子类目列常为纯数字 ID：用同行规格属性「简称」映射为可读名称。""" - m: dict[str, str] = {} - for r in rows: - cid = _cell(r, _K_CAT_COL).strip() - if not cid.isdigit(): - continue - if cid in m: - continue - sn = _shortname_from_prop(_cell(r, _K_PROP_COL)) - if sn: - m[cid] = sn - return m - - -def _md_cell(s: str, max_len: int = 120) -> str: - t = (s or "").replace("\r\n", " ").replace("\n", " ").replace("|", "/") - t = " ".join(t.split()) - return (t[:max_len] + "…") if max_len > 0 and len(t) > max_len else t - - -def _read_csv_rows(path: Path) -> tuple[list[str], list[dict[str, str]]]: - if not path.is_file(): - return [], [] - raw = path.read_text(encoding="utf-8-sig") - lines = raw.splitlines() - if not lines: - return [], [] - rdr = csv.DictReader(lines) - fn = rdr.fieldnames or [] - return list(fn), list(rdr) - - -def _float_price(s: str) -> float | None: - if not (s or "").strip(): - return None - m = re.search(r"(\d+(?:\.\d+)?)", str(s).replace(",", "")) - if not m: - return None - try: - return float(m.group(1)) - except ValueError: - return None - - -def _collect_prices(rows: list[dict[str, str]]) -> list[float]: - out: list[float] = [] - price_keys = ( - "detail_price_final", - "标价(jdPrice,jdPriceText,realPrice)", - "券后到手价(couponPrice,subsidyPrice,finalPrice.estimatedPrice,priceShow)", - ) - for row in rows: - for k in price_keys: - p = _float_price(_cell(row, k)) - if p is not None and 0 < p < 1_000_000: - out.append(p) - break - return out - - -def _comment_keyword_hits( - rows: list[dict[str, str]], - focus_words: tuple[str, ...], -) -> Counter[str]: - c: Counter[str] = Counter() - texts: list[str] = [] - for row in rows: - t = _cell(row, "tagCommentContent") - if t: - texts.append(t) - blob = "\n".join(texts) - for w in focus_words: - if len(w) == 1: - continue - n = blob.count(w) - if n: - c[w] += n - return c - - -def _merge_comment_previews(merged_rows: list[dict[str, str]]) -> str: - parts: list[str] = [] - for row in merged_rows: - p = _cell(row, "comment_preview") - if p: - parts.append(p) - return "\n".join(parts) - - -def _iter_comment_text_units( - comment_rows: list[dict[str, str]], - merged_rows: list[dict[str, str]], -) -> list[str]: - """逐条评价正文；无 flat 评论时用合并表 comment_preview 按行兜底。""" - out: list[str] = [] - for row in comment_rows: - t = _cell(row, "tagCommentContent") - if t: - out.append(t) - if out: - return out - for row in merged_rows: - p = _cell(row, "comment_preview") - if p: - out.append(p) - return out - - -_POS_LEX = ( - "好", - "赞", - "满意", - "回购", - "推荐", - "不错", - "喜欢", - "香", - "实惠", - "值得", - "棒", - "鲜嫩", - "好吃", - "划算", - "正品", - "好评", -) -_NEG_LEX = ( - "差", - "烂", - "难吃", - "失望", - "假", - "骗", - "退货", - "不建议", - "硬", - "糟糕", - "难用", - "臭", - "差评", - "不好", - "难喝", - "发霉", -) -# 条形图/摘要用：多字短语优先，避免只显示「硬、差」等单字 -_POS_LEXEME_DETAIL = ( - "已经回购很多次", - "还会再买", - "值得回购", - "推荐购买", - "性价比很高", - "性价比不错", - "物美价廉", - "物流很快", - "包装很用心", - "包装完好", - "口感很好", - "味道不错", - "很好吃", - "香而不腻", - "饱腹感不错", - "控糖很友好", - "低糖很适合", - "代餐很方便", - "品质很稳定", - "值得信赖", -) -_NEG_LEXEME_DETAIL = ( - "口感偏硬", - "口感很硬", - "口感发粘", - "太甜了", - "甜得发腻", - "甜度过高", - "不太好吃", - "很难吃", - "味道很奇怪", - "有股怪味", - "一股异味", - "包装破损", - "漏气受潮", - "日期不新鲜", - "临期产品", - "质量很差", - "不值这个价", - "与描述不符", - "疑似假货", - "发货特别慢", - "物流太慢了", - "售后很差", - "退款很麻烦", - "不建议购买", - "不会再买", -) - - -def _lex_tuple_classify(*parts: tuple[str, ...]) -> tuple[str, ...]: - seen: set[str] = set() - out: list[str] = [] - for tup in parts: - for w in tup: - w = (w or "").strip() - if w and w not in seen: - seen.add(w) - out.append(w) - out.sort(key=len, reverse=True) - return tuple(out) - - -_POS_CLASS = _lex_tuple_classify(_POS_LEX, _POS_LEXEME_DETAIL) -_NEG_CLASS = _lex_tuple_classify(_NEG_LEX, _NEG_LEXEME_DETAIL) -_POS_LEX_HITS = tuple(sorted(_POS_LEXEME_DETAIL, key=len, reverse=True)) -_NEG_LEX_HITS = tuple(sorted(_NEG_LEXEME_DETAIL, key=len, reverse=True)) - - -def _lexeme_hits_in_texts( - texts: list[str], lexemes: tuple[str, ...] -) -> list[dict[str, Any]]: - """每条文本内同一短语只计 1 次；``lexemes`` 宜按长度降序以优先匹配更长表述。""" - c: Counter[str] = Counter() - for raw in texts: - s = (raw or "").strip() - if not s: - continue - seen_line: set[str] = set() - for k in lexemes: - if not k or k not in s: - continue - if k in seen_line: - continue - seen_line.add(k) - c[k] += 1 - return [{"word": w, "texts_matched": n} for w, n in c.most_common(18)] - - -def _comment_sentiment_lexicon(texts: list[str]) -> dict[str, Any]: - """ - 基于预设词表对每条文本做正/负向粗判（非深度学习）；同条同时含正负词时计为「混合」。 - 短语级词频仅在对应语境下统计（条形图用 ``_POS_LEX_HITS`` / ``_NEG_LEX_HITS``）。 - """ - pos_only = neg_only = mixed = neutral = 0 - corpus_pos_mixed: list[str] = [] - corpus_neg_mixed: list[str] = [] - for t in texts: - s = (t or "").strip() - if not s: - neutral += 1 - continue - hp = any(k in s for k in _POS_CLASS) - hn = any(k in s for k in _NEG_CLASS) - if hp and hn: - mixed += 1 - corpus_pos_mixed.append(s) - corpus_neg_mixed.append(s) - elif hp: - pos_only += 1 - corpus_pos_mixed.append(s) - elif hn: - neg_only += 1 - corpus_neg_mixed.append(s) - else: - neutral += 1 - total = len(texts) - pos_lex = _lexeme_hits_in_texts(corpus_pos_mixed, _POS_LEX_HITS) - neg_lex = _lexeme_hits_in_texts(corpus_neg_mixed, _NEG_LEX_HITS) - return { - "method": "keyword_lexicon", - "text_units": total, - "positive_only": pos_only, - "negative_only": neg_only, - "mixed_positive_and_negative": mixed, - "neutral_or_empty": neutral, - "positive_lexicon_sample": list(_POS_LEX[:10]) + list(_POS_LEXEME_DETAIL[:5]), - "negative_lexicon_sample": list(_NEG_LEX[:10]) + list(_NEG_LEXEME_DETAIL[:5]), - "positive_tone_lexeme_hits": pos_lex, - "negative_tone_lexeme_hits": neg_lex, - "lexeme_scope_note": ( - "「正向短语」仅在命中正向词表的评价条内统计（含混合条）；" - "「负向短语」仅在命中负向词表的评价条内统计（含混合条）；每条每短语最多计 1 次；" - "统计的是预设口语片段，非分词模型。" - ), - } - - -def build_comment_sentiment_llm_payload( - texts: list[str], - *, - max_samples_per_tone: int = 14, - max_chars_per_review: int = 240, -) -> dict[str, Any]: - """ - 供大模型做正/负向语义归纳：附规则统计与**去重后的评价原文抽样**（与 §8.2 词表分桶一致）。 - """ - pos_only_texts: list[str] = [] - neg_only_texts: list[str] = [] - mixed_texts: list[str] = [] - for t in texts: - s = (t or "").strip() - if not s: - continue - hp = any(k in s for k in _POS_CLASS) - hn = any(k in s for k in _NEG_CLASS) - if hp and hn: - mixed_texts.append(s) - elif hp: - pos_only_texts.append(s) - elif hn: - neg_only_texts.append(s) - - def _sample(seq: list[str]) -> list[str]: - out: list[str] = [] - seen: set[str] = set() - for raw in seq: - if raw in seen: - continue - seen.add(raw) - if len(raw) > max_chars_per_review: - out.append(raw[:max_chars_per_review] + "…") - else: - out.append(raw) - if len(out) >= max_samples_per_tone: - break - return out - - lex = _comment_sentiment_lexicon(texts) - return { - "comment_sentiment_lexicon": lex, - "sample_reviews_positive_biased": _sample(pos_only_texts), - "sample_reviews_negative_biased": _sample(neg_only_texts), - "sample_reviews_mixed_tone": _sample(mixed_texts)[:8], - } - - -def _mermaid_pie_focus_keywords(hits: Counter[str], *, top_k: int = 8) -> str: - """关注词全局 Top 的 Mermaid pie（便于渲染或导出工具识别）。""" - top = hits.most_common(top_k) - if not top: - return "" - rest = list(hits.most_common()) - if len(rest) > top_k: - others_n = sum(n for _, n in rest[top_k:]) - else: - others_n = 0 - lines = ["```mermaid", 'pie title 关注词命中次数（全局 Top，子串计数）'] - for w, n in top: - if n <= 0: - continue - label = (w or "?").replace('"', "'").replace("\n", " ")[:18] - lines.append(f' "{label}（{n}）" : {n}') - if others_n > 0: - lines.append(f' "其余词合计" : {others_n}') - lines.append("```") - return "\n".join(lines) - - -def _comment_scenario_counts( - texts: list[str], - scenario_groups: tuple[tuple[str, tuple[str, ...]], ...], -) -> tuple[Counter[str], int]: - """每组统计「至少命中一个触发词」的条数。返回 (各组条数, 有效文本条数)。""" - c: Counter[str] = Counter() - n = len(texts) - for blob in texts: - for label, triggers in scenario_groups: - if any(t in blob for t in triggers): - c[label] += 1 - return c, n - - -def _scenario_summary_bullets(counter: Counter[str], n_texts: int, top_k: int = 5) -> list[str]: - if n_texts <= 0 or not counter: - return [] - ordered = counter.most_common() - lines: list[str] = [] - head = ordered[:top_k] - parts = [] - for label, cnt in head: - pct = 100.0 * cnt / n_texts - parts.append(f"「{label}」约 **{cnt}** 条（占有效文本 **{pct:.0f}%**）") - lines.append( - "用户自述的用途/场景（基于预设词组，**非语义分类**）：" + "；".join(parts) + "。" - ) - tail = [lbl for lbl, n in ordered[top_k:] if n > 0] - if tail: - lines.append(f"另有提及较少的场景标签：{'、'.join(tail)}。") - return lines - - -def _sku_to_matrix_group_map( - merged_rows: list[dict[str, str]], sku_header: str -) -> dict[str, str]: - m: dict[str, str] = {} - for row in merged_rows: - sku = _cell(row, sku_header).strip() - if sku: - m[sku] = _competitor_matrix_group_key(row) - return m - - -def _comment_text_units_for_matrix_group( - gname: str, - merged_rows: list[dict[str, str]], - comment_rows_in_group: list[dict[str, str]], - sku_header: str, -) -> list[str]: - """某细类下的评价正文列表；无 flat 时用该细类合并行的 comment_preview。""" - texts: list[str] = [] - for row in comment_rows_in_group: - t = _cell(row, "tagCommentContent") - if t: - texts.append(t) - if texts: - return texts - for row in merged_rows: - if _competitor_matrix_group_key(row) != gname: - continue - p = _cell(row, "comment_preview") - if p: - texts.append(p) - return texts - - -def _group_keyword_hits( - comment_rows_in_group: list[dict[str, str]], - texts_fallback: list[str], - *, - focus_words: tuple[str, ...], -) -> Counter[str]: - h = _comment_keyword_hits(comment_rows_in_group, focus_words) - if h: - return h - if not texts_fallback: - return Counter() - blob = "\n".join(texts_fallback) - c: Counter[str] = Counter() - for w in focus_words: - if len(w) < 2: - continue - n = blob.count(w) - if n: - c[w] += n - return c - - -def _consumer_feedback_by_matrix_group( - *, - merged_rows: list[dict[str, str]], - comment_rows: list[dict[str, str]], - sku_header: str, -) -> list[tuple[str, list[dict[str, str]], list[str]]]: - """ - 与 §5 矩阵同序的细类列表；每项为 (细类名, 该类的 comments_flat 行, 用于场景统计的文本单元)。 - 评价 SKU 不在深入样本时归入「未归类（评价 SKU 无对应深入样本）」。 - """ - if not merged_rows: - if not comment_rows: - return [] - texts = _iter_comment_text_units(comment_rows, []) - return [ - ( - "未归类（无深入合并表）", - list(comment_rows), - texts, - ) - ] - - sku_map = _sku_to_matrix_group_map(merged_rows, sku_header) - by_g: dict[str, list[dict[str, str]]] = {} - for row in comment_rows: - sku = _cell(row, "sku").strip() - g = sku_map.get(sku, "未归类（评价 SKU 无对应深入样本）") - by_g.setdefault(g, []).append(row) - - out: list[tuple[str, list[dict[str, str]], list[str]]] = [] - used: set[str] = set() - for gname, _ in _merged_rows_grouped_for_matrix(merged_rows): - cr = by_g.get(gname, []) - tu = _comment_text_units_for_matrix_group( - gname, merged_rows, cr, sku_header - ) - out.append((gname, cr, tu)) - used.add(gname) - for gname, cr in sorted(by_g.items(), key=lambda x: (-len(x[1]), x[0])): - if gname in used: - continue - tu = _comment_text_units_for_matrix_group( - gname, merged_rows, cr, sku_header - ) - out.append((gname, cr, tu)) - return out - - -def _run_batch_label(run_dir: Path) -> str: - name = run_dir.name - m = re.match(r"^(\d{8})_(\d{6})_", name) - if m: - return f"{m.group(1)} {m.group(2)}" - return name - - -def _resolve_existing_run_dir(raw: str | Path | None) -> Path | None: - if raw is None: - return None - s = str(raw).strip() - if not s: - return None - p = Path(s).expanduser() - if not p.is_absolute(): - p = (Path.cwd() / p).resolve() - else: - p = p.resolve() - return p - - -def _infer_keyword(run_dir: Path, meta: dict[str, Any] | None) -> str: - if meta: - k = str(meta.get("keyword") or "").strip() - if k: - return k - m = re.match(r"^\d{8}_\d{6}_(.+)$", run_dir.name) - if m: - return m.group(1).strip() - return "" - - -def _pc_search_result_count_from_raw( - run_dir: Path, -) -> tuple[int | None, str, list[int], int, int]: - """ - 从 ``pc_search_raw/*.json`` 读取 ``data.resultCount``（京东 PC 搜索接口返回的检索命中规模）。 - 多文件时取众数；返回 (众数值, data.listKeyWord 首见值, 出现过的不同取值升序, 解析到的样本文件数)。 - """ - raw_dir = run_dir / "pc_search_raw" - if not raw_dir.is_dir(): - return None, "", [], 0, 0 - counts: list[int] = [] - list_kw = "" - n_files = 0 - for p in sorted(raw_dir.glob("*.json")): - try: - obj = json.loads(p.read_text(encoding="utf-8")) - except (json.JSONDecodeError, OSError, UnicodeError): - continue - n_files += 1 - if not isinstance(obj, dict): - continue - data = obj.get("data") - if not isinstance(data, dict): - continue - rc = data.get("resultCount") - val: int | None = None - if isinstance(rc, int) and not isinstance(rc, bool) and rc >= 0: - val = rc - elif isinstance(rc, str) and rc.strip().isdigit(): - val = int(rc.strip()) - if val is not None: - counts.append(val) - lk = data.get("listKeyWord") - if isinstance(lk, str) and lk.strip() and not list_kw: - list_kw = lk.strip() - if not counts: - return None, list_kw, [], n_files, 0 - consensus_rc, _freq = Counter(counts).most_common(1)[0] - uniques = sorted(set(counts)) - return consensus_rc, list_kw, uniques, n_files, len(counts) - - -def _brand_cr(cnames: list[str]) -> tuple[float | None, float | None, str, str]: - """按名称计数返回 (第一大主体份额, 前三合计份额, 头部标签, 头部占比展示字符串)。""" - if not cnames: - return None, None, "", "" - cnt = Counter(cnames) - total = sum(cnt.values()) - if total <= 0: - return None, None, "", "" - mc = cnt.most_common() - top1_n = mc[0][1] if mc else 0 - top1 = mc[0][0] if mc else "" - cr1 = top1_n / total - top3_n = sum(n for _, n in mc[:3]) - cr3 = top3_n / total - return cr1, cr3, top1, f"{100.0 * top1_n / total:.1f}%" - - -def _price_stats_extended(prices: list[float]) -> dict[str, Any]: - if not prices: - return {} - out: dict[str, Any] = { - "min": min(prices), - "max": max(prices), - "mean": statistics.mean(prices), - "n": len(prices), - } - if len(prices) >= 2: - out["stdev"] = statistics.stdev(prices) - if len(prices) >= 2: - out["median"] = statistics.median(prices) - if len(prices) >= 4: - s = sorted(prices) - n = len(s) - mid = n // 2 - lower = s[:mid] if n % 2 else s[:mid] - upper = s[mid + 1 :] if n % 2 else s[mid:] - out["q1"] = statistics.median(lower) if lower else s[0] - out["q3"] = statistics.median(upper) if upper else s[-1] - return out - - -def _search_list_proxies(rows: list[dict[str, str]]) -> dict[str, Any]: - """ - 基于 pc_search_export 的「列表可见度」指标，**不是**全渠道零售额或 TAM。 - """ - sku_k = "SKU(skuId)" - shop_k = "店铺名(shopName)" - page_k = "页码(page)" - cat_k = "类目(leafCategory,cid3Name,catid)" - skus: set[str] = set() - shops: set[str] = set() - pages: set[str] = set() - cats: set[str] = set() - for r in rows: - s = _cell(r, sku_k) - if s: - skus.add(s) - sh = _cell(r, shop_k) - if sh: - shops.add(sh) - pg = _cell(r, page_k) - if pg: - pages.add(pg) - c = _cell(r, cat_k) - if c: - cats.add(c) - prices = _collect_prices(rows) - pst = _price_stats_extended(prices) - return { - "total_rows": len(rows), - "unique_skus": len(skus), - "unique_shops": len(shops), - "unique_pages": len(pages), - "page_span": (min((int(p) for p in pages if p.isdigit()), default=None), max((int(p) for p in pages if p.isdigit()), default=None)), - "unique_leaf_cats": len(cats), - "list_price_stats": pst, - } - - -def _category_mix(rows: list[dict[str, str]]) -> list[tuple[str, int]]: - cats: list[str] = [] - for r in rows: - c = _category_cell(r) - if c: - cats.append(c.split(">")[0].strip() if ">" in c else c[:80]) - return Counter(cats).most_common(8) - - -def _category_mix_search_export(rows: list[dict[str, str]]) -> list[tuple[str, int]]: - """PC 搜索导出：优先可读类目名；纯数字 ID 时用「简称」聚合，避免展示无意义类目码。""" - id_names = _search_export_catid_to_shortname_map(rows) - labels: list[str] = [] - for r in rows: - c = _cell(r, _K_CAT_COL).strip() - p = _cell(r, _K_PROP_COL) - if c and not c.isdigit(): - labels.append(c[:80]) - continue - if c.isdigit(): - labels.append( - id_names.get(c) - or _shortname_from_prop(p) - or "未解析类目（列表仅有内部编码且无简称）" - ) - continue - m = re.search(r"类目[:：]\s*(\d+)", p) - if m: - cid = m.group(1) - labels.append( - id_names.get(cid) - or _shortname_from_prop(p) - or "未解析类目（列表仅有内部编码且无简称）" - ) - else: - sn = _shortname_from_prop(p) - if sn: - labels.append(sn) - return Counter(labels).most_common(12) - - -def _structure_shops(rows: list[dict[str, str]], *, list_export: bool) -> list[str]: - if list_export: - return [_cell(r, "店铺名(shopName)") for r in rows if _cell(r, "店铺名(shopName)")] - out: list[str] = [] - for r in rows: - s = _cell(r, "detail_shop_name") or _cell(r, "店铺名(shopName)") - if s: - out.append(s) - return out - - -def _structure_brands(rows: list[dict[str, str]], *, list_export: bool) -> list[str]: - if list_export: - k = "店铺信息标题(shopInfoTitle,brandName)" - return [_cell(r, k) for r in rows if _cell(r, k)] - return [_cell(r, "detail_brand") for r in rows if _cell(r, "detail_brand")] - - -def _structure_category_mix( - rows: list[dict[str, str]], *, list_export: bool -) -> list[tuple[str, int]]: - if list_export: - return _category_mix_search_export(rows) - return _category_mix(rows) - - -def _competitor_matrix_group_key(row: dict[str, str]) -> str: - """ - 竞品矩阵分组：使「饼干」「面条」等同细类同表。 - - 路径 ≥4 段：取倒数第二段（如 … > 面条 > 挂面 → 面条）。 - - 路径 3 段：取中间段（如 休闲食品 > 饼干 > 粗粮饼干 → 饼干）。 - - 路径 2 段：取第二段；1 段：取该段。 - """ - c = _category_cell(row) - if not c: - return "未归类（无类目路径）" - parts = [p.strip() for p in c.replace("＞", ">").split(">") if p.strip()] - if not parts: - return "未归类（无类目路径）" - if len(parts) >= 4: - return parts[-2] - if len(parts) >= 3: - return parts[1] - if len(parts) >= 2: - return parts[1] - return parts[0] - - -def _merged_rows_grouped_for_matrix( - merged_rows: list[dict[str, str]], -) -> list[tuple[str, list[dict[str, str]]]]: - buckets: dict[str, list[dict[str, str]]] = {} - for row in merged_rows: - k = _competitor_matrix_group_key(row) - buckets.setdefault(k, []).append(row) - - def sort_key(item: tuple[str, list[dict[str, str]]]) -> tuple[int, int, str]: - name, rows = item - miss = name.startswith("未归类") - return (1 if miss else 0, -len(rows), name) - - return sorted(buckets.items(), key=sort_key) - - -def _is_ingredient_url_blob(s: str) -> bool: - """详情主图 URL 串（分号分隔）或单列以 http 开头。""" - t = (s or "").strip() - if not t: - return False - if t.startswith(("http://", "https://")): - return True - head = t[:400] - if ("https://" in head or "http://" in head) and ( - ";" in t or len(t) > 180 or t.count("http") >= 2 - ): - return True - return False - - -def _ingredients_from_product_attributes(attrs: str) -> str: - m = re.search(r"配料(?:表)?[:：]\s*([^;；]+)", attrs or "") - return m.group(1).strip() if m else "" - - -def _ingredients_single_line(s: str) -> str: - """与 ``AI_crawler.normalize_ingredients_text_for_csv`` 一致：多行配料压成一行（行间 ``；``），便于表格/CSV。""" - t = (s or "").replace("\r\n", "\n").replace("\r", "\n").strip() - if not t: - return "" - lines = [ln.strip() for ln in t.split("\n") if ln.strip()] - if len(lines) <= 1: - return lines[0] if lines else "" - return "；".join(lines) - - -def _matrix_ingredients_cell(row: dict[str, str], *, max_len: int = 420) -> str: - """ - 优先 ``detail_body_ingredients``（配料 OCR/文本）；旧合并表可能为 ``detail_body_image_urls``。 - 若为 URL 串则尝试 ``detail_product_attributes`` 中的「配料/配料表：」片段。 - """ - raw = _cell(row, "detail_body_ingredients", "detail_body_image_urls") - if raw and not _is_ingredient_url_blob(raw): - return _md_cell(_ingredients_single_line(raw), max_len) - from_attr = _ingredients_from_product_attributes( - _cell(row, "detail_product_attributes") - ) - if from_attr: - return _md_cell(from_attr, max_len) - if raw and _is_ingredient_url_blob(raw): - return _md_cell( - "（详情长图链接，无配料正文；可在采集侧开启配料识别后重新跑批次）", - max_len, - ) - return "—" - - -def _competitor_matrix_md_line( - row: dict[str, str], *, sku_header: str, title_h: str -) -> str: - sku = _md_cell(_cell(row, sku_header), 14) - title = _md_cell(_cell(row, title_h), 56) - brand = _md_cell(_cell(row, "detail_brand"), 16) - pj = _md_cell(_cell(row, "标价(jdPrice,jdPriceText,realPrice)"), 10) - df = _md_cell(_cell(row, "detail_price_final"), 10) - shop = _md_cell(_cell(row, "店铺名(shopName)", "detail_shop_name"), 22) - sell = _md_cell(_cell(row, "卖点(sellingPoint)"), 36) - rank = _md_cell( - _cell(row, "榜单类文案(标签/腰带/标题数组中的榜、TOP 等)"), 28 - ) - cat = _md_cell(_category_cell(row), 24) - ing = _matrix_ingredients_cell(row) - cc = _md_cell(_cell(row, "评价量(commentFuzzy)"), 10) - prev = _md_cell(_cell(row, "comment_preview"), 72) - return ( - f"| {sku} | {title} | {brand} | {pj} | {df} | {shop} | {sell} | {rank} | " - f"{cat} | {ing} | {cc} | {prev} |" - ) - - -def _strategy_hints( - *, - cr1: float | None, - pst: dict[str, Any], - hits: Counter[str], - n_comments: int, - scen_counts: Counter[str], - scen_n_texts: int, -) -> list[str]: - """基于规则的「提示性」结论，均标注待验证。""" - hints: list[str] = [] - if cr1 is not None and cr1 >= 0.45: - hints.append( - "样本内品牌集中度较高（第一大品牌份额偏高），头部玩家占据显著曝光；新原料/解决方案宜明确差异化价值主张（**需线下渠道与招商信息交叉验证**）。" - ) - elif cr1 is not None and cr1 < 0.25: - hints.append( - "样本内品牌较分散，品类或关键词下竞争格局未固化，存在定位与叙事空间（**需扩大样本页数与关键词矩阵验证**）。" - ) - if pst.get("stdev") and pst.get("mean") and pst["mean"] > 0: - cv = pst["stdev"] / pst["mean"] - if cv > 0.35: - hints.append( - "价格离散度较高，同时存在偏低价与偏高价陈列，可分别对标「性价比带」与「品质/功能带」竞品（**终端到手价受促销影响，非成本结构**）。" - ) - if hits: - top = hits.most_common(3) - top_s = "、".join(w for w, _ in top) - hints.append( - f"评价文本中「{top_s}」等主题出现较多，可作为消费者沟通与产品卖点的假设输入（**非严格主题模型，建议人工抽样复核**）。" - ) - if n_comments < 5: - hints.append( - "有效评价样本偏少，消费者洞察部分仅作方向参考，正式结论建议加大 SKU 数或评论分页。" - ) - if scen_n_texts >= 5 and scen_counts: - top_lbl, top_n = scen_counts.most_common(1)[0] - share = top_n / scen_n_texts - if share >= 0.25: - hints.append( - f"用途/场景中「{top_lbl}」在约 {100 * share:.0f}% 的有效评价自述中出现，可作为沟通场景与卖点的优先假设（**词组规则，建议抽样核对原句**）。" - ) - if not hints: - hints.append( - "当前样本下自动规则未触发强信号；请结合业务目标人工解读对比矩阵与原始 CSV。" - ) - return hints - - -def _embed_chart(run_dir: Path, filename: str, caption: str = "") -> list[str]: - """若 ``report_assets/`` 存在则返回插图 Markdown 片段。""" - if not (run_dir / "report_assets" / filename).is_file(): - return [] - cap = (caption or "").strip() - out: list[str] = [] - if cap: - out.append(f"*{cap}*") - out.append("") - out.append(f"![](report_assets/{filename})") - out.append("") - return out - - -def _scenario_group_asset_slug(group: str, index: int) -> str: - """与 ``pipeline.report_charts`` 中场景分组图文件名规则一致（勿改格式）。""" - raw = (group or "").strip() - core = re.sub(r"[^\w\u4e00-\u9fff-]", "", raw)[:20] - if not core: - core = "group" - return f"i{index:02d}_{core}" - - -def _scenario_group_usage_bar_filename(group: str, index: int) -> str: - """场景/用途按细类条形图（横轴为占有效文本比例 %，与报告表格口径一致）。""" - slug = _scenario_group_asset_slug(group, index) - return f"chart_usage_scenarios_bar__{slug}.png" - - -def _focus_keywords_group_bar_filename(group: str, index: int) -> str: - slug = _scenario_group_asset_slug(group, index) - return f"chart_focus_keywords_bar__{slug}.png" - - -def _lines_4_reading_brand( - *, - cr1: float | None, - cr3: float | None, - top: str, - brand_rows_n: int, - n_structure: int, -) -> list[str]: - if cr1 is None or not (top or "").strip(): - return [] - lines = [ - "", - "**数据解读（规则摘要）**：", - "", - f"- 在含品牌字段的 **{brand_rows_n}** 条列表行（占本章结构样本 **{n_structure}** 行）中，" - f"「{_md_cell(top.strip(), 36)}」曝光约占 **{100 * cr1:.1f}%**（按行计，同一 SKU 多行会重复计）。", - ] - if cr3 is not None: - lines.append( - f"- 前三品牌合计约 **{100 * cr3:.1f}%**；若该比例偏高，说明搜索页品牌集中度高，" - "新品需搭配清晰的差异定位与资源投放，避免与头部在泛词下正面撞车。" - ) - lines.append("") - return lines - - -def _lines_4_reading_shop( - *, - cr1: float | None, - cr3: float | None, - top: str, - shop_rows_n: int, - n_structure: int, -) -> list[str]: - if not shop_rows_n: - return [] - lines = [ - "", - "**数据解读（规则摘要）**：", - "", - f"- 含店铺名的列表行共 **{shop_rows_n}** 条（结构样本 **{n_structure}** 行），反映搜索曝光下的店铺格局。", - ] - if cr1 is not None and (top or "").strip(): - lines.append( - f"- 第一大店铺「{_md_cell(top.strip(), 40)}」约占 **{100 * cr1:.1f}%**；" - "该指标刻画的是**列表可见度**而非销量，适合用于判断货架被哪些店铺占据。" - ) - if cr3 is not None: - lines.append( - f"- 前三店铺合计约 **{100 * cr3:.1f}%**；若集中度高，可考虑从店铺矩阵、旗舰店/专营店布局等角度拆解竞争。" - ) - lines.append("") - return lines - - -def _lines_4_reading_category( - cm_structure: list[tuple[str, int]], -) -> list[str]: - if not cm_structure: - return [] - total = sum(c for _, c in cm_structure) - if total <= 0: - return [] - top_lbl, top_c = cm_structure[0] - share = top_c / total - lines = [ - "", - "**数据解读（规则摘要）**：", - "", - f"- 列表侧可读类目/简称共 **{len(cm_structure)}** 种取值，合计 **{total}** 行；" - f"其中「{_md_cell(top_lbl, 40)}」行数最多，约占 **{100 * share:.1f}%**。", - "- 若头部类目占比极高，说明当前关键词下货架被少数品类定义；跨品类机会需结合商详矩阵（§5）再核对。", - "", - "| 类目/简称（Top 5） | 列表行数 | 占本章结构样本 |", - "| --- | ---: | ---: |", - ] - for lbl, cnt in cm_structure[:5]: - lines.append( - f"| {_md_cell(lbl, 36)} | {cnt} | {100 * cnt / total:.1f}% |" - ) - lines.append("") - return lines - - -def build_competitor_markdown( - *, - run_dir: Path, - keyword: str, - merged_rows: list[dict[str, str]], - search_export_rows: list[dict[str, str]], - comment_rows: list[dict[str, str]], - meta: dict[str, Any] | None, - report_config: dict[str, Any] | None = None, - llm_sentiment_section_md: str | None = None, -) -> str: - focus_words, scenario_groups, external_rows = resolve_report_tuning(report_config) - sku_header = "SKU(skuId)" - title_h = "标题(wareName)" - batch = _run_batch_label(run_dir) - n_sku = len(merged_rows) - n_cmt = len(comment_rows) - - list_export = len(search_export_rows) > 0 - structure_rows = search_export_rows if list_export else merged_rows - n_structure = len(structure_rows) - shops_s = _structure_shops(structure_rows, list_export=list_export) - brands_s = _structure_brands(structure_rows, list_export=list_export) - cr1_shop, cr3_shop, top_shop_s, _ = _brand_cr(shops_s) - cr1_list_brand, cr3_list_brand, top_list_brand, _ = _brand_cr(brands_s) - cm_structure = _structure_category_mix(structure_rows, list_export=list_export) - min_brand_rows = max(5, int(0.02 * n_structure)) if n_structure else 5 - - brands_deep = [_cell(r, "detail_brand") for r in merged_rows if _cell(r, "detail_brand")] - cr1_deep, cr3_deep, top_brand_deep, _top_share_deep = _brand_cr(brands_deep) - cr1_hints = ( - cr1_shop if list_export and cr1_shop is not None else cr1_deep - ) - - pst_merged = _price_stats_extended(_collect_prices(merged_rows)) - pst_list = ( - _price_stats_extended(_collect_prices(search_export_rows)) - if list_export - else {} - ) - # 价格分析（§2 要点、§6、策略提示）：优先「列表全量」；无列表或无解析价时再用合并表深入样本 - pst = ( - pst_list - if list_export and pst_list.get("n", 0) > 0 - else pst_merged - ) - price_analysis_basis_cn = ( - f"PC 搜索列表导出共 **{len(search_export_rows)}** 行中的展示价（标价/券后等）" - if list_export and pst_list.get("n", 0) > 0 - else f"已深入抓取的 **{n_sku}** 个 SKU 合并数据中的展示价" - ) - - hits = _comment_keyword_hits(comment_rows, focus_words) - if not hits: - blob = _merge_comment_previews(merged_rows) - for w in focus_words: - if len(w) < 2: - continue - n = blob.count(w) - if n: - hits[w] += n - - comment_texts = _iter_comment_text_units(comment_rows, merged_rows) - sentiment_lex = _comment_sentiment_lexicon(comment_texts) - scen_counts, scen_n_texts = _comment_scenario_counts( - comment_texts, scenario_groups - ) - - feedback_groups = _consumer_feedback_by_matrix_group( - merged_rows=merged_rows, - comment_rows=comment_rows, - sku_header=sku_header, - ) - matrix_groups_for_exec = _merged_rows_grouped_for_matrix(merged_rows) - multi_feedback_cat = len(matrix_groups_for_exec) >= 2 - - ( - api_rc, - api_list_kw, - api_rc_uniques, - api_raw_json_n, - api_rc_n_values, - ) = _pc_search_result_count_from_raw(run_dir) - - has_external_market = bool(external_rows) - - lines: list[str] = [ - f"# 竞品分析报告（京东 PC 渠道）", - "", - f"> **监测主题**：{keyword} ", - f"> **数据批次**：{batch} ", - f"> **报告生成**：自动化草稿，**仅供内部研讨**，不构成市场承诺或投资建议。", - "", - "---", - "", - "## 一、研究范围、数据来源与局限", - "", - "### 1.1 研究范围", - "", - f"- **搜索关键词**：「{keyword}」", - f"- **分析对象**：流水线拉取的 **{n_sku}** 个 SKU（搜索排序靠前子样本，非全站普查）。", - ] - if meta: - lines.append( - f"- **搜索列表页**：逻辑第 **{meta.get('page_start')}** 页至第 **{meta.get('page_to')}** 页；" - f"搜索导出共 **{meta.get('pc_search_export_rows', '—')}** 行（含未深入拉详情的商品）。" - ) - lines.extend( - [ - "", - "### 1.2 数据来源", - "", - "- **渠道**：京东 PC 端公开商品列表、商详与评价等可访问数据。", - "- **可追溯**：原始表格与接口响应保存在本批次任务输出目录，供内部复核；对外分享请脱敏。", - "", - "### 1.3 方法说明（指标含义）", - "", - "- **价格**：自页面「标价 / 券后价 / 详情价」等抽取的**展示价**，含促销与规格差异，**不等于**出厂价或成本。**第六章** 在具备可用的搜索列表导出时，优先以**列表全量**统计；否则使用**已深入 SKU** 的合并数据。", - "- **品牌/店铺集中度（第四章）**：有列表全量时按列表行计店铺与品牌占比；无列表导出时按深入 SKU 合并表估算。", - "- **评价主题词**：对评价正文做**预设词表子串计数**，非分词主题模型，适合扫方向，**需抽样人工验证**。", - "- **用途/场景**：对每条评价独立判断是否命中预设场景词；一条可计入多个场景，统计的是「提及该场景的评价条数」而非用户数。", - "- **用户画像（第八章）**：正负面粗判含**口语短语**级摘录；关注词与场景**仅按细类**以条形图展示（场景图为**占该细类有效文本比例 %**）；见 §8.3～8.4。", - "- **检索结果规模**：来自京东 PC 搜索返回的「结果条数」类指标，表示平台侧申报的匹配数量级，**不等于**动销、库存或独立 SKU 数。", - "", - "### 1.4 主要局限", - "", - "- 仅覆盖 **京东 PC**，不含天猫、抖音、线下、B2B 原料端。", - "- 样本量由本次抓取上限与搜索页数决定，**结论外推需谨慎**。", - "- 详情配料与宣称以页面展示为准，**与真实配方可能不一致**（合规与实测另议）。", - ( - "- **行业零售额、TAM、CAGR 等**：无法从本批次数据推导；本报告已纳入任务中配置的第三方摘录，见 **§3.5**。" - if has_external_market - else "- **行业零售额、TAM、CAGR 等**：无法从本批次数据推导；本报告未纳入外部摘录（可在任务报告调参中维护市场信息表）。" - ), - "", - "---", - "", - "## 二、执行摘要（要点）", - "", - ] - ) - - exec_bullets: list[str] = [] - exec_bullets.append( - f"在关键词「{keyword}」下，本次深入分析 **{n_sku}** 个 SKU，关联评价文本 **{n_cmt}** 条。" - ) - if list_export and cr1_shop is not None and top_shop_s: - src = f"列表全量 **{n_structure}** 行" - if cr3_shop is not None: - exec_bullets.append( - f"竞争结构（{src}，§4）：**店铺** 第一大店铺份额 ≈ **{100 * cr1_shop:.1f}%**（「{top_shop_s}」），" - f"前三店铺合计份额 ≈ **{100 * cr3_shop:.1f}%**（按列表行计，同一 SKU 多行会重复计）。" - ) - else: - exec_bullets.append( - f"竞争结构（{src}，§4）：**店铺** 第一大店铺份额 ≈ **{100 * cr1_shop:.1f}%**（「{top_shop_s}」）。" - ) - elif not list_export and cr1_deep is not None and top_brand_deep: - if cr3_deep is not None: - exec_bullets.append( - f"竞争结构（无列表导出，§4 用深入合并表）：**品牌** 第一大品牌份额 ≈ **{100 * cr1_deep:.1f}%**（「{top_brand_deep}」），" - f"前三品牌合计份额 ≈ **{100 * cr3_deep:.1f}%**。" - ) - else: - exec_bullets.append( - f"竞争结构（无列表导出，§4 用深入合并表）：**品牌** 第一大品牌份额 ≈ **{100 * cr1_deep:.1f}%**（「{top_brand_deep}」）。" - ) - if ( - list_export - and len(brands_s) >= min_brand_rows - and cr1_list_brand is not None - and top_list_brand - ): - if cr3_list_brand is not None: - exec_bullets.append( - f"同批列表中**品牌信息有效** **{len(brands_s)}** 条：**品牌** 第一大品牌份额 ≈ **{100 * cr1_list_brand:.1f}%**（「{top_list_brand}」），" - f"前三品牌合计份额 ≈ **{100 * cr3_list_brand:.1f}%**。" - ) - else: - exec_bullets.append( - f"同批列表中**品牌信息有效** **{len(brands_s)}** 条：**品牌** 第一大品牌份额 ≈ **{100 * cr1_list_brand:.1f}%**（「{top_list_brand}」）。" - ) - elif list_export and cr1_deep is not None and top_brand_deep and not brands_s: - exec_bullets.append( - f"列表导出缺少品牌标题字段，**深入 {n_sku} SKU** 商详品牌第一大品牌份额 ≈ **{100 * cr1_deep:.1f}%**（「{top_brand_deep}」），供与 §5 矩阵对照。" - ) - if pst: - price_src_short = ( - "（列表全量）" - if list_export and pst_list.get("n", 0) > 0 - else "（深入样本）" - ) - exec_bullets.append( - f"展示价格{price_src_short}：可解析价格 **{pst['n']}** 个观测，区间约 **{pst['min']:.2f}～{pst['max']:.2f}** 元，" - f"中位数 **{pst.get('median', pst['mean']):.2f}** 元。" - ) - if multi_feedback_cat and (hits or scen_n_texts > 0): - exec_bullets.append( - "评价侧写（关注词、用途/场景）已按 **§5 同款细类** 分节，见 **§8.3～8.4**。" - ) - elif hits: - top3 = "、".join(f"「{w}」({n})" for w, n in hits.most_common(3)) - exec_bullets.append(f"评价侧写（词频）：{top3}。") - if scen_n_texts > 0 and scen_counts and not multi_feedback_cat: - top_s = scen_counts.most_common(4) - frag = "；".join(f"{lbl} **{n}** 条" for lbl, n in top_s) - exec_bullets.append(f"用途/场景（评价自述，可多选）：{frag}（有效文本 **{scen_n_texts}** 条）。") - if api_rc is not None: - exec_bullets.append( - f"PC 搜索返回的检索结果规模约 **{api_rc:,}**（站内匹配条数量级，见 §3.2；非零售额口径）。" - ) - for b in exec_bullets: - lines.append(f"- {b}") - if not exec_bullets: - lines.append("- 当前批次可汇总要点较少（以正文各节实际输出为准）。") - - proxy = _search_list_proxies(search_export_rows) if search_export_rows else {} - lines.extend(["", "---", "", "## 三、整体市场观察（渠道可见度 proxy · 非官方规模）", ""]) - lines.extend( - [ - "### 3.1 与「市场规模」的区别", - "", - "- **官方/行业市场规模**（如全国零售额、品类增速、渗透率）通常来自 **Euromonitor、行业协会、上市公司年报、券商研报** 等；**不能**用京东搜索返回条数或 SKU 数直接等同。", - "- **§3.2** 使用搜索接口返回的**检索结果规模**字段；**§3.3～3.4** 描述本次导出的列表行、去重 SKU/店铺及列表价，用作 **proxy（参照）**，外推全市场需谨慎。", - "", - "### 3.2 接口返回的检索规模", - "", - ] - ) - if api_rc is not None: - lines.append( - f"- 根据本批次保存的搜索原始响应解析：监测词「**{keyword}**」下，平台申报的检索匹配规模约 **{api_rc:,}**。" - ) - if api_list_kw: - lines.append( - f"- 同批响应中的列表关键词：**{api_list_kw}**（可与监测词对照是否一致）。" - ) - if len(api_rc_uniques) > 1: - nums = "、".join(f"{u:,}" for u in api_rc_uniques) - lines.append( - f"- 注：多份原始响应中该规模字段曾出现不同取值（{nums}），正文取**众数** **{api_rc:,}**（共 {api_rc_n_values} 次有效读取）。" - ) - elif api_raw_json_n > 0: - lines.append( - f"- 已扫描 **{api_raw_json_n}** 份原始响应并完成读取。" - ) - lines.extend( - [ - "- **含义**：平台对该关键词给出的**检索匹配条数量级**，用于感受站内商品池「宽度」；可能含不同类目/规格条目，**不等于**独立 SKU 数、动销或 GMV，且会随索引与运营策略变化。", - "", - ] - ) - else: - lines.append( - "*未能从本批次搜索原始响应中解析到有效的检索规模字段（目录缺失、无可用响应或字段为空）。*" - ) - lines.append("") - - lines.extend(["### 3.3 搜索列表规模（本次抓取范围内的可见 SKU / 店铺）", ""]) - if proxy.get("total_rows", 0) > 0: - pmin, pmax = proxy.get("page_span") or (None, None) - span_txt = ( - f"页码（去重）约 **{pmin}～{pmax}** 页" - if pmin is not None and pmax is not None - else "页码字段缺失或无法解析" - ) - lines.extend( - [ - f"- **列表导出行数**：**{proxy['total_rows']}** 行。", - f"- **去重 SKU 数**：**{proxy['unique_skus']}**；**去重店铺数**：**{proxy['unique_shops']}**；{span_txt}。", - f"- **列表中去重叶子类目代码/片段数**（粗略）：**{proxy['unique_leaf_cats']}**（同一关键词下品类宽度 proxy）。", - "", - ] - ) - lpst = proxy.get("list_price_stats") or {} - lines.extend(["### 3.4 列表端展示价（全导出，非仅深入样本）", ""]) - if lpst: - lines.extend( - [ - f"- 自列表「标价 / 券后价」解析到 **{lpst['n']}** 个数值价；" - f"区间约 **{lpst['min']:.2f}～{lpst['max']:.2f}** 元，" - f"中位数 **{float(lpst.get('median', lpst['mean'])):.2f}** 元。", - "- **说明**：第六章价格统计表已与上表同源（均为列表全量，条件满足时）；若正文第六章标注为合并表样本，则因无可用列表价而退化。深入 SKU 的详情价可与列表价对照。", - "", - ] - ) - else: - lines.append("*列表导出中未能解析出数值价格。*") - lines.append("") - else: - lines.append( - "*未读到可用的搜索列表导出或文件为空；§3.3～3.4 无列表侧数据。*" - ) - lines.append("") - lines.extend(["### 3.4 列表端展示价（全导出）", "", "*无列表数据。*", ""]) - - if external_rows: - lines.extend( - [ - "### 3.5 外部市场规模与行业信息（运行配置摘录）", - "", - "以下为本次任务报告调参中维护的**第三方市场摘录**，可与 §3.2 检索规模及 §3.3～3.4 列表参照对照使用；口径与真实性以原出处为准。", - "", - "| 指标 | 数值与口径 | 来源 | 年份 |", - "| --- | --- | --- | --- |", - ] - ) - for a, b, c, d in external_rows: - lines.append( - f"| {_md_cell(a, 40)} | {_md_cell(b, 48)} | {_md_cell(c, 36)} | {_md_cell(d, 12)} |" - ) - lines.append("") - - ch4_heading = ( - "## 四、市场与竞争结构（PC 搜索列表全量）" - if list_export - else "## 四、市场与竞争结构（深入合并表 · 无列表导出）" - ) - lines.extend(["", "---", "", ch4_heading, ""]) - if list_export: - lines.append( - f"基于**搜索列表导出**共 **{n_structure}** 行，与 §3.3 一致；" - f"集中度按**列表行**计数（同一 SKU 多次曝光则重复计）。" - ) - else: - lines.append( - f"*未读到可用列表全量行，以下退化为**深入 SKU 合并样本** **{n_structure}** 行。*" - ) - lines.append("") - - lines.extend(["### 4.1 品牌分布与集中度", ""]) - brand_rows_n = len(brands_s) - show_list_brand_cr = list_export and brand_rows_n >= min_brand_rows - show_merged_brand_cr = not list_export and brand_rows_n > 0 - if (show_list_brand_cr or show_merged_brand_cr) and cr1_list_brand is not None: - lines.append("| 指标 | 数值 |") - lines.append("| --- | --- |") - lines.append(f"| 含品牌字段的列表行数 | {brand_rows_n} |") - lines.append( - f"| 第一大品牌份额（按行计） | {100 * cr1_list_brand:.1f}%（{_md_cell(top_list_brand, 36)}） |" - ) - if cr3_list_brand is not None: - lines.append(f"| 前三品牌合计份额（按行计） | {100 * cr3_list_brand:.1f}% |") - lines.append("") - lines.extend( - _embed_chart( - run_dir, - "chart_brand_rows_pie.png", - "品牌列表曝光占比（扇形图，Top 段合并为「其他」；与上表集中度同源）", - ) - ) - lines.extend( - _lines_4_reading_brand( - cr1=cr1_list_brand, - cr3=cr3_list_brand, - top=top_list_brand or "", - brand_rows_n=brand_rows_n, - n_structure=n_structure, - ) - ) - lines.append( - "*更细行数分布见结构化摘要 JSON 的 ``list_brand_mix_top``。*" - ) - elif list_export: - lines.append( - f"*列表导出中店铺/品牌标题有效 **{brand_rows_n}** 条，" - f"低于建议阈值（≥{min_brand_rows}），品牌集中度未展开。**店铺结构见 §4.2**；" - f"商详品牌在 **§5**。*" - ) - else: - lines.append("*深入子样本无可用品牌字段。*") - lines.append("") - - lines.extend(["### 4.2 店铺分布与集中度", ""]) - shop_rows_n = len(shops_s) - if shops_s: - lines.append("| 指标 | 数值 |") - lines.append("| --- | --- |") - lines.append(f"| 含店铺名的行数 | {shop_rows_n} |") - if cr1_shop is not None and top_shop_s: - lines.append( - f"| 第一大店铺份额（按行计） | {100 * cr1_shop:.1f}%（{_md_cell(top_shop_s, 40)}） |" - ) - if cr3_shop is not None: - lines.append(f"| 前三店铺合计份额（按行计） | {100 * cr3_shop:.1f}% |") - lines.append("") - lines.extend( - _embed_chart( - run_dir, - "chart_shop_rows_pie.png", - "店铺列表曝光占比（扇形图；与上表同源）", - ) - ) - lines.extend( - _lines_4_reading_shop( - cr1=cr1_shop, - cr3=cr3_shop, - top=top_shop_s or "", - shop_rows_n=shop_rows_n, - n_structure=n_structure, - ) - ) - lines.append( - "*更细店铺行数分布见结构化摘要 ``list_shop_mix_top``。*" - ) - else: - lines.append("*无店铺字段。*") - lines.append("") - - lines.extend(["### 4.3 类目/叶子类目（列表字段 Top）", ""]) - if cm_structure: - lines.extend( - _embed_chart( - run_dir, - "chart_category_mix_pie.png", - "类目/可读名称分布（扇形图；已用列表「简称」替代裸类目码）", - ) - ) - lines.extend(_lines_4_reading_category(cm_structure)) - lines.append( - "*完整类目行数见结构化摘要 ``category_mix_top``。*" - ) - else: - lines.append("*无类目列或无法解析。*") - lines.append("") - - lines.extend( - [ - "---", - "", - "## 五、竞品对比矩阵（按细分类目分组）", - "", - "优先按商详**类目路径**列分组：**三级路径**取中间一段（如 … > **饼干** > 粗粮饼干），" - "**四级及以上**取倒数第二段（如 … > **面条** > 挂面）。若该列为空，退化为搜索列表中的类目或规格属性；仍无则「未归类」。全量合并模式下另有更多商详字段可供核对。", - "", - "维度说明：**产品**（标题/规格）、**价格**（列表展示）、**渠道**（京东店铺）、**推广**（卖点/榜单文案）、" - "**类目**、**配料表**（见下）、**声量**（评价量与摘要）。", - "", - "**配料表**：优先使用配料正文列（开启配料视觉解析时为识别出的文字）；" - "仅有详情长图链接时列内会提示；若商详参数含「配料/配料表：」则摘录该段。" - "均为页面信息摘录，**以包装实物与法规标签为准**。", - "", - ] - ) - matrix_header = [ - "| SKU | 产品（标题） | 品牌 | 标价 | 详情价 | 渠道（店铺） | 推广（卖点） | 榜单/标签 | 类目 | 配料表 | 评价量(搜索) | 消费者反馈摘要 |", - "| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |", - ] - grouped_matrix = _merged_rows_grouped_for_matrix(merged_rows) - if not grouped_matrix: - lines.append("*无合并表 SKU。*") - lines.append("") - for gname, grows in grouped_matrix: - lines.append(f"### {gname}（**{len(grows)}** 款）") - lines.append("") - lines.extend(matrix_header) - grows_sorted = sorted(grows, key=lambda r: _cell(r, sku_header) or "") - for row in grows_sorted: - lines.append( - _competitor_matrix_md_line( - row, sku_header=sku_header, title_h=title_h - ) - ) - lines.append("") - - ch6_price_title = ( - "## 六、价格分析（PC 搜索列表全量）" - if list_export and pst_list.get("n", 0) > 0 - else "## 六、价格分析（深入 SKU 合并表 · 无可用列表价或未导出列表）" - ) - lines.extend(["---", "", ch6_price_title, ""]) - lines.append(f"- **统计基础**：{price_analysis_basis_cn}。") - if ( - list_export - and pst_list.get("n", 0) > 0 - and pst_merged.get("n", 0) > 0 - ): - lines.append( - f"- **对照**：合并表深入样本可解析价 **{pst_merged['n']}** 个观测，中位数约 **{float(pst_merged.get('median', pst_merged['mean'])):.2f}** 元（与上表样本范围不同，仅作对照）。" - ) - lines.append("") - if pst: - price_tbl = [ - "| 统计量 | 数值（元） | 说明 |", - "| --- | --- | --- |", - f"| 样本量 | {pst['n']} | 与统计基础一致 |", - f"| 最小值 | {pst['min']:.2f} | |", - ] - if "q1" in pst: - price_tbl.append(f"| 下四分位 Q1 | {float(pst['q1']):.2f} | |") - else: - price_tbl.append("| 下四分位 Q1 | — | 样本不足 4 个 |") - price_tbl.append( - f"| 中位数 | {float(pst.get('median', pst['mean'])):.2f} | |" - ) - if "q3" in pst: - price_tbl.append(f"| 上四分位 Q3 | {float(pst['q3']):.2f} | |") - else: - price_tbl.append("| 上四分位 Q3 | — | 样本不足 4 个 |") - price_tbl.extend( - [ - f"| 最大值 | {pst['max']:.2f} | |", - f"| 均值 | {pst['mean']:.2f} | |", - ] - ) - if "stdev" in pst: - price_tbl.append(f"| 标准差 | {pst['stdev']:.2f} | 离散程度 |") - lines.extend(price_tbl) - lines.append("") - lines.append( - "**解读提示**：价差大通常反映规格、组合装、品牌溢价或促销差异；B 端定价策略需结合成本与渠道单独建模。" - ) - else: - lines.append("*当前样本无可用数值价格，本节不展开统计表。*") - lines.append("") - - attrs: list[str] = [] - for row in merged_rows: - a = _cell(row, "detail_product_attributes") - if a and a not in attrs: - attrs.append(a) - lines.extend(["---", "", "## 七、产品与宣称（商详参数摘要）", ""]) - if attrs: - for i, a in enumerate(attrs[:12], 1): - lines.append(f"{i}. {_md_cell(a, 600)}") - else: - lines.append("*无可用商详参数摘要。*") - lines.append("") - - lines.extend( - [ - "---", - "", - "## 八、消费者反馈与用户画像（按细分类目）", - "", - "### 8.1 方法", - "", - "- **细类划分**：与 **§5 竞品矩阵** 相同，依据商详类目路径解析为「饼干 / 西式糕点 / …」等（规则见 §5 章首说明）。", - "- **归因**：每条评价按其 SKU 对应到深入样本，再映射到该 SKU 所属细类；SKU 不在合并表中的评价单独归入说明性分组。", - "- **正负面粗判（§8.2）**：先以关键词规则与图表做粗分；若任务开启 **llm_comment_sentiment**，可附**大模型对抽样原文的语义归纳**（与规则统计互补）。", - "- **关注词按细类（§8.3）**：对组内评价正文做子串计数并出条形图；若无逐条正文则用该细类下评价摘要列拼接兜底；与配置关注词及联想扩展同源。", - "- **用途/场景按细类（§8.4）**：对组内每条有效文本独立扫描**本次任务生效的场景词组**（来自报告调参或系统默认），一条可属多场景；条形图横轴为**占该细类有效文本比例 %**（多标签下各比例可相加大于 100%）。", - "", - "### 8.2 评价正负面粗判（关键词规则）", - "", - f"- **有效文本条数**：{sentiment_lex.get('text_units', 0)}（与 §8.1 归因口径一致）。", - f"- **偏正向（仅命中正向词表）**：{sentiment_lex.get('positive_only', 0)} 条；" - f"**偏负向（仅命中负向词表）**：{sentiment_lex.get('negative_only', 0)} 条；" - f"**混合（同条兼含正/负词）**：{sentiment_lex.get('mixed_positive_and_negative', 0)} 条；" - f"**中性或空文本**：{sentiment_lex.get('neutral_or_empty', 0)} 条。", - "- **说明**：词表为方向性粗判，讽刺、省略与错别字会导致误判；正式结论请**人工抽样**阅读原文。", - ] - ) - _scope = (sentiment_lex.get("lexeme_scope_note") or "").strip() - if _scope: - lines.append(f"- **词根统计口径**：{_scope}") - lines.extend(["", ""]) - lines.extend( - _embed_chart( - run_dir, - "chart_sentiment_overview_pie.png", - "评价语气四象限占比（扇形图；与上表条数一致）", - ) - ) - lines.extend( - _embed_chart( - run_dir, - "chart_positive_lexemes_bar.png", - "正向评价里**最常出现的口语短语**（在偏正向或混合评价条内统计；条形图）", - ) - ) - lines.extend( - _embed_chart( - run_dir, - "chart_negative_lexemes_bar.png", - "负向评价里**最常出现的口语短语**（在偏负向或混合评价条内统计；条形图）", - ) - ) - pos_h = sentiment_lex.get("positive_tone_lexeme_hits") or [] - neg_h = sentiment_lex.get("negative_tone_lexeme_hits") or [] - if pos_h: - frag = "；".join( - f"「{x.get('word', '')}」{x.get('texts_matched', 0)} 条" - for x in pos_h[:6] - if isinstance(x, dict) - ) - lines.append(f"- **正向语境高频短语（摘要）**：{frag}。") - if neg_h: - frag_n = "；".join( - f"「{x.get('word', '')}」{x.get('texts_matched', 0)} 条" - for x in neg_h[:6] - if isinstance(x, dict) - ) - lines.append(f"- **负向语境高频短语（摘要）**：{frag_n}。") - _llm_s = (llm_sentiment_section_md or "").strip() - if _llm_s: - lines.extend( - [ - "", - "#### 大模型解读（正/负向评价要点）", - "", - "> **说明**：基于与上节**同一分桶规则**抽样的评价原文，由大模型做语义归纳，与关键词条数、条形图**互补**；具体措辞以原评论为准。", - "", - _llm_s, - ] - ) - lines.append("") - lines.extend( - [ - "### 8.3 关注词频次（按细类）", - "", - ] - ) - if not feedback_groups: - lines.append("*无评价数据可归组。*") - lines.append("") - else: - for gi, (gname, cr_g, texts_g) in enumerate(feedback_groups): - n_flat = len(cr_g) - lines.append(f"#### {gname}") - lines.append("") - lines.append( - f"- **本细类逐条评价**：{n_flat} 条；**用于统计的有效文本条数**：{len(texts_g)}。" - ) - lines.append("") - hits_g = _group_keyword_hits(cr_g, texts_g, focus_words=focus_words) - if hits_g: - lines.extend( - _embed_chart( - run_dir, - _focus_keywords_group_bar_filename(gname, gi), - f"「{_md_cell(gname, 24)}」细类 · 关注词子串命中次数（条形图；同一评价可出现多次，为次数而非去重条数）", - ) - ) - else: - lines.append("*该细类无命中或无数文本。*") - lines.append("") - - lines.extend( - [ - "### 8.4 用途与使用场景（按细类）", - "", - "每条评价文本（或兜底预览）独立扫描场景词组；若命中某组内**任一**关键词则该组 +1，同一条可计入多组；" - "**不等于**购买动机调研结论，建议结合原句抽样阅读。", - "", - ] - ) - if not feedback_groups: - lines.append("*无评价数据可归组。*") - lines.append("") - else: - for gi, (gname, cr_g, texts_g) in enumerate(feedback_groups): - scen_g, scen_ng = _comment_scenario_counts(texts_g, scenario_groups) - lines.append(f"#### {gname}") - lines.append("") - if scen_ng <= 0: - lines.append("*该细类下无可用评价正文。*") - lines.append("") - continue - lines.append(f"- **有效评价文本条数**：{scen_ng}") - lines.append("") - if scen_g: - lines.extend( - _embed_chart( - run_dir, - _scenario_group_usage_bar_filename(gname, gi), - f"「{_md_cell(gname, 24)}」细类 · 场景/用途（条形图：**横轴 = 提及条数 ÷ 本细类有效文本条数**，" - f"柱尾标注「条数 · 占比」；有效文本 **{scen_ng}** 条）", - ) - ) - for para in _scenario_summary_bullets(scen_g, scen_ng): - lines.append(para) - lines.append("") - else: - lines.append("*未命中预设场景词组。*") - lines.append("") - - lines.extend( - [ - "---", - "", - "## 九、策略与机会提示（假设清单，待验证）", - "", - "以下为基于本批次数据的**规则化提示**，用于内部脑暴与假设生成，**不可替代**定性访谈与渠道调研。", - "", - ] - ) - for h in _strategy_hints( - cr1=cr1_hints, - pst=pst, - hits=hits, - n_comments=n_cmt, - scen_counts=scen_counts, - scen_n_texts=scen_n_texts, - ): - lines.append(f"- {h}") - lines.append("") - - lines.extend( - [ - "---", - "", - "## 附录 A：数据留存说明", - "", - "- 本批次**任务输出目录**内保存：搜索列表导出、深入 SKU 合并表、商详与评价相关表格，以及搜索/商详原始响应与运行参数快照，供内部复核与复算。", - "- 对外演示或转发前请按公司规范做**脱敏**处理。", - "", - "---", - "", - "*本报告由系统自动汇总生成；定稿前请业务交叉核对数据与结论。*", - "", - ] - ) - return "\n".join(lines) - - -def _sanitize_json_numbers(obj: Any) -> Any: - """浮点 NaN/Inf 无法 JSON 序列化，统一转 None 或圆角。""" - if isinstance(obj, float): - if math.isnan(obj) or math.isinf(obj): - return None - return round(obj, 6) - if isinstance(obj, dict): - return {k: _sanitize_json_numbers(v) for k, v in obj.items()} - if isinstance(obj, list): - return [_sanitize_json_numbers(x) for x in obj] - return obj - - -def build_competitor_brief( - *, - run_dir: Path, - keyword: str, - merged_rows: list[dict[str, str]], - search_export_rows: list[dict[str, str]], - comment_rows: list[dict[str, str]], - meta: dict[str, Any] | None, - report_config: dict[str, Any] | None = None, -) -> dict[str, Any]: - """ - 与 ``build_competitor_markdown`` 共用统计口径，输出可 JSON 序列化的结构化竞品摘要（**规则驱动**，无 LLM）。 - """ - focus_words, scenario_groups, _ext = resolve_report_tuning(report_config) - sku_header = "SKU(skuId)" - title_h = "标题(wareName)" - batch = _run_batch_label(run_dir) - n_sku = len(merged_rows) - n_cmt = len(comment_rows) - - list_export = len(search_export_rows) > 0 - structure_rows = search_export_rows if list_export else merged_rows - n_structure = len(structure_rows) - shops_s = _structure_shops(structure_rows, list_export=list_export) - brands_s = _structure_brands(structure_rows, list_export=list_export) - cr1_shop, cr3_shop, top_shop_s, top_shop_share = _brand_cr(shops_s) - cr1_list_brand, cr3_list_brand, top_list_brand, _ = _brand_cr(brands_s) - cm_structure = _structure_category_mix(structure_rows, list_export=list_export) - min_brand_rows = max(5, int(0.02 * n_structure)) if n_structure else 5 - - brands_deep = [ - _cell(r, "detail_brand") for r in merged_rows if _cell(r, "detail_brand") - ] - cr1_deep, cr3_deep, top_brand_deep, top_brand_deep_share = _brand_cr( - brands_deep - ) - cr1_hints = cr1_shop if list_export and cr1_shop is not None else cr1_deep - - pst_merged = _price_stats_extended(_collect_prices(merged_rows)) - pst_list = ( - _price_stats_extended(_collect_prices(search_export_rows)) - if list_export - else {} - ) - pst = ( - pst_list - if list_export and pst_list.get("n", 0) > 0 - else pst_merged - ) - price_stats_source = ( - "pc_search_export_all_rows" - if list_export and pst_list.get("n", 0) > 0 - else "keyword_pipeline_merged" - ) - - hits = _comment_keyword_hits(comment_rows, focus_words) - if not hits: - blob = _merge_comment_previews(merged_rows) - for w in focus_words: - if len(w) < 2: - continue - n = blob.count(w) - if n: - hits[w] += n - - comment_texts = _iter_comment_text_units(comment_rows, merged_rows) - comment_sentiment_lexicon = _comment_sentiment_lexicon(comment_texts) - scen_counts, scen_n_texts = _comment_scenario_counts( - comment_texts, scenario_groups - ) - - ( - api_rc, - api_list_kw, - api_rc_uniques, - api_raw_json_n, - _api_rc_n_values, - ) = _pc_search_result_count_from_raw(run_dir) - - proxy = _search_list_proxies(search_export_rows) if search_export_rows else {} - - hints = _strategy_hints( - cr1=cr1_hints, - pst=pst, - hits=hits, - n_comments=n_cmt, - scen_counts=scen_counts, - scen_n_texts=scen_n_texts, - ) - - matrix_groups: list[dict[str, Any]] = [] - for gname, mrows in _merged_rows_grouped_for_matrix(merged_rows): - items: list[dict[str, str]] = [] - for row in mrows: - items.append( - { - "sku_id": _cell(row, sku_header), - "title": _cell(row, title_h), - "brand": _cell(row, "detail_brand"), - "list_price_show": _cell( - row, "标价(jdPrice,jdPriceText,realPrice)" - ), - "coupon_or_detail_price": _cell( - row, - "券后到手价(couponPrice,subsidyPrice,finalPrice.estimatedPrice,priceShow)", - ), - "detail_price_final": _cell(row, "detail_price_final"), - "shop": _cell( - row, "店铺名(shopName)", "detail_shop_name" - ), - "category": _category_cell(row), - "selling_point": _cell(row, "卖点(sellingPoint)")[:240], - "comment_fuzzy": _cell(row, "评价量(commentFuzzy)"), - } - ) - matrix_groups.append( - {"group": gname, "sku_count": len(items), "skus": items} - ) - - feedback_by_group: list[dict[str, Any]] = [] - usage_scenarios_by_matrix_group: list[dict[str, Any]] = [] - for gi, (gname, cr, tu) in enumerate( - _consumer_feedback_by_matrix_group( - merged_rows=merged_rows, - comment_rows=comment_rows, - sku_header=sku_header, - ) - ): - gh = _group_keyword_hits(cr, tu, focus_words=focus_words) - scen_g, scen_n_g = _comment_scenario_counts(tu, scenario_groups) - slug_fb = _scenario_group_asset_slug(gname, gi) - feedback_by_group.append( - { - "group": gname, - "matrix_group_index": gi, - "chart_slug": slug_fb, - "comment_rows": len(cr), - "effective_comment_text_units": len(tu), - "focus_keyword_hits": [ - {"word": w, "count": n} for w, n in gh.most_common(24) - ], - "scenarios_top": [ - { - "scenario": s, - "count": n, - "share_of_text_units": ( - n / scen_n_g if scen_n_g else 0.0 - ), - } - for s, n in scen_g.most_common(6) - ] - if scen_n_g - else [], - } - ) - if scen_n_g > 0 and scen_g: - usage_scenarios_by_matrix_group.append( - { - "group": gname, - "matrix_group_index": gi, - "chart_slug": slug_fb, - "effective_text_units": scen_n_g, - "scenarios": [ - { - "scenario": s, - "count": int(n), - "share_of_text_units": ( - float(n) / scen_n_g if scen_n_g else 0.0 - ), - } - for s, n in scen_g.most_common() - if n > 0 - ], - } - ) - - meta_slice: dict[str, Any] = {} - if meta: - for k in ( - "page_start", - "page_to", - "max_skus_config", - "pc_search_export_rows", - "merged_rows", - "scenario_filter_enabled", - "merged_csv_mode", - ): - if k in meta: - meta_slice[k] = meta[k] - - list_brand_block: dict[str, Any] | None - if len(brands_s) >= min_brand_rows: - list_brand_block = { - "cr1": cr1_list_brand, - "cr3": cr3_list_brand, - "top_label": top_list_brand, - } - else: - list_brand_block = None - - out: dict[str, Any] = { - "schema_version": 1, - "keyword": keyword, - "batch_label": batch, - "run_dir": str(run_dir.resolve()), - "scope": { - "merged_sku_count": n_sku, - "comment_flat_rows": n_cmt, - "structure_source_rows": n_structure, - "uses_pc_search_list_export": list_export, - }, - "meta": meta_slice or None, - "pc_search_raw": { - "result_count_consensus": api_rc, - "list_keyword": api_list_kw or None, - "result_count_uniques": api_rc_uniques, - "raw_json_files_scanned": api_raw_json_n, - }, - "list_visibility_proxy": proxy, - "concentration": { - "shops_from_list": { - "cr1": cr1_shop, - "cr3": cr3_shop, - "top_label": top_shop_s, - "top_share_pct": top_shop_share, - }, - "list_brand_field": list_brand_block, - "detail_brand_among_merged": { - "cr1": cr1_deep, - "cr3": cr3_deep, - "top_label": top_brand_deep, - "top_share_pct": top_brand_deep_share, - }, - }, - "category_mix_top": [ - {"label": lbl, "count": cnt} for lbl, cnt in cm_structure - ], - "list_brand_mix_top": [ - {"label": k, "count": v} - for k, v in Counter( - b for b in brands_s if (b or "").strip() - ).most_common(24) - ], - "list_shop_mix_top": [ - {"label": k, "count": v} - for k, v in Counter( - s for s in shops_s if (s or "").strip() - ).most_common(24) - ], - "price_stats": pst, - "price_stats_source": price_stats_source, - "price_stats_merged_sample": pst_merged, - "price_stats_list_export": pst_list if list_export else {}, - "comment_focus_keywords": [ - {"word": w, "count": n} for w, n in hits.most_common(24) - ], - "usage_scenarios": [ - { - "scenario": lbl, - "count": n, - "share_of_text_units": ( - n / scen_n_texts if scen_n_texts else 0.0 - ), - } - for lbl, n in scen_counts.most_common(16) - ], - "usage_scenarios_denominator": scen_n_texts, - "usage_scenarios_by_matrix_group": usage_scenarios_by_matrix_group, - "strategy_hints": hints, - "matrix_by_group": matrix_groups, - "consumer_feedback_by_matrix_group": feedback_by_group, - "comment_sentiment_lexicon": comment_sentiment_lexicon, - "notes": [ - "与在线分析报告各章统计口径一致；主题词与场景为预设词表，非 NLP 主题模型。", - "价格来自页面展示字段抽取，含促销与规格差异。", - "comment_sentiment_lexicon 为关键词粗判，非深度学习情感模型。", - ], - } - return _sanitize_json_numbers(out) - - -def main() -> None: - try: - if hasattr(sys.stdout, "reconfigure"): - sys.stdout.reconfigure(encoding="utf-8", errors="replace") - if hasattr(sys.stderr, "reconfigure"): - sys.stderr.reconfigure(encoding="utf-8", errors="replace") - except Exception: - pass - - existing = _resolve_existing_run_dir(EXISTING_RUN_DIR) - meta_path_early = (existing / kpl.FILE_RUN_META_JSON) if existing else None - meta_early: dict[str, Any] | None = None - if meta_path_early and meta_path_early.is_file(): - try: - meta_early = json.loads(meta_path_early.read_text(encoding="utf-8")) - except json.JSONDecodeError: - meta_early = None - - if existing: - if not existing.is_dir(): - print(f"[竞品报告] EXISTING_RUN_DIR 不是目录: {existing}", file=sys.stderr) - sys.exit(2) - kw = (KEYWORD or "").strip() or _infer_keyword(existing, meta_early) - if not kw: - print( - "[竞品报告] 仅分析已有目录时，请配置 KEYWORD，或保留 run_meta.json 的 keyword，" - "或使目录名为 YYYYMMDD_HHMMSS_关键词", - file=sys.stderr, - ) - sys.exit(2) - run_dir = existing - print(f"[竞品报告] 使用已有目录（不抓取）: {run_dir}", file=sys.stderr) - else: - kw = (KEYWORD or "").strip() - if not kw: - print("[竞品报告] 全量抓取时请在本文件顶部配置 KEYWORD", file=sys.stderr) - sys.exit(2) - - backup: dict[str, Any] = {} - try: - if OVERRIDE_MAX_SKUS is not None: - backup["MAX_SKUS"] = kpl.MAX_SKUS - kpl.MAX_SKUS = max(1, int(OVERRIDE_MAX_SKUS)) - if OVERRIDE_PAGE_START is not None: - backup["PAGE_START"] = kpl.PAGE_START - kpl.PAGE_START = max(1, int(OVERRIDE_PAGE_START)) - if OVERRIDE_PAGE_TO is not None: - backup["PAGE_TO"] = kpl.PAGE_TO - kpl.PAGE_TO = max(1, int(OVERRIDE_PAGE_TO)) - - print(f"[竞品报告] 关键词={kw!r}，开始流水线…", file=sys.stderr) - run_dir = kpl.main(keyword=kw) - finally: - for name, val in backup.items(): - setattr(kpl, name, val) - - merged_path = run_dir / kpl.FILE_MERGED_CSV - comments_path = run_dir / kpl.FILE_COMMENTS_FLAT_CSV - meta_path = run_dir / kpl.FILE_RUN_META_JSON - - _, merged_rows = _read_csv_rows(merged_path) - _, search_export_rows = _read_csv_rows(run_dir / kpl.FILE_PC_SEARCH_CSV) - _, comment_rows = _read_csv_rows(comments_path) - meta: dict[str, Any] | None = meta_early if existing else None - if meta is None and meta_path.is_file(): - try: - meta = json.loads(meta_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - meta = None - - md = build_competitor_markdown( - run_dir=run_dir, - keyword=kw, - merged_rows=merged_rows, - search_export_rows=search_export_rows, - comment_rows=comment_rows, - meta=meta, - ) - out_md = run_dir / "competitor_analysis.md" - out_md.write_text(md, encoding="utf-8") - print(f"[竞品报告] 运行目录: {run_dir}", file=sys.stderr) - print(f"[竞品报告] 已写: {out_md}", file=sys.stderr) +_backend = Path(__file__).resolve().parents[2] +if str(_backend) not in sys.path: + sys.path.insert(0, str(_backend)) +from pipeline.competitor_report import jd_report as _jd # noqa: E402 if __name__ == "__main__": - main() + _jd.main() diff --git a/backend/crawler_copy/jd_pc_search/jd_keyword_pipeline.py b/backend/crawler_copy/jd_pc_search/jd_keyword_pipeline.py index 6ec95f1..5637115 100644 --- a/backend/crawler_copy/jd_pc_search/jd_keyword_pipeline.py +++ b/backend/crawler_copy/jd_pc_search/jd_keyword_pipeline.py @@ -16,7 +16,8 @@ 每次运行默认在 ``data/JD/pipeline_runs/<时间戳>_<关键词>/`` 下集中写入：合并表、 PC 搜索导出 CSV、评价扁平 CSV、详情汇总 CSV（``detail_ware_export.csv``）、 -各 SKU 规整 JSON（``detail/ware_{sku}_response.json``），以及（可选）pc_search 原始包与请求记录。 +各 SKU 规整 JSON（``detail/ware_{sku}_response.json``）、 +购买者优惠摘要 JSON（``buyer_offer_profiles/ware_{sku}_buyer_profile.json``），以及（可选）pc_search 原始包与请求记录。 合并表 ``keyword_pipeline_merged.csv`` 默认 ``MERGED_CSV_MODE=lean``：搜索全列 + **竞品报告/入库实际用到的商详子集**（见 ``_MERGED_LEAN_DETAIL_FIELDNAMES``）+ 评论摘要；全量商详扁平请设 ``MERGED_CSV_MODE="full"``（``WARE_BUSINESS_MERGE_FIELDNAMES``）。 ``detail_ware_export.csv`` 默认 ``DETAIL_WARE_CSV_MODE=lean``，为 ``skuId`` + 与合并表一致的商详子集（品牌/到手价/店铺/类目/参数/配料）；全列请设 ``DETAIL_WARE_CSV_MODE="full"``。 @@ -24,17 +25,15 @@ PC 搜索导出 CSV、评价扁平 CSV、详情汇总 CSV（``detail_ware_export 默认启用 **应用场景筛选**（``brief_content.txt`` 4.1 中式面点/主食 + 4.2 烘焙）：仅命中关键词的 SKU 进入详情与评论队列；词表见 ``scenario_filter.py``。``SCENARIO_FILTER_ENABLED=False`` 可关闭；``SCENARIO_FILTER_PC_SEARCH_CSV="filtered"`` 可使导出 CSV 与筛选后列表一致。 各 SKU 完整接口 JSON 仍在 ``detail/ware_{sku}_response.json``。 -端到端竞品速览 Markdown：配置 ``jd_competitor_report.py`` 顶部 ``KEYWORD`` 后执行 ``python jd_competitor_report.py``（内部调用本模块 ``main(keyword=...)``）。 +端到端竞品速览 Markdown：在 ``backend`` 下配置 ``pipeline.competitor_report.jd_report`` 顶部 ``KEYWORD`` 后执行 ``python -m pipeline.competitor_report.jd_report``；或执行本目录兼容入口 ``python jd_competitor_report.py``（调用同一 ``jd_report.main``）。 """ from __future__ import annotations -import csv import json import random import sys import time -from io import StringIO from pathlib import Path from types import SimpleNamespace from typing import Any @@ -110,6 +109,8 @@ FILE_PC_SEARCH_CSV = "pc_search_export.csv" FILE_COMMENTS_FLAT_CSV = "comments_flat.csv" FILE_DETAIL_WARE_CSV = "detail_ware_export.csv" FILE_RUN_META_JSON = "run_meta.json" +# 与 ``detail/`` 同级：购买者视角优惠摘要（非原始接口 JSON） +DIR_BUYER_OFFER_PROFILES = "buyer_offer_profiles" # MERGED_CSV_MODE：``lean`` 时合并表为搜索全列 + 商详子集（``_MERGED_LEAN_DETAIL_FIELDNAMES``）+ 评论摘要；``full`` 为搜索全列 + ``WARE_BUSINESS_MERGE_FIELDNAMES`` 全量 MERGED_CSV_MODE = "lean" # DETAIL_WARE_CSV_MODE：``lean`` 时 ``detail_ware_export.csv`` 为 ``skuId`` + lean 商详子集；``full`` 为完整详情扁平列（含 http_status 与各 detail_*） @@ -136,12 +137,21 @@ for _p in (_SEARCH_DIR, _COMMENT_DIR, _DETAIL_DIR): if s not in sys.path: sys.path.insert(0, s) +_BACKEND_ROOT = Path(__file__).resolve().parents[2] +if str(_BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(_BACKEND_ROOT)) + from collect_pc_search_items import ( # noqa: E402 SearchCollectionCancelled, collect_pc_search_export_rows, ) from common.jd_delay_utils import parse_request_delay_range # noqa: E402 from scenario_filter import filter_rows_by_scenario # noqa: E402 +from jd_detail_buyer_extraction import ( # noqa: E402 + buyer_promo_text_from_profile, + buyer_ranking_line_from_profile, + extract_buyer_offer_profile_from_json_text, +) from jd_detail_ware_business_requests import ( # noqa: E402 DETAIL_WARE_LEAN_CSV_FIELDNAMES, WARE_BUSINESS_MERGE_FIELDNAMES, @@ -164,68 +174,19 @@ from jd_h5_item_comment_requests import ( # noqa: E402 ) from jd_h5_search_requests import ( # noqa: E402 CSV_FIELDS, - JD_EXPORT_COLUMN_HEADERS, jd_row_to_export, ) - - -_SKU_CSV_HEADER = JD_EXPORT_COLUMN_HEADERS["sku_id"] - -_MERGED_EXTRA_FIELDS = ( - ["pipeline_keyword"] - + list(WARE_BUSINESS_MERGE_FIELDNAMES) - + ["comment_count", "comment_preview"] +from jd_pipeline_export import ( # noqa: E402 + SKU_CSV_HEADER, + comment_fields_from_rows, + dedupe_comment_rows, + finalize_merged_row_for_disk, + write_detail_ware_csv, + write_merged_csv, + write_pc_search_export_csv, + write_run_meta_json, ) -# lean 合并表·商详块（jd_competitor_report + ingest + 配料）；须与 pipeline/csv_schema.MERGED_LEAN_DETAIL_KEYS 一致 -_MERGED_LEAN_DETAIL_FIELDNAMES: tuple[str, ...] = ( - "detail_brand", - "detail_price_final", - "detail_shop_name", - "detail_category_path", - "detail_product_attributes", - "detail_body_ingredients", -) - -# 合并表精简列：搜索列与 jd_h5_search_requests 一致 + 上表商详子集 + 评论摘要 -_MERGED_LEAN_FIELDNAMES: tuple[str, ...] = ( - "pipeline_keyword", - "SKU(skuId)", - "主商品ID(wareId)", - "标题(wareName)", - "标价(jdPrice,jdPriceText,realPrice)", - "券后到手价(couponPrice,subsidyPrice,finalPrice.estimatedPrice,priceShow)", - "原价(oriPrice,originalPrice,marketPrice)", - "卖点(sellingPoint)", - "榜单类文案(标签/腰带/标题数组中的榜、TOP 等)", - "评价量(commentFuzzy)", - "销量楼层(commentSalesFloor)", - "店铺名(shopName)", - "商品链接(toUrl,clickUrl,item.m.jd.com)", - "主图(imageurl,imageUrl)", - "规格属性(propertyList,color,catid,shortName)", - "类目(leafCategory,cid3Name,catid)", - "搜索词(keyword)", - "页码(page)", - *_MERGED_LEAN_DETAIL_FIELDNAMES, - "comment_count", - "comment_preview", -) - - -def _merged_csv_fieldnames() -> list[str]: - if (MERGED_CSV_MODE or "lean").strip().lower() == "full": - return list(CSV_FIELDS) + [ - f for f in _MERGED_EXTRA_FIELDS if f not in CSV_FIELDS - ] - return list(_MERGED_LEAN_FIELDNAMES) - - -def _detail_ware_csv_fieldnames() -> list[str]: - if (DETAIL_WARE_CSV_MODE or "lean").strip().lower() == "full": - return list(WARE_PARSED_CSV_FIELDNAMES) - return list(DETAIL_WARE_LEAN_CSV_FIELDNAMES) - def _sleep_range(spec: str, label: str) -> None: try: @@ -239,33 +200,6 @@ def _sleep_range(spec: str, label: str) -> None: time.sleep(t) -def _dedupe_comment_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: - """按 commentId 去重（跨首屏 + 多页列表）。""" - seen: set[str] = set() - out: list[dict[str, Any]] = [] - for r in rows: - cid = str(r.get("commentId") or "").strip() - if cid: - if cid in seen: - continue - seen.add(cid) - out.append(r) - return out - - -def _comment_fields_from_rows(rows: list[dict[str, Any]]) -> dict[str, str]: - previews: list[str] = [] - for r in rows[:8]: - t = str(r.get("tagCommentContent") or "").strip() - if t: - previews.append(t[:400]) - joined = " | ".join(previews)[:4000] - return { - "comment_count": str(len(rows)), - "comment_preview": joined, - } - - def _loads_json(text: str) -> Any: try: return json.loads(text) @@ -312,7 +246,7 @@ def main(keyword: str | None = None) -> Path: """ 跑完整条流水线。``keyword`` 非空时覆盖文件内 ``KEYWORD``；返回本次运行目录。 - 供 ``jd_competitor_report`` 等脚本 ``import`` 调用；命令行仍执行 ``main()`` 无参。 + 供 ``pipeline.competitor_report.jd_report`` 等脚本 ``import`` 调用；命令行仍执行 ``main()`` 无参。 """ try: if hasattr(sys.stdout, "reconfigure"): @@ -384,9 +318,9 @@ def main(keyword: str | None = None) -> Path: _ingredient_vision_ok = False if EXTRACT_INGREDIENTS_FROM_DETAIL_BODY_IMAGES: try: - import AI_crawler as _ac_mod # noqa: WPS433 + import pipeline.openai_gateway as _ac_mod # noqa: WPS433 - _ac_mod._resolve_credentials(None, None, None) + _ac_mod.resolve_credentials(None, None, None) _ingredient_vision_ok = True except Exception as e: print( @@ -474,7 +408,7 @@ def main(keyword: str | None = None) -> Path: skus_ordered: list[str] = [] seen: set[str] = set() for row in export_rows_for_skus: - sid = str(row.get(_SKU_CSV_HEADER) or "").strip() + sid = str(row.get(SKU_CSV_HEADER) or "").strip() if not sid or sid in seen: continue seen.add(sid) @@ -491,13 +425,7 @@ def main(keyword: str | None = None) -> Path: stop_pipeline = True search_csv_path = run_dir / FILE_PC_SEARCH_CSV - sbuf = StringIO() - sw = csv.DictWriter( - sbuf, fieldnames=list(CSV_FIELDS), extrasaction="ignore" - ) - sw.writeheader() - sw.writerows(rows_for_search_csv) - search_csv_path.write_text("\ufeff" + sbuf.getvalue(), encoding="utf-8") + write_pc_search_export_csv(search_csv_path, rows_for_search_csv) print( f"[流水线] 已写 PC 搜索导出 {search_csv_path}", file=sys.stderr, @@ -505,6 +433,8 @@ def main(keyword: str | None = None) -> Path: detail_dir = run_dir / "detail" detail_dir.mkdir(parents=True, exist_ok=True) + buyer_prof_dir = run_dir / DIR_BUYER_OFFER_PROFILES + buyer_prof_dir.mkdir(parents=True, exist_ok=True) detail_ctx = browser.new_context( user_agent=_JD_DETAIL_UA, @@ -528,12 +458,12 @@ def main(keyword: str | None = None) -> Path: ( r for r in export_rows_full - if str(r.get(_SKU_CSV_HEADER) or "").strip() == sku + if str(r.get(SKU_CSV_HEADER) or "").strip() == sku ), {}, ) merged: dict[str, str] = {k: str(search_row.get(k) or "") for k in CSV_FIELDS} - merged["pipeline_keyword"] = kw + merged["流水线关键词"] = kw if stop_pipeline or _pipeline_cancel_requested(): stop_pipeline = True @@ -627,6 +557,23 @@ def main(keyword: str | None = None) -> Path: (detail_dir / f"ware_{sku}_response.json").write_text( response_body, encoding="utf-8" ) + rline, ptext = "", "" + if response_body.strip(): + try: + _prof = extract_buyer_offer_profile_from_json_text(response_body) + rline = buyer_ranking_line_from_profile(_prof) + ptext = buyer_promo_text_from_profile(_prof) + (buyer_prof_dir / f"ware_{sku}_buyer_profile.json").write_text( + json.dumps(_prof, ensure_ascii=False, indent=2) + "\n", + encoding="utf-8", + ) + except Exception as e: + print( + f"[流水线] sku={sku} 购买者摘要写入失败: {e}", + file=sys.stderr, + ) + merged["buyer_ranking_line"] = rline + merged["buyer_promo_text"] = ptext _d_ing = str(merged.get("detail_body_ingredients") or "").strip() _d_src = str( merged.get("detail_body_ingredients_source_url") or "" @@ -649,6 +596,8 @@ def main(keyword: str | None = None) -> Path: d_text or "", detail_body_ingredients=_d_ing, detail_body_ingredients_source_url=_d_src, + buyer_ranking_line=rline, + buyer_promo_text=ptext, ) ) @@ -656,6 +605,7 @@ def main(keyword: str | None = None) -> Path: stop_pipeline = True merged["comment_count"] = "0" merged["comment_preview"] = "" + finalize_merged_row_for_disk(merged) merged_rows.append(merged) break @@ -665,6 +615,7 @@ def main(keyword: str | None = None) -> Path: stop_pipeline = True merged["comment_count"] = "0" merged["comment_preview"] = "" + finalize_merged_row_for_disk(merged) merged_rows.append(merged) break @@ -678,6 +629,7 @@ def main(keyword: str | None = None) -> Path: except SystemExit: merged["comment_count"] = "0" merged["comment_preview"] = "" + finalize_merged_row_for_disk(merged) merged_rows.append(merged) continue @@ -685,6 +637,7 @@ def main(keyword: str | None = None) -> Path: stop_pipeline = True merged["comment_count"] = "0" merged["comment_preview"] = "" + finalize_merged_row_for_disk(merged) merged_rows.append(merged) break @@ -783,8 +736,8 @@ def main(keyword: str | None = None) -> Path: "firstCommentGuid，仅保留首屏评价", file=sys.stderr, ) - comment_rows = _dedupe_comment_rows(comment_rows) - merged.update(_comment_fields_from_rows(comment_rows)) + comment_rows = dedupe_comment_rows(comment_rows) + merged.update(comment_fields_from_rows(comment_rows)) all_comment_rows.extend(comment_rows) except Exception as e: print( @@ -794,6 +747,7 @@ def main(keyword: str | None = None) -> Path: merged["comment_count"] = "0" merged["comment_preview"] = "" + finalize_merged_row_for_disk(merged) merged_rows.append(merged) print(f"[流水线] [{idx + 1}/{len(skus_ordered)}] sku={sku} OK", file=sys.stderr) if stop_pipeline: @@ -810,35 +764,49 @@ def main(keyword: str | None = None) -> Path: browser.close() out_path = run_dir / FILE_MERGED_CSV - fieldnames = _merged_csv_fieldnames() - buf = StringIO() - w = csv.DictWriter(buf, fieldnames=fieldnames, extrasaction="ignore") - w.writeheader() - w.writerows(merged_rows) - out_path.write_text("\ufeff" + buf.getvalue(), encoding="utf-8") + _, merged_col_count = write_merged_csv( + out_path, + merged_rows, + merged_csv_mode=MERGED_CSV_MODE, + ) print( f"[流水线] 已写合并表 {out_path} 共 {len(merged_rows)} 行 " - f"（MERGED_CSV_MODE={MERGED_CSV_MODE!r}，{len(fieldnames)} 列）", + f"（MERGED_CSV_MODE={MERGED_CSV_MODE!r}，{merged_col_count} 列）", file=sys.stderr, ) detail_csv_path = run_dir / FILE_DETAIL_WARE_CSV - detail_csv_path.parent.mkdir(parents=True, exist_ok=True) - detail_fn = _detail_ware_csv_fieldnames() - with detail_csv_path.open("w", encoding="utf-8-sig", newline="") as dcf: - dw = csv.DictWriter( - dcf, - fieldnames=detail_fn, - extrasaction="ignore", - ) - dw.writeheader() - dw.writerows(detail_csv_rows) + _, detail_col_count = write_detail_ware_csv( + detail_csv_path, + detail_csv_rows, + detail_ware_csv_mode=DETAIL_WARE_CSV_MODE, + ) print( f"[流水线] 已写详情扁平表 {detail_csv_path} 共 {len(detail_csv_rows)} 行 " - f"（DETAIL_WARE_CSV_MODE={DETAIL_WARE_CSV_MODE!r}，{len(detail_fn)} 列）", + f"（DETAIL_WARE_CSV_MODE={DETAIL_WARE_CSV_MODE!r}，{detail_col_count} 列）", file=sys.stderr, ) + _backend_root = Path(__file__).resolve().parents[2] + if str(_backend_root) not in sys.path: + sys.path.insert(0, str(_backend_root)) + try: + from pipeline.jd.buyer_offer_export_csv import ( # noqa: WPS433 + export_buyer_offer_with_detail_csv, + ) + + export_buyer_offer_with_detail_csv(run_dir) + print( + f"[流水线] 已写 {DIR_BUYER_OFFER_PROFILES}/" + "buyer_offer_with_detail.csv（与 detail_ware 列一致，含购买者摘要）", + file=sys.stderr, + ) + except Exception as e: + print( + f"[流水线] 跳过 buyer_offer_with_detail.csv：{e}", + file=sys.stderr, + ) + comments_path = run_dir / FILE_COMMENTS_FLAT_CSV write_comments_flat_csv(comments_path, all_comment_rows) print( @@ -864,18 +832,16 @@ def main(keyword: str | None = None) -> Path: "pc_search_export_rows_full": len(export_rows_full), "merged_rows": len(merged_rows), "merged_csv_mode": (MERGED_CSV_MODE or "lean").strip().lower(), - "merged_csv_column_count": len(fieldnames), + "merged_csv_column_count": merged_col_count, "detail_ware_csv_mode": (DETAIL_WARE_CSV_MODE or "lean").strip().lower(), - "detail_ware_csv_column_count": len(detail_fn), + "detail_ware_csv_column_count": detail_col_count, "comment_flat_rows": len(all_comment_rows), "detail_ware_csv_rows": len(detail_csv_rows), + "buyer_offer_profiles_dir": DIR_BUYER_OFFER_PROFILES, "with_comment_list": bool(WITH_COMMENT_LIST), "list_pages": (LIST_PAGES or "").strip(), } - (run_dir / FILE_RUN_META_JSON).write_text( - json.dumps(meta, ensure_ascii=False, indent=2) + "\n", - encoding="utf-8", - ) + write_run_meta_json(run_dir / FILE_RUN_META_JSON, meta) if stop_pipeline: print("[流水线] 已按请求终止（已写出当前进度）", file=sys.stderr) raise PipelineCancelled(run_dir) diff --git a/backend/crawler_copy/jd_pc_search/jd_pipeline_export.py b/backend/crawler_copy/jd_pc_search/jd_pipeline_export.py new file mode 100644 index 0000000..cedceb6 --- /dev/null +++ b/backend/crawler_copy/jd_pc_search/jd_pipeline_export.py @@ -0,0 +1,171 @@ +# -*- coding: utf-8 -*- +""" +流水线**落盘层**：合并表 / PC 搜索导出 / 详情扁平 CSV 的列名、行规范化与 UTF-8 BOM 写入。 + +与 ``jd_keyword_pipeline`` 中的 **采集编排**（Playwright、请求、合并内存行）分离，便于单独阅读与单测。 +""" +from __future__ import annotations + +import csv +import json +from io import StringIO +from pathlib import Path +from typing import Any + +from pipeline.csv.schema import ( # noqa: E402 + MERGED_CSV_COLUMNS, + remap_merged_row_english_detail_keys_to_csv_headers, +) +from jd_detail_ware_business_requests import ( # noqa: E402 + DETAIL_WARE_LEAN_CSV_FIELDNAMES, + WARE_BUSINESS_MERGE_FIELDNAMES, + WARE_PARSED_CSV_FIELDNAMES, +) +from jd_h5_search_requests import CSV_FIELDS, JD_EXPORT_COLUMN_HEADERS # noqa: E402 + +SKU_CSV_HEADER = JD_EXPORT_COLUMN_HEADERS["sku_id"] + +_MERGED_EXTRA_FIELDS = ( + ["pipeline_keyword"] + + list(WARE_BUSINESS_MERGE_FIELDNAMES) + + ["comment_count", "comment_preview"] +) + + +def finalize_merged_row_for_disk(merged: dict[str, str]) -> None: + """英文内部键 → 中文 CSV 列名；评论摘要列名。""" + remap_merged_row_english_detail_keys_to_csv_headers(merged) + if "comment_count" in merged: + merged["评论条数"] = str(merged.pop("comment_count") or "") + if "comment_preview" in merged: + merged["评价摘要"] = str(merged.pop("comment_preview") or "") + + +def merged_csv_fieldnames(merged_csv_mode: str) -> list[str]: + if (merged_csv_mode or "lean").strip().lower() == "full": + return list(CSV_FIELDS) + [ + f for f in _MERGED_EXTRA_FIELDS if f not in CSV_FIELDS + ] + return list(MERGED_CSV_COLUMNS) + + +def normalize_merged_rows_for_export(rows: list[dict[str, str]]) -> None: + """ + 整合表落盘前：搜索侧「榜单类文案」与「榜单排名」去掉 ``榜单/曝光：`` 前缀， + 与 ``strip_buyer_ranking_line_prefix`` / 入库规则一致。 + """ + from pipeline.csv.schema import strip_buyer_ranking_line_prefix # noqa: WPS433 + + hot_key = "榜单类文案" + rank_key = "榜单排名" + for merged in rows: + if merged.get(hot_key): + merged[hot_key] = strip_buyer_ranking_line_prefix(merged[hot_key]) + merged[rank_key] = strip_buyer_ranking_line_prefix(merged.get(rank_key) or "") + + +def detail_ware_csv_fieldnames(detail_ware_csv_mode: str) -> list[str]: + if (detail_ware_csv_mode or "lean").strip().lower() == "full": + return list(WARE_PARSED_CSV_FIELDNAMES) + return list(DETAIL_WARE_LEAN_CSV_FIELDNAMES) + + +def dedupe_comment_rows(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: + """按 commentId 去重（跨首屏 + 多页列表）。""" + seen: set[str] = set() + out: list[dict[str, Any]] = [] + for r in rows: + cid = str(r.get("commentId") or "").strip() + if cid: + if cid in seen: + continue + seen.add(cid) + out.append(r) + return out + + +def comment_fields_from_rows(rows: list[dict[str, Any]]) -> dict[str, str]: + previews: list[str] = [] + for r in rows[:8]: + t = str(r.get("tagCommentContent") or "").strip() + if t: + previews.append(t[:400]) + joined = " | ".join(previews)[:4000] + return { + "comment_count": str(len(rows)), + "comment_preview": joined, + } + + +def write_pc_search_export_csv( + path: Path, rows: list[dict[str, str]] +) -> None: + """写入 ``pc_search_export.csv``（UTF-8 BOM + 全列）。""" + sbuf = StringIO() + sw = csv.DictWriter( + sbuf, fieldnames=list(CSV_FIELDS), extrasaction="ignore" + ) + sw.writeheader() + sw.writerows(rows) + path.write_text("\ufeff" + sbuf.getvalue(), encoding="utf-8") + + +def write_merged_csv( + path: Path, + merged_rows: list[dict[str, str]], + *, + merged_csv_mode: str, +) -> tuple[list[str], int]: + """ + 写入合并表；返回 (fieldnames, 列数) 供 ``run_meta`` 使用。 + """ + fieldnames = merged_csv_fieldnames(merged_csv_mode) + normalize_merged_rows_for_export(merged_rows) + buf = StringIO() + w = csv.DictWriter(buf, fieldnames=fieldnames, extrasaction="ignore") + w.writeheader() + w.writerows(merged_rows) + path.write_text("\ufeff" + buf.getvalue(), encoding="utf-8") + return fieldnames, len(fieldnames) + + +def write_detail_ware_csv( + path: Path, + detail_csv_rows: list[dict[str, str]], + *, + detail_ware_csv_mode: str, +) -> tuple[list[str], int]: + """写入 ``detail_ware_export.csv``；返回 (fieldnames, 列数)。""" + path.parent.mkdir(parents=True, exist_ok=True) + detail_fn = detail_ware_csv_fieldnames(detail_ware_csv_mode) + with path.open("w", encoding="utf-8-sig", newline="") as dcf: + dw = csv.DictWriter( + dcf, + fieldnames=detail_fn, + extrasaction="ignore", + ) + dw.writeheader() + dw.writerows(detail_csv_rows) + return detail_fn, len(detail_fn) + + +def write_run_meta_json(path: Path, meta: dict[str, Any]) -> None: + path.write_text( + json.dumps(meta, ensure_ascii=False, indent=2) + "\n", + encoding="utf-8", + ) + + +__all__ = [ + "SKU_CSV_HEADER", + "comment_fields_from_rows", + "dedupe_comment_rows", + "detail_ware_csv_fieldnames", + "finalize_merged_row_for_disk", + "merged_csv_fieldnames", + "normalize_merged_rows_for_export", + "write_detail_ware_csv", + "write_merged_csv", + "write_pc_search_export_csv", + "write_run_meta_json", +] diff --git a/backend/crawler_copy/jd_pc_search/scenario_filter.py b/backend/crawler_copy/jd_pc_search/scenario_filter.py index 0f83258..3fa208a 100644 --- a/backend/crawler_copy/jd_pc_search/scenario_filter.py +++ b/backend/crawler_copy/jd_pc_search/scenario_filter.py @@ -10,14 +10,21 @@ from __future__ import annotations +import sys +from pathlib import Path from typing import Any +_BACKEND_ROOT = Path(__file__).resolve().parents[2] +if str(_BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(_BACKEND_ROOT)) +from pipeline.csv.schema import JD_SEARCH_CSV_HEADERS # noqa: E402 + # 与 CSV 导出列名一致（jd_h5_search_requests.CSV_FIELDS 子集） _SCENARIO_TEXT_FIELDS: tuple[str, ...] = ( - "标题(wareName)", - "卖点(sellingPoint)", - "类目(leafCategory,cid3Name,catid)", - "规格属性(propertyList,color,catid,shortName)", + JD_SEARCH_CSV_HEADERS["title"], + JD_SEARCH_CSV_HEADERS["selling_point"], + JD_SEARCH_CSV_HEADERS["leaf_category"], + JD_SEARCH_CSV_HEADERS["attributes"], ) # 4.1 中式（米）面点及主食（含常见同义/细分） diff --git a/backend/crawler_copy/jd_pc_search/search/jd_h5_search_parse.py b/backend/crawler_copy/jd_pc_search/search/jd_h5_search_parse.py new file mode 100644 index 0000000..5e63743 --- /dev/null +++ b/backend/crawler_copy/jd_pc_search/search/jd_h5_search_parse.py @@ -0,0 +1,1310 @@ +# -*- coding: utf-8 -*- +""" +京东 PC 搜索响应**解析**：JSON/HTML → 商品行、游标步进、重试判定。 + +HTTP、Node 签 URL 与 CLI 见 ``jd_h5_search_requests``。 +""" +from __future__ import annotations + +import html as html_module +import json +import re +import sys +from pathlib import Path +from typing import Any +from urllib.parse import parse_qs, urlparse + +_BACKEND_ROOT = Path(__file__).resolve().parents[3] +if str(_BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(_BACKEND_ROOT)) +from pipeline.csv.schema import JD_SEARCH_CSV_HEADERS as JD_EXPORT_COLUMN_HEADERS # noqa: E402 + +_JD_PC_SEARCH_DIR = Path(__file__).resolve().parent + +# PC 搜索：与浏览器一致，**每逻辑页固定 2 包** pc_search（body.page 连续 +1：第 L 页为 2L-1 与 2L）。 +# **Δs = max(1, 自然位条数-1)**（wareList 非广告计自然位）。跳过前序屏时同样每逻辑页 2 包推进游标。 +# CLI 的 --page 是**逻辑页**，不是请求 body.page（避免混淆）。 +JD_PC_SEARCH_ITEMS_PER_PAGE = 60 +JD_PC_SEARCH_CHUNKS_PER_LOGICAL_PAGE = 2 +# 重试仍失败时推进 s 的兜底（与常见两包 Δs≈21～22 一致） +JD_PC_SEARCH_FALLBACK_S_STEP = 22 + +def pc_search_response_is_empty_ware_list(body: str) -> bool: + """合法 JSON 且 ``data.wareList`` 为明确空数组时视为「已无更多商品」，不宜再强推游标。""" + p = _loads_json_or_jsonp((body or "").strip()) + if not isinstance(p, dict): + return False + data = p.get("data") + return isinstance(data, dict) and data.get("wareList") == [] + + +def pc_search_should_retry_fetch( + body: str, *, has_rows: bool, s_step: int +) -> bool: + """ + 是否应对同一 body.page / s 重发请求。 + 空体、非 JSON 短包等视为瞬时故障；明确的 ``data.wareList == []`` 不重试。 + """ + if has_rows or s_step > 0: + return False + s = (body or "").strip() + if not s: + return True + p = _loads_json_or_jsonp(s) + if p is None: + return len(s) < 12000 + if not isinstance(p, dict): + return False + data = p.get("data") + if isinstance(data, dict) and data.get("wareList") == []: + return False + return True + +def jd_pc_api_body_page_first_pack(logical_page_1based: int) -> int: + """逻辑页 L（从 1 起）第一包请求里的 body.page = 2L - 1。""" + L = max(1, int(logical_page_1based)) + return 2 * L - 1 + + +def _jd_row_count_for_page(rows: list[dict[str, str]], page: int) -> int: + ps = str(page) + return sum(1 for r in rows if (r.get("page") or "").strip() == ps) +# 与淘宝 CANONICAL_FIELDS 基本一致，但不导出 features / promotion_tags（用户侧去重简化）； +# 末尾追加 platform / keyword / page。 +JD_ITEM_CSV_FIELDS = ( + "item_id", + "sku_id", + "title", + # "title_plain", + "price", + "coupon_price", + "original_price", + "selling_point", + "comment_sales_floor", + "total_sales", + "hot_list_rank", + "comment_count", + "shop_name", + "shop_url", + "shop_info_url", + "location", + "detail_url", + "image", + "seckill_info", + "attributes", + "leaf_category", + # "same_count", + # "relation_score", + # "is_p4p", + "platform", + "keyword", + "page", +) + +CSV_FIELDS = tuple(JD_EXPORT_COLUMN_HEADERS[k] for k in JD_ITEM_CSV_FIELDS) + + +def jd_row_to_export(row: dict[str, str]) -> dict[str, str]: + """内部键 → 导出列名；无内容则不写入该键（JSON 稀疏对象；CSV 缺列按空单元格写出）。""" + out: dict[str, str] = {} + for k in JD_ITEM_CSV_FIELDS: + v = str(row.get(k, "") or "").strip() + if not v: + continue + out[JD_EXPORT_COLUMN_HEADERS[k]] = v + return out + + +def _jd_empty_export_row() -> dict[str, str]: + return {k: "" for k in JD_ITEM_CSV_FIELDS} +def _human_text(s: str, max_len: int = 2000) -> str: + if not s: + return "" + t = re.sub(r"<[^>]+>", " ", s) + t = html_module.unescape(t) + t = " ".join(t.split()).strip() + return t[:max_len] + + +def _safe_url(u: str) -> str: + u = (u or "").strip() + if not u: + return "" + if u.startswith("//"): + return "https:" + u + if u.startswith("/"): + return "https://so.m.jd.com" + u + if u.startswith("http://"): + return "https://" + u[7:] + return u + + +# 搜索接口里常见仅返回 jfs/...，需拼到 360buyimg 下（与 PC 列表 n2/s480x480 规格一致；img10–img14 等 CDN 可互换） +_JD_PRODUCT_IMG_HOST = "https://img13.360buyimg.com" + + +def _jd_product_image_url(u: str) -> str: + """ + 主图补全为可访问 URL。 + 例：jfs/t1/402762/.../xxx.jpg → https://img13.360buyimg.com/n2/s480x480_jfs/t1/402762/.../xxx.jpg + """ + u = (u or "").strip() + if not u: + return "" + if u.startswith("//"): + return "https:" + u + if u.startswith("http://"): + return "https://" + u[7:] + if u.startswith("https://"): + return u + low = u.lower() + if "360buyimg.com" in low and not re.match(r"^https?://", u, re.I): + return "https://" + u.lstrip("/") + p = u.lstrip("/") + if p.startswith("jfs/"): + return f"{_JD_PRODUCT_IMG_HOST}/n2/s480x480_{p}" + if re.match(r"^n[0-9]+/", p): + return f"{_JD_PRODUCT_IMG_HOST}/{p}" + return u +def _detect_blocked(html_text: str) -> str | None: + if not (html_text and html_text.strip()): + return None + t = html_text.lower() + if "当前人数过多" in html_text or "前往京东app" in t or "前往京东APP" in html_text: + return ( + "服务端限流/风控提示（纯文本）。search.action 易风控；" + "含 h5st 的 api 请用 search/jd_search_playwright.py，或粘贴完整 URL（--url）；" + "或 jd_low_gi_playwright.py 整页抓取。" + ) + if "plogin.m.jd.com" in t or "passport.jd.com" in t: + return "疑似被要求登录（跳转到登录域）。" + # pc_search 正常 JSON 里字段名常含 risk/verify 等子串，宽松匹配会误判。 + payload = _loads_json_or_jsonp(html_text) + if payload is not None: + raw_probe: list[dict[str, Any]] = [] + _walk_collect_jd_wares(payload, raw_probe) + if raw_probe: + return None + if re.search( + r"risk\.jd\.com|riskhandler|/risk/|sev\.jd|verify\.jd\.com|" + r"sliderverify|slide_verify|securityverify", + html_text, + re.I, + ): + return "疑似进入风控/验证页（risk/verify）。" + elif "risk" in t and ("handler" in t or "verify" in t): + return "疑似进入风控/验证页（risk/verify）。" + if "验证码" in html_text or "安全验证" in html_text: + return "疑似需要验证码/安全验证。" + return None +def strip_jsonp(body: str) -> Any: + """去掉 JSONP 包裹，解析为 Python 对象（与 taobao strip_jsonp 思路一致）。""" + text = body.strip().rstrip(";") + m = re.match(r"^[a-zA-Z_$][a-zA-Z0-9_$]*\((.*)\)\s*$", text, re.DOTALL) + if not m: + raise ValueError("响应不是预期的 JSONP 格式") + return json.loads(m.group(1)) + + +def _loads_json_or_jsonp(text: str) -> Any | None: + s = text.strip() + if not s: + return None + if s.startswith("{") or s.startswith("["): + try: + return json.loads(s) + except json.JSONDecodeError: + return None + try: + return strip_jsonp(s) + except (json.JSONDecodeError, ValueError): + return None +JD_SKU_KEYS = ("wareId", "skuId", "itemId", "wid", "productId") +JD_TITLE_KEYS = ("wareName", "wname", "name", "title", "skuName") +# pc_search_searchWare 常见：jdPrice / jdPriceText / realPrice +JD_PRICE_KEYS = ( + "jdPrice", + "jdPriceText", + "realPrice", + "price", + "priceText", + "p", + "salePrice", + "purchasePrice", +) +JD_IMG_KEYS = ("imageurl", "imageUrl", "imgUrl", "picUrl", "image") +JD_ORIGINAL_PRICE_KEYS = ( + "oriPrice", + "originalPrice", + "marketPrice", + "linePrice", + "mPrice", + "maoPrice", + "strikePrice", +) +# finalPrice 为对象，见 _jd_parse_final_price；勿在此放 dict 键名 +JD_COUPON_KEYS = ("couponPrice", "subsidyPrice", "promoPrice") +JD_DETAIL_URL_KEYS = ( + "toUrl", + "clickUrl", + "wareUrl", + "href", + "itemUrl", + "detailUrl", + "skuUrl", + "pUrl", +) +JD_SHOP_URL_KEYS = ("shopUrl", "storeUrl", "shopURL", "jumpUrl") +JD_LOC_KEYS = ("deliveryAddress", "area", "procity", "stockAddress", "sendAddr") +JD_TAG_LIST_KEYS = ( + "wareBeltList", + "beltList", + "labelList", + "tagList", + "wareTags", + "tags", + "icons", + "serviceIcons", + "iconList", + "iconList1", + "iconList2", + "iconList3", + "iconList4", + "textTagList", + "beltAttrs", + "serviceTags", + "benefitList", + "sellPoints", + "commonTagList", +) + +# 单块对象里含 title:[str,…]（如 newRegionFloor） +JD_TAG_NEST_DICT_KEYS = ( + "newRegionFloor", + "midTagList", + "paragraphInfo", + "floatLayerInfo", + "drawer", +) + +# JSON 里常见 title:["高膳食纤维","低GI食品"]、title:["礼盒装…热卖榜第1名"] 等字符串数组，需整树收集 +JD_LIST_STRING_TEXT_KEYS = frozenset( + { + "title", + "subtitle", + "subtitles", + "sellpoint", + "sellpoints", + "usp", + "usps", + "textlist", + "textlists", + "wordlist", + "keywords", + "highlight", + "highlights", + "sellingpoint", + "sellingpoints", + "featurelist", + "pointlist", + "points", + "shorttitle", + "shorttitles", + "recommendreason", + "reasonlist", + "icontexts", + "icontext", + "tagtexts", + "descs", + "labels", + } +) +def _sval_jd(d: dict[str, Any], keys: tuple[str, ...]) -> str: + for k in keys: + v = d.get(k) + if v is None or isinstance(v, (dict, list)): + continue + t = str(v).strip() + if t: + return t + return "" + + +def _jd_flatten_ware(d: dict[str, Any]) -> dict[str, Any]: + out = dict(d) + for nest_key in ("wareInfo", "product", "skuInfo", "item", "main", "content", "base"): + inner = out.get(nest_key) + if isinstance(inner, dict): + for k, v in inner.items(): + if k not in out or out[k] in (None, "", [], {}): + out[k] = v + ex = out.get("exContent") or out.get("excontent") + if isinstance(ex, dict): + for k, v in ex.items(): + if k not in out or out[k] in (None, "", [], {}): + out[k] = v + return out + + +def _jd_norm_key(s: str) -> str: + """去重用：压缩空白，便于合并「相同文案、空白不同」的重复。""" + return " ".join(str(s).split()).strip() + + +def _jd_unique_ordered(strings: list[str]) -> list[str]: + seen: set[str] = set() + out: list[str] = [] + for x in strings: + t = _jd_norm_key(x) + if len(t) < 2: + continue + k = t.casefold() + if k in seen: + continue + seen.add(k) + out.append(t) + return out + + +def _jd_benefit_list_title_lines(d: dict[str, Any]) -> list[str]: + """benefitList[].title：PC 搜索里 sellingPoint 常为 null，卖点多在此。""" + bl = d.get("benefitList") + if not isinstance(bl, list): + return [] + lines: list[str] = [] + for it in bl[:30]: + if not isinstance(it, dict): + continue + t = it.get("title") + if isinstance(t, list): + for x in t[:25]: + if isinstance(x, str) and x.strip(): + lines.append(_jd_norm_key(x)) + elif isinstance(t, str) and t.strip(): + lines.append(_jd_norm_key(t)) + return lines + + +def _jd_sell_points_lines(d: dict[str, Any]) -> list[str]: + """sellPoints：字符串列表或对象列表。""" + sp = d.get("sellPoints") + if not isinstance(sp, list): + return [] + out: list[str] = [] + for x in sp[:25]: + if isinstance(x, str) and x.strip(): + out.append(_jd_norm_key(x)) + elif isinstance(x, dict): + t = _sval_jd(x, ("text", "title", "name", "desc")) + if t: + out.append(_jd_norm_key(t)) + return out + + +def _jd_selling_point_norm_set(d: dict[str, Any]) -> set[str]: + """sellingPoint、benefitList.title、sellPoints 规范化键，用于树遍历去重。""" + out: set[str] = set() + sp = d.get("sellingPoint") + if isinstance(sp, list): + for x in sp: + if isinstance(x, str) and x.strip(): + out.add(_jd_norm_key(x).casefold()) + for t in _jd_benefit_list_title_lines(d): + out.add(t.casefold()) + for t in _jd_sell_points_lines(d): + out.add(t.casefold()) + return out + + +def _jd_iter_tag_strings(d: dict[str, Any]) -> list[str]: + seen: set[str] = set() + out: list[str] = [] + sp_dup = _jd_selling_point_norm_set(d) + + def add(s: str, *, skip_if_selling_dup: bool = False) -> None: + t = _jd_norm_key(s) + if len(t) < 2: + return + if skip_if_selling_dup and sp_dup and t.casefold() in sp_dup: + return + k = t.casefold() + if k in seen: + return + seen.add(k) + out.append(t) + + sub_keys = ( + "text", + "name", + "title", + "desc", + "msg", + "beltMsg", + "labelName", + "typeName", + "showName", + "content", + "beltTitle", + ) + for key in JD_TAG_LIST_KEYS: + v = d.get(key) + is_benefit = key == "benefitList" + if isinstance(v, list): + for it in v[:30]: + if isinstance(it, str): + add(it) + elif isinstance(it, dict): + for sk in sub_keys: + t = it.get(sk) + sk_skip = is_benefit and sk == "title" + if isinstance(t, list): + for x in t[:25]: + if isinstance(x, str) and x.strip(): + add(x.strip(), skip_if_selling_dup=sk_skip) + elif isinstance(t, str) and t.strip(): + add(t.strip(), skip_if_selling_dup=sk_skip) + break + elif isinstance(v, dict): + for sk in sub_keys: + t = v.get(sk) + if isinstance(t, list): + for x in t[:25]: + if isinstance(x, str) and x.strip(): + add(x.strip()) + elif isinstance(t, str) and t.strip(): + add(t.strip()) + break + + for key in JD_TAG_NEST_DICT_KEYS: + v = d.get(key) + if not isinstance(v, dict): + continue + tl = v.get("title") + if isinstance(tl, list): + for x in tl[:25]: + if isinstance(x, str) and x.strip(): + add(x.strip(), skip_if_selling_dup=True) + elif isinstance(tl, str) and tl.strip(): + add(tl.strip(), skip_if_selling_dup=True) + + mtl = d.get("midTagList") + if isinstance(mtl, list): + for it in mtl[:25]: + if not isinstance(it, dict): + continue + for sk in ("text", "name", "title", "desc", "msg"): + t = it.get(sk) + if isinstance(t, str) and t.strip(): + add(t.strip()) + break + return out + + +def _jd_collect_list_string_fragments( + root: Any, + *, + selling_norm_skip: set[str] | frozenset[str] | None = None, + max_depth: int = 14, + max_list_len: int = 40, + max_branch_lists: int = 80, +) -> tuple[list[str], list[str]]: + """ + 从整棵 JSON 子树收集「键在 JD_LIST_STRING_TEXT_KEYS 且值为字符串数组」的文案。 + + 返回 (逐条短句, 每组用 · 拼接的整组)，供 hot_list_rank 等使用。 + """ + skip_sp = selling_norm_skip or set() + individuals: list[str] = [] + groups: list[str] = [] + seen_ind: set[str] = set() + seen_grp: set[str] = set() + list_walk_count = 0 + + def add_ind(s: str) -> None: + t = _jd_norm_key(s) + if len(t) < 2: + return + k = t.casefold() + if skip_sp and k in skip_sp: + return + if k in seen_ind: + return + seen_ind.add(k) + individuals.append(t) + + def walk(obj: Any, depth: int) -> None: + nonlocal list_walk_count + if depth > max_depth or obj is None: + return + if isinstance(obj, dict): + for k, v in obj.items(): + lk = str(k).lower() + if ( + isinstance(v, list) + and lk in JD_LIST_STRING_TEXT_KEYS + and list_walk_count < max_branch_lists + ): + list_walk_count += 1 + elems: list[str] = [] + for x in v[:max_list_len]: + if isinstance(x, str) and x.strip(): + t = _jd_norm_key(x) + if skip_sp and t.casefold() in skip_sp: + continue + elems.append(t) + add_ind(t) + elif isinstance(x, dict): + t = _sval_jd( + x, + ("text", "name", "title", "desc", "value", "content"), + ) + if t: + t = _jd_norm_key(t) + if skip_sp and t.casefold() in skip_sp: + continue + elems.append(t) + add_ind(t) + if elems: + gline = " · ".join(elems) + gk = gline.casefold() + if gk not in seen_grp: + seen_grp.add(gk) + groups.append(gline) + walk(v, depth + 1) + elif isinstance(obj, list): + for x in obj[:90]: + walk(x, depth + 1) + + walk(root, 0) + return individuals, groups + + +# commentSalesFloor 等 attr+text 格式化时跳过明显非文案类 attr +JD_ATTR_TEXT_SKIP_ATTRS = frozenset( + { + "wareid", + "skuid", + "itemid", + "imageurl", + "imgurl", + "href", + "url", + "cid", + "shopid", + "venderid", + } +) + + +def _jd_is_rank_text(s: str) -> bool: + t = s.strip() + if not t: + return False + if "榜" in t: + return True + if "TOP" in t.upper() and len(t) <= 48: + return True + if re.search(r"第\s*\d+\s*名", t): + return True + if "热销" in t and len(t) <= 36: + return True + return False + + +def _jd_format_price_show_dict(ps: dict[str, Any]) -> tuple[str, str]: + parts: list[str] = [] + coupon = "" + for k, v in ps.items(): + if isinstance(v, str) and v.strip(): + parts.append(f"{k}={v.strip()[:100]}") + elif isinstance(v, (int, float)) and not isinstance(v, bool): + parts.append(f"{k}={v}") + line = " | ".join(parts)[:500] + for ck in ("couponPrice", "purchasePrice", "finalPrice", "subsidyPrice"): + v = ps.get(ck) + if isinstance(v, str) and v.strip(): + coupon = v.strip()[:80] + break + if isinstance(v, (int, float)) and not isinstance(v, bool): + coupon = str(v) + break + return line, coupon + + +def _jd_parse_final_price(d: dict[str, Any]) -> tuple[str, str]: + """ + pc 搜索 ware.finalPrice：到手价 + estimatedPrice。 + 返回 (estimatedPrice 字符串, 展示用「到手价:25.5」)。 + """ + for fk in ("finalPrice", "mockFinalPrice", "intervalNewPrice"): + fp = d.get(fk) + if not isinstance(fp, dict): + continue + est = fp.get("estimatedPrice") + if est is None: + est = fp.get("price") + tit = fp.get("title") + tit_s = str(tit).strip() if tit is not None else "" + if not tit_s: + tit_s = "到手价" + es = str(est).strip() if est is not None else "" + if not es: + continue + return es, f"{tit_s}:{es}" + return "", "" + + +def _jd_selling_point_text(d: dict[str, Any]) -> str: + """优先根字段 sellingPoint；为空时用 benefitList / sellPoints（接口常返回 sellingPoint=null）。""" + lines: list[str] = [] + sp = d.get("sellingPoint") + if isinstance(sp, list): + lines = [ + _jd_norm_key(x) for x in sp[:25] if isinstance(x, str) and x.strip() + ] + if not lines: + lines = _jd_benefit_list_title_lines(d) + _jd_sell_points_lines(d) + return _human_text(" | ".join(_jd_unique_ordered(lines)), 1200) + + +def _jd_format_comment_sales_floor(d: dict[str, Any]) -> str: + """仅根字段 commentSalesFloor：[{attr,text},…]。""" + csf = d.get("commentSalesFloor") + if not isinstance(csf, list): + return "" + parts: list[str] = [] + for el in csf[:30]: + if not isinstance(el, dict): + continue + text = el.get("text") or el.get("msg") or el.get("desc") + if not isinstance(text, str) or not text.strip(): + continue + t = _jd_norm_key(text) + attr = el.get("attr") or el.get("type") or el.get("key") or "" + if isinstance(attr, str) and attr.strip(): + lk = attr.strip().lower() + if lk in JD_ATTR_TEXT_SKIP_ATTRS: + parts.append(t) + else: + parts.append(f"{attr.strip()}:{t}") + else: + parts.append(t) + return _human_text(" | ".join(parts), 800) + + +def _jd_seckill_text(d: dict[str, Any]) -> str: + for k in ("seckillInfo", "secKill", "secKillInfo", "miaosha", "flashSale"): + v = d.get(k) + if isinstance(v, str) and v.strip(): + return v.strip()[:400] + if isinstance(v, dict): + t = _sval_jd(v, ("text", "title", "name", "status")) + if t: + return t[:400] + return "" + + +def _jd_attributes_line(d: dict[str, Any]) -> str: + chunks: list[str] = [] + for pl_key in ("propertyList", "properties", "wareProps", "props"): + pl = d.get(pl_key) + if not isinstance(pl, list): + continue + for el in pl[:25]: + if isinstance(el, dict): + k = str(el.get("name") or el.get("key") or "").strip() + v = str(el.get("value") or el.get("text") or "").strip() + if k and v: + chunks.append(f"{k}:{v}") + elif isinstance(el, str) and el.strip(): + chunks.append(el.strip()[:120]) + if chunks: + break + return " | ".join(chunks)[:4000] + + +def _jd_attributes_line_full(d: dict[str, Any]) -> str: + """属性行 + pc 搜索常见 color / catid / shortName。""" + base = _jd_attributes_line(d) + extra: list[str] = [] + c = d.get("color") + if isinstance(c, str) and c.strip(): + extra.append(f"颜色规格:{c.strip()}") + cid = d.get("catid") or d.get("cid3") or d.get("cid3Name") + if cid is not None and str(cid).strip(): + extra.append(f"类目:{str(cid).strip()}") + sn = d.get("shortName") + if isinstance(sn, str) and sn.strip(): + extra.append(f"简称:{sn.strip()}") + parts = [p for p in extra if p] + if base: + parts.append(base) + return " | ".join(parts)[:4000] + + +def _jd_is_p4p_flag(d: dict[str, Any]) -> str: + for k in ("isAdv", "isAd", "isP4p", "is_p4p", "adFlag"): + v = d.get(k) + if v is None: + continue + return str(v).strip()[:20] + if d.get("extension_id") or d.get("extensionId") or d.get("adLog"): + return "true" + return "" + + +def _jd_is_explicit_pc_search_ad_ware(d0: dict[str, Any]) -> bool: + """ + 仅显式广告标记。extensionId/adLog 在自然商品上也常见，不能用来扣减 body.s。 + """ + for k in ("isAdv", "isAd", "isP4p", "is_p4p", "adFlag"): + v = d0.get(k) + if v is None: + continue + s = str(v).strip().lower() + if s in ("1", "true", "yes", "y"): + return True + return False + + +def _pc_search_s_delta_from_natural_slot_count(n_natural: int, slot_total: int) -> int: + """ + pc_search 两包满屏：下一包 ``body.s = 上一包 s + Δ``，抓包为 **Δ = 自然位条数 - 1**。 + 例：首包 ``s=1``、``wareList`` 非广告 22 条 → 次包 ``s=22``（即 ``1+21``）。 + """ + cnt = n_natural if n_natural > 0 else max(0, slot_total) + return max(1, cnt - 1) + + +def _pc_search_body_s_step_from_raw_ware_list(raw_list: list[dict[str, Any]]) -> int: + """ + 无 ``wareList`` 时的兜底：树遍历去重 SKU + 去显式广告得自然位数，再套 ``Δ=自然位-1``。 + """ + seen_sku: set[str] = set() + n = 0 + for obj in raw_list: + d0 = _jd_flatten_ware(obj) + if _jd_is_explicit_pc_search_ad_ware(d0): + continue + sku = _sval_jd(d0, JD_SKU_KEYS) + if sku.isdigit() and len(sku) >= 5: + if sku in seen_sku: + continue + seen_sku.add(sku) + n += 1 + return _pc_search_s_delta_from_natural_slot_count(n, len(raw_list)) + + +def _pc_search_s_step_from_payload_data_ware_list(payload: Any) -> int | None: + """ + 与抓包 ``response1.js`` + 第二包请求（``page=2,s=22``）一致： + ``data.wareList`` 内非广告条数为 ``N`` 时，**``s`` 增量为 ``max(1, N-1)``**。 + """ + if not isinstance(payload, dict): + return None + data = payload.get("data") + if not isinstance(data, dict): + return None + wl = data.get("wareList") + if not isinstance(wl, list) or len(wl) == 0: + return None + if not all(isinstance(x, dict) for x in wl): + return None + n = 0 + for w in wl: + d0 = _jd_flatten_ware(w) + if _jd_is_explicit_pc_search_ad_ware(d0): + continue + n += 1 + return _pc_search_s_delta_from_natural_slot_count(n, len(wl)) + + +def pc_search_ware_list_slot_count_from_body(text: str) -> int | None: + """Return len(data.wareList) from pc_search JSON/JSONP body, or None.""" + payload = _loads_json_or_jsonp(text) + if not isinstance(payload, dict): + return None + data = payload.get("data") + if not isinstance(data, dict): + return None + wl = data.get("wareList") + return len(wl) if isinstance(wl, list) else None + + +def _jd_coerce_int(v: Any) -> int | None: + if v is None or isinstance(v, bool): + return None + if isinstance(v, int): + return v + if isinstance(v, str) and v.strip(): + t = v.strip() + if t.lstrip("-").isdigit(): + try: + return int(t) + except ValueError: + return None + return None + + +def _extract_pc_search_next_s_from_payload( + payload: Any, *, request_page: int, request_s: int +) -> int | None: + want_p = request_page + 1 + cands: list[int] = [] + + def walk(o: Any) -> None: + if isinstance(o, dict): + p = _jd_coerce_int(o.get("page")) + if p is None: + p = _jd_coerce_int(o.get("pageNo")) or _jd_coerce_int( + o.get("pageIndex") + ) + s_v = _jd_coerce_int(o.get("s")) + if p == want_p and s_v is not None and s_v > request_s: + cands.append(s_v) + for v in o.values(): + walk(v) + elif isinstance(o, list): + for x in o: + walk(x) + + walk(payload) + return min(cands) if cands else None + + +def _looks_like_jd_ware(d: dict[str, Any]) -> bool: + d0 = _jd_flatten_ware(d) + sku = _sval_jd(d0, JD_SKU_KEYS) + if not (sku.isdigit() and len(sku) >= 5): + return False + title = _sval_jd(d0, JD_TITLE_KEYS) + return len(title) >= 2 + + +def _normalize_jd_api_row(d: dict[str, Any], *, keyword: str, page: int) -> dict[str, str]: + """ + 将 pc_search_searchWare 单条 ware（及同类结构）规整为与淘宝 CSV 对齐的宽表。 + + 主路径字段：wareId/skuId、wareName、jdPrice/jdPriceText、oriPrice、finalPrice（到手价）、 + sellingPoint、commentFuzzy（评价量）、commentSalesFloor、benefitList/newRegionFloor、 + iconList1–4、promotionSet、wareBuried、shopId/shopName、color/catid、isAdv/extensionId 等； + 仍保留树遍历兜底以兼容字段变更。 + """ + d = _jd_flatten_ware(d) + sku = _sval_jd(d, JD_SKU_KEYS) + ware = _sval_jd(d, ("wareId", "wid")) + item_id = ware if ware else sku + + title = _human_text(_sval_jd(d, JD_TITLE_KEYS), 2000) + price = _human_text(_sval_jd(d, JD_PRICE_KEYS), 120) + orig = _human_text(_sval_jd(d, JD_ORIGINAL_PRICE_KEYS), 120) + coupon_p = _human_text(_sval_jd(d, JD_COUPON_KEYS), 80) + fp_coupon, _ = _jd_parse_final_price(d) + if fp_coupon: + if not coupon_p: + coupon_p = fp_coupon + elif fp_coupon not in coupon_p: + coupon_p = _human_text(f"{coupon_p}/{fp_coupon}", 120) + + ps = d.get("priceShow") + _, ps_coupon = ( + _jd_format_price_show_dict(ps) if isinstance(ps, dict) else ("", "") + ) + if not coupon_p and ps_coupon: + coupon_p = ps_coupon + + selling_point = _jd_selling_point_text(d) + comment_count = _human_text( + _sval_jd( + d, + ("commentFuzzy", "comment_fuzzy", "cmtFuzzy", "evaluationFuzzy"), + ), + 120, + ) + comment_sales_floor = _jd_format_comment_sales_floor(d) + total_sales = _human_text( + _sval_jd( + d, + ("totalSales", "total_sales", "TotalSales"), + ), + 400, + ) + + tag_strs = _jd_iter_tag_strings(d) + arr_ind, _arr_groups = _jd_collect_list_string_fragments( + d, selling_norm_skip=_jd_selling_point_norm_set(d) + ) + tag_seen_norm = {t.casefold() for t in tag_strs} + extra_from_arrays = [ + x for x in arr_ind if _jd_norm_key(x).casefold() not in tag_seen_norm + ] + + merged_for_signals = _jd_unique_ordered(tag_strs + extra_from_arrays) + rank_parts = _jd_unique_ordered( + [t for t in merged_for_signals if _jd_is_rank_text(t)] + ) + + hot_list_rank = _human_text(" | ".join(rank_parts), 600) + + shop = _human_text( + _sval_jd(d, ("shopName", "venderName", "storeName", "shop_name")), 200 + ) + shop_url = _safe_url(_sval_jd(d, JD_SHOP_URL_KEYS))[:2000] + if not shop_url: + sid = d.get("shopId") + if sid is not None and str(sid).strip(): + shop_url = f"https://mall.jd.com/index-{str(sid).strip()}.html"[:2000] + shop_info_url = _safe_url(_sval_jd(d, ("shopInfoUrl", "brandUrl")))[:2000] + + loc = _human_text(_sval_jd(d, JD_LOC_KEYS), 200) + + detail_raw = _sval_jd(d, JD_DETAIL_URL_KEYS) + if detail_raw: + detail_url = _safe_url(detail_raw)[:2500] + else: + detail_url = ( + f"https://item.m.jd.com/product/{sku}.html" if sku else "" + ) + + img_raw = _sval_jd( + d, + JD_IMG_KEYS + ("squareImage", "squarePic", "imgDfsUrl"), + ) + image = _jd_product_image_url(img_raw)[:1200] if img_raw else "" + + seckill = _jd_seckill_text(d) + attributes = _jd_attributes_line_full(d) + leaf = str( + d.get("leafCategory") or d.get("cid3Name") or d.get("catid") or "" + ).strip()[:80] + samec = str(d.get("sameStyleCount") or d.get("sameCount") or "").strip()[:40] + rel = str(d.get("relationScore") or d.get("score") or "").strip()[:40] + is_p4p = _jd_is_p4p_flag(d) + + out = _jd_empty_export_row() + out["item_id"] = item_id[:80] + out["sku_id"] = sku[:80] + out["title"] = title + out["title_plain"] = title + out["price"] = price + out["coupon_price"] = coupon_p + out["original_price"] = orig + out["selling_point"] = selling_point + out["comment_sales_floor"] = comment_sales_floor + out["total_sales"] = total_sales + out["hot_list_rank"] = hot_list_rank + out["comment_count"] = comment_count + out["shop_name"] = shop + out["shop_url"] = shop_url + out["shop_info_url"] = shop_info_url + out["location"] = loc + out["detail_url"] = detail_url + out["image"] = image + out["seckill_info"] = _human_text(seckill, 400) + out["attributes"] = attributes + out["leaf_category"] = leaf + out["same_count"] = samec + out["relation_score"] = rel + out["is_p4p"] = is_p4p + out["platform"] = "京东" + out["keyword"] = keyword + out["page"] = str(page) + return out + + +def _walk_collect_jd_wares(obj: Any, acc: list[dict[str, Any]]) -> None: + if isinstance(obj, dict): + if _looks_like_jd_ware(obj): + acc.append(obj) + for v in obj.values(): + _walk_collect_jd_wares(v, acc) + elif isinstance(obj, list): + for x in obj: + _walk_collect_jd_wares(x, acc) + + +def _parse_jd_json_payload_rows_and_ware_slots( + payload: Any, *, keyword: str, page: int +) -> tuple[list[dict[str, str]], int, int]: + """ + 一次遍历：导出商品行、树遍历 ware 数（兜底）、以及 **body.s 用的自然位步长**。 + """ + raw_list: list[dict[str, Any]] = [] + _walk_collect_jd_wares(payload, raw_list) + seen: set[str] = set() + rows: list[dict[str, str]] = [] + for d in raw_list: + row = _normalize_jd_api_row(d, keyword=keyword, page=page) + sku = row.get("sku_id", "") + if not sku or sku in seen: + continue + seen.add(sku) + rows.append(row) + s_slots = len(raw_list) + s_scroll = _pc_search_body_s_step_from_raw_ware_list(raw_list) + return rows, s_slots, s_scroll + + +def parse_items_from_jd_json_payload(payload: Any, *, keyword: str, page: int) -> list[dict[str, str]]: + rows, _, _ = _parse_jd_json_payload_rows_and_ware_slots( + payload, keyword=keyword, page=page + ) + return rows + + +def parse_items_and_pc_search_s_step_from_response_body( + text: str, + *, + keyword: str, + page: int, + request_api_page: int | None = None, + request_body_s: int | None = None, +) -> tuple[list[dict[str, str]], int]: + """ + 解析商品列表，并给出本次响应对应的 **body.s 游标增量**。 + + JSON：优先顺序：① 响应内嵌下一跳 ``s``；② ``data.wareList`` 得自然位 ``N`` → **Δs=max(1,N-1)**； + ③ 树遍历兜底，同上 Δ 规则；无 ``wareList`` 时对 ``len(rows)`` 亦用 Δ。 + HTML：增量仍为解析行数（非 pc_search 两包逻辑）。 + """ + payload = _loads_json_or_jsonp(text) + if payload is not None: + rows, s_slots, s_scroll = _parse_jd_json_payload_rows_and_ware_slots( + payload, keyword=keyword, page=page + ) + step_api: int | None = None + if request_api_page is not None and request_body_s is not None: + next_s = _extract_pc_search_next_s_from_payload( + payload, + request_page=request_api_page, + request_s=request_body_s, + ) + if next_s is not None: + d = next_s - request_body_s + if d > 0: + step_api = d + step_wl = _pc_search_s_step_from_payload_data_ware_list(payload) + if rows: + if step_api is not None: + step = step_api + elif step_wl is not None: + step = step_wl + else: + step = s_scroll if s_scroll > 0 else _pc_search_s_delta_from_natural_slot_count( + len(rows), len(rows) + ) + return rows, max(1, step) + if s_slots > 0: + if step_api is not None: + step = step_api + elif step_wl is not None: + step = step_wl + else: + step = ( + s_scroll + if s_scroll > 0 + else _pc_search_s_delta_from_natural_slot_count(0, s_slots) + ) + return [], max(1, step) + # 结构与 ware 识别不匹配时，与 parse_items_from_response_body 一样再试 HTML + rows = parse_items_from_html(text, keyword=keyword, page=page) + return rows, (len(rows) if rows else 0) + + +def parse_items_from_response_body(text: str, *, keyword: str, page: int) -> list[dict[str, str]]: + """先尝试 JSON/JSONP（client.action），失败再按 HTML 解析。""" + payload = _loads_json_or_jsonp(text) + if payload is not None: + rows = parse_items_from_jd_json_payload(payload, keyword=keyword, page=page) + if rows: + return rows + return parse_items_from_html(text, keyword=keyword, page=page) + + +def _jd_minimal_html_row( + *, + keyword: str, + page: int, + sku_id: str, + title: str, + price: str, + detail_url: str, + shop_name: str, + comment_count: str, + image: str, +) -> dict[str, str]: + out = _jd_empty_export_row() + out["item_id"] = sku_id + out["sku_id"] = sku_id + out["title"] = title + out["title_plain"] = title + out["price"] = price + out["detail_url"] = detail_url + out["shop_name"] = shop_name + out["comment_count"] = comment_count + out["image"] = image + out["platform"] = "京东" + out["keyword"] = keyword + out["page"] = str(page) + return out + + +def _collect_items_from_json_like( + html_text: str, keyword: str, page: int +) -> list[dict[str, str]]: + """ + 策略 A：从内嵌 JSON/脚本片段中用正则抓取 wareId/wareName/price 等。 + 该策略对结构变更相对鲁棒，但字段名可能变化，因此做多套模板。 + """ + items: list[dict[str, str]] = [] + + # 常见字段组合：wareId + wareName + jdPrice / price / priceText + patterns: list[re.Pattern[str]] = [ + re.compile( + r'"wareId"\s*:\s*"?(?P\d{5,20})"?' + r'[\s\S]{0,1200}?' + r'"wareName"\s*:\s*"(?P[^"]{2,300})"' + r'[\s\S]{0,1200}?' + r'"(?:jdPrice|price|priceText|mainPrice)"\s*:\s*"?(?P<price>[\d.]{1,12})"?', + re.I, + ), + re.compile( + r'"skuId"\s*:\s*"?(?P<sku>\d{5,20})"?' + r'[\s\S]{0,1200}?' + r'"title"\s*:\s*"(?P<title>[^"]{2,300})"' + r'[\s\S]{0,1200}?' + r'"(?:price|priceText|jdPrice)"\s*:\s*"?(?P<price>[\d.]{1,12})"?', + re.I, + ), + # 兜底：只要 sku + title，价格缺失也接受 + re.compile( + r'"(?:wareId|skuId)"\s*:\s*"?(?P<sku>\d{5,20})"?' + r'[\s\S]{0,1200}?' + r'"(?:wareName|title|name)"\s*:\s*"(?P<title>[^"]{2,300})"', + re.I, + ), + ] + + seen: set[str] = set() + for pat in patterns: + for m in pat.finditer(html_text): + sku = (m.groupdict().get("sku") or "").strip() + title = _human_text(m.groupdict().get("title") or "", 300) + price = (m.groupdict().get("price") or "").strip() + if not sku or not title: + continue + if sku in seen: + continue + seen.add(sku) + detail = f"https://item.m.jd.com/product/{sku}.html" + items.append( + _jd_minimal_html_row( + keyword=keyword, + page=page, + sku_id=sku, + title=title, + price=price, + detail_url=detail, + shop_name="", + comment_count="", + image="", + ) + ) + if items: + # 命中一套 pattern 后就不再叠加下一套，避免重复/误配 + break + return items + + +def _collect_items_from_dom(html_text: str, keyword: str, page: int) -> list[dict[str, str]]: + """ + 策略 B：从 DOM/属性中抓取 data-sku + title/price/href。 + 不依赖 BeautifulSoup（零依赖），但对结构变化更敏感。 + """ + items: list[dict[str, str]] = [] + seen: set[str] = set() + + # 以 data-sku 为锚点，截取一个窗口做二次抽取 + for m in re.finditer(r'data-sku\s*=\s*"(?P<sku>\d{5,20})"', html_text, re.I): + sku = (m.group("sku") or "").strip() + if not sku or sku in seen: + continue + seen.add(sku) + + win = html_text[m.start() : m.start() + 4500] + + # 链接 + href = "" + mh = re.search(r'href\s*=\s*"([^"]+)"', win, re.I) + if mh: + href = _safe_url(mh.group(1)) + if not href: + href = f"https://item.m.jd.com/product/{sku}.html" + + # 标题 + title = "" + for tp in ( + r'title\s*=\s*"([^"]{2,300})"', + r'alt\s*=\s*"([^"]{2,300})"', + r'data-name\s*=\s*"([^"]{2,300})"', + ): + mt = re.search(tp, win, re.I) + if mt: + title = _human_text(mt.group(1), 300) + break + + # 价格（HTML 上可能是 ¥xx.xx / ¥xx.xx） + price = "" + mp = re.search(r"(?:¥|¥)\s*([\d.]{1,12})", win) + if mp: + price = mp.group(1).strip() + + # 店铺（H5 列表可能没有） + shop = "" + ms = re.search(r'data-shopname\s*=\s*"([^"]{2,80})"', win, re.I) + if ms: + shop = _human_text(ms.group(1), 80) + + # 图片 + image = "" + mi = re.search(r'(?:data-lazy-img|data-img|src)\s*=\s*"([^"]+\.(?:jpg|jpeg|png|webp)[^"]*)"', win, re.I) + if mi: + image = _jd_product_image_url(mi.group(1))[:1200] + + if not title: + # 标题拿不到时宁可跳过，避免充斥“空标题” + continue + + items.append( + _jd_minimal_html_row( + keyword=keyword, + page=page, + sku_id=sku, + title=title, + price=price, + detail_url=href[:2000], + shop_name=shop, + comment_count="", + image=image, + ) + ) + + return items + + +def parse_items_from_html(html_text: str, *, keyword: str, page: int) -> list[dict[str, str]]: + # 优先尝试 JSON-like（更稳），再退回 DOM + parsed = _collect_items_from_json_like(html_text, keyword, page) + if not parsed: + parsed = _collect_items_from_dom(html_text, keyword, page) + + seen: set[str] = set() + rows: list[dict[str, str]] = [] + for row in parsed: + sku = (row.get("sku_id") or "").strip() + if not sku or sku in seen: + continue + seen.add(sku) + rows.append(row) + return rows + diff --git a/backend/crawler_copy/jd_pc_search/search/jd_h5_search_requests.py b/backend/crawler_copy/jd_pc_search/search/jd_h5_search_requests.py index e6a9ee4..abfc57b 100644 --- a/backend/crawler_copy/jd_pc_search/search/jd_h5_search_requests.py +++ b/backend/crawler_copy/jd_pc_search/search/jd_h5_search_requests.py @@ -34,6 +34,9 @@ python jd_h5_search_requests.py --q 低GI --page 1 --csv --out ../../data/jd_h5_p1.csv python jd_h5_search_requests.py --url "https://api.m.jd.com/..." + +**模块划分**：响应 JSON/HTML 解析、商品行扁平化、游标与重试判定在 ``jd_h5_search_parse.py``； +本文件保留 Node 签 URL、``requests``/Header、CLI ``main``，并从解析模块 re-export ``CSV_FIELDS`` 等以兼容旧导入。 """ from __future__ import annotations @@ -58,60 +61,34 @@ import requests _JD_PKG_ROOT = Path(__file__).resolve().parent.parent if str(_JD_PKG_ROOT) not in sys.path: sys.path.insert(0, str(_JD_PKG_ROOT)) +_BACKEND_ROOT = Path(__file__).resolve().parents[3] +if str(_BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(_BACKEND_ROOT)) from common.jd_delay_utils import parse_request_delay_range, sleep_pc_search_request_gap +from pipeline.csv.schema import JD_SEARCH_CSV_HEADERS as JD_EXPORT_COLUMN_HEADERS # noqa: E402 _JD_PC_SEARCH_DIR = Path(__file__).resolve().parent +if str(_JD_PC_SEARCH_DIR) not in sys.path: + sys.path.insert(0, str(_JD_PC_SEARCH_DIR)) -# PC 搜索：与浏览器一致，**每逻辑页固定 2 包** pc_search（body.page 连续 +1：第 L 页为 2L-1 与 2L）。 -# **Δs = max(1, 自然位条数-1)**（wareList 非广告计自然位）。跳过前序屏时同样每逻辑页 2 包推进游标。 -# CLI 的 --page 是**逻辑页**，不是请求 body.page（避免混淆）。 -JD_PC_SEARCH_ITEMS_PER_PAGE = 60 -JD_PC_SEARCH_CHUNKS_PER_LOGICAL_PAGE = 2 -# 重试仍失败时推进 s 的兜底（与常见两包 Δs≈21～22 一致） -JD_PC_SEARCH_FALLBACK_S_STEP = 22 - - -def pc_search_response_is_empty_ware_list(body: str) -> bool: - """合法 JSON 且 ``data.wareList`` 为明确空数组时视为「已无更多商品」，不宜再强推游标。""" - p = _loads_json_or_jsonp((body or "").strip()) - if not isinstance(p, dict): - return False - data = p.get("data") - return isinstance(data, dict) and data.get("wareList") == [] - - -def pc_search_should_retry_fetch( - body: str, *, has_rows: bool, s_step: int -) -> bool: - """ - 是否应对同一 body.page / s 重发请求。 - 空体、非 JSON 短包等视为瞬时故障；明确的 ``data.wareList == []`` 不重试。 - """ - if has_rows or s_step > 0: - return False - s = (body or "").strip() - if not s: - return True - p = _loads_json_or_jsonp(s) - if p is None: - return len(s) < 12000 - if not isinstance(p, dict): - return False - data = p.get("data") - if isinstance(data, dict) and data.get("wareList") == []: - return False - return True - - -def jd_pc_api_body_page_first_pack(logical_page_1based: int) -> int: - """逻辑页 L（从 1 起）第一包请求里的 body.page = 2L - 1。""" - L = max(1, int(logical_page_1based)) - return 2 * L - 1 - - -def _jd_row_count_for_page(rows: list[dict[str, str]], page: int) -> int: - ps = str(page) - return sum(1 for r in rows if (r.get("page") or "").strip() == ps) +from jd_h5_search_parse import ( # noqa: E402 + CSV_FIELDS, + JD_PC_SEARCH_CHUNKS_PER_LOGICAL_PAGE, + JD_PC_SEARCH_FALLBACK_S_STEP, + JD_PC_SEARCH_ITEMS_PER_PAGE, + JD_SKU_KEYS, + _detect_blocked, + _jd_flatten_ware, + _jd_row_count_for_page, + _sval_jd, + jd_pc_api_body_page_first_pack, + jd_row_to_export, + parse_items_and_pc_search_s_step_from_response_body, + parse_items_from_response_body, + pc_search_response_is_empty_ware_list, + pc_search_should_retry_fetch, + pc_search_ware_list_slot_count_from_body, +) def export_pc_search_request_json( @@ -327,136 +304,8 @@ def read_url_list_file(path: str) -> list[str]: return _noncomment_lines(Path(path).read_text(encoding="utf-8")) -# 与淘宝 CANONICAL_FIELDS 基本一致，但不导出 features / promotion_tags（用户侧去重简化）； -# 末尾追加 platform / keyword / page。 -JD_ITEM_CSV_FIELDS = ( - "item_id", - "sku_id", - "title", - # "title_plain", - "price", - "coupon_price", - "original_price", - "selling_point", - "comment_sales_floor", - "hot_list_rank", - "comment_count", - "shop_name", - "shop_url", - "shop_info_url", - "location", - "detail_url", - "image", - "seckill_info", - "attributes", - "leaf_category", - # "same_count", - # "relation_score", - # "is_p4p", - "platform", - "keyword", - "page", -) - -# 导出列名：中文说明(JSON 中主要原始字段名)，便于对照接口 -JD_EXPORT_COLUMN_HEADERS: dict[str, str] = { - "item_id": "主商品ID(wareId)", - "sku_id": "SKU(skuId)", - "title": "标题(wareName)", - # "title_plain": "标题纯文本(wareName)", - "price": "标价(jdPrice,jdPriceText,realPrice)", - "coupon_price": "券后到手价(couponPrice,subsidyPrice,finalPrice.estimatedPrice,priceShow)", - "original_price": "原价(oriPrice,originalPrice,marketPrice)", - "selling_point": "卖点(sellingPoint)", - "comment_sales_floor": "销量楼层(commentSalesFloor)", - "hot_list_rank": "榜单类文案(标签/腰带/标题数组中的榜、TOP 等)", - "comment_count": "评价量(commentFuzzy)", - "shop_name": "店铺名(shopName)", - "shop_url": "店铺链接(shopUrl,shopId)", - "shop_info_url": "店铺信息链接(shopInfoUrl,brandUrl)", - "location": "地域(deliveryAddress,area,procity)", - "detail_url": "商品链接(toUrl,clickUrl,item.m.jd.com)", - "image": "主图(imageurl,imageUrl)", - # "video_cover": "视频封面(videoImage,videoPic)", - # "video_dimension": "视频比例(videoRatio)", - "seckill_info": "秒杀(seckillInfo,secKill)", - "attributes": "规格属性(propertyList,color,catid,shortName)", - "leaf_category": "类目(leafCategory,cid3Name,catid)", - # "same_count": "同款数(sameStyleCount,sameCount)", - # "relation_score": "相关度(relationScore,score)", - # "is_p4p": "广告位(isAdv,isAd,extensionId)", - "platform": "平台(platform)", - "keyword": "搜索词(keyword)", - "page": "页码(page)", -} - -CSV_FIELDS = tuple(JD_EXPORT_COLUMN_HEADERS[k] for k in JD_ITEM_CSV_FIELDS) -def jd_row_to_export(row: dict[str, str]) -> dict[str, str]: - """内部键 → 导出列名；无内容则不写入该键（JSON 稀疏对象；CSV 缺列按空单元格写出）。""" - out: dict[str, str] = {} - for k in JD_ITEM_CSV_FIELDS: - v = str(row.get(k, "") or "").strip() - if not v: - continue - out[JD_EXPORT_COLUMN_HEADERS[k]] = v - return out - - -def _jd_empty_export_row() -> dict[str, str]: - return {k: "" for k in JD_ITEM_CSV_FIELDS} - - -def _human_text(s: str, max_len: int = 2000) -> str: - if not s: - return "" - t = re.sub(r"<[^>]+>", " ", s) - t = html_module.unescape(t) - t = " ".join(t.split()).strip() - return t[:max_len] - - -def _safe_url(u: str) -> str: - u = (u or "").strip() - if not u: - return "" - if u.startswith("//"): - return "https:" + u - if u.startswith("/"): - return "https://so.m.jd.com" + u - if u.startswith("http://"): - return "https://" + u[7:] - return u - - -# 搜索接口里常见仅返回 jfs/...，需拼到 360buyimg 下（与 PC 列表 n2/s480x480 规格一致；img10–img14 等 CDN 可互换） -_JD_PRODUCT_IMG_HOST = "https://img13.360buyimg.com" - - -def _jd_product_image_url(u: str) -> str: - """ - 主图补全为可访问 URL。 - 例：jfs/t1/402762/.../xxx.jpg → https://img13.360buyimg.com/n2/s480x480_jfs/t1/402762/.../xxx.jpg - """ - u = (u or "").strip() - if not u: - return "" - if u.startswith("//"): - return "https:" + u - if u.startswith("http://"): - return "https://" + u[7:] - if u.startswith("https://"): - return u - low = u.lower() - if "360buyimg.com" in low and not re.match(r"^https?://", u, re.I): - return "https://" + u.lstrip("/") - p = u.lstrip("/") - if p.startswith("jfs/"): - return f"{_JD_PRODUCT_IMG_HOST}/n2/s480x480_{p}" - if re.match(r"^n[0-9]+/", p): - return f"{_JD_PRODUCT_IMG_HOST}/{p}" - return u def build_h5_search_url(keyword: str, page: int) -> str: @@ -477,1146 +326,12 @@ def build_h5_search_url(keyword: str, page: int) -> str: return base + "?" + urlencode(q, safe=":%/,+") -def _detect_blocked(html_text: str) -> str | None: - if not (html_text and html_text.strip()): - return None - t = html_text.lower() - if "当前人数过多" in html_text or "前往京东app" in t or "前往京东APP" in html_text: - return ( - "服务端限流/风控提示（纯文本）。search.action 易风控；" - "含 h5st 的 api 请用 search/jd_search_playwright.py，或粘贴完整 URL（--url）；" - "或 jd_low_gi_playwright.py 整页抓取。" - ) - if "plogin.m.jd.com" in t or "passport.jd.com" in t: - return "疑似被要求登录（跳转到登录域）。" - # pc_search 正常 JSON 里字段名常含 risk/verify 等子串，宽松匹配会误判。 - payload = _loads_json_or_jsonp(html_text) - if payload is not None: - raw_probe: list[dict[str, Any]] = [] - _walk_collect_jd_wares(payload, raw_probe) - if raw_probe: - return None - if re.search( - r"risk\.jd\.com|riskhandler|/risk/|sev\.jd|verify\.jd\.com|" - r"sliderverify|slide_verify|securityverify", - html_text, - re.I, - ): - return "疑似进入风控/验证页（risk/verify）。" - elif "risk" in t and ("handler" in t or "verify" in t): - return "疑似进入风控/验证页（risk/verify）。" - if "验证码" in html_text or "安全验证" in html_text: - return "疑似需要验证码/安全验证。" - return None -def strip_jsonp(body: str) -> Any: - """去掉 JSONP 包裹，解析为 Python 对象（与 taobao strip_jsonp 思路一致）。""" - text = body.strip().rstrip(";") - m = re.match(r"^[a-zA-Z_$][a-zA-Z0-9_$]*\((.*)\)\s*$", text, re.DOTALL) - if not m: - raise ValueError("响应不是预期的 JSONP 格式") - return json.loads(m.group(1)) -def _loads_json_or_jsonp(text: str) -> Any | None: - s = text.strip() - if not s: - return None - if s.startswith("{") or s.startswith("["): - try: - return json.loads(s) - except json.JSONDecodeError: - return None - try: - return strip_jsonp(s) - except (json.JSONDecodeError, ValueError): - return None -JD_SKU_KEYS = ("wareId", "skuId", "itemId", "wid", "productId") -JD_TITLE_KEYS = ("wareName", "wname", "name", "title", "skuName") -# pc_search_searchWare 常见：jdPrice / jdPriceText / realPrice -JD_PRICE_KEYS = ( - "jdPrice", - "jdPriceText", - "realPrice", - "price", - "priceText", - "p", - "salePrice", - "purchasePrice", -) -JD_IMG_KEYS = ("imageurl", "imageUrl", "imgUrl", "picUrl", "image") -JD_ORIGINAL_PRICE_KEYS = ( - "oriPrice", - "originalPrice", - "marketPrice", - "linePrice", - "mPrice", - "maoPrice", - "strikePrice", -) -# finalPrice 为对象，见 _jd_parse_final_price；勿在此放 dict 键名 -JD_COUPON_KEYS = ("couponPrice", "subsidyPrice", "promoPrice") -JD_DETAIL_URL_KEYS = ( - "toUrl", - "clickUrl", - "wareUrl", - "href", - "itemUrl", - "detailUrl", - "skuUrl", - "pUrl", -) -JD_SHOP_URL_KEYS = ("shopUrl", "storeUrl", "shopURL", "jumpUrl") -JD_LOC_KEYS = ("deliveryAddress", "area", "procity", "stockAddress", "sendAddr") -JD_TAG_LIST_KEYS = ( - "wareBeltList", - "beltList", - "labelList", - "tagList", - "wareTags", - "tags", - "icons", - "serviceIcons", - "iconList", - "iconList1", - "iconList2", - "iconList3", - "iconList4", - "textTagList", - "beltAttrs", - "serviceTags", - "benefitList", - "sellPoints", - "commonTagList", -) - -# 单块对象里含 title:[str,…]（如 newRegionFloor） -JD_TAG_NEST_DICT_KEYS = ( - "newRegionFloor", - "midTagList", - "paragraphInfo", - "floatLayerInfo", - "drawer", -) - -# JSON 里常见 title:["高膳食纤维","低GI食品"]、title:["礼盒装…热卖榜第1名"] 等字符串数组，需整树收集 -JD_LIST_STRING_TEXT_KEYS = frozenset( - { - "title", - "subtitle", - "subtitles", - "sellpoint", - "sellpoints", - "usp", - "usps", - "textlist", - "textlists", - "wordlist", - "keywords", - "highlight", - "highlights", - "sellingpoint", - "sellingpoints", - "featurelist", - "pointlist", - "points", - "shorttitle", - "shorttitles", - "recommendreason", - "reasonlist", - "icontexts", - "icontext", - "tagtexts", - "descs", - "labels", - } -) - - -def _sval_jd(d: dict[str, Any], keys: tuple[str, ...]) -> str: - for k in keys: - v = d.get(k) - if v is None or isinstance(v, (dict, list)): - continue - t = str(v).strip() - if t: - return t - return "" - - -def _jd_flatten_ware(d: dict[str, Any]) -> dict[str, Any]: - out = dict(d) - for nest_key in ("wareInfo", "product", "skuInfo", "item", "main", "content", "base"): - inner = out.get(nest_key) - if isinstance(inner, dict): - for k, v in inner.items(): - if k not in out or out[k] in (None, "", [], {}): - out[k] = v - ex = out.get("exContent") or out.get("excontent") - if isinstance(ex, dict): - for k, v in ex.items(): - if k not in out or out[k] in (None, "", [], {}): - out[k] = v - return out - - -def _jd_norm_key(s: str) -> str: - """去重用：压缩空白，便于合并「相同文案、空白不同」的重复。""" - return " ".join(str(s).split()).strip() - - -def _jd_unique_ordered(strings: list[str]) -> list[str]: - seen: set[str] = set() - out: list[str] = [] - for x in strings: - t = _jd_norm_key(x) - if len(t) < 2: - continue - k = t.casefold() - if k in seen: - continue - seen.add(k) - out.append(t) - return out - - -def _jd_benefit_list_title_lines(d: dict[str, Any]) -> list[str]: - """benefitList[].title：PC 搜索里 sellingPoint 常为 null，卖点多在此。""" - bl = d.get("benefitList") - if not isinstance(bl, list): - return [] - lines: list[str] = [] - for it in bl[:30]: - if not isinstance(it, dict): - continue - t = it.get("title") - if isinstance(t, list): - for x in t[:25]: - if isinstance(x, str) and x.strip(): - lines.append(_jd_norm_key(x)) - elif isinstance(t, str) and t.strip(): - lines.append(_jd_norm_key(t)) - return lines - - -def _jd_sell_points_lines(d: dict[str, Any]) -> list[str]: - """sellPoints：字符串列表或对象列表。""" - sp = d.get("sellPoints") - if not isinstance(sp, list): - return [] - out: list[str] = [] - for x in sp[:25]: - if isinstance(x, str) and x.strip(): - out.append(_jd_norm_key(x)) - elif isinstance(x, dict): - t = _sval_jd(x, ("text", "title", "name", "desc")) - if t: - out.append(_jd_norm_key(t)) - return out - - -def _jd_selling_point_norm_set(d: dict[str, Any]) -> set[str]: - """sellingPoint、benefitList.title、sellPoints 规范化键，用于树遍历去重。""" - out: set[str] = set() - sp = d.get("sellingPoint") - if isinstance(sp, list): - for x in sp: - if isinstance(x, str) and x.strip(): - out.add(_jd_norm_key(x).casefold()) - for t in _jd_benefit_list_title_lines(d): - out.add(t.casefold()) - for t in _jd_sell_points_lines(d): - out.add(t.casefold()) - return out - - -def _jd_iter_tag_strings(d: dict[str, Any]) -> list[str]: - seen: set[str] = set() - out: list[str] = [] - sp_dup = _jd_selling_point_norm_set(d) - - def add(s: str, *, skip_if_selling_dup: bool = False) -> None: - t = _jd_norm_key(s) - if len(t) < 2: - return - if skip_if_selling_dup and sp_dup and t.casefold() in sp_dup: - return - k = t.casefold() - if k in seen: - return - seen.add(k) - out.append(t) - - sub_keys = ( - "text", - "name", - "title", - "desc", - "msg", - "beltMsg", - "labelName", - "typeName", - "showName", - "content", - "beltTitle", - ) - for key in JD_TAG_LIST_KEYS: - v = d.get(key) - is_benefit = key == "benefitList" - if isinstance(v, list): - for it in v[:30]: - if isinstance(it, str): - add(it) - elif isinstance(it, dict): - for sk in sub_keys: - t = it.get(sk) - sk_skip = is_benefit and sk == "title" - if isinstance(t, list): - for x in t[:25]: - if isinstance(x, str) and x.strip(): - add(x.strip(), skip_if_selling_dup=sk_skip) - elif isinstance(t, str) and t.strip(): - add(t.strip(), skip_if_selling_dup=sk_skip) - break - elif isinstance(v, dict): - for sk in sub_keys: - t = v.get(sk) - if isinstance(t, list): - for x in t[:25]: - if isinstance(x, str) and x.strip(): - add(x.strip()) - elif isinstance(t, str) and t.strip(): - add(t.strip()) - break - - for key in JD_TAG_NEST_DICT_KEYS: - v = d.get(key) - if not isinstance(v, dict): - continue - tl = v.get("title") - if isinstance(tl, list): - for x in tl[:25]: - if isinstance(x, str) and x.strip(): - add(x.strip(), skip_if_selling_dup=True) - elif isinstance(tl, str) and tl.strip(): - add(tl.strip(), skip_if_selling_dup=True) - - mtl = d.get("midTagList") - if isinstance(mtl, list): - for it in mtl[:25]: - if not isinstance(it, dict): - continue - for sk in ("text", "name", "title", "desc", "msg"): - t = it.get(sk) - if isinstance(t, str) and t.strip(): - add(t.strip()) - break - return out - - -def _jd_collect_list_string_fragments( - root: Any, - *, - selling_norm_skip: set[str] | frozenset[str] | None = None, - max_depth: int = 14, - max_list_len: int = 40, - max_branch_lists: int = 80, -) -> tuple[list[str], list[str]]: - """ - 从整棵 JSON 子树收集「键在 JD_LIST_STRING_TEXT_KEYS 且值为字符串数组」的文案。 - - 返回 (逐条短句, 每组用 · 拼接的整组)，供 hot_list_rank 等使用。 - """ - skip_sp = selling_norm_skip or set() - individuals: list[str] = [] - groups: list[str] = [] - seen_ind: set[str] = set() - seen_grp: set[str] = set() - list_walk_count = 0 - - def add_ind(s: str) -> None: - t = _jd_norm_key(s) - if len(t) < 2: - return - k = t.casefold() - if skip_sp and k in skip_sp: - return - if k in seen_ind: - return - seen_ind.add(k) - individuals.append(t) - - def walk(obj: Any, depth: int) -> None: - nonlocal list_walk_count - if depth > max_depth or obj is None: - return - if isinstance(obj, dict): - for k, v in obj.items(): - lk = str(k).lower() - if ( - isinstance(v, list) - and lk in JD_LIST_STRING_TEXT_KEYS - and list_walk_count < max_branch_lists - ): - list_walk_count += 1 - elems: list[str] = [] - for x in v[:max_list_len]: - if isinstance(x, str) and x.strip(): - t = _jd_norm_key(x) - if skip_sp and t.casefold() in skip_sp: - continue - elems.append(t) - add_ind(t) - elif isinstance(x, dict): - t = _sval_jd( - x, - ("text", "name", "title", "desc", "value", "content"), - ) - if t: - t = _jd_norm_key(t) - if skip_sp and t.casefold() in skip_sp: - continue - elems.append(t) - add_ind(t) - if elems: - gline = " · ".join(elems) - gk = gline.casefold() - if gk not in seen_grp: - seen_grp.add(gk) - groups.append(gline) - walk(v, depth + 1) - elif isinstance(obj, list): - for x in obj[:90]: - walk(x, depth + 1) - - walk(root, 0) - return individuals, groups - - -# commentSalesFloor 等 attr+text 格式化时跳过明显非文案类 attr -JD_ATTR_TEXT_SKIP_ATTRS = frozenset( - { - "wareid", - "skuid", - "itemid", - "imageurl", - "imgurl", - "href", - "url", - "cid", - "shopid", - "venderid", - } -) - - -def _jd_is_rank_text(s: str) -> bool: - t = s.strip() - if not t: - return False - if "榜" in t: - return True - if "TOP" in t.upper() and len(t) <= 48: - return True - if re.search(r"第\s*\d+\s*名", t): - return True - if "热销" in t and len(t) <= 36: - return True - return False - - -def _jd_format_price_show_dict(ps: dict[str, Any]) -> tuple[str, str]: - parts: list[str] = [] - coupon = "" - for k, v in ps.items(): - if isinstance(v, str) and v.strip(): - parts.append(f"{k}={v.strip()[:100]}") - elif isinstance(v, (int, float)) and not isinstance(v, bool): - parts.append(f"{k}={v}") - line = " | ".join(parts)[:500] - for ck in ("couponPrice", "purchasePrice", "finalPrice", "subsidyPrice"): - v = ps.get(ck) - if isinstance(v, str) and v.strip(): - coupon = v.strip()[:80] - break - if isinstance(v, (int, float)) and not isinstance(v, bool): - coupon = str(v) - break - return line, coupon - - -def _jd_parse_final_price(d: dict[str, Any]) -> tuple[str, str]: - """ - pc 搜索 ware.finalPrice：到手价 + estimatedPrice。 - 返回 (estimatedPrice 字符串, 展示用「到手价:25.5」)。 - """ - for fk in ("finalPrice", "mockFinalPrice", "intervalNewPrice"): - fp = d.get(fk) - if not isinstance(fp, dict): - continue - est = fp.get("estimatedPrice") - if est is None: - est = fp.get("price") - tit = fp.get("title") - tit_s = str(tit).strip() if tit is not None else "" - if not tit_s: - tit_s = "到手价" - es = str(est).strip() if est is not None else "" - if not es: - continue - return es, f"{tit_s}:{es}" - return "", "" - - -def _jd_selling_point_text(d: dict[str, Any]) -> str: - """优先根字段 sellingPoint；为空时用 benefitList / sellPoints（接口常返回 sellingPoint=null）。""" - lines: list[str] = [] - sp = d.get("sellingPoint") - if isinstance(sp, list): - lines = [ - _jd_norm_key(x) for x in sp[:25] if isinstance(x, str) and x.strip() - ] - if not lines: - lines = _jd_benefit_list_title_lines(d) + _jd_sell_points_lines(d) - return _human_text(" | ".join(_jd_unique_ordered(lines)), 1200) - - -def _jd_format_comment_sales_floor(d: dict[str, Any]) -> str: - """仅根字段 commentSalesFloor：[{attr,text},…]。""" - csf = d.get("commentSalesFloor") - if not isinstance(csf, list): - return "" - parts: list[str] = [] - for el in csf[:30]: - if not isinstance(el, dict): - continue - text = el.get("text") or el.get("msg") or el.get("desc") - if not isinstance(text, str) or not text.strip(): - continue - t = _jd_norm_key(text) - attr = el.get("attr") or el.get("type") or el.get("key") or "" - if isinstance(attr, str) and attr.strip(): - lk = attr.strip().lower() - if lk in JD_ATTR_TEXT_SKIP_ATTRS: - parts.append(t) - else: - parts.append(f"{attr.strip()}:{t}") - else: - parts.append(t) - return _human_text(" | ".join(parts), 800) - - -def _jd_seckill_text(d: dict[str, Any]) -> str: - for k in ("seckillInfo", "secKill", "secKillInfo", "miaosha", "flashSale"): - v = d.get(k) - if isinstance(v, str) and v.strip(): - return v.strip()[:400] - if isinstance(v, dict): - t = _sval_jd(v, ("text", "title", "name", "status")) - if t: - return t[:400] - return "" - - -def _jd_attributes_line(d: dict[str, Any]) -> str: - chunks: list[str] = [] - for pl_key in ("propertyList", "properties", "wareProps", "props"): - pl = d.get(pl_key) - if not isinstance(pl, list): - continue - for el in pl[:25]: - if isinstance(el, dict): - k = str(el.get("name") or el.get("key") or "").strip() - v = str(el.get("value") or el.get("text") or "").strip() - if k and v: - chunks.append(f"{k}:{v}") - elif isinstance(el, str) and el.strip(): - chunks.append(el.strip()[:120]) - if chunks: - break - return " | ".join(chunks)[:4000] - - -def _jd_attributes_line_full(d: dict[str, Any]) -> str: - """属性行 + pc 搜索常见 color / catid / shortName。""" - base = _jd_attributes_line(d) - extra: list[str] = [] - c = d.get("color") - if isinstance(c, str) and c.strip(): - extra.append(f"颜色规格:{c.strip()}") - cid = d.get("catid") or d.get("cid3") or d.get("cid3Name") - if cid is not None and str(cid).strip(): - extra.append(f"类目:{str(cid).strip()}") - sn = d.get("shortName") - if isinstance(sn, str) and sn.strip(): - extra.append(f"简称:{sn.strip()}") - parts = [p for p in extra if p] - if base: - parts.append(base) - return " | ".join(parts)[:4000] - - -def _jd_is_p4p_flag(d: dict[str, Any]) -> str: - for k in ("isAdv", "isAd", "isP4p", "is_p4p", "adFlag"): - v = d.get(k) - if v is None: - continue - return str(v).strip()[:20] - if d.get("extension_id") or d.get("extensionId") or d.get("adLog"): - return "true" - return "" - - -def _jd_is_explicit_pc_search_ad_ware(d0: dict[str, Any]) -> bool: - """ - 仅显式广告标记。extensionId/adLog 在自然商品上也常见，不能用来扣减 body.s。 - """ - for k in ("isAdv", "isAd", "isP4p", "is_p4p", "adFlag"): - v = d0.get(k) - if v is None: - continue - s = str(v).strip().lower() - if s in ("1", "true", "yes", "y"): - return True - return False - - -def _pc_search_s_delta_from_natural_slot_count(n_natural: int, slot_total: int) -> int: - """ - pc_search 两包满屏：下一包 ``body.s = 上一包 s + Δ``，抓包为 **Δ = 自然位条数 - 1**。 - 例：首包 ``s=1``、``wareList`` 非广告 22 条 → 次包 ``s=22``（即 ``1+21``）。 - """ - cnt = n_natural if n_natural > 0 else max(0, slot_total) - return max(1, cnt - 1) - - -def _pc_search_body_s_step_from_raw_ware_list(raw_list: list[dict[str, Any]]) -> int: - """ - 无 ``wareList`` 时的兜底：树遍历去重 SKU + 去显式广告得自然位数，再套 ``Δ=自然位-1``。 - """ - seen_sku: set[str] = set() - n = 0 - for obj in raw_list: - d0 = _jd_flatten_ware(obj) - if _jd_is_explicit_pc_search_ad_ware(d0): - continue - sku = _sval_jd(d0, JD_SKU_KEYS) - if sku.isdigit() and len(sku) >= 5: - if sku in seen_sku: - continue - seen_sku.add(sku) - n += 1 - return _pc_search_s_delta_from_natural_slot_count(n, len(raw_list)) - - -def _pc_search_s_step_from_payload_data_ware_list(payload: Any) -> int | None: - """ - 与抓包 ``response1.js`` + 第二包请求（``page=2,s=22``）一致： - ``data.wareList`` 内非广告条数为 ``N`` 时，**``s`` 增量为 ``max(1, N-1)``**。 - """ - if not isinstance(payload, dict): - return None - data = payload.get("data") - if not isinstance(data, dict): - return None - wl = data.get("wareList") - if not isinstance(wl, list) or len(wl) == 0: - return None - if not all(isinstance(x, dict) for x in wl): - return None - n = 0 - for w in wl: - d0 = _jd_flatten_ware(w) - if _jd_is_explicit_pc_search_ad_ware(d0): - continue - n += 1 - return _pc_search_s_delta_from_natural_slot_count(n, len(wl)) - - -def pc_search_ware_list_slot_count_from_body(text: str) -> int | None: - """Return len(data.wareList) from pc_search JSON/JSONP body, or None.""" - payload = _loads_json_or_jsonp(text) - if not isinstance(payload, dict): - return None - data = payload.get("data") - if not isinstance(data, dict): - return None - wl = data.get("wareList") - return len(wl) if isinstance(wl, list) else None - - -def _jd_coerce_int(v: Any) -> int | None: - if v is None or isinstance(v, bool): - return None - if isinstance(v, int): - return v - if isinstance(v, str) and v.strip(): - t = v.strip() - if t.lstrip("-").isdigit(): - try: - return int(t) - except ValueError: - return None - return None - - -def _extract_pc_search_next_s_from_payload( - payload: Any, *, request_page: int, request_s: int -) -> int | None: - want_p = request_page + 1 - cands: list[int] = [] - - def walk(o: Any) -> None: - if isinstance(o, dict): - p = _jd_coerce_int(o.get("page")) - if p is None: - p = _jd_coerce_int(o.get("pageNo")) or _jd_coerce_int( - o.get("pageIndex") - ) - s_v = _jd_coerce_int(o.get("s")) - if p == want_p and s_v is not None and s_v > request_s: - cands.append(s_v) - for v in o.values(): - walk(v) - elif isinstance(o, list): - for x in o: - walk(x) - - walk(payload) - return min(cands) if cands else None - - -def _looks_like_jd_ware(d: dict[str, Any]) -> bool: - d0 = _jd_flatten_ware(d) - sku = _sval_jd(d0, JD_SKU_KEYS) - if not (sku.isdigit() and len(sku) >= 5): - return False - title = _sval_jd(d0, JD_TITLE_KEYS) - return len(title) >= 2 - - -def _normalize_jd_api_row(d: dict[str, Any], *, keyword: str, page: int) -> dict[str, str]: - """ - 将 pc_search_searchWare 单条 ware（及同类结构）规整为与淘宝 CSV 对齐的宽表。 - - 主路径字段：wareId/skuId、wareName、jdPrice/jdPriceText、oriPrice、finalPrice（到手价）、 - sellingPoint、commentFuzzy（评价量）、commentSalesFloor、benefitList/newRegionFloor、 - iconList1–4、promotionSet、wareBuried、shopId/shopName、color/catid、isAdv/extensionId 等； - 仍保留树遍历兜底以兼容字段变更。 - """ - d = _jd_flatten_ware(d) - sku = _sval_jd(d, JD_SKU_KEYS) - ware = _sval_jd(d, ("wareId", "wid")) - item_id = ware if ware else sku - - title = _human_text(_sval_jd(d, JD_TITLE_KEYS), 2000) - price = _human_text(_sval_jd(d, JD_PRICE_KEYS), 120) - orig = _human_text(_sval_jd(d, JD_ORIGINAL_PRICE_KEYS), 120) - coupon_p = _human_text(_sval_jd(d, JD_COUPON_KEYS), 80) - fp_coupon, _ = _jd_parse_final_price(d) - if fp_coupon: - if not coupon_p: - coupon_p = fp_coupon - elif fp_coupon not in coupon_p: - coupon_p = _human_text(f"{coupon_p}/{fp_coupon}", 120) - - ps = d.get("priceShow") - _, ps_coupon = ( - _jd_format_price_show_dict(ps) if isinstance(ps, dict) else ("", "") - ) - if not coupon_p and ps_coupon: - coupon_p = ps_coupon - - selling_point = _jd_selling_point_text(d) - comment_count = _human_text( - _sval_jd( - d, - ("commentFuzzy", "comment_fuzzy", "cmtFuzzy", "evaluationFuzzy"), - ), - 120, - ) - comment_sales_floor = _jd_format_comment_sales_floor(d) - - tag_strs = _jd_iter_tag_strings(d) - arr_ind, _arr_groups = _jd_collect_list_string_fragments( - d, selling_norm_skip=_jd_selling_point_norm_set(d) - ) - tag_seen_norm = {t.casefold() for t in tag_strs} - extra_from_arrays = [ - x for x in arr_ind if _jd_norm_key(x).casefold() not in tag_seen_norm - ] - - merged_for_signals = _jd_unique_ordered(tag_strs + extra_from_arrays) - rank_parts = _jd_unique_ordered( - [t for t in merged_for_signals if _jd_is_rank_text(t)] - ) - - hot_list_rank = _human_text(" | ".join(rank_parts), 600) - - shop = _human_text( - _sval_jd(d, ("shopName", "venderName", "storeName", "shop_name")), 200 - ) - shop_url = _safe_url(_sval_jd(d, JD_SHOP_URL_KEYS))[:2000] - if not shop_url: - sid = d.get("shopId") - if sid is not None and str(sid).strip(): - shop_url = f"https://mall.jd.com/index-{str(sid).strip()}.html"[:2000] - shop_info_url = _safe_url(_sval_jd(d, ("shopInfoUrl", "brandUrl")))[:2000] - - loc = _human_text(_sval_jd(d, JD_LOC_KEYS), 200) - - detail_raw = _sval_jd(d, JD_DETAIL_URL_KEYS) - if detail_raw: - detail_url = _safe_url(detail_raw)[:2500] - else: - detail_url = ( - f"https://item.m.jd.com/product/{sku}.html" if sku else "" - ) - - img_raw = _sval_jd( - d, - JD_IMG_KEYS + ("squareImage", "squarePic", "imgDfsUrl"), - ) - image = _jd_product_image_url(img_raw)[:1200] if img_raw else "" - - seckill = _jd_seckill_text(d) - attributes = _jd_attributes_line_full(d) - leaf = str( - d.get("leafCategory") or d.get("cid3Name") or d.get("catid") or "" - ).strip()[:80] - samec = str(d.get("sameStyleCount") or d.get("sameCount") or "").strip()[:40] - rel = str(d.get("relationScore") or d.get("score") or "").strip()[:40] - is_p4p = _jd_is_p4p_flag(d) - - out = _jd_empty_export_row() - out["item_id"] = item_id[:80] - out["sku_id"] = sku[:80] - out["title"] = title - out["title_plain"] = title - out["price"] = price - out["coupon_price"] = coupon_p - out["original_price"] = orig - out["selling_point"] = selling_point - out["comment_sales_floor"] = comment_sales_floor - out["hot_list_rank"] = hot_list_rank - out["comment_count"] = comment_count - out["shop_name"] = shop - out["shop_url"] = shop_url - out["shop_info_url"] = shop_info_url - out["location"] = loc - out["detail_url"] = detail_url - out["image"] = image - out["seckill_info"] = _human_text(seckill, 400) - out["attributes"] = attributes - out["leaf_category"] = leaf - out["same_count"] = samec - out["relation_score"] = rel - out["is_p4p"] = is_p4p - out["platform"] = "京东" - out["keyword"] = keyword - out["page"] = str(page) - return out - - -def _walk_collect_jd_wares(obj: Any, acc: list[dict[str, Any]]) -> None: - if isinstance(obj, dict): - if _looks_like_jd_ware(obj): - acc.append(obj) - for v in obj.values(): - _walk_collect_jd_wares(v, acc) - elif isinstance(obj, list): - for x in obj: - _walk_collect_jd_wares(x, acc) - - -def _parse_jd_json_payload_rows_and_ware_slots( - payload: Any, *, keyword: str, page: int -) -> tuple[list[dict[str, str]], int, int]: - """ - 一次遍历：导出商品行、树遍历 ware 数（兜底）、以及 **body.s 用的自然位步长**。 - """ - raw_list: list[dict[str, Any]] = [] - _walk_collect_jd_wares(payload, raw_list) - seen: set[str] = set() - rows: list[dict[str, str]] = [] - for d in raw_list: - row = _normalize_jd_api_row(d, keyword=keyword, page=page) - sku = row.get("sku_id", "") - if not sku or sku in seen: - continue - seen.add(sku) - rows.append(row) - s_slots = len(raw_list) - s_scroll = _pc_search_body_s_step_from_raw_ware_list(raw_list) - return rows, s_slots, s_scroll - - -def parse_items_from_jd_json_payload(payload: Any, *, keyword: str, page: int) -> list[dict[str, str]]: - rows, _, _ = _parse_jd_json_payload_rows_and_ware_slots( - payload, keyword=keyword, page=page - ) - return rows - - -def parse_items_and_pc_search_s_step_from_response_body( - text: str, - *, - keyword: str, - page: int, - request_api_page: int | None = None, - request_body_s: int | None = None, -) -> tuple[list[dict[str, str]], int]: - """ - 解析商品列表，并给出本次响应对应的 **body.s 游标增量**。 - - JSON：优先顺序：① 响应内嵌下一跳 ``s``；② ``data.wareList`` 得自然位 ``N`` → **Δs=max(1,N-1)**； - ③ 树遍历兜底，同上 Δ 规则；无 ``wareList`` 时对 ``len(rows)`` 亦用 Δ。 - HTML：增量仍为解析行数（非 pc_search 两包逻辑）。 - """ - payload = _loads_json_or_jsonp(text) - if payload is not None: - rows, s_slots, s_scroll = _parse_jd_json_payload_rows_and_ware_slots( - payload, keyword=keyword, page=page - ) - step_api: int | None = None - if request_api_page is not None and request_body_s is not None: - next_s = _extract_pc_search_next_s_from_payload( - payload, - request_page=request_api_page, - request_s=request_body_s, - ) - if next_s is not None: - d = next_s - request_body_s - if d > 0: - step_api = d - step_wl = _pc_search_s_step_from_payload_data_ware_list(payload) - if rows: - if step_api is not None: - step = step_api - elif step_wl is not None: - step = step_wl - else: - step = s_scroll if s_scroll > 0 else _pc_search_s_delta_from_natural_slot_count( - len(rows), len(rows) - ) - return rows, max(1, step) - if s_slots > 0: - if step_api is not None: - step = step_api - elif step_wl is not None: - step = step_wl - else: - step = ( - s_scroll - if s_scroll > 0 - else _pc_search_s_delta_from_natural_slot_count(0, s_slots) - ) - return [], max(1, step) - # 结构与 ware 识别不匹配时，与 parse_items_from_response_body 一样再试 HTML - rows = parse_items_from_html(text, keyword=keyword, page=page) - return rows, (len(rows) if rows else 0) - - -def parse_items_from_response_body(text: str, *, keyword: str, page: int) -> list[dict[str, str]]: - """先尝试 JSON/JSONP（client.action），失败再按 HTML 解析。""" - payload = _loads_json_or_jsonp(text) - if payload is not None: - rows = parse_items_from_jd_json_payload(payload, keyword=keyword, page=page) - if rows: - return rows - return parse_items_from_html(text, keyword=keyword, page=page) - - -def _jd_minimal_html_row( - *, - keyword: str, - page: int, - sku_id: str, - title: str, - price: str, - detail_url: str, - shop_name: str, - comment_count: str, - image: str, -) -> dict[str, str]: - out = _jd_empty_export_row() - out["item_id"] = sku_id - out["sku_id"] = sku_id - out["title"] = title - out["title_plain"] = title - out["price"] = price - out["detail_url"] = detail_url - out["shop_name"] = shop_name - out["comment_count"] = comment_count - out["image"] = image - out["platform"] = "京东" - out["keyword"] = keyword - out["page"] = str(page) - return out - - -def _collect_items_from_json_like( - html_text: str, keyword: str, page: int -) -> list[dict[str, str]]: - """ - 策略 A：从内嵌 JSON/脚本片段中用正则抓取 wareId/wareName/price 等。 - 该策略对结构变更相对鲁棒，但字段名可能变化，因此做多套模板。 - """ - items: list[dict[str, str]] = [] - - # 常见字段组合：wareId + wareName + jdPrice / price / priceText - patterns: list[re.Pattern[str]] = [ - re.compile( - r'"wareId"\s*:\s*"?(?P<sku>\d{5,20})"?' - r'[\s\S]{0,1200}?' - r'"wareName"\s*:\s*"(?P<title>[^"]{2,300})"' - r'[\s\S]{0,1200}?' - r'"(?:jdPrice|price|priceText|mainPrice)"\s*:\s*"?(?P<price>[\d.]{1,12})"?', - re.I, - ), - re.compile( - r'"skuId"\s*:\s*"?(?P<sku>\d{5,20})"?' - r'[\s\S]{0,1200}?' - r'"title"\s*:\s*"(?P<title>[^"]{2,300})"' - r'[\s\S]{0,1200}?' - r'"(?:price|priceText|jdPrice)"\s*:\s*"?(?P<price>[\d.]{1,12})"?', - re.I, - ), - # 兜底：只要 sku + title，价格缺失也接受 - re.compile( - r'"(?:wareId|skuId)"\s*:\s*"?(?P<sku>\d{5,20})"?' - r'[\s\S]{0,1200}?' - r'"(?:wareName|title|name)"\s*:\s*"(?P<title>[^"]{2,300})"', - re.I, - ), - ] - - seen: set[str] = set() - for pat in patterns: - for m in pat.finditer(html_text): - sku = (m.groupdict().get("sku") or "").strip() - title = _human_text(m.groupdict().get("title") or "", 300) - price = (m.groupdict().get("price") or "").strip() - if not sku or not title: - continue - if sku in seen: - continue - seen.add(sku) - detail = f"https://item.m.jd.com/product/{sku}.html" - items.append( - _jd_minimal_html_row( - keyword=keyword, - page=page, - sku_id=sku, - title=title, - price=price, - detail_url=detail, - shop_name="", - comment_count="", - image="", - ) - ) - if items: - # 命中一套 pattern 后就不再叠加下一套，避免重复/误配 - break - return items - - -def _collect_items_from_dom(html_text: str, keyword: str, page: int) -> list[dict[str, str]]: - """ - 策略 B：从 DOM/属性中抓取 data-sku + title/price/href。 - 不依赖 BeautifulSoup（零依赖），但对结构变化更敏感。 - """ - items: list[dict[str, str]] = [] - seen: set[str] = set() - - # 以 data-sku 为锚点，截取一个窗口做二次抽取 - for m in re.finditer(r'data-sku\s*=\s*"(?P<sku>\d{5,20})"', html_text, re.I): - sku = (m.group("sku") or "").strip() - if not sku or sku in seen: - continue - seen.add(sku) - - win = html_text[m.start() : m.start() + 4500] - - # 链接 - href = "" - mh = re.search(r'href\s*=\s*"([^"]+)"', win, re.I) - if mh: - href = _safe_url(mh.group(1)) - if not href: - href = f"https://item.m.jd.com/product/{sku}.html" - - # 标题 - title = "" - for tp in ( - r'title\s*=\s*"([^"]{2,300})"', - r'alt\s*=\s*"([^"]{2,300})"', - r'data-name\s*=\s*"([^"]{2,300})"', - ): - mt = re.search(tp, win, re.I) - if mt: - title = _human_text(mt.group(1), 300) - break - - # 价格（HTML 上可能是 ¥xx.xx / ¥xx.xx） - price = "" - mp = re.search(r"(?:¥|¥)\s*([\d.]{1,12})", win) - if mp: - price = mp.group(1).strip() - - # 店铺（H5 列表可能没有） - shop = "" - ms = re.search(r'data-shopname\s*=\s*"([^"]{2,80})"', win, re.I) - if ms: - shop = _human_text(ms.group(1), 80) - - # 图片 - image = "" - mi = re.search(r'(?:data-lazy-img|data-img|src)\s*=\s*"([^"]+\.(?:jpg|jpeg|png|webp)[^"]*)"', win, re.I) - if mi: - image = _jd_product_image_url(mi.group(1))[:1200] - - if not title: - # 标题拿不到时宁可跳过，避免充斥“空标题” - continue - - items.append( - _jd_minimal_html_row( - keyword=keyword, - page=page, - sku_id=sku, - title=title, - price=price, - detail_url=href[:2000], - shop_name=shop, - comment_count="", - image=image, - ) - ) - - return items - - -def parse_items_from_html(html_text: str, *, keyword: str, page: int) -> list[dict[str, str]]: - # 优先尝试 JSON-like（更稳），再退回 DOM - parsed = _collect_items_from_json_like(html_text, keyword, page) - if not parsed: - parsed = _collect_items_from_dom(html_text, keyword, page) - - seen: set[str] = set() - rows: list[dict[str, str]] = [] - for row in parsed: - sku = (row.get("sku_id") or "").strip() - if not sku or sku in seen: - continue - seen.add(sku) - rows.append(row) - return rows - def _normalize_url_keep_query(url: str, **override: str) -> str: """ diff --git a/backend/manage.py b/backend/manage.py index 8e7ac79..ffea21e 100644 --- a/backend/manage.py +++ b/backend/manage.py @@ -2,11 +2,34 @@ """Django's command-line utility for administrative tasks.""" import os import sys +from pathlib import Path + + +def _require_pipeline_migration_files() -> None: + """0013 依赖 0012；缺文件时 MigrationLoader 报 NodeNotFoundError，此处给出可操作的提示。""" + base = Path(__file__).resolve().parent + required = ( + base / "pipeline" / "migrations" / "0012_job_pause_checkpoint.py", + base / "pipeline" / "migrations" / "0013_rebuild_pipelinejobcheckpoint.py", + ) + missing = [p for p in required if not p.is_file()] + if not missing: + return + print( + "Missing pipeline migration file(s) (clone/pull 不完整或误删):\n" + + "\n".join(f" - {p}" for p in missing) + + "\n\nRestore from Git, e.g.\n" + " git checkout HEAD -- pipeline/migrations/0012_job_pause_checkpoint.py " + "pipeline/migrations/0013_rebuild_pipelinejobcheckpoint.py\n", + file=sys.stderr, + ) + sys.exit(1) def main(): """Run administrative tasks.""" os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'config.settings') + _require_pipeline_migration_files() try: from django.core.management import execute_from_command_line except ImportError as exc: diff --git a/backend/pipeline/__init__.py b/backend/pipeline/__init__.py new file mode 100644 index 0000000..b472b78 --- /dev/null +++ b/backend/pipeline/__init__.py @@ -0,0 +1,2 @@ +# -*- coding: utf-8 -*- +"""Django 应用 ``pipeline``：任务、数据集、竞品报告与 CSV 规范（含 ``pipeline.csv`` 子包）。""" diff --git a/backend/pipeline/competitor_report/__init__.py b/backend/pipeline/competitor_report/__init__.py new file mode 100644 index 0000000..4bcfaa2 --- /dev/null +++ b/backend/pipeline/competitor_report/__init__.py @@ -0,0 +1 @@ +"""京东竞品分析报告：CSV 解析、统计与 Markdown/简报（归属 ``pipeline``，与爬虫采集分离）。""" diff --git a/backend/pipeline/competitor_report/comment_sentiment.py b/backend/pipeline/competitor_report/comment_sentiment.py new file mode 100644 index 0000000..9148ad0 --- /dev/null +++ b/backend/pipeline/competitor_report/comment_sentiment.py @@ -0,0 +1,513 @@ +"""评价关键词命中、星级与口语词表、情感 lexicon、大模型情感 payload。""" +from __future__ import annotations + +import hashlib +import random +import re +from collections import Counter +from typing import Any + +from pipeline.csv.schema import MERGED_FIELD_TO_CSV_HEADER + +from .constants import ( + _COMMENT_CSV_BODY, + _COMMENT_CSV_SCORE, + _COMMENT_SCORE_NEG_MAX, + _COMMENT_SCORE_POS_MIN, +) +from .csv_io import _cell + + +def _comment_keyword_hits( + rows: list[dict[str, str]], + focus_words: tuple[str, ...], +) -> Counter[str]: + c: Counter[str] = Counter() + texts: list[str] = [] + for row in rows: + t = _cell(row, _COMMENT_CSV_BODY, "tagCommentContent") + if t: + texts.append(t) + blob = "\n".join(texts) + for w in focus_words: + if len(w) == 1: + continue + n = blob.count(w) + if n: + c[w] += n + return c + + +def _merge_comment_previews(merged_rows: list[dict[str, str]]) -> str: + parts: list[str] = [] + for row in merged_rows: + p = _cell( + row, + MERGED_FIELD_TO_CSV_HEADER["comment_preview"], + "comment_preview", + ) + if p: + parts.append(p) + return "\n".join(parts) + + +def _parse_comment_score(val: Any) -> int | None: + """解析 ``commentScore`` /「评分」列；期望京东 1～5 星，非法或空返回 None。""" + s = str(val or "").strip() + if not s: + return None + m = re.match(r"^\s*(\d+(?:\.\d+)?)", s) + if not m: + return None + try: + x = float(m.group(1)) + except ValueError: + return None + if x < 1 or x > 5: + return None + return int(round(x)) + + +def _iter_comment_text_units_and_scores( + comment_rows: list[dict[str, str]], + merged_rows: list[dict[str, str]], +) -> tuple[list[str], list[int | None]]: + """逐条评价正文与同序评分（无则为 None）；无 flat 评论时用合并表 comment_preview 按行兜底（评分均为 None）。""" + texts: list[str] = [] + scores: list[int | None] = [] + for row in comment_rows: + t = _cell(row, _COMMENT_CSV_BODY, "tagCommentContent") + if not t: + continue + texts.append(t) + scores.append(_parse_comment_score(_cell(row, _COMMENT_CSV_SCORE, "commentScore"))) + if texts: + return texts, scores + for row in merged_rows: + p = _cell( + row, + MERGED_FIELD_TO_CSV_HEADER["comment_preview"], + "comment_preview", + ) + if p: + texts.append(p) + scores.append(None) + return texts, scores + + +def _iter_comment_text_units( + comment_rows: list[dict[str, str]], + merged_rows: list[dict[str, str]], +) -> list[str]: + """逐条评价正文；无 flat 评论时用合并表 comment_preview 按行兜底。""" + texts, _ = _iter_comment_text_units_and_scores(comment_rows, merged_rows) + return texts + + +_POS_LEX = ( + "好", + "赞", + "满意", + "回购", + "推荐", + "不错", + "喜欢", + "香", + "实惠", + "值得", + "棒", + "鲜嫩", + "好吃", + "划算", + "正品", + "好评", +) +_NEG_LEX = ( + "差", + "烂", + "难吃", + "失望", + "假", + "骗", + "退货", + "不建议", + "糟糕", + "难用", + "臭", + "差评", + "不好", + "难喝", + "发霉", +) +# 条形图/摘要用：多字短语优先，避免只显示「硬、差」等单字 +_POS_LEXEME_DETAIL = ( + "已经回购很多次", + "还会再买", + "值得回购", + "推荐购买", + "性价比很高", + "性价比不错", + "物美价廉", + "物流很快", + "包装很用心", + "包装完好", + # 包装/物流等维度：若未命中下列句式，条形图仍可能偏低；关注词「包装」等反映提及频率 + "包装不错", + "包装很好", + "独立包装", + "小包装很方便", + "包装精美", + "密封性好", + "包装严实", + "快递很快", + "口感很好", + "味道不错", + "很好吃", + "香而不腻", + "饱腹感不错", + "控糖很友好", + "低糖很适合", + "代餐很方便", + "品质很稳定", + "值得信赖", + "软硬适中", +) +_NEG_LEXEME_DETAIL = ( + # 质地（保留；与分量类并列，避免统计只突出硬而不计少量） + "口感偏硬", + "口感很硬", + "咬不动", + "发硬", + "硬邦邦", + "口感发粘", + # 分量/规格（常见生活化抱怨，与条形图、§8.2 归纳对齐） + "分量少", + "量太少", + "太少了", + "不够吃", + "一袋很少", + "比想象少", + "克重不足", + "太甜了", + "甜得发腻", + "甜度过高", + "不太好吃", + "很难吃", + "味道很奇怪", + "有股怪味", + "一股异味", + "包装破损", + "漏气受潮", + "日期不新鲜", + "临期产品", + "质量很差", + "不值这个价", + "与描述不符", + "疑似假货", + "发货特别慢", + "物流太慢了", + "售后很差", + "退款很麻烦", + "不建议购买", + "不会再买", +) + + +def _lex_tuple_classify(*parts: tuple[str, ...]) -> tuple[str, ...]: + seen: set[str] = set() + out: list[str] = [] + for tup in parts: + for w in tup: + w = (w or "").strip() + if w and w not in seen: + seen.add(w) + out.append(w) + out.sort(key=len, reverse=True) + return tuple(out) + + +_POS_CLASS = _lex_tuple_classify(_POS_LEX, _POS_LEXEME_DETAIL) +_NEG_CLASS = _lex_tuple_classify(_NEG_LEX, _NEG_LEXEME_DETAIL) +_POS_LEX_HITS = tuple(sorted(_POS_LEXEME_DETAIL, key=len, reverse=True)) +_NEG_LEX_HITS = tuple(sorted(_NEG_LEXEME_DETAIL, key=len, reverse=True)) + + +def _lexeme_hits_in_texts( + texts: list[str], lexemes: tuple[str, ...] +) -> list[dict[str, Any]]: + """每条文本内同一短语只计 1 次；``lexemes`` 宜按长度降序以优先匹配更长表述。""" + c: Counter[str] = Counter() + for raw in texts: + s = (raw or "").strip() + if not s: + continue + seen_line: set[str] = set() + for k in lexemes: + if not k or k not in s: + continue + if k in seen_line: + continue + seen_line.add(k) + c[k] += 1 + return [{"word": w, "texts_matched": n} for w, n in c.most_common(18)] + + +def _keyword_sentiment_quadrant(stripped: str) -> str: + """``pos_only`` | ``neg_only`` | ``mixed`` | ``neutral``（空文本为 neutral）。""" + if not stripped: + return "neutral" + hp = any(k in stripped for k in _POS_CLASS) + hn = any(k in stripped for k in _NEG_CLASS) + if hp and hn: + return "mixed" + if hp: + return "pos_only" + if hn: + return "neg_only" + return "neutral" + + +def _sentiment_quadrant_for_row( + stripped: str, + score: int | None, + *, + use_score_column: bool, +) -> str: + if not stripped: + return "neutral" + if use_score_column and score is not None: + if score <= _COMMENT_SCORE_NEG_MAX: + return "neg_only" + if score >= _COMMENT_SCORE_POS_MIN: + return "pos_only" + return "neutral" + return _keyword_sentiment_quadrant(stripped) + + +def _include_in_positive_lexeme_corpus( + stripped: str, + score: int | None, + *, + use_score_column: bool, +) -> bool: + """口语短语正向统计语境：评分模式下为 4～5 星；否则为命中正向词（含原「混合」条）。""" + if not stripped: + return False + if use_score_column and score is not None: + return score >= _COMMENT_SCORE_POS_MIN + hp = any(k in stripped for k in _POS_CLASS) + return hp + + +def _include_in_negative_lexeme_corpus( + stripped: str, + score: int | None, + *, + use_score_column: bool, +) -> bool: + """口语短语负向统计语境：评分模式下为 1～2 星；否则为命中负向词（含原「混合」条）。""" + if not stripped: + return False + if use_score_column and score is not None: + return score <= _COMMENT_SCORE_NEG_MAX + hn = any(k in stripped for k in _NEG_CLASS) + return hn + + +def _comment_sentiment_lexicon( + texts: list[str], + scores: list[int | None] | None = None, +) -> dict[str, Any]: + """ + 正/负向粗判（非深度学习）： + + - 若 ``scores`` 与 ``texts`` 等长且**至少有一条非空评分**，则**先按 1～5 星分桶**，再在对应子集内统计 + 正向/负向口语短语（条形图）；无评分或非法评分的行仍按**关键词子串**粗判。 + - 否则：与旧版一致，**仅关键词**划分四象限与短语语境。 + """ + use_score_column = bool( + scores is not None + and len(scores) == len(texts) + and any(s is not None for s in scores) + ) + pos_only = neg_only = mixed = neutral = 0 + corpus_pos_mixed: list[str] = [] + corpus_neg_mixed: list[str] = [] + for i, t in enumerate(texts): + s = (t or "").strip() + sc: int | None = None + if use_score_column: + sc = scores[i] if scores is not None and i < len(scores) else None + if not s: + neutral += 1 + continue + quad = _sentiment_quadrant_for_row(s, sc, use_score_column=use_score_column) + if quad == "pos_only": + pos_only += 1 + elif quad == "neg_only": + neg_only += 1 + elif quad == "mixed": + mixed += 1 + else: + neutral += 1 + if _include_in_positive_lexeme_corpus(s, sc, use_score_column=use_score_column): + corpus_pos_mixed.append(s) + if _include_in_negative_lexeme_corpus(s, sc, use_score_column=use_score_column): + corpus_neg_mixed.append(s) + total = len(texts) + pos_lex = _lexeme_hits_in_texts(corpus_pos_mixed, _POS_LEX_HITS) + neg_lex = _lexeme_hits_in_texts(corpus_neg_mixed, _NEG_LEX_HITS) + method = "score_then_lexeme" if use_score_column else "keyword_lexicon" + base_note = ( + "「正向短语」与「负向短语」条形图统计的是预设口语片段在**对应语境**下的命中条数，" + "每条每短语最多计 1 次；非分词模型。" + ) + if use_score_column: + scope_extra = ( + "当前批次启用了**评分列**：四象限以星级为主（1～2 星偏负、4～5 星偏正、3 星为中评、空文本为中性）；" + "「正向口语短语」仅在 **4～5 星** 评价条内统计；「负向口语短语」仅在 **1～2 星** 评价条内统计；" + "无评分行仍按关键词子串归入四象限并参与短语语境。" + "条形图不是全文情感或某维度的完整满意度；未收录说法仍可能出现在关注词与语义池。" + ) + else: + scope_extra = ( + "「正向短语」仅在命中正向词表的评价条内统计（含关键词混合条）；" + "「负向短语」仅在命中负向词表的评价条内统计（含关键词混合条）。" + "条形图表示的是「预设短语命中条数」，不是全文情感或某维度（如包装、物流）的完整满意度；" + "若用户用「盒子不错」「没压坏」等未收录说法，仍可能落在关注词「包装」子串与语义池原文中。" + "预设表无法覆盖全部说法（如「一袋就一点点」），须结合语义池原文。" + ) + return { + "method": method, + "text_units": total, + "positive_only": pos_only, + "negative_only": neg_only, + "mixed_positive_and_negative": mixed, + "neutral_or_empty": neutral, + "positive_lexicon_sample": list(_POS_LEX[:10]) + list(_POS_LEXEME_DETAIL[:5]), + "negative_lexicon_sample": list(_NEG_LEX[:10]) + list(_NEG_LEXEME_DETAIL[:5]), + "positive_tone_lexeme_hits": pos_lex, + "negative_tone_lexeme_hits": neg_lex, + "lexeme_scope_note": base_note + scope_extra, + } + + +def build_comment_sentiment_llm_payload( + texts: list[str], + *, + scores: list[int | None] | None = None, + attributed_texts: list[str] | None = None, + max_samples_positive: int = 16, + max_samples_negative: int = 30, + max_samples_mixed: int = 10, + max_chars_per_review: int = 300, + semantic_pool_max: int = 40, + shuffle_seed: str = "", +) -> dict[str, Any]: + """ + 供大模型做正/负向语义归纳：附规则统计、按**评分优先或关键词**归类后的抽样，以及 **sample_reviews_semantic_pool** + （全量去重后的评价句确定性洗牌抽样，供模型结合语境自行判断褒贬）。 + + ``sentiment_bucket_method``：有有效评分列时为 ``score_then_lexeme``，否则为 ``keyword_substring_heuristic``； + ``comment_sentiment_lexicon`` 与各象限计数一致（竞品报告与 brief **已不再**发布同口径图）；正文归纳仍以整句语义为准。 + """ + use_score_column = bool( + scores is not None + and len(scores) == len(texts) + and any(s is not None for s in scores) + ) + pos_only_texts: list[str] = [] + neg_only_texts: list[str] = [] + mixed_texts: list[str] = [] + use_attr = ( + attributed_texts is not None + and len(attributed_texts) == len(texts) + ) + all_unique_disp: list[str] = [] + seen_unique: set[str] = set() + for i, t in enumerate(texts): + s = (t or "").strip() + if not s: + continue + disp = ( + (attributed_texts[i] or s).strip() + if use_attr + else s + ) + if disp and disp not in seen_unique: + seen_unique.add(disp) + all_unique_disp.append(disp) + sc = scores[i] if use_score_column and scores is not None else None + quad = _sentiment_quadrant_for_row(s, sc, use_score_column=use_score_column) + if quad == "mixed": + mixed_texts.append(disp) + elif quad == "pos_only": + pos_only_texts.append(disp) + elif quad == "neg_only": + neg_only_texts.append(disp) + + def _semantic_pool(seq: list[str], cap: int) -> list[str]: + """去重列表的洗牌子样本；shuffle_seed 非空时按种子固定顺序以便同任务可复现。""" + if not seq or cap <= 0: + return [] + work = list(seq) + if (shuffle_seed or "").strip(): + h = hashlib.sha256(shuffle_seed.encode("utf-8")).digest() + rnd = random.Random(int.from_bytes(h[:8], "big")) + rnd.shuffle(work) + out: list[str] = [] + for raw in work: + if len(raw) > max_chars_per_review: + out.append(raw[:max_chars_per_review] + "…") + else: + out.append(raw) + if len(out) >= cap: + break + return out + + semantic_pool = _semantic_pool(all_unique_disp, semantic_pool_max) + + def _sample(seq: list[str], cap: int) -> list[str]: + out: list[str] = [] + seen: set[str] = set() + for raw in seq: + if raw in seen: + continue + seen.add(raw) + if len(raw) > max_chars_per_review: + out.append(raw[:max_chars_per_review] + "…") + else: + out.append(raw) + if len(out) >= cap: + break + return out + + lex = _comment_sentiment_lexicon(texts, scores) + pos_h = lex.get("positive_tone_lexeme_hits") or [] + neg_h = lex.get("negative_tone_lexeme_hits") or [] + pos_h_top = [x for x in pos_h[:12] if isinstance(x, dict)] + neg_h_top = [x for x in neg_h[:12] if isinstance(x, dict)] + bucket_method = ( + "score_then_lexeme" if use_score_column else "keyword_substring_heuristic" + ) + return { + "comment_sentiment_lexicon": lex, + "positive_lexeme_hits_top": pos_h_top, + "negative_lexeme_hits_top": neg_h_top, + "sentiment_bucket_method": bucket_method, + "sample_reviews_semantic_pool": semantic_pool, + "sample_reviews_positive_biased": _sample(pos_only_texts, max_samples_positive), + "sample_reviews_negative_biased": _sample(neg_only_texts, max_samples_negative), + "sample_reviews_mixed_tone": _sample(mixed_texts, max_samples_mixed), + } + + +__all__ = [ + "build_comment_sentiment_llm_payload", + "_comment_keyword_hits", + "_comment_sentiment_lexicon", + "_iter_comment_text_units", + "_iter_comment_text_units_and_scores", + "_merge_comment_previews", + "_parse_comment_score", +] diff --git a/backend/pipeline/competitor_report/config.py b/backend/pipeline/competitor_report/config.py new file mode 100644 index 0000000..3053c0c --- /dev/null +++ b/backend/pipeline/competitor_report/config.py @@ -0,0 +1,109 @@ +"""``report_config`` JSON → 关注词、场景组、外部市场表行。""" +from __future__ import annotations + +from typing import Any + +from .constants import ( + COMMENT_FOCUS_WORDS, + COMMENT_SCENARIO_GROUPS, + EXTERNAL_MARKET_TABLE_ROWS, +) + + +def _normalize_focus_words(raw: Any) -> tuple[str, ...]: + if not isinstance(raw, list) or not raw: + return COMMENT_FOCUS_WORDS + out: list[str] = [] + for x in raw[:120]: + s = str(x).strip() + if len(s) > 48: + s = s[:48] + if s: + out.append(s) + return tuple(out) if out else COMMENT_FOCUS_WORDS + + +def _normalize_scenario_groups( + raw: Any, +) -> tuple[tuple[str, tuple[str, ...]], ...]: + if not isinstance(raw, list) or not raw: + return COMMENT_SCENARIO_GROUPS + parsed: list[tuple[str, tuple[str, ...]]] = [] + for item in raw[:40]: + label = "" + triggers: list[str] = [] + if isinstance(item, dict): + label = str(item.get("label") or "").strip()[:80] + tr = item.get("triggers") + if isinstance(tr, list): + for t in tr[:48]: + s = str(t).strip() + if len(s) > 48: + s = s[:48] + if s: + triggers.append(s) + elif isinstance(item, (list, tuple)) and len(item) >= 2: + label = str(item[0]).strip()[:80] + tr = item[1] + if isinstance(tr, (list, tuple)): + for t in tr[:48]: + s = str(t).strip() + if len(s) > 48: + s = s[:48] + if s: + triggers.append(s) + if label and triggers: + parsed.append((label, tuple(triggers))) + return tuple(parsed) if parsed else COMMENT_SCENARIO_GROUPS + + +def _normalize_external_market_rows( + raw: Any, +) -> tuple[tuple[str, str, str, str], ...]: + if not isinstance(raw, list) or not raw: + return EXTERNAL_MARKET_TABLE_ROWS + rows: list[tuple[str, str, str, str]] = [] + + def _four_cells(x: Any) -> tuple[str, str, str, str] | None: + if isinstance(x, (list, tuple)) and len(x) >= 4: + return tuple(str(c)[:500] for c in x[:4]) + if isinstance(x, dict): + a = str(x.get("indicator") or x.get("a") or "").strip()[:500] + b = str(x.get("value_and_scope") or x.get("b") or "").strip()[:500] + c = str(x.get("source") or x.get("c") or "").strip()[:500] + d = str(x.get("year") or x.get("d") or "").strip()[:500] + if any((a, b, c, d)): + return (a, b, c, d) + return None + + for item in raw[:24]: + r = _four_cells(item) + if r: + rows.append(r) + return tuple(rows) if rows else EXTERNAL_MARKET_TABLE_ROWS + + +def resolve_report_tuning( + report_config: dict[str, Any] | None, +) -> tuple[ + tuple[str, ...], + tuple[tuple[str, tuple[str, ...]], ...], + tuple[tuple[str, str, str, str], ...], +]: + if not report_config: + return COMMENT_FOCUS_WORDS, COMMENT_SCENARIO_GROUPS, EXTERNAL_MARKET_TABLE_ROWS + return ( + _normalize_focus_words(report_config.get("comment_focus_words")), + _normalize_scenario_groups(report_config.get("comment_scenario_groups")), + _normalize_external_market_rows( + report_config.get("external_market_table_rows") + ), + ) + + +__all__ = [ + "resolve_report_tuning", + "_normalize_external_market_rows", + "_normalize_focus_words", + "_normalize_scenario_groups", +] diff --git a/backend/pipeline/competitor_report/constants.py b/backend/pipeline/competitor_report/constants.py new file mode 100644 index 0000000..2954ae7 --- /dev/null +++ b/backend/pipeline/competitor_report/constants.py @@ -0,0 +1,141 @@ +"""CSV 表头键、运行默认调参与关注词/场景配置（与 ``pipeline.competitor_report.jd_report`` 顶层一致）。""" +from __future__ import annotations + +from pipeline.csv.schema import ( + COMMENT_CSV_COLUMNS, + JD_SEARCH_CSV_HEADERS, + MERGED_FIELD_TO_CSV_HEADER, +) + +_JD_LIST_PRICE_KEY = JD_SEARCH_CSV_HEADERS["price"] +_COUPON_SHOW_PRICE_KEY = JD_SEARCH_CSV_HEADERS["coupon_price"] +_ORIGINAL_LIST_PRICE_KEY = JD_SEARCH_CSV_HEADERS["original_price"] +_SELLING_POINT_KEY = JD_SEARCH_CSV_HEADERS["selling_point"] +_RANK_TAGLINE_KEY = JD_SEARCH_CSV_HEADERS["hot_list_rank"] + +# 历史批次 CSV 表头（括号英文）；新批次为纯中文，读取时新键优先 +_LEGACY_JD_LIST_PRICE_KEY = "标价(jdPrice,jdPriceText,realPrice)" +_LEGACY_COUPON_SHOW_PRICE_KEY = ( + "券后到手价(couponPrice,subsidyPrice,finalPrice.estimatedPrice,priceShow)" +) +_LEGACY_SHOP_NAME_KEY = "店铺名(shopName)" +_LEGACY_RANK_TAGLINE_KEY = "榜单类文案(标签/腰带/标题数组中的榜、TOP 等)" +_LEGACY_COMMENT_FUZZ_KEY = "评价量(commentFuzzy)" +_LEGACY_SELLING_POINT_KEY = "卖点(sellingPoint)" +_LIST_BRAND_TITLE_HEADER = "店铺信息标题" +_LEGACY_LIST_BRAND_TITLE_KEY = "店铺信息标题(shopInfoTitle,brandName)" + +_DETAIL_PRICE_FINAL_CSV_KEYS: tuple[str, ...] = ( + MERGED_FIELD_TO_CSV_HEADER["detail_price_final"], + "detail_price_final", +) +_LIST_PRICE_AND_COUPON_KEYS: tuple[str, ...] = ( + *_DETAIL_PRICE_FINAL_CSV_KEYS, + _JD_LIST_PRICE_KEY, + _LEGACY_JD_LIST_PRICE_KEY, + _COUPON_SHOW_PRICE_KEY, + _LEGACY_COUPON_SHOW_PRICE_KEY, +) + +# 报告摘录「标价」：列表标价优先，缺省时用商详到手价列兜底 +_LIST_SHOW_PRICE_CELL_KEYS: tuple[str, ...] = ( + _JD_LIST_PRICE_KEY, + _LEGACY_JD_LIST_PRICE_KEY, + MERGED_FIELD_TO_CSV_HEADER["detail_price_final"], + "detail_price_final", +) + +_MERGED_SHOP_CELL_KEYS: tuple[str, ...] = ( + MERGED_FIELD_TO_CSV_HEADER["detail_shop_name"], + "detail_shop_name", + JD_SEARCH_CSV_HEADERS["shop_name"], + _LEGACY_SHOP_NAME_KEY, +) + +_COMMENT_FUZZ_KEYS: tuple[str, ...] = ( + MERGED_FIELD_TO_CSV_HEADER["comment_fuzzy"], + _LEGACY_COMMENT_FUZZ_KEY, +) + +_COMMENT_CSV_SKU = COMMENT_CSV_COLUMNS[0] +_COMMENT_CSV_BODY = COMMENT_CSV_COLUMNS[3] +_COMMENT_CSV_SCORE = COMMENT_CSV_COLUMNS[7] # 「评分」→ commentScore + +# 评价星级分桶（comment_sentiment 模块）：仍用于可选的情感 LLM 载荷；报告正文与 brief 已不再输出同口径图 +_COMMENT_SCORE_NEG_MAX = 2 # 1～2 星 → 偏负向 +_COMMENT_SCORE_POS_MIN = 4 # 4～5 星 → 偏正向（3 星为中评，归入中性） + +_DETAIL_CATEGORY_PATH_KEY = MERGED_FIELD_TO_CSV_HEADER["detail_category_path"] +_K_CAT_COL = JD_SEARCH_CSV_HEADERS["leaf_category"] +_K_PROP_COL = JD_SEARCH_CSV_HEADERS["attributes"] + +EXTERNAL_MARKET_TABLE_ROWS: tuple[tuple[str, str, str, str], ...] = () + +COMMENT_FOCUS_WORDS: tuple[str, ...] = ( + "口感", + "甜", + "糖", + "血糖", + "控糖", + "低糖", + "无糖", + "饱腹", + "升糖", + "GI", + "gi", + "孕妇", + "老人", + "糖尿病", + "价格", + "贵", + "便宜", + "回购", + "包装", + "物流", + "分量", + "量少", + "克重", +) + +COMMENT_SCENARIO_GROUPS: tuple[tuple[str, tuple[str, ...]], ...] = ( + ("早餐/代餐", ("早餐", "代餐", "早饭", "当早餐", "当早饭", "早上吃", "晨起")), + ("零食/加餐/解馋", ("零食", "加餐", "嘴馋", "小零食", "解馋", "垫肚子", "饿了", "肚子饿", "两餐之间", "间食")), + ("控糖/血糖相关", ("控糖", "血糖高", "升糖", "糖友", "糖尿病", "孕期控糖", "妊娠糖", "血糖")), + ("孕期/育儿", ("孕期", "孕妇", "怀孕", "产妇", "坐月子", "哺乳", "给宝宝", "给娃", "孩子吃", "小孩吃", "宝宝吃")), + ("健身/减脂", ("减肥", "减脂", "瘦身", "健身", "卡路里", "热量低", "低脂")), + ("长辈/家庭", ("老人", "爸妈", "父母", "长辈", "爷爷奶奶", "给家里")), + ("办公/外出", ("办公室", "上班吃", "出门", "外出", "随身带", "包里", "便携")), + ("送礼/囤货", ("送礼", "送人", "囤货", "年货")), + ("夜宵/熬夜", ("夜宵", "熬夜", "晚上饿")), +) + +__all__ = [ + "_COMMENT_CSV_BODY", + "_COMMENT_CSV_SCORE", + "_COMMENT_CSV_SKU", + "COMMENT_FOCUS_WORDS", + "COMMENT_SCENARIO_GROUPS", + "EXTERNAL_MARKET_TABLE_ROWS", + "_COMMENT_FUZZ_KEYS", + "_COMMENT_SCORE_NEG_MAX", + "_COMMENT_SCORE_POS_MIN", + "_COUPON_SHOW_PRICE_KEY", + "_DETAIL_CATEGORY_PATH_KEY", + "_DETAIL_PRICE_FINAL_CSV_KEYS", + "_JD_LIST_PRICE_KEY", + "_K_CAT_COL", + "_K_PROP_COL", + "_LEGACY_COUPON_SHOW_PRICE_KEY", + "_LEGACY_JD_LIST_PRICE_KEY", + "_LEGACY_LIST_BRAND_TITLE_KEY", + "_LEGACY_RANK_TAGLINE_KEY", + "_LEGACY_SELLING_POINT_KEY", + "_LEGACY_SHOP_NAME_KEY", + "_LIST_BRAND_TITLE_HEADER", + "_LIST_PRICE_AND_COUPON_KEYS", + "_LIST_SHOW_PRICE_CELL_KEYS", + "_MERGED_SHOP_CELL_KEYS", + "_ORIGINAL_LIST_PRICE_KEY", + "_RANK_TAGLINE_KEY", + "_SELLING_POINT_KEY", +] diff --git a/backend/pipeline/competitor_report/consumer_feedback.py b/backend/pipeline/competitor_report/consumer_feedback.py new file mode 100644 index 0000000..4cc351f --- /dev/null +++ b/backend/pipeline/competitor_report/consumer_feedback.py @@ -0,0 +1,161 @@ +"""评价与合并表按细类矩阵对齐：带前缀行、SKU→细类映射、按细类消费者反馈分组。""" +from __future__ import annotations + +from pipeline.csv.schema import MERGED_FIELD_TO_CSV_HEADER + +from .comment_sentiment import _iter_comment_text_units +from .constants import ( + _COMMENT_CSV_BODY, + _COMMENT_CSV_SKU, + _MERGED_SHOP_CELL_KEYS, +) +from .csv_io import _cell, _md_cell +from .matrix_group import _competitor_matrix_group_key, _merged_rows_grouped_for_matrix + + +def _comment_lines_with_product_context( + comment_rows: list[dict[str, str]], + merged_rows: list[dict[str, str]], + *, + sku_header: str, + title_h: str, +) -> list[str]: + """与 ``comment_rows`` 顺序对齐：带细类/SKU/品名/店铺前缀，供可选的情感/评论 LLM 载荷抽样。""" + sku_meta: dict[str, tuple[str, str, str]] = {} + for row in merged_rows: + sku = _cell(row, sku_header).strip() + if not sku: + continue + gk = _competitor_matrix_group_key(row) + if not gk: + continue + sku_meta[sku] = ( + gk, + _cell(row, title_h), + _cell(row, *_MERGED_SHOP_CELL_KEYS), + ) + out: list[str] = [] + for cr in comment_rows: + txt = _cell(cr, _COMMENT_CSV_BODY, "tagCommentContent") + if not txt: + continue + sku = _cell(cr, _COMMENT_CSV_SKU, "sku").strip() + meta = sku_meta.get(sku) + if meta: + gname, tit, shop = meta + prefix = ( + f"【细类：{gname}｜SKU：{sku}｜品名：{_md_cell(tit, 80)}｜" + f"店铺：{_md_cell(shop, 40)}】" + ) + out.append(prefix + txt) + else: + out.append(txt) + return out + + +def _sku_to_matrix_group_map( + merged_rows: list[dict[str, str]], sku_header: str +) -> dict[str, str]: + m: dict[str, str] = {} + for row in merged_rows: + sku = _cell(row, sku_header).strip() + if not sku: + continue + gk = _competitor_matrix_group_key(row) + if gk: + m[sku] = gk + return m + + +def _comment_text_units_for_matrix_group( + gname: str, + merged_rows: list[dict[str, str]], + comment_rows_in_group: list[dict[str, str]], + sku_header: str, +) -> list[str]: + """某细类下的评价正文列表；无 flat 时用该细类合并行的 comment_preview。""" + texts: list[str] = [] + for row in comment_rows_in_group: + t = _cell(row, _COMMENT_CSV_BODY, "tagCommentContent") + if t: + texts.append(t) + if texts: + return texts + for row in merged_rows: + if _competitor_matrix_group_key(row) != gname: + continue + p = _cell( + row, + MERGED_FIELD_TO_CSV_HEADER["comment_preview"], + "comment_preview", + ) + if p: + texts.append(p) + return texts + + +def _consumer_feedback_by_matrix_group( + *, + merged_rows: list[dict[str, str]], + comment_rows: list[dict[str, str]], + sku_header: str, +) -> list[tuple[str, list[dict[str, str]], list[str]]]: + """ + 与 §5 矩阵同序的细类列表；每项为 (细类名, 该类的 comments_flat 行, 用于场景统计的文本单元)。 + 评价 SKU 不在深入样本时归入「未归类（评价 SKU 无对应深入样本）」。 + """ + if not merged_rows: + if not comment_rows: + return [] + texts = _iter_comment_text_units(comment_rows, []) + return [ + ( + "未归类（无深入合并表）", + list(comment_rows), + texts, + ) + ] + + sku_map = _sku_to_matrix_group_map(merged_rows, sku_header) + merged_by_sku: dict[str, dict[str, str]] = {} + for row in merged_rows: + s = _cell(row, sku_header).strip() + if s: + merged_by_sku[s] = row + by_g: dict[str, list[dict[str, str]]] = {} + for row in comment_rows: + sku = _cell(row, _COMMENT_CSV_SKU, "sku").strip() + g = sku_map.get(sku) + if g: + by_g.setdefault(g, []).append(row) + continue + if sku and sku in merged_by_sku: + # 深入样本存在但缺 detail_category_path（或路径无法解析为可读细类）：不参与按细类分析 + continue + by_g.setdefault("未归类（评价 SKU 无对应深入样本）", []).append(row) + + out: list[tuple[str, list[dict[str, str]], list[str]]] = [] + used: set[str] = set() + for gname, _ in _merged_rows_grouped_for_matrix(merged_rows): + cr = by_g.get(gname, []) + tu = _comment_text_units_for_matrix_group( + gname, merged_rows, cr, sku_header + ) + out.append((gname, cr, tu)) + used.add(gname) + for gname, cr in sorted(by_g.items(), key=lambda x: (-len(x[1]), x[0])): + if gname in used: + continue + tu = _comment_text_units_for_matrix_group( + gname, merged_rows, cr, sku_header + ) + out.append((gname, cr, tu)) + return out + + +__all__ = [ + "_comment_lines_with_product_context", + "_comment_text_units_for_matrix_group", + "_consumer_feedback_by_matrix_group", + "_sku_to_matrix_group_map", +] diff --git a/backend/pipeline/competitor_report/csv_io.py b/backend/pipeline/competitor_report/csv_io.py new file mode 100644 index 0000000..7f0c7ed --- /dev/null +++ b/backend/pipeline/competitor_report/csv_io.py @@ -0,0 +1,99 @@ +"""CSV 行读取与单元格、价格抽取等通用辅助。""" +from __future__ import annotations + +import csv +import re +from pathlib import Path + +from .constants import ( + _DETAIL_CATEGORY_PATH_KEY, + _K_CAT_COL, + _K_PROP_COL, + _LIST_PRICE_AND_COUPON_KEYS, +) + + +def _cell(row: dict[str, str], *keys: str) -> str: + for k in keys: + v = str(row.get(k) or "").strip() + if v: + return v + return "" + + +def _shortname_from_prop(prop: str) -> str: + m = re.search(r"简称[:：]\s*([^|]+)", prop or "") + return m.group(1).strip()[:120] if m else "" + + +def _detail_category_path_cell(row: dict[str, str]) -> str: + """细类矩阵与按细类评价统计仅以该列为准；空则视为商详类目不完整。""" + return _cell(row, _DETAIL_CATEGORY_PATH_KEY, "detail_category_path") + + +def _search_export_catid_to_shortname_map(rows: list[dict[str, str]]) -> dict[str, str]: + """列表导出中叶子类目列常为纯数字 ID：用同行规格属性「简称」映射为可读名称。""" + m: dict[str, str] = {} + for r in rows: + cid = _cell(r, _K_CAT_COL).strip() + if not cid.isdigit(): + continue + if cid in m: + continue + sn = _shortname_from_prop(_cell(r, _K_PROP_COL)) + if sn: + m[cid] = sn + return m + + +def _md_cell(s: str, max_len: int = 120) -> str: + t = (s or "").replace("\r\n", " ").replace("\n", " ").replace("|", "/") + t = " ".join(t.split()) + return (t[:max_len] + "…") if max_len > 0 and len(t) > max_len else t + + +def _read_csv_rows(path: Path) -> tuple[list[str], list[dict[str, str]]]: + if not path.is_file(): + return [], [] + raw = path.read_text(encoding="utf-8-sig") + lines = raw.splitlines() + if not lines: + return [], [] + rdr = csv.DictReader(lines) + fn = rdr.fieldnames or [] + return list(fn), list(rdr) + + +def _float_price(s: str) -> float | None: + if not (s or "").strip(): + return None + m = re.search(r"(\d+(?:\.\d+)?)", str(s).replace(",", "")) + if not m: + return None + try: + return float(m.group(1)) + except ValueError: + return None + + +def _collect_prices(rows: list[dict[str, str]]) -> list[float]: + out: list[float] = [] + for row in rows: + for k in _LIST_PRICE_AND_COUPON_KEYS: + p = _float_price(_cell(row, k)) + if p is not None and 0 < p < 1_000_000: + out.append(p) + break + return out + + +__all__ = [ + "_cell", + "_collect_prices", + "_detail_category_path_cell", + "_float_price", + "_md_cell", + "_read_csv_rows", + "_search_export_catid_to_shortname_map", + "_shortname_from_prop", +] diff --git a/backend/pipeline/competitor_report/ingredients.py b/backend/pipeline/competitor_report/ingredients.py new file mode 100644 index 0000000..9dbbfad --- /dev/null +++ b/backend/pipeline/competitor_report/ingredients.py @@ -0,0 +1,42 @@ +"""配料列清洗：与 ``AI_crawler.normalize_ingredients_text_for_csv`` 等口径对齐。""" +from __future__ import annotations + +import re + + +def _is_ingredient_url_blob(s: str) -> bool: + """详情主图 URL 串（分号分隔）或单列以 http 开头。""" + t = (s or "").strip() + if not t: + return False + if t.startswith(("http://", "https://")): + return True + head = t[:400] + if ("https://" in head or "http://" in head) and ( + ";" in t or len(t) > 180 or t.count("http") >= 2 + ): + return True + return False + + +def _ingredients_from_product_attributes(attrs: str) -> str: + m = re.search(r"配料(?:表)?[:：]\s*([^;；]+)", attrs or "") + return m.group(1).strip() if m else "" + + +def _ingredients_single_line(s: str) -> str: + """与 ``AI_crawler.normalize_ingredients_text_for_csv`` 一致：多行配料压成一行（行间 ``；``），便于表格/CSV。""" + t = (s or "").replace("\r\n", "\n").replace("\r", "\n").strip() + if not t: + return "" + lines = [ln.strip() for ln in t.split("\n") if ln.strip()] + if len(lines) <= 1: + return lines[0] if lines else "" + return "；".join(lines) + + +__all__ = [ + "_ingredients_from_product_attributes", + "_ingredients_single_line", + "_is_ingredient_url_blob", +] diff --git a/backend/pipeline/competitor_report/jd_report.py b/backend/pipeline/competitor_report/jd_report.py new file mode 100644 index 0000000..0dccf0a --- /dev/null +++ b/backend/pipeline/competitor_report/jd_report.py @@ -0,0 +1,1188 @@ +# -*- coding: utf-8 -*- +""" +关键词 → 调用 ``jd_keyword_pipeline`` 全链路采集 → 生成 **标准化竞品分析报告**（Markdown）。 + +报告结构对齐常见竞品分析框架：研究范围与方法、执行摘要、**整体市场观察（列表可见度参考）**、 +市场与竞争结构、**按细分类目分组的竞品对比矩阵**、价格分析（含规则化价差/活动信号与可选 **细类价盘·促销** 大模型归纳）、**按细分类目的消费者反馈与用户画像**、**策略与机会提示**（固定读者说明，指向产品中按细类「策略制定」；不再默认生成本章大模型长文）与附录；并明确数据边界。 +若运行配置中提供了外部市场规模摘录（``EXTERNAL_MARKET_TABLE_ROWS``），则追加对应表格小节；否则不输出占位行。 + +依赖：全量抓取时与 ``crawler_copy/jd_pc_search/jd_keyword_pipeline.py`` 相同（Node、h5st、Playwright、``common/jd_cookie.txt``）。 +**仅复用已有目录生成报告时**不需要跑浏览器，只需该目录下已有 CSV / ``run_meta.json``。 + +实现位于 ``pipeline.competitor_report.jd_report``；爬虫仅在 ``crawler_copy/jd_pc_search``。 + +用法： + +- **重新抓取并出报告**：``EXISTING_RUN_DIR = None``，配置 ``KEYWORD``（及可选 ``OVERRIDE_*``），在 ``backend`` 目录下执行 + ``python -m pipeline.competitor_report.jd_report``；或沿用爬虫目录下的兼容入口 ``python jd_competitor_report.py``（见该文件说明）。 +- **只分析已有批次**：将 ``EXISTING_RUN_DIR`` 设为 ``pipeline_runs/<时间戳>_<关键词>/`` 的绝对或相对路径（相对当前工作目录）， + 再执行同一命令；**不重新抓取**。关键词优先用本文件 ``KEYWORD``，否则读 ``run_meta.json`` 的 ``keyword``，再否则从目录名 + ``YYYYMMDD_HHMMSS_<词>`` 推断。 + +流水线其余参数（评论分页、延迟等）仍在 ``jd_keyword_pipeline.py`` 顶部配置。 + +输出：在对应运行目录下覆盖写入 ``competitor_analysis.md``。 +""" + +from __future__ import annotations + +import hashlib +import json +import math +import random +import re +import sys +from collections import Counter +from pathlib import Path +from typing import Any + +# 竞品报告依赖爬虫副本中的 ``jd_keyword_pipeline``（采集）；本文件归属 pipeline（解析与成稿）。 +_BACKEND = Path(__file__).resolve().parent.parent.parent +_CRAWLER_JD = _BACKEND / "crawler_copy" / "jd_pc_search" +if str(_CRAWLER_JD) not in sys.path: + sys.path.insert(0, str(_CRAWLER_JD)) + +import jd_keyword_pipeline as kpl # noqa: E402 +from pipeline.csv.schema import ( # noqa: E402 + COMMENT_CSV_COLUMNS, + JD_SEARCH_CSV_HEADERS, + MERGED_FIELD_TO_CSV_HEADER, + merged_csv_effective_total_sales, +) + +from pipeline.competitor_report.config import * # noqa: F403 +from pipeline.competitor_report.constants import * # noqa: F403 +from pipeline.competitor_report.csv_io import * # noqa: F403 +from pipeline.competitor_report.price_promo import ( # noqa: E402 + _analyze_price_promotions, + _markdown_price_promotion_section, +) +from pipeline.competitor_report.comment_sentiment import ( # noqa: E402 + _iter_comment_text_units_and_scores, + _parse_comment_score, + build_comment_sentiment_llm_payload, +) +from pipeline.competitor_report.llm_group_payloads import ( # noqa: E402 + build_comment_groups_llm_payload, + build_matrix_groups_llm_payload, + build_price_groups_llm_payload, + build_promo_groups_llm_payload, +) +from pipeline.competitor_report.matrix_group import ( # noqa: E402 + _category_mix, + _competitor_matrix_group_key, + _merged_rows_grouped_for_matrix, +) +from pipeline.competitor_report.price_stats import _price_stats_extended # noqa: E402 +from pipeline.competitor_report.consumer_feedback import ( # noqa: E402 + _comment_lines_with_product_context, + _consumer_feedback_by_matrix_group, + _sku_to_matrix_group_map, +) +from pipeline.competitor_report.list_mix import ( # noqa: E402 + _brand_cr, + _counter_mix_top_rows_with_remainder, + _search_list_proxies, + _shop_concentration_by_unique_sku, + _structure_brands, + _structure_names_for_pie_counter, + _structure_shops, +) +from pipeline.competitor_report.matrix_md import ( # noqa: E402 + _competitor_matrix_md_line, + _matrix_ingredients_cell, +) +from pipeline.competitor_report.report_md_helpers import ( # noqa: E402 + _embed_chart, + _lines_4_reading_brand, + _lines_4_reading_shop, + _matrix_prices_sales_chart_filename, + _scenario_group_asset_slug, + _strategy_hints, +) +from pipeline.competitor_report.run_context import ( # noqa: E402 + _infer_keyword, + _pc_search_result_count_from_raw, + _resolve_existing_run_dir, + _run_batch_label, +) + +# --------------------------------------------------------------------------- +# 运行配置（按需改这里；外部市场表等见 constants） +# --------------------------------------------------------------------------- +# KEYWORD：京东 PC 搜索词；全量抓取时必填。「仅已有目录」模式下可留空，改从 run_meta / 目录名推断。 +KEYWORD = "低GI" +# 已有流水线目录（含 keyword_pipeline_merged.csv 等）时设为路径则**不重新抓取**，只生成 competitor_analysis.md。 +EXISTING_RUN_DIR = None +# EXISTING_RUN_DIR = r"data\JD\pipeline_runs\20260408_144606_低GI" # 相对数据根或绝对路径 +# 以下非 None 时仅本次运行临时覆盖 jd_keyword_pipeline 中同名变量（不改 pipeline 文件） +OVERRIDE_MAX_SKUS: int | None = None +OVERRIDE_PAGE_START: int | None = None +OVERRIDE_PAGE_TO: int | None = None + +# 报告正文面向读者：不写配置键名、文件名、环境变量名或原始异常栈。 +_CH8_PROBE_ENABLE_HINT = ( + "请在任务配置中开启「第八章评论文本探针」并重新生成报告，以输出词频、共现与主题等开放词表分析。" +) +def _strategy_opportunities_reader_fixed_lines() -> list[str]: + """ + 第九章不再默认输出全任务大模型长文时的固定读者说明（与产品「策略制定」按细类生成分工）。 + """ + return [ + "本任务**监测与归纳**见前文各章（执行摘要、整体观察、集中度、竞品矩阵、价格与促销、消费者反馈等）。" + "**可执行的市场策略稿**请在产品中通过「**策略制定**」**按矩阵细类**逐类生成：与同任务数据及第五章划分对齐，避免仅凭全关键词检索池做一句泛化结论。", + "", + "本报告**默认不再**附加全任务大模型「策略与机会」长文；若个别历史批次仍含该正文，以当时任务配置与落稿为准。", + "", + ] + + +def build_competitor_markdown( + *, + run_dir: Path, + keyword: str, + merged_rows: list[dict[str, str]], + search_export_rows: list[dict[str, str]], + comment_rows: list[dict[str, str]], + meta: dict[str, Any] | None, + report_config: dict[str, Any] | None = None, + llm_sentiment_section_md: str | None = None, + llm_matrix_section_md: str | None = None, + llm_price_groups_section_md: str | None = None, + llm_promo_groups_section_md: str | None = None, + llm_scenario_groups_section_md: str | None = None, + llm_comment_groups_section_md: str | None = None, + llm_strategy_opportunities_section_md: str | None = None, + chapter8_text_mining_probe_section_md: str | None = None, +) -> str: + (external_rows,) = resolve_report_tuning(report_config) + _ch8_probe_sec = (chapter8_text_mining_probe_section_md or "").strip() + sku_header = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + title_h = MERGED_FIELD_TO_CSV_HEADER["title"] + batch = _run_batch_label(run_dir) + n_sku = len(merged_rows) + n_cmt = len(comment_rows) + n_sku_pathed = sum(1 for r in merged_rows if _detail_category_path_cell(r)) + n_sku_matrix = sum(1 for r in merged_rows if _competitor_matrix_group_key(r)) + + list_export = len(search_export_rows) > 0 + structure_rows = search_export_rows if list_export else merged_rows + n_structure = len(structure_rows) + shops_s = _structure_shops(structure_rows, list_export=list_export) + brands_s = _structure_brands(structure_rows, list_export=list_export) + shops_for_cr = _structure_names_for_pie_counter(shops_s) + brands_for_cr = _structure_names_for_pie_counter(brands_s) + cr1_shop, cr3_shop, top_shop_s, _ = _brand_cr(shops_for_cr) + shop_unique_sku_basis = ( + _shop_concentration_by_unique_sku(search_export_rows) + if list_export + else None + ) + cr1_list_brand, cr3_list_brand, top_list_brand, _ = _brand_cr(brands_for_cr) + # §4.3 类目分布：深入合并表（与 §5 竞品矩阵同一细类划分，非搜索列表行） + cm_structure = _category_mix(merged_rows, top_k=12) + min_brand_rows = max(5, int(0.02 * n_structure)) if n_structure else 5 + + brands_deep = [ + _cell(r, MERGED_FIELD_TO_CSV_HEADER["detail_brand"], "detail_brand") + for r in merged_rows + if _cell(r, MERGED_FIELD_TO_CSV_HEADER["detail_brand"], "detail_brand") + ] + cr1_deep, cr3_deep, top_brand_deep, _top_share_deep = _brand_cr(brands_deep) + cr1_hints = ( + cr1_shop if list_export and cr1_shop is not None else cr1_deep + ) + + pst_merged = _price_stats_extended(_collect_prices(merged_rows)) + pst_list = ( + _price_stats_extended(_collect_prices(search_export_rows)) + if list_export + else {} + ) + # 价格分析（§2 要点、§6、策略提示）：优先「列表全量」；无列表或无解析价时再用合并表深入样本 + pst = ( + pst_list + if list_export and pst_list.get("n", 0) > 0 + else pst_merged + ) + price_analysis_basis_cn = ( + f"PC 搜索列表导出共 **{len(search_export_rows)}** 行中的展示价（标价/券后等）" + if list_export and pst_list.get("n", 0) > 0 + else f"已深入抓取的 **{n_sku}** 个 SKU 合并数据中的展示价" + ) + promo_rows = ( + search_export_rows + if list_export and pst_list.get("n", 0) > 0 + else merged_rows + ) + promo_sig = _analyze_price_promotions(promo_rows) + + feedback_groups = _consumer_feedback_by_matrix_group( + merged_rows=merged_rows, + comment_rows=comment_rows, + sku_header=sku_header, + ) + matrix_groups_for_exec = _merged_rows_grouped_for_matrix(merged_rows) + multi_feedback_cat = len(matrix_groups_for_exec) >= 2 + + ( + api_rc, + api_list_kw, + api_rc_uniques, + api_raw_json_n, + api_rc_n_values, + ) = _pc_search_result_count_from_raw(run_dir) + + has_external_market = bool(external_rows) + + lines: list[str] = [ + f"# 竞品分析报告（京东 PC 渠道）", + "", + f"> **监测主题**：{keyword} ", + f"> **数据批次**：{batch} ", + f"> **报告生成**：自动化草稿，**仅供内部研讨**，不构成市场承诺或投资建议。", + "", + "---", + "", + "## 一、研究范围、数据来源与局限", + "", + "### 1.1 研究范围", + "", + f"- **搜索关键词**：「{keyword}」", + f"- **分析对象**：本次采集流程选取的 **{n_sku}** 个 SKU（搜索排序靠前子样本，非全站普查）。", + ] + if n_sku: + n_sku_nop = n_sku - n_sku_pathed + n_sku_unparsed = n_sku_pathed - n_sku_matrix + lines.append( + f"- **细类分析范围**：**{n_sku_matrix}** 个 SKU 具备参与**第五至第八章**分析所需的**商品详情页类目路径**" + f"（且能读出常见细类名称，如饼干、挂面等）；另有 **{n_sku_nop}** 个商品缺少该信息、" + f"**{n_sku_unparsed}** 个虽有路径但读不出细类名称，**未纳入**细类矩阵与按细类的评价统计。" + ) + if meta: + lines.append( + f"- **搜索列表页**：逻辑第 **{meta.get('page_start')}** 页至第 **{meta.get('page_to')}** 页；" + f"搜索导出共 **{meta.get('pc_search_export_rows', '—')}** 行（含未深入拉详情的商品）。" + ) + lines.extend( + [ + "", + "### 1.2 数据来源", + "", + "- **渠道**：京东 PC 端公开商品列表、商详与评价等可访问数据。", + "- **可追溯**：原始表格与接口响应保存在本批次任务输出目录，供内部复核；对外分享请脱敏。", + "", + "### 1.3 方法说明（指标含义）", + "", + "- **价格**：自页面「标价 / 券后价 / 详情价」等抽取的**展示价**，含促销与规格差异，**不等于**出厂价或成本。**第六章** 在具备可用的搜索列表导出时，优先以**列表全量**统计；否则使用**已深入 SKU** 的合并数据；**第六章第一节** 归纳标价与券后价差等**列表侧展示价差信号**（不对卖点/腰带字段做预设关键词扫描）。", + "- **品牌/店铺集中度（第四章）**：有列表全量时按列表行计店铺与品牌占比；无列表导出时按深入 SKU 合并表估算。", + "- **评价文本**：**不再**使用「预设关注词 / 预设场景词组」子串计数作为报告或简报主指标；**第八章第二节**以中文分词与开放词表统计（词频、关键词突出度、共现、主题探索等，可选词云及大模型解读）为主，结论须结合抽样原文理解。", + ( + "- **用户画像（第八章）**：**不再**使用星级子集内**预设口语短语**条形图、正负面扇形图及同口径摘要；**第八章第二节**为上述**文本补充分析**（探针）。" + if _ch8_probe_sec + else f"- **用户画像（第八章）**：若未启用探针，第二节仅说明方法并列出按细类评价条数；{_CH8_PROBE_ENABLE_HINT}" + ), + "- **细类划分（第五至第八章）**：**仅**依据合并表中的**商品详情页类目路径**；该信息缺失或无法读出细类名称的 SKU **不参与**竞品矩阵与按细类评价统计（相关评价条亦**不进入**按细类图表）。", + "- **检索结果规模**：来自京东 PC 搜索返回的「结果条数」类指标，表示平台侧申报的匹配数量级，**不等于**动销、库存或独立 SKU 数。", + "", + "### 1.4 主要局限", + "", + "- 仅覆盖 **京东 PC**，不含天猫、抖音、线下、B2B 原料端。", + "- 样本量由本次抓取上限与搜索页数决定，**结论外推需谨慎**。", + "- 详情配料与宣称以页面展示为准，**与真实配方可能不一致**（合规与实测另议）。", + ( + "- **行业零售额、TAM、CAGR 等**：无法从本批次数据推导；本报告已纳入任务中配置的第三方摘录，见 **第三章第五节**。" + if has_external_market + else "- **行业零售额、TAM、CAGR 等**：无法从本批次数据推导；本报告未纳入外部摘录（可在任务报告调参中维护市场信息表）。" + ), + "", + "---", + "", + "## 二、执行摘要（要点）", + "", + ] + ) + + exec_bullets: list[str] = [] + exec_bullets.append( + f"在关键词「{keyword}」下，本次深入分析 **{n_sku}** 个 SKU，关联评价文本 **{n_cmt}** 条。" + ) + if list_export and cr1_shop is not None and top_shop_s: + src = f"列表全量 **{n_structure}** 行" + if cr3_shop is not None: + exec_bullets.append( + f"竞争结构（{src}，第四章）：**店铺** 第一大店铺份额 ≈ **{100 * cr1_shop:.1f}%**（「{top_shop_s}」），" + f"前三店铺合计份额 ≈ **{100 * cr3_shop:.1f}%**（**按列表行计**，同一 SKU 多行会重复计；**勿与「去重 SKU 款数占比」或市占混淆**）。" + ) + else: + exec_bullets.append( + f"竞争结构（{src}，第四章）：**店铺** 第一大店铺份额 ≈ **{100 * cr1_shop:.1f}%**（「{top_shop_s}」；**按列表行计**）。" + ) + if shop_unique_sku_basis and shop_unique_sku_basis.get("n_unique_skus"): + u = shop_unique_sku_basis + fs = u.get("first_share") + lab = (u.get("top_label") or "").strip() + if fs is not None and lab: + exec_bullets.append( + f"**对照（按去重 SKU）**：同批列表共 **{u['n_unique_skus']}** 个去重 SKU，第一大店铺「{lab}」" + f"约占 **{100 * float(fs):.1f}%**（每 SKU 计 1 次；**不是**全渠道销量或市占）。" + ) + elif not list_export and cr1_deep is not None and top_brand_deep: + if cr3_deep is not None: + exec_bullets.append( + f"竞争结构（无列表导出，第四章用深入合并表）：**品牌** 第一大品牌份额 ≈ **{100 * cr1_deep:.1f}%**（「{top_brand_deep}」），" + f"前三品牌合计份额 ≈ **{100 * cr3_deep:.1f}%**。" + ) + else: + exec_bullets.append( + f"竞争结构（无列表导出，第四章用深入合并表）：**品牌** 第一大品牌份额 ≈ **{100 * cr1_deep:.1f}%**（「{top_brand_deep}」）。" + ) + if ( + list_export + and len(brands_for_cr) >= min_brand_rows + and cr1_list_brand is not None + and top_list_brand + ): + if cr3_list_brand is not None: + exec_bullets.append( + f"同批列表中**品牌信息有效** **{len(brands_for_cr)}** 条：**品牌** 第一大品牌份额 ≈ **{100 * cr1_list_brand:.1f}%**（「{top_list_brand}」），" + f"前三品牌合计份额 ≈ **{100 * cr3_list_brand:.1f}%**。" + ) + else: + exec_bullets.append( + f"同批列表中**品牌信息有效** **{len(brands_for_cr)}** 条：**品牌** 第一大品牌份额 ≈ **{100 * cr1_list_brand:.1f}%**（「{top_list_brand}」）。" + ) + elif list_export and cr1_deep is not None and top_brand_deep and not brands_for_cr: + exec_bullets.append( + f"列表导出缺少品牌标题字段，**深入 {n_sku} SKU** 商详品牌第一大品牌份额 ≈ **{100 * cr1_deep:.1f}%**（「{top_brand_deep}」），供与第五章矩阵对照。" + ) + if pst: + price_src_short = ( + "（列表全量）" + if list_export and pst_list.get("n", 0) > 0 + else "（深入样本）" + ) + exec_bullets.append( + f"展示价格{price_src_short}：可解析价格 **{pst['n']}** 个观测，区间约 **{pst['min']:.2f}～{pst['max']:.2f}** 元，" + f"中位数 **{pst.get('median', pst['mean']):.2f}** 元。" + ) + wb = int(promo_sig.get("rows_with_both_list_and_coupon") or 0) + sh = promo_sig.get("share_coupon_below_list_when_both") + med = promo_sig.get("median_discount_pct_when_coupon_below") + if wb >= 3 and isinstance(sh, (int, float)) and sh >= 0.08 and med is not None: + exec_bullets.append( + f"列表侧约 **{100.0 * float(sh):.0f}%** 可对齐行呈现「券后/到手」**低于**「标价」，展示价差中位数约 **{float(med):.1f}%**（**第六章第一节** 活动与话术摘录）。" + ) + if multi_feedback_cat and n_cmt > 0: + if _ch8_probe_sec: + exec_bullets.append( + "评价侧写已按**第五章同一细类划分**呈现，见**第八章第二节**（文本挖掘探针）。" + ) + else: + exec_bullets.append( + f"评价已按**第五章同一细类**归组；**第八章第二节**定量主题分析依赖探针，{_CH8_PROBE_ENABLE_HINT}" + ) + if api_rc is not None: + exec_bullets.append( + f"PC 搜索返回的检索结果规模约 **{api_rc:,}**（站内匹配条数量级，见第三章第二节；**不是**零售额或动销统计）。" + ) + for b in exec_bullets: + lines.append(f"- {b}") + if not exec_bullets: + lines.append("- 当前批次可汇总要点较少（以正文各节实际输出为准）。") + + proxy = _search_list_proxies(search_export_rows) if search_export_rows else {} + lines.extend(["", "---", "", "## 三、整体市场观察（渠道可见度参考，非官方市场规模）", ""]) + lines.extend( + [ + "### 3.1 与「市场规模」的区别", + "", + "- **官方/行业市场规模**（如全国零售额、品类增速、渗透率）通常来自 **Euromonitor、行业协会、上市公司年报、券商研报** 等；**不能**用京东搜索返回条数或 SKU 数直接等同。", + "- **第三章第二节** 使用搜索接口返回的**结果条数**；**第三章第三、四节** 描述本次导出的列表行、去重 SKU/店铺及列表价，仅作**参照**，外推全市场需谨慎。", + "", + "### 3.2 接口返回的检索规模", + "", + ] + ) + if api_rc is not None: + lines.append( + f"- 根据本批次保存的搜索原始响应解析：监测词「**{keyword}**」下，平台申报的检索匹配规模约 **{api_rc:,}**。" + ) + if api_list_kw: + lines.append( + f"- 同批响应中的列表关键词：**{api_list_kw}**（可与监测词对照是否一致）。" + ) + if len(api_rc_uniques) > 1: + nums = "、".join(f"{u:,}" for u in api_rc_uniques) + lines.append( + f"- 注：多份原始响应中该规模字段曾出现不同取值（{nums}），正文取**众数** **{api_rc:,}**（共 {api_rc_n_values} 次有效读取）。" + ) + elif api_raw_json_n > 0: + lines.append( + f"- 已扫描 **{api_raw_json_n}** 份原始响应并完成读取。" + ) + lines.extend( + [ + "- **含义**：平台对该关键词给出的**检索匹配条数量级**，用于感受站内商品池「宽度」；可能含不同类目/规格条目，**不等于**独立 SKU 数、动销或 GMV，且会随索引与运营策略变化。", + "", + ] + ) + else: + lines.append( + "*未能从本批次搜索原始响应中解析到有效的检索规模字段（目录缺失、无可用响应或字段为空）。*" + ) + lines.append("") + + lines.extend(["### 3.3 搜索列表规模（本次抓取范围内的可见 SKU / 店铺）", ""]) + if proxy.get("total_rows", 0) > 0: + pmin, pmax = proxy.get("page_span") or (None, None) + span_txt = ( + f"页码（去重）约 **{pmin}～{pmax}** 页" + if pmin is not None and pmax is not None + else "页码字段缺失或无法解析" + ) + lines.extend( + [ + f"- **列表导出行数**：**{proxy['total_rows']}** 行。", + f"- **去重 SKU 数**：**{proxy['unique_skus']}**；**去重店铺数**：**{proxy['unique_shops']}**；{span_txt}。", + f"- **列表中去重叶子类目代码/片段数**（粗略）：**{proxy['unique_leaf_cats']}**（同一关键词下品类宽度的参考）。", + "", + ] + ) + lpst = proxy.get("list_price_stats") or {} + lines.extend(["### 3.4 列表端展示价（全导出，非仅深入样本）", ""]) + if lpst: + lines.extend( + [ + f"- 自列表「标价 / 券后价」解析到 **{lpst['n']}** 个数值价；" + f"区间约 **{lpst['min']:.2f}～{lpst['max']:.2f}** 元，" + f"中位数 **{float(lpst.get('median', lpst['mean'])):.2f}** 元。", + "- **说明**：第六章价格统计表已与上表同源（均为列表全量，条件满足时）；若正文第六章标注为合并表样本，则因无可用列表价而退化。深入 SKU 的详情价可与列表价对照。", + "", + ] + ) + else: + lines.append("*列表导出中未能解析出数值价格。*") + lines.append("") + else: + lines.append( + "*未读到可用的搜索列表导出或文件为空；第三章第三、四节无列表侧数据。*" + ) + lines.append("") + lines.extend(["### 3.4 列表端展示价（全导出）", "", "*无列表数据。*", ""]) + + if external_rows: + lines.extend( + [ + "### 3.5 外部市场规模与行业信息（运行配置摘录）", + "", + "以下为本次任务报告调参中维护的**第三方市场摘录**，可与第三章第二节检索规模及第三章第三、四节列表参照对照使用；**指标含义与真实性以原出处为准**。", + "", + "| 指标 | 数值与说明 | 来源 | 年份 |", + "| --- | --- | --- | --- |", + ] + ) + for a, b, c, d in external_rows: + lines.append( + f"| {_md_cell(a, 40)} | {_md_cell(b, 48)} | {_md_cell(c, 36)} | {_md_cell(d, 12)} |" + ) + lines.append("") + + ch4_heading = ( + "## 四、市场与竞争结构（PC 搜索列表全量）" + if list_export + else "## 四、市场与竞争结构（深入合并表 · 无列表导出）" + ) + lines.extend(["", "---", "", ch4_heading, ""]) + if list_export: + lines.append( + f"基于**搜索列表导出**共 **{n_structure}** 行，与第三章第三节一致；" + f"集中度按**列表行**计数（同一 SKU 多次曝光则重复计）。" + ) + else: + lines.append( + f"*未读到可用列表全量行，以下退化为**深入 SKU 合并样本** **{n_structure}** 行。*" + ) + lines.append("") + + lines.extend(["### 4.1 品牌分布与集中度", ""]) + brand_rows_n = len(brands_for_cr) + show_list_brand_cr = list_export and brand_rows_n >= min_brand_rows + show_merged_brand_cr = not list_export and brand_rows_n > 0 + if (show_list_brand_cr or show_merged_brand_cr) and cr1_list_brand is not None: + lines.extend( + _embed_chart( + run_dir, + "chart_brand_rows_pie.png", + "品牌列表曝光占比（扇形图；按整理后的品牌名计数，与结构化摘要中的品牌占比统计一致；" + "长尾并入「（其余品牌）」；扇形内再合并为「其他」）", + ) + ) + lines.extend( + _lines_4_reading_brand( + cr1=cr1_list_brand, + cr3=cr3_list_brand, + top=top_list_brand or "", + brand_rows_n=brand_rows_n, + n_structure=n_structure, + ) + ) + lines.append( + "*更细的品牌行数分布见本任务「结构化摘要」数据包。*" + ) + elif list_export: + lines.append( + f"*列表导出中店铺/品牌标题有效 **{brand_rows_n}** 条，" + f"低于建议阈值（≥{min_brand_rows}），品牌集中度未展开。**店铺结构见第四章第二节**；" + f"商详品牌在**第五章**。*" + ) + else: + lines.append("*深入子样本无可用品牌字段。*") + lines.append("") + + lines.extend(["### 4.2 店铺分布与集中度", ""]) + shop_rows_n = len(shops_for_cr) + if shop_rows_n: + lines.extend( + _embed_chart( + run_dir, + "chart_shop_rows_pie.png", + "店铺列表曝光占比（扇形图；**按列表行计**的店铺名计数，与结构化摘要一致；若与「按去重 SKU」口径差异大，见下文规则解读；" + "长尾并入「（其余店铺）」；扇形内再合并为「其他」）", + ) + ) + lines.extend( + _lines_4_reading_shop( + cr1=cr1_shop, + cr3=cr3_shop, + top=top_shop_s or "", + shop_rows_n=shop_rows_n, + n_structure=n_structure, + unique_sku_basis=shop_unique_sku_basis, + ) + ) + lines.append( + "*更细的店铺行数分布见本任务「结构化摘要」数据包。*" + ) + else: + lines.append("*无店铺字段。*") + lines.append("") + + lines.extend(["### 4.3 细分类目分布（深入合并表 · 与第五章矩阵同一细类划分）", ""]) + if cm_structure and n_sku_matrix > 0: + lines.extend( + _embed_chart( + run_dir, + "chart_category_mix_pie.png", + "细类标签分布（扇形图；依据合并表中的商品详情页类目路径，与第五章一致；" + "Top 12 以外的细类在统计时并入「（其余细类）」；扇形图内再合并为「其他」）", + ) + ) + lines.append( + "*完整类目分布见界面「数据摘要」或简报包中的数据文件。*" + ) + else: + lines.append( + "*深入合并表中无具备可解析商品详情页类目路径的 SKU，本小节不展示扇形图；请核对商详抓取与合并字段。*" + ) + lines.append("") + + lines.extend( + [ + "---", + "", + "## 五、竞品对比矩阵（按细分类目分组）", + "", + "分组**仅**依据合并表中的**商品详情页类目路径**（京东商详中的类目层级）：**三级路径**取中间一段（如 … > **饼干** > 粗粮饼干），" + "**四级及以上**取倒数第二段（如 … > **面条** > 挂面）。**路径缺失**或各段均为内部编码、**读不出常见细类名称**的 SKU **不进入**本矩阵，亦**不参与**第八章按细类的评价统计。", + "", + "**读图方式**：每个细类下为**并列横向条形图**（左：**展示价**（元）；右：**销量**（来自搜索列表页「已售」等销量文案，如「已售50万+」计为 **50 万**）），" + "纵轴为**产品标题**（与本节各附图一致）。**SKU、店铺、配料与评价摘要等明细不列入正文**，详见本批次导出的合并数据表。", + "", + ] + ) + grouped_matrix = _merged_rows_grouped_for_matrix(merged_rows) + if not grouped_matrix: + if merged_rows: + lines.append( + "*深入合并表有条目，但均无可用商品详情页类目路径（或路径无法解析为可读细类），故无法生成细类矩阵；" + "第五至第八章中依赖矩阵的按细类统计相应为空。请核对商详抓取与合并字段。*" + ) + else: + lines.append("*无合并表 SKU。*") + lines.append("") + for gi, (gname, grows) in enumerate(grouped_matrix): + lines.append(f"### {gname}（**{len(grows)}** 款）") + lines.append("") + mx_chart = _matrix_prices_sales_chart_filename(gname, gi) + lines.extend( + _embed_chart( + run_dir, + mx_chart, + f"「{_md_cell(gname, 20)}」· 展示价与销量（页面「已售」销量文案）；纵轴为产品标题。", + ) + ) + if not (run_dir / "report_assets" / mx_chart).is_file(): + lines.append( + "*（本细类矩阵价量散点图尚未生成：请确认已执行报告出图流程，或重新生成报告。）*" + ) + lines.append("") + lines.append("") + + _llm_mx = (llm_matrix_section_md or "").strip() + if _llm_mx: + lines.extend( + [ + "", + "#### 细类要点归纳（大模型，与上文条形图互补）", + "", + "> **说明**：与第五章相同的细类划分下归纳卖点与配料共性；**具体 SKU、价格与条形图以正文为准**，SKU 级明细见合并表 CSV。", + "", + _llm_mx, + "", + ] + ) + + ch6_price_title = ( + "## 六、价格分析（PC 搜索列表全量）" + if list_export and pst_list.get("n", 0) > 0 + else "## 六、价格分析（深入 SKU 合并表 · 无可用列表价或未导出列表）" + ) + lines.extend(["---", "", ch6_price_title, ""]) + lines.append(f"- **统计基础**：{price_analysis_basis_cn}。") + if ( + list_export + and pst_list.get("n", 0) > 0 + and pst_merged.get("n", 0) > 0 + ): + lines.append( + f"- **对照**：合并表深入样本可解析价 **{pst_merged['n']}** 个观测，中位数约 **{float(pst_merged.get('median', pst_merged['mean'])):.2f}** 元（与上表样本范围不同，仅作对照）。" + ) + lines.append("") + if pst: + price_tbl = [ + "| 统计量 | 数值（元） | 说明 |", + "| --- | --- | --- |", + f"| 样本量 | {pst['n']} | 与统计基础一致 |", + f"| 最小值 | {pst['min']:.2f} | |", + ] + if "q1" in pst: + price_tbl.append(f"| 下四分位 Q1 | {float(pst['q1']):.2f} | |") + else: + price_tbl.append("| 下四分位 Q1 | — | 样本不足 4 个 |") + price_tbl.append( + f"| 中位数 | {float(pst.get('median', pst['mean'])):.2f} | |" + ) + if "q3" in pst: + price_tbl.append(f"| 上四分位 Q3 | {float(pst['q3']):.2f} | |") + else: + price_tbl.append("| 上四分位 Q3 | — | 样本不足 4 个 |") + price_tbl.extend( + [ + f"| 最大值 | {pst['max']:.2f} | |", + f"| 均值 | {pst['mean']:.2f} | |", + ] + ) + if "stdev" in pst: + price_tbl.append(f"| 标准差 | {pst['stdev']:.2f} | 离散程度 |") + lines.extend(price_tbl) + lines.append("") + lines.append( + "**解读提示**：价差大通常反映规格、组合装、品牌溢价或促销差异；B 端定价策略需结合成本与渠道单独建模。" + ) + lines.append("") + lines.extend(_markdown_price_promotion_section(promo_sig)) + else: + lines.append("*当前样本无可用数值价格，本节不展开统计表。*") + lines.append("") + lines.extend(_markdown_price_promotion_section(promo_sig)) + lines.append("") + + _llm_pr = (llm_price_groups_section_md or "").strip() + if _llm_pr: + lines.extend( + [ + "", + "#### 细类价盘要点归纳（大模型，与第六章量化表互补）", + "", + "> **说明**：侧重价带与标价/券后关系的可读叙述；**数值以正文分位数表为准**。", + "", + _llm_pr, + "", + ] + ) + + _llm_po = (llm_promo_groups_section_md or "").strip() + if _llm_po: + lines.extend( + [ + "", + "#### 细类促销与活动要点归纳（大模型，与第六章第一节及价盘互补）", + "", + "> **说明**：依据合并表「促销摘要」及榜单相关字段（如「榜单排名」「榜单类文案」）等**页面展示摘录**；" + "不采用列表「卖点/腰带」类字段作归纳依据（多为固定词表匹配，口径偏粗）。归纳券/补贴/新人/榜单曝光等活动形态，**不**替代第五章的配料/宣称归纳。**具体以页面与 CSV 为准**。", + "", + _llm_po, + "", + ] + ) + + _sec8_open: list[str] = [ + "---", + "", + "## 八、消费者反馈与用户画像（按细分类目）", + "", + "### 8.1 方法", + "", + "- **细类划分**：与**第五章「竞品矩阵」**相同，**仅**依据合并表中的**商品详情页类目路径**解析为「饼干 / 西式糕点 / …」等（规则见第五章开头说明）。", + "- **归因**：每条评价按其 SKU 对应到深入样本，再映射到该 SKU 所属细类；SKU 不在合并表中的评价单独归入说明性分组；**在合并表中但该 SKU 缺少类目路径或读不出细类名称的，该评价不进入按细类统计**（与第五章**同一条排除规则**）。", + ( + "- **评论文本补充分析（第八章第二节）**：本任务已用中文分词与统计工具做了开放词表分析（词频、关键词突出度、词对共现、主题归纳等，可选词云），**不再**在报告中使用「星级子集内预设口语短语」条形图、正负面扇形图及同口径摘要；**不再**使用「预设关注词 + 预设场景词组」子串计数与并列条图。" + if _ch8_probe_sec + else f"- **评论文本补充分析（第八章第二节）**：本任务**未**嵌入探针正文；**不再**输出预设关注词/场景子串统计图。{_CH8_PROBE_ENABLE_HINT}" + ), + "", + ] + lines.extend(_sec8_open) + if _ch8_probe_sec: + lines.extend( + [ + "### 8.2 评论文本补充分析（词频、关键词与共现、主题归纳）", + "", + "> **说明**：开放词表统计与分词主题探索；**勿**与已废弃的预设口语短语条形图混读；插图位于本批次报告附图文件夹中。", + "", + _ch8_probe_sec, + "", + ] + ) + else: + lines.extend( + [ + "### 8.2 评论文本补充分析（未启用探针时的占位）", + "", + "> **说明**：本版本**不再**生成「预设关注词 + 预设场景」子串统计及旧版并列条形图。" + f"开放词表、词频、共现与主题等分析需{_CH8_PROBE_ENABLE_HINT}", + "", + ] + ) + if not feedback_groups: + lines.append("*无评价数据可归组。*") + lines.append("") + else: + for _gi, (gname, cr_g, texts_g) in enumerate(feedback_groups): + n_flat = len(cr_g) + lines.append(f"#### {gname}") + lines.append("") + lines.append( + f"- **本细类逐条评价**：{n_flat} 条；**有效文本单元**：{len(texts_g)}。" + ) + lines.append("") + + _llm_cg = (llm_comment_groups_section_md or "").strip() + if _llm_cg: + lines.extend( + [ + "", + "#### 细类评论要点归纳（大模型，基于评价摘录）", + "", + "> **说明**：仅依据输入中的评价短摘录与文本单元归纳，**不**依赖预设关注词表或场景触发词统计。", + "", + _llm_cg, + "", + ] + ) + + _llm_sent = (llm_sentiment_section_md or "").strip() + if _llm_sent: + lines.extend( + [ + "", + "### 8.3 评价正/负向主题（按细类 · 大模型）", + "", + "> **说明**：与 **8.2 评论文本补充分析（探针）** 及 **「细类评论要点归纳（大模型）」** 并列；每段仅使用该细类下评价原文做语境归纳（如正向体验主题、负向评价主题归因）。" + "**不替代**探针的开放词表与专用口径，也不替代「细类评论要点归纳」的写法与载荷。", + "", + _llm_sent, + "", + ] + ) + + lines.extend(["---", "", "## 九、策略与机会提示（假设清单，待验证）", ""]) + _llm_st = (llm_strategy_opportunities_section_md or "").strip() + if _llm_st: + lines.extend( + [ + "基于本任务结构化摘要（价盘、集中度、评价与场景、促销信号等）的**假设性策略归纳**；数字与明细以前文及 CSV 为准，定稿前请结合贵司成本、渠道与合规复核。", + "", + "#### 策略与机会建议（大模型）", + "", + _llm_st, + "", + ] + ) + else: + lines.extend(_strategy_opportunities_reader_fixed_lines()) + + lines.extend( + [ + "---", + "", + "## 附录 A：数据留存说明", + "", + "- 本批次**任务输出目录**内保存：搜索列表导出、深入 SKU 合并表、商详与评价相关表格，以及搜索/商详原始响应与运行参数快照，供内部复核与复算。", + "- 对外演示或转发前请按公司规范做**脱敏**处理。", + "", + "---", + "", + "*本报告由系统自动汇总生成；定稿前请业务交叉核对数据与结论。*", + "", + ] + ) + return "\n".join(lines) + + +def _sanitize_json_numbers(obj: Any) -> Any: + """浮点 NaN/Inf 无法 JSON 序列化，统一转 None 或圆角。""" + if isinstance(obj, float): + if math.isnan(obj) or math.isinf(obj): + return None + return round(obj, 6) + if isinstance(obj, dict): + return {k: _sanitize_json_numbers(v) for k, v in obj.items()} + if isinstance(obj, list): + return [_sanitize_json_numbers(x) for x in obj] + return obj + + +def build_competitor_brief( + *, + run_dir: Path, + keyword: str, + merged_rows: list[dict[str, str]], + search_export_rows: list[dict[str, str]], + comment_rows: list[dict[str, str]], + meta: dict[str, Any] | None, + report_config: dict[str, Any] | None = None, +) -> dict[str, Any]: + """ + 与 ``build_competitor_markdown`` 共用**同一套统计规则**，输出可 JSON 序列化的结构化竞品摘要（**规则驱动**，无 LLM）。 + """ + (_ext,) = resolve_report_tuning(report_config) + sku_header = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + title_h = MERGED_FIELD_TO_CSV_HEADER["title"] + batch = _run_batch_label(run_dir) + n_sku = len(merged_rows) + n_cmt = len(comment_rows) + n_sku_matrix = sum(1 for r in merged_rows if _competitor_matrix_group_key(r)) + + list_export = len(search_export_rows) > 0 + structure_rows = search_export_rows if list_export else merged_rows + n_structure = len(structure_rows) + shops_s = _structure_shops(structure_rows, list_export=list_export) + brands_s = _structure_brands(structure_rows, list_export=list_export) + shops_for_cr = _structure_names_for_pie_counter(shops_s) + brands_for_cr = _structure_names_for_pie_counter(brands_s) + cr1_shop, cr3_shop, top_shop_s, top_shop_share = _brand_cr(shops_for_cr) + shop_unique_sku_basis = ( + _shop_concentration_by_unique_sku(search_export_rows) + if list_export + else None + ) + cr1_list_brand, cr3_list_brand, top_list_brand, _ = _brand_cr(brands_for_cr) + cm_structure = _category_mix(merged_rows, top_k=12) + min_brand_rows = max(5, int(0.02 * n_structure)) if n_structure else 5 + + brands_deep = [ + _cell(r, MERGED_FIELD_TO_CSV_HEADER["detail_brand"], "detail_brand") + for r in merged_rows + if _cell(r, MERGED_FIELD_TO_CSV_HEADER["detail_brand"], "detail_brand") + ] + cr1_deep, cr3_deep, top_brand_deep, top_brand_deep_share = _brand_cr( + brands_deep + ) + cr1_hints = cr1_shop if list_export and cr1_shop is not None else cr1_deep + + pst_merged = _price_stats_extended(_collect_prices(merged_rows)) + pst_list = ( + _price_stats_extended(_collect_prices(search_export_rows)) + if list_export + else {} + ) + pst = ( + pst_list + if list_export and pst_list.get("n", 0) > 0 + else pst_merged + ) + price_stats_source = ( + "pc_search_export_all_rows" + if list_export and pst_list.get("n", 0) > 0 + else "keyword_pipeline_merged" + ) + promo_rows_brief = ( + search_export_rows + if list_export and pst_list.get("n", 0) > 0 + else merged_rows + ) + price_promotion_signals = _analyze_price_promotions(promo_rows_brief) + + comment_texts, _comment_scores = _iter_comment_text_units_and_scores( + comment_rows, merged_rows + ) + + ( + api_rc, + api_list_kw, + api_rc_uniques, + api_raw_json_n, + _api_rc_n_values, + ) = _pc_search_result_count_from_raw(run_dir) + + proxy = _search_list_proxies(search_export_rows) if search_export_rows else {} + + hints = _strategy_hints( + cr1=cr1_hints, + pst=pst, + n_comments=n_cmt, + ) + + matrix_groups: list[dict[str, Any]] = [] + for gname, mrows in _merged_rows_grouped_for_matrix(merged_rows): + items: list[dict[str, str]] = [] + for row in mrows: + items.append( + { + "sku_id": _cell(row, sku_header), + "title": _cell(row, title_h), + "brand": _cell( + row, + MERGED_FIELD_TO_CSV_HEADER["detail_brand"], + "detail_brand", + ), + "list_price_show": _cell( + row, *_LIST_SHOW_PRICE_CELL_KEYS + ), + "coupon_or_detail_price": _cell( + row, + _COUPON_SHOW_PRICE_KEY, + _LEGACY_COUPON_SHOW_PRICE_KEY, + ), + "detail_price_final": _cell(row, *_DETAIL_PRICE_FINAL_CSV_KEYS), + "shop": _cell(row, *_MERGED_SHOP_CELL_KEYS), + "category": _detail_category_path_cell(row), + "selling_point": _cell( + row, _SELLING_POINT_KEY, _LEGACY_SELLING_POINT_KEY + )[:240], + "comment_fuzzy": _cell(row, *_COMMENT_FUZZ_KEYS), + "total_sales": merged_csv_effective_total_sales(row), + } + ) + matrix_groups.append( + {"group": gname, "sku_count": len(items), "skus": items} + ) + + feedback_by_group: list[dict[str, Any]] = [] + for gi, (gname, cr, tu) in enumerate( + _consumer_feedback_by_matrix_group( + merged_rows=merged_rows, + comment_rows=comment_rows, + sku_header=sku_header, + ) + ): + slug_fb = _scenario_group_asset_slug(gname, gi) + feedback_by_group.append( + { + "group": gname, + "matrix_group_index": gi, + "chart_slug": slug_fb, + "comment_rows": len(cr), + "effective_comment_text_units": len(tu), + } + ) + + meta_slice: dict[str, Any] = {} + if meta: + for k in ( + "page_start", + "page_to", + "max_skus_config", + "pc_search_export_rows", + "merged_rows", + "scenario_filter_enabled", + "merged_csv_mode", + ): + if k in meta: + meta_slice[k] = meta[k] + + list_brand_block: dict[str, Any] | None + if len(brands_for_cr) >= min_brand_rows: + list_brand_block = { + "first_share": cr1_list_brand, + "top_three_combined_share": cr3_list_brand, + "top_label": top_list_brand, + } + else: + list_brand_block = None + + out: dict[str, Any] = { + "schema_version": 1, + "keyword": keyword, + "batch_label": batch, + "run_dir": str(run_dir.resolve()), + "scope": { + "merged_sku_count": n_sku, + "comment_flat_rows": n_cmt, + "structure_source_rows": n_structure, + "uses_pc_search_list_export": list_export, + "category_mix_source": "keyword_pipeline_merged", + "category_mix_valid_matrix_sku_count": n_sku_matrix, + }, + "meta": meta_slice or None, + "pc_search_raw": { + "result_count_consensus": api_rc, + "list_keyword": api_list_kw or None, + "result_count_uniques": api_rc_uniques, + "raw_json_files_scanned": api_raw_json_n, + }, + "list_visibility_proxy": proxy, + "concentration": { + "shops_from_list": { + "first_share": cr1_shop, + "top_three_combined_share": cr3_shop, + "top_label": top_shop_s, + "top_share_pct": top_shop_share, + "unique_sku_basis": shop_unique_sku_basis, + }, + "list_brand_field": list_brand_block, + "detail_brand_among_merged": { + "first_share": cr1_deep, + "top_three_combined_share": cr3_deep, + "top_label": top_brand_deep, + "top_share_pct": top_brand_deep_share, + }, + }, + "category_mix_top": [ + {"label": lbl, "count": cnt} for lbl, cnt in cm_structure + ], + "list_brand_mix_top": [ + {"label": k, "count": v} + for k, v in _counter_mix_top_rows_with_remainder( + brands_s, + top_n=24, + remainder_label="（其余品牌）", + ) + ], + "list_shop_mix_top": [ + {"label": k, "count": v} + for k, v in _counter_mix_top_rows_with_remainder( + shops_s, + top_n=24, + remainder_label="（其余店铺）", + ) + ], + "price_stats": pst, + "price_stats_source": price_stats_source, + "price_stats_merged_sample": pst_merged, + "price_stats_list_export": pst_list if list_export else {}, + "price_promotion_signals": price_promotion_signals, + "comment_focus_keywords": [], + "usage_scenarios": [], + "usage_scenarios_denominator": len(comment_texts), + "usage_scenarios_by_matrix_group": [], + "strategy_hints": hints, + "matrix_by_group": matrix_groups, + "consumer_feedback_by_matrix_group": feedback_by_group, + "notes": [ + "与在线分析报告各章**计数规则**一致；**不再**输出预设关注词/场景子串统计，评论侧主题以第八章文本挖掘探针（若启用）为准。", + "价格来自页面展示字段抽取，含促销与规格差异；促销与标价对齐等为启发式摘录，仅供对照。", + "「集中度」中：默认按**列表行**计数（同一 SKU 多页曝光会重复计）；另有按**去重 SKU** 计数的对照口径。二者均**不是**销量、库存或全渠道市场份额。", + ], + } + return _sanitize_json_numbers(out) + + +def main() -> None: + try: + if hasattr(sys.stdout, "reconfigure"): + sys.stdout.reconfigure(encoding="utf-8", errors="replace") + if hasattr(sys.stderr, "reconfigure"): + sys.stderr.reconfigure(encoding="utf-8", errors="replace") + except Exception: + pass + + existing = _resolve_existing_run_dir(EXISTING_RUN_DIR) + meta_path_early = (existing / kpl.FILE_RUN_META_JSON) if existing else None + meta_early: dict[str, Any] | None = None + if meta_path_early and meta_path_early.is_file(): + try: + meta_early = json.loads(meta_path_early.read_text(encoding="utf-8")) + except json.JSONDecodeError: + meta_early = None + + if existing: + if not existing.is_dir(): + print(f"[竞品报告] EXISTING_RUN_DIR 不是目录: {existing}", file=sys.stderr) + sys.exit(2) + kw = (KEYWORD or "").strip() or _infer_keyword(existing, meta_early) + if not kw: + print( + "[竞品报告] 仅分析已有目录时，请配置 KEYWORD，或保留 run_meta.json 的 keyword，" + "或使目录名为 YYYYMMDD_HHMMSS_关键词", + file=sys.stderr, + ) + sys.exit(2) + run_dir = existing + print(f"[竞品报告] 使用已有目录（不抓取）: {run_dir}", file=sys.stderr) + else: + kw = (KEYWORD or "").strip() + if not kw: + print("[竞品报告] 全量抓取时请在本文件顶部配置 KEYWORD", file=sys.stderr) + sys.exit(2) + + backup: dict[str, Any] = {} + try: + if OVERRIDE_MAX_SKUS is not None: + backup["MAX_SKUS"] = kpl.MAX_SKUS + kpl.MAX_SKUS = max(1, int(OVERRIDE_MAX_SKUS)) + if OVERRIDE_PAGE_START is not None: + backup["PAGE_START"] = kpl.PAGE_START + kpl.PAGE_START = max(1, int(OVERRIDE_PAGE_START)) + if OVERRIDE_PAGE_TO is not None: + backup["PAGE_TO"] = kpl.PAGE_TO + kpl.PAGE_TO = max(1, int(OVERRIDE_PAGE_TO)) + + print(f"[竞品报告] 关键词={kw!r}，开始流水线…", file=sys.stderr) + run_dir = kpl.main(keyword=kw) + finally: + for name, val in backup.items(): + setattr(kpl, name, val) + + merged_path = run_dir / kpl.FILE_MERGED_CSV + comments_path = run_dir / kpl.FILE_COMMENTS_FLAT_CSV + meta_path = run_dir / kpl.FILE_RUN_META_JSON + + _, merged_rows = _read_csv_rows(merged_path) + _, search_export_rows = _read_csv_rows(run_dir / kpl.FILE_PC_SEARCH_CSV) + _, comment_rows = _read_csv_rows(comments_path) + meta: dict[str, Any] | None = meta_early if existing else None + if meta is None and meta_path.is_file(): + try: + meta = json.loads(meta_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + meta = None + + md = build_competitor_markdown( + run_dir=run_dir, + keyword=kw, + merged_rows=merged_rows, + search_export_rows=search_export_rows, + comment_rows=comment_rows, + meta=meta, + ) + out_md = run_dir / "competitor_analysis.md" + out_md.write_text(md, encoding="utf-8") + print(f"[竞品报告] 运行目录: {run_dir}", file=sys.stderr) + print(f"[竞品报告] 已写: {out_md}", file=sys.stderr) + + +if __name__ == "__main__": + main() diff --git a/backend/pipeline/competitor_report/list_mix.py b/backend/pipeline/competitor_report/list_mix.py new file mode 100644 index 0000000..d0f64d9 --- /dev/null +++ b/backend/pipeline/competitor_report/list_mix.py @@ -0,0 +1,175 @@ +"""列表可见度代理指标、品牌/店铺扇图用的名称列表与计数。""" +from __future__ import annotations + +from collections import Counter +from typing import Any + +from pipeline.csv.schema import JD_SEARCH_CSV_HEADERS, MERGED_FIELD_TO_CSV_HEADER + +from .constants import ( + _LEGACY_LIST_BRAND_TITLE_KEY, + _LEGACY_SHOP_NAME_KEY, + _LIST_BRAND_TITLE_HEADER, + _MERGED_SHOP_CELL_KEYS, +) +from .csv_io import _cell, _collect_prices +from .price_stats import _price_stats_extended + + +def _structure_names_for_pie_counter(row_names: list[str]) -> list[str]: + """ + 与 ``_counter_mix_top_rows_with_remainder`` / 列表品牌·店铺扇图同一套规则： + 按 strip 后的名称逐行保留一条，便于 ``_brand_cr`` 与饼图 Counter 一致。 + """ + return [(x or "").strip() for x in row_names if (x or "").strip()] + + +def _brand_cr(cnames: list[str]) -> tuple[float | None, float | None, str, str]: + """按名称计数返回 (第一大主体份额, 前三合计份额, 头部标签, 头部占比展示字符串)。""" + if not cnames: + return None, None, "", "" + cnt = Counter(cnames) + total = sum(cnt.values()) + if total <= 0: + return None, None, "", "" + mc = cnt.most_common() + top1_n = mc[0][1] if mc else 0 + top1 = mc[0][0] if mc else "" + cr1 = top1_n / total + top3_n = sum(n for _, n in mc[:3]) + cr3 = top3_n / total + return cr1, cr3, top1, f"{100.0 * top1_n / total:.1f}%" + + +def _counter_mix_top_rows_with_remainder( + row_names: list[str], *, top_n: int, remainder_label: str +) -> list[tuple[str, int]]: + """ + 与列表品牌/店铺扇图一致：按 strip 后的名称计数；``most_common(top_n)`` 未覆盖的长尾合并为 + ``remainder_label``，保证各块 count 之和等于可统计行数（与 ``_structure_names_for_pie_counter`` 总条数一致）。 + """ + c = Counter((x or "").strip() for x in row_names if (x or "").strip()) + if not c: + return [] + total = sum(c.values()) + common = c.most_common(top_n) + accounted = sum(v for _, v in common) + rest = total - accounted + out: list[tuple[str, int]] = list(common) + if rest > 0: + out.append((remainder_label, rest)) + return out + + +def _search_list_proxies(rows: list[dict[str, str]]) -> dict[str, Any]: + """ + 基于 pc_search_export 的「列表可见度」指标，**不是**全渠道零售额或 TAM。 + """ + sku_k = JD_SEARCH_CSV_HEADERS["sku_id"] + shop_k = JD_SEARCH_CSV_HEADERS["shop_name"] + page_k = JD_SEARCH_CSV_HEADERS["page"] + cat_k = JD_SEARCH_CSV_HEADERS["leaf_category"] + skus: set[str] = set() + shops: set[str] = set() + pages: set[str] = set() + cats: set[str] = set() + for r in rows: + s = _cell(r, sku_k) + if s: + skus.add(s) + sh = _cell(r, shop_k) + if sh: + shops.add(sh) + pg = _cell(r, page_k) + if pg: + pages.add(pg) + c = _cell(r, cat_k) + if c: + cats.add(c) + prices = _collect_prices(rows) + pst = _price_stats_extended(prices) + return { + "total_rows": len(rows), + "unique_skus": len(skus), + "unique_shops": len(shops), + "unique_pages": len(pages), + "page_span": (min((int(p) for p in pages if p.isdigit()), default=None), max((int(p) for p in pages if p.isdigit()), default=None)), + "unique_leaf_cats": len(cats), + "list_price_stats": pst, + } + + +def _shop_concentration_by_unique_sku( + rows: list[dict[str, str]], +) -> dict[str, Any] | None: + """ + 列表导出专用：按 **去重 SKU** 归属店铺后计集中度（每个 SKU 计 1 次；店铺名取该 SKU **首次出现** 的列表行）。 + + 与 ``_structure_names_for_pie_counter(_structure_shops(...))`` 的「按列表行计」可形成对照： + 同一 SKU 在多页重复曝光时，行占比会高于去重 SKU 占比。**不是**销量、库存或全渠道市场份额。 + """ + if not rows: + return None + sku_k = JD_SEARCH_CSV_HEADERS["sku_id"] + shop_k = JD_SEARCH_CSV_HEADERS["shop_name"] + sku_to_shop: dict[str, str] = {} + for r in rows: + sid = _cell(r, sku_k) + sh = _cell(r, shop_k, _LEGACY_SHOP_NAME_KEY) + if not sid or not sh: + continue + sid = sid.strip() + sh = sh.strip() + if not sid or not sh: + continue + if sid not in sku_to_shop: + sku_to_shop[sid] = sh + if not sku_to_shop: + return None + c1, c3, top, _ = _brand_cr(list(sku_to_shop.values())) + return { + "first_share": c1, + "top_three_combined_share": c3, + "top_label": top, + "n_unique_skus": len(sku_to_shop), + } + + +def _structure_shops(rows: list[dict[str, str]], *, list_export: bool) -> list[str]: + if list_export: + return [ + _cell(r, JD_SEARCH_CSV_HEADERS["shop_name"], _LEGACY_SHOP_NAME_KEY) + for r in rows + if _cell(r, JD_SEARCH_CSV_HEADERS["shop_name"], _LEGACY_SHOP_NAME_KEY) + ] + out: list[str] = [] + for r in rows: + s = _cell(r, *_MERGED_SHOP_CELL_KEYS) + if s: + out.append(s) + return out + + +def _structure_brands(rows: list[dict[str, str]], *, list_export: bool) -> list[str]: + if list_export: + return [ + _cell(r, _LIST_BRAND_TITLE_HEADER, _LEGACY_LIST_BRAND_TITLE_KEY) + for r in rows + if _cell(r, _LIST_BRAND_TITLE_HEADER, _LEGACY_LIST_BRAND_TITLE_KEY) + ] + return [ + _cell(r, MERGED_FIELD_TO_CSV_HEADER["detail_brand"], "detail_brand") + for r in rows + if _cell(r, MERGED_FIELD_TO_CSV_HEADER["detail_brand"], "detail_brand") + ] + + +__all__ = [ + "_brand_cr", + "_counter_mix_top_rows_with_remainder", + "_search_list_proxies", + "_shop_concentration_by_unique_sku", + "_structure_brands", + "_structure_names_for_pie_counter", + "_structure_shops", +] diff --git a/backend/pipeline/competitor_report/llm_group_payloads.py b/backend/pipeline/competitor_report/llm_group_payloads.py new file mode 100644 index 0000000..3cc7fa0 --- /dev/null +++ b/backend/pipeline/competitor_report/llm_group_payloads.py @@ -0,0 +1,396 @@ +"""按细类矩阵分组的 LLM 载荷（矩阵/价盘/促销/评价/场景）。""" +from __future__ import annotations + +from collections import Counter +from typing import Any + +from pipeline.csv.schema import JD_SEARCH_CSV_HEADERS, MERGED_FIELD_TO_CSV_HEADER + +from .comment_sentiment import _comment_keyword_hits +from .constants import ( + _COMMENT_CSV_BODY, + _COMMENT_CSV_SKU, + _COUPON_SHOW_PRICE_KEY, + _DETAIL_PRICE_FINAL_CSV_KEYS, + _LEGACY_COUPON_SHOW_PRICE_KEY, + _LEGACY_RANK_TAGLINE_KEY, + _LEGACY_SELLING_POINT_KEY, + _LIST_SHOW_PRICE_CELL_KEYS, + _MERGED_SHOP_CELL_KEYS, + _RANK_TAGLINE_KEY, + _SELLING_POINT_KEY, +) +from .csv_io import _cell, _collect_prices, _md_cell +from .ingredients import _ingredients_from_product_attributes, _ingredients_single_line +from .matrix_group import _competitor_matrix_group_key, _merged_rows_grouped_for_matrix +from .price_stats import _price_stats_extended + + +def _comment_scenario_counts( + texts: list[str], + scenario_groups: tuple[tuple[str, tuple[str, ...]], ...], +) -> tuple[Counter[str], int]: + """每组统计「至少命中一个触发词」的条数。返回 (各组条数, 有效文本条数)。""" + c: Counter[str] = Counter() + n = len(texts) + for blob in texts: + for label, triggers in scenario_groups: + if any(t in blob for t in triggers): + c[label] += 1 + return c, n + + +def _text_hits_scenario_triggers( + text: str, + scenario_groups: tuple[tuple[str, tuple[str, ...]], ...], +) -> bool: + blob = text or "" + for _lbl, triggers in scenario_groups: + if any(t in blob for t in triggers): + return True + return False + + +def _group_keyword_hits( + comment_rows_in_group: list[dict[str, str]], + texts_fallback: list[str], + *, + focus_words: tuple[str, ...], +) -> Counter[str]: + h = _comment_keyword_hits(comment_rows_in_group, focus_words) + if h: + return h + if not texts_fallback: + return Counter() + blob = "\n".join(texts_fallback) + c: Counter[str] = Counter() + for w in focus_words: + if len(w) < 2: + continue + n = blob.count(w) + if n: + c[w] += n + return c + + +def _matrix_excerpt_line_for_llm(row: dict[str, str], title_h: str) -> str: + title = _md_cell(_cell(row, title_h), 100) + sp = _md_cell(_cell(row, _SELLING_POINT_KEY, _LEGACY_SELLING_POINT_KEY), 120) + ing_raw = _ingredients_from_product_attributes( + _cell( + row, + MERGED_FIELD_TO_CSV_HEADER["detail_product_attributes"], + "detail_product_attributes", + ) + ) + ing = _md_cell(_ingredients_single_line(ing_raw), 100) if ing_raw else "" + chunks: list[str] = [] + if title: + chunks.append(title) + if sp: + chunks.append(f"卖点:{sp}") + if ing: + chunks.append(f"配料:{ing}") + return "｜".join(chunks) if chunks else "（无标题摘录）" + + +def _listing_price_snippet_for_llm(row: dict[str, str], title_h: str) -> str: + title = _md_cell(_cell(row, title_h), 72) + lp = _cell(row, *_LIST_SHOW_PRICE_CELL_KEYS) + cp = _cell(row, _COUPON_SHOW_PRICE_KEY, _LEGACY_COUPON_SHOW_PRICE_KEY) + dp = _cell(row, *_DETAIL_PRICE_FINAL_CSV_KEYS) + return f"{title}｜标价:{lp}｜券后:{cp}｜详情价:{dp}" + + +def build_matrix_groups_llm_payload( + merged_rows: list[dict[str, str]], + *, + title_h: str, + sku_header: str = "", +) -> list[dict[str, Any]]: + """供 ``generate_matrix_group_summaries_llm``：与 §5 细类划分一致。""" + _ = sku_header + if not merged_rows: + return [] + out: list[dict[str, Any]] = [] + for gname, grows in _merged_rows_grouped_for_matrix(merged_rows): + prices = _collect_prices(grows) + pst = _price_stats_extended(prices) if prices else {"n": 0} + lines = [_matrix_excerpt_line_for_llm(r, title_h) for r in grows[:24]] + out.append( + { + "group": gname, + "sku_count": len(grows), + "price_stats": pst, + "lines": lines, + } + ) + return out + + +def build_price_groups_llm_payload( + merged_rows: list[dict[str, str]], + *, + title_h: str, + sku_header: str = "", +) -> list[dict[str, Any]]: + """供 ``generate_price_group_summaries_llm``。""" + _ = sku_header + if not merged_rows: + return [] + out: list[dict[str, Any]] = [] + for gname, grows in _merged_rows_grouped_for_matrix(merged_rows): + prices = _collect_prices(grows) + pst = _price_stats_extended(prices) if prices else {"n": 0} + snippets = [_listing_price_snippet_for_llm(r, title_h) for r in grows[:16]] + out.append( + { + "group": gname, + "sku_count": len(grows), + "price_stats": pst, + "listing_snippets": snippets, + } + ) + return out + + +def _promo_snippet_for_llm(row: dict[str, str], title_h: str) -> str: + """单条 SKU：合并表「促销摘要」「榜单排名」「榜单类文案」摘录，供促销 LLM 用（不含列表卖点/腰带列，避免固定词表匹配的粗口径）。""" + title = _md_cell(_cell(row, title_h), 56) + promo = _cell( + row, + MERGED_FIELD_TO_CSV_HEADER["buyer_promo_text"], + "buyer_promo_text", + ) + br = _cell( + row, + MERGED_FIELD_TO_CSV_HEADER["buyer_ranking_line"], + "buyer_ranking_line", + ) + belt = _cell(row, _RANK_TAGLINE_KEY, _LEGACY_RANK_TAGLINE_KEY) + parts: list[str] = [title] + if promo.strip(): + parts.append( + f"{MERGED_FIELD_TO_CSV_HEADER['buyer_promo_text']}:{_md_cell(promo, 360)}" + ) + if br.strip(): + parts.append( + f"{MERGED_FIELD_TO_CSV_HEADER['buyer_ranking_line']}:{_md_cell(br, 120)}" + ) + if belt.strip(): + parts.append( + f"{JD_SEARCH_CSV_HEADERS['hot_list_rank']}:{_md_cell(belt, 80)}" + ) + return "｜".join(parts) if len(parts) > 1 else (parts[0] if parts else "") + + +def build_promo_groups_llm_payload( + merged_rows: list[dict[str, str]], + *, + title_h: str, + sku_header: str = "", +) -> list[dict[str, Any]]: + """供 ``generate_promo_group_summaries_llm``：与 §5/§6 细类划分一致。""" + _ = sku_header + if not merged_rows: + return [] + out: list[dict[str, Any]] = [] + for gname, grows in _merged_rows_grouped_for_matrix(merged_rows): + snippets = [_promo_snippet_for_llm(r, title_h) for r in grows[:16]] + nonempty = sum( + 1 + for r in grows + if _cell( + r, + MERGED_FIELD_TO_CSV_HEADER["buyer_promo_text"], + "buyer_promo_text", + ).strip() + ) + out.append( + { + "group": gname, + "sku_count": len(grows), + "rows_with_buyer_promo_text": nonempty, + "promo_snippets": snippets, + } + ) + return out + + +def build_comment_groups_llm_payload( + *, + feedback_groups: list[tuple[str, list[dict[str, str]], list[str]]], + merged_rows: list[dict[str, str]], + sku_header: str, + title_h: str, +) -> list[dict[str, Any]]: + """供 ``generate_comment_group_summaries_llm``：仅含评价文本单元与短摘录，不含关注词子串计数摘要。""" + if not feedback_groups: + return [] + sku_meta: dict[str, tuple[str, str, str]] = {} + for row in merged_rows: + sku = _cell(row, sku_header).strip() + if not sku: + continue + gk = _competitor_matrix_group_key(row) + if not gk: + continue + sku_meta[sku] = ( + gk, + _cell(row, title_h), + _cell(row, *_MERGED_SHOP_CELL_KEYS), + ) + out: list[dict[str, Any]] = [] + for gname, cr, tu in feedback_groups: + if not tu and not cr: + continue + snippets: list[str] = [] + for row in cr[:48]: + txt = _cell(row, _COMMENT_CSV_BODY, "tagCommentContent") + if not txt: + continue + sku = _cell(row, _COMMENT_CSV_SKU, "sku").strip() + meta = sku_meta.get(sku) + if meta: + sg, tit, shop = meta + prefix = ( + f"【细类：{sg}｜SKU：{sku}｜品名：{_md_cell(tit, 60)}｜" + f"店铺：{_md_cell(shop, 28)}】" + ) + snippets.append(prefix + txt[:300]) + else: + snippets.append( + f"【细类：{gname}｜SKU：{sku or '—'}】" + txt[:320] + ) + if len(snippets) >= 16: + break + eff = tu[:28] + if len(tu) > 28: + eff = list(eff) + [f"…共 {len(tu)} 条有效文本，此处截断"] + out.append( + { + "group": gname, + "comment_flat_rows": f"评价行 {len(cr)}；有效文本单元 {len(tu)}", + "effective_text_lines": eff, + "sample_text_snippets": snippets, + } + ) + return out + + +def build_scenario_groups_llm_payload( + *, + feedback_groups: list[tuple[str, list[dict[str, str]], list[str]]], + scenario_groups: tuple[tuple[str, tuple[str, ...]], ...], + merged_rows: list[dict[str, str]], + sku_header: str, + title_h: str, +) -> dict[str, Any]: + """供 ``generate_scenario_group_summaries_llm``；计数与 **§8.2** 关注词/场景路径下图右栏（场景）一致。""" + if not feedback_groups: + return {} + sku_meta: dict[str, tuple[str, str, str]] = {} + for row in merged_rows: + sku = _cell(row, sku_header).strip() + if not sku: + continue + gk = _competitor_matrix_group_key(row) + if not gk: + continue + sku_meta[sku] = ( + gk, + _cell(row, title_h), + _cell(row, *_MERGED_SHOP_CELL_KEYS), + ) + lexicon = [ + {"label": lbl, "trigger_examples": list(trigs[:12])} + for lbl, trigs in scenario_groups + ] + groups_out: list[dict[str, Any]] = [] + for gname, cr, tu in feedback_groups: + if not tu and not cr: + continue + scen_g, scen_ng = _comment_scenario_counts(tu, scenario_groups) + dist: list[dict[str, Any]] = [] + for lbl, n in scen_g.most_common(): + if n <= 0: + continue + dist.append( + { + "scenario": lbl, + "mention_rows": int(n), + "share_of_effective_texts": round( + float(n) / float(scen_ng), 4 + ) + if scen_ng > 0 + else 0.0, + } + ) + snippets: list[str] = [] + for row in cr: + txt = _cell(row, _COMMENT_CSV_BODY, "tagCommentContent") + if not txt: + continue + if not _text_hits_scenario_triggers(txt, scenario_groups): + continue + sku = _cell(row, _COMMENT_CSV_SKU, "sku").strip() + meta = sku_meta.get(sku) + if meta: + sg, tit, shop = meta + prefix = ( + f"【细类：{sg}\uff5cSKU：{sku}\uff5c品名：{_md_cell(tit, 60)}\uff5c" + f"店铺：{_md_cell(shop, 28)}】" + ) + snippets.append(prefix + txt[:300]) + else: + snippets.append( + f"【细类：{gname}\uff5cSKU：{sku or '—'}】" + txt[:320] + ) + if len(snippets) >= 16: + break + if len(snippets) < 5: + for row in cr: + txt = _cell(row, _COMMENT_CSV_BODY, "tagCommentContent") + if not txt: + continue + sku = _cell(row, _COMMENT_CSV_SKU, "sku").strip() + meta = sku_meta.get(sku) + if meta: + sg, tit, shop = meta + prefix = ( + f"【细类：{sg}\uff5cSKU：{sku}\uff5c品名：{_md_cell(tit, 60)}\uff5c" + f"店铺：{_md_cell(shop, 28)}】" + ) + snippets.append(prefix + txt[:260]) + else: + snippets.append( + f"【细类：{gname}\uff5cSKU：{sku or '—'}】" + txt[:280] + ) + if len(snippets) >= 10: + break + groups_out.append( + { + "group": gname, + "effective_text_count": int(scen_ng), + "scenario_distribution": dist[:18], + "sample_text_snippets": snippets, + } + ) + if not groups_out: + return {} + return {"scenario_lexicon": lexicon, "groups": groups_out} + + +__all__ = [ + "build_comment_groups_llm_payload", + "build_matrix_groups_llm_payload", + "build_price_groups_llm_payload", + "build_promo_groups_llm_payload", + "build_scenario_groups_llm_payload", + "_comment_scenario_counts", + "_group_keyword_hits", + "_listing_price_snippet_for_llm", + "_matrix_excerpt_line_for_llm", + "_promo_snippet_for_llm", + "_text_hits_scenario_triggers", +] diff --git a/backend/pipeline/competitor_report/matrix_group.py b/backend/pipeline/competitor_report/matrix_group.py new file mode 100644 index 0000000..bfd14a1 --- /dev/null +++ b/backend/pipeline/competitor_report/matrix_group.py @@ -0,0 +1,76 @@ +"""竞品矩阵细类键：与 ``pipeline.jd.matrix_group_label`` 及 §5 矩阵/扇图同源。""" +from __future__ import annotations + +from collections import Counter + +from pipeline.jd.matrix_group_label import ( + matrix_group_label_from_detail_path as _matrix_group_label_from_path, +) + +from .csv_io import _detail_category_path_cell + + +def _matrix_group_label_from_detail_path(row: dict[str, str]) -> str: + return _matrix_group_label_from_path(_detail_category_path_cell(row)) + + +def _competitor_matrix_group_key(row: dict[str, str]) -> str: + """ + 竞品矩阵分组：§5 / §8 / 统计图共用。 + **仅**依据 ``detail_category_path``；列为空或路径段均为无意义编码时不参与矩阵（返回空串）。 + """ + return _matrix_group_label_from_detail_path(row) + + +def _merged_rows_grouped_for_matrix( + merged_rows: list[dict[str, str]], +) -> list[tuple[str, list[dict[str, str]]]]: + buckets: dict[str, list[dict[str, str]]] = {} + for row in merged_rows: + k = _competitor_matrix_group_key(row) + if not k: + continue + buckets.setdefault(k, []).append(row) + + def sort_key(item: tuple[str, list[dict[str, str]]]) -> tuple[int, int, str]: + name, rows = item + miss = name.startswith("未归类") + return (1 if miss else 0, -len(rows), name) + + return sorted(buckets.items(), key=sort_key) + + +def _category_mix( + rows: list[dict[str, str]], *, top_k: int = 12 +) -> list[tuple[str, int]]: + """ + 按「可读细类标签」统计 SKU 分布（与 §5 ``_competitor_matrix_group_key`` 同源）； + 仅含 ``detail_category_path`` 可解析为展示名的行。 + + 返回 ``most_common(top_k)``，并将未列入 Top K 的款数合并为「（其余细类）」， + 使各块 SKU 数之和等于有效矩阵 SKU 总数（与扇形图、简报 ``category_mix_top`` 一致）。 + """ + labels: list[str] = [] + for r in rows: + k = _matrix_group_label_from_detail_path(r) + if k: + labels.append(k) + if not labels: + return [] + c = Counter(labels) + common = c.most_common(top_k) + accounted = sum(v for _, v in common) + total = sum(c.values()) + rest = total - accounted + out: list[tuple[str, int]] = list(common) + if rest > 0: + out.append(("（其余细类）", rest)) + return out + + +__all__ = [ + "_category_mix", + "_competitor_matrix_group_key", + "_matrix_group_label_from_detail_path", + "_merged_rows_grouped_for_matrix", +] diff --git a/backend/pipeline/competitor_report/matrix_md.py b/backend/pipeline/competitor_report/matrix_md.py new file mode 100644 index 0000000..7faa510 --- /dev/null +++ b/backend/pipeline/competitor_report/matrix_md.py @@ -0,0 +1,87 @@ +"""竞品矩阵 Markdown 行：配料格与整行管道表单元。""" +from __future__ import annotations + +from pipeline.csv.schema import MERGED_FIELD_TO_CSV_HEADER, merged_csv_effective_total_sales + +from .constants import ( + _COMMENT_FUZZ_KEYS, + _DETAIL_PRICE_FINAL_CSV_KEYS, + _LEGACY_RANK_TAGLINE_KEY, + _LEGACY_SELLING_POINT_KEY, + _LIST_SHOW_PRICE_CELL_KEYS, + _MERGED_SHOP_CELL_KEYS, + _RANK_TAGLINE_KEY, + _SELLING_POINT_KEY, +) +from .csv_io import _cell, _detail_category_path_cell, _md_cell +from .ingredients import ( + _ingredients_from_product_attributes, + _ingredients_single_line, + _is_ingredient_url_blob, +) + + +def _matrix_ingredients_cell(row: dict[str, str], *, max_len: int = 420) -> str: + """ + 优先 ``detail_body_ingredients``（配料 OCR/文本）；旧合并表可能为 ``detail_body_image_urls``。 + 若为 URL 串则尝试 ``detail_product_attributes`` 中的「配料/配料表：」片段。 + """ + raw = _cell( + row, + MERGED_FIELD_TO_CSV_HEADER["detail_body_ingredients"], + "detail_body_ingredients", + "detail_body_image_urls", + ) + if raw and not _is_ingredient_url_blob(raw): + return _md_cell(_ingredients_single_line(raw), max_len) + from_attr = _ingredients_from_product_attributes( + _cell( + row, + MERGED_FIELD_TO_CSV_HEADER["detail_product_attributes"], + "detail_product_attributes", + ) + ) + if from_attr: + return _md_cell(from_attr, max_len) + if raw and _is_ingredient_url_blob(raw): + return _md_cell( + "（详情长图链接，无配料正文；可在采集侧开启配料识别后重新跑批次）", + max_len, + ) + return "—" + + +def _competitor_matrix_md_line( + row: dict[str, str], *, sku_header: str, title_h: str +) -> str: + sku = _md_cell(_cell(row, sku_header), 14) + title = _md_cell(_cell(row, title_h), 56) + brand = _md_cell( + _cell(row, MERGED_FIELD_TO_CSV_HEADER["detail_brand"], "detail_brand"), 16 + ) + pj = _md_cell(_cell(row, *_LIST_SHOW_PRICE_CELL_KEYS), 10) + df = _md_cell(_cell(row, *_DETAIL_PRICE_FINAL_CSV_KEYS), 10) + shop = _md_cell(_cell(row, *_MERGED_SHOP_CELL_KEYS), 22) + sell = _md_cell(_cell(row, _SELLING_POINT_KEY, _LEGACY_SELLING_POINT_KEY), 36) + rank = _md_cell( + _cell(row, _RANK_TAGLINE_KEY, _LEGACY_RANK_TAGLINE_KEY), 28 + ) + cat = _md_cell(_detail_category_path_cell(row), 24) + ing = _matrix_ingredients_cell(row) + ts_eff = merged_csv_effective_total_sales(row) + cc = _md_cell(ts_eff or _cell(row, *_COMMENT_FUZZ_KEYS), 14) + prev = _md_cell( + _cell( + row, + MERGED_FIELD_TO_CSV_HEADER["comment_preview"], + "comment_preview", + ), + 72, + ) + return ( + f"| {sku} | {title} | {brand} | {pj} | {df} | {shop} | {sell} | {rank} | " + f"{cat} | {ing} | {cc} | {prev} |" + ) + + +__all__ = ["_competitor_matrix_md_line", "_matrix_ingredients_cell"] diff --git a/backend/pipeline/competitor_report/price_promo.py b/backend/pipeline/competitor_report/price_promo.py new file mode 100644 index 0000000..c24e453 --- /dev/null +++ b/backend/pipeline/competitor_report/price_promo.py @@ -0,0 +1,147 @@ +"""列表/合并行上的标价与券后价差统计，及第六章 6.1 Markdown 片段。""" +from __future__ import annotations + +import statistics +from typing import Any + +from .constants import ( + _COUPON_SHOW_PRICE_KEY, + _JD_LIST_PRICE_KEY, + _LEGACY_COUPON_SHOW_PRICE_KEY, + _LEGACY_JD_LIST_PRICE_KEY, + _LEGACY_RANK_TAGLINE_KEY, + _LEGACY_SELLING_POINT_KEY, + _ORIGINAL_LIST_PRICE_KEY, + _RANK_TAGLINE_KEY, + _SELLING_POINT_KEY, +) +from .csv_io import _cell, _float_price + + +def _analyze_price_promotions(rows: list[dict[str, str]]) -> dict[str, Any]: + """ + 从列表或合并行中归纳「标价 vs 券后/到手」等价差信号， + 供第六章第一节与结构化摘要使用（按**页面展示价**字段归纳，非结算实付）。 + """ + n = len(rows) + with_jd = with_cp = with_both = 0 + coupon_below = 0 + pct_offs: list[float] = [] + ori_above_list = 0 + for row in rows: + jd = _float_price(_cell(row, _JD_LIST_PRICE_KEY, _LEGACY_JD_LIST_PRICE_KEY)) + cp = _float_price( + _cell(row, _COUPON_SHOW_PRICE_KEY, _LEGACY_COUPON_SHOW_PRICE_KEY) + ) + ori = _float_price(_cell(row, _ORIGINAL_LIST_PRICE_KEY)) + if jd is not None and jd > 0: + with_jd += 1 + if cp is not None and cp > 0: + with_cp += 1 + if jd is not None and cp is not None and jd > 0 and cp > 0: + with_both += 1 + if cp + 1e-6 < jd: + coupon_below += 1 + pct_offs.append((jd - cp) / jd * 100.0) + if ( + ori is not None + and jd is not None + and ori > 0 + and jd > 0 + and ori > jd + 1e-6 + ): + ori_above_list += 1 + + selling_nonempty = sum( + 1 + for r in rows + if _cell(r, _SELLING_POINT_KEY, _LEGACY_SELLING_POINT_KEY).strip() + ) + rank_nonempty = sum( + 1 + for r in rows + if _cell(r, _RANK_TAGLINE_KEY, _LEGACY_RANK_TAGLINE_KEY).strip() + ) + + median_pct = statistics.median(pct_offs) if pct_offs else None + mean_pct = statistics.mean(pct_offs) if pct_offs else None + share_below = ( + (coupon_below / with_both) if with_both else None + ) + + return { + "row_count": n, + "rows_with_list_price": with_jd, + "rows_with_coupon_price": with_cp, + "rows_with_both_list_and_coupon": with_both, + "rows_coupon_below_list_price": coupon_below, + "share_coupon_below_list_when_both": share_below, + "median_discount_pct_when_coupon_below": median_pct, + "mean_discount_pct_when_coupon_below": mean_pct, + "rows_original_price_above_list_price": ori_above_list, + "rows_selling_point_nonempty": selling_nonempty, + "rows_rank_tagline_nonempty": rank_nonempty, + "promo_keyword_row_hits_top": [], + } + + +def _markdown_price_promotion_section(p: dict[str, Any]) -> list[str]: + """第六章第一节：优惠活动与价差信号（Markdown 行列表）。""" + lines: list[str] = [ + "### 6.1 优惠活动与价差信号（页面展示摘录）", + "", + "- **统计范围**：与上节价量统计**同一批行**；比较的是列表/合并表中的**展示标价**与**展示券后/到手价**（字段见表头），" + "反映页面呈现的活动与券信息，**不等于**用户结算实付或历史最低价。", + "", + ] + wb = int(p.get("rows_with_both_list_and_coupon") or 0) + if wb <= 0: + lines.append( + "- **标价与券后价可对齐比较**的有效行不足，本节以展示价与券后/到手字段的可得信息为主。" + ) + lines.append("") + else: + cb = int(p.get("rows_coupon_below_list_price") or 0) + sh = p.get("share_coupon_below_list_when_both") + med = p.get("median_discount_pct_when_coupon_below") + mean = p.get("mean_discount_pct_when_coupon_below") + lines.append( + f"- **同时解析到标价与券后/到手价** 的行：**{wb}**；其中展示「到手/券后」**严格低于**「标价」的行：**{cb}**" + + ( + f"（占可对齐行的 **{100.0 * float(sh):.1f}%**）" + if isinstance(sh, (int, float)) + else "" + ) + + "。" + ) + if med is not None: + frag_mean = ( + f"，平均价差约 **{float(mean):.1f}%**" if mean is not None else "" + ) + lines.append( + f"- **价差力度（仅「券后低于标价」子集）**：展示价差的中位数约 **{float(med):.1f}%**（相对标价）{frag_mean}；" + "通常对应满减、券、限时价等在列表上的叠加呈现。" + ) + elif cb > 0: + lines.append( + "- **价差**：存在「券后低于标价」样本，但条数较少，未给出稳健分位数；建议结合第五章矩阵中的单品对照。" + ) + lines.append("") + oa = int(p.get("rows_original_price_above_list_price") or 0) + if oa > 0: + lines.append( + f"- **划线原价高于当前标价** 的行约 **{oa}** 条（常见「划线价 + 当前价」促销陈列，具体以页面为准）。" + ) + lines.append("") + lines.append( + "- **说明**：本节**不对**列表「卖点/腰带」等字段做预设促销关键词行级统计；" + "活动形态归纳以第六章「细类促销与活动要点归纳」为准（若已生成）。" + ) + lines.append("") + return lines + + +__all__ = [ + "_analyze_price_promotions", + "_markdown_price_promotion_section", +] diff --git a/backend/pipeline/competitor_report/price_stats.py b/backend/pipeline/competitor_report/price_stats.py new file mode 100644 index 0000000..0b818b2 --- /dev/null +++ b/backend/pipeline/competitor_report/price_stats.py @@ -0,0 +1,32 @@ +"""合并表价格列的汇总统计（价盘/LLM 载荷等共用）。""" +from __future__ import annotations + +import statistics +from typing import Any + + +def _price_stats_extended(prices: list[float]) -> dict[str, Any]: + if not prices: + return {} + out: dict[str, Any] = { + "min": min(prices), + "max": max(prices), + "mean": statistics.mean(prices), + "n": len(prices), + } + if len(prices) >= 2: + out["stdev"] = statistics.stdev(prices) + if len(prices) >= 2: + out["median"] = statistics.median(prices) + if len(prices) >= 4: + s = sorted(prices) + n = len(s) + mid = n // 2 + lower = s[:mid] if n % 2 else s[:mid] + upper = s[mid + 1 :] if n % 2 else s[mid:] + out["q1"] = statistics.median(lower) if lower else s[0] + out["q3"] = statistics.median(upper) if upper else s[-1] + return out + + +__all__ = ["_price_stats_extended"] diff --git a/backend/pipeline/competitor_report/report_md_helpers.py b/backend/pipeline/competitor_report/report_md_helpers.py new file mode 100644 index 0000000..03b92d8 --- /dev/null +++ b/backend/pipeline/competitor_report/report_md_helpers.py @@ -0,0 +1,216 @@ +"""报告 Markdown 片段：Mermaid、场景摘要、规则策略提示、插图路径与解读段落。""" +from __future__ import annotations + +import re +from collections import Counter +from pathlib import Path +from typing import Any + +from .csv_io import _md_cell + + +def _mermaid_pie_focus_keywords(hits: Counter[str], *, top_k: int = 8) -> str: + """关注词全局 Top 的 Mermaid pie（便于渲染或导出工具识别）。""" + top = hits.most_common(top_k) + if not top: + return "" + rest = list(hits.most_common()) + if len(rest) > top_k: + others_n = sum(n for _, n in rest[top_k:]) + else: + others_n = 0 + lines = ["```mermaid", 'pie title 关注词命中次数（全局 Top，子串计数）'] + for w, n in top: + if n <= 0: + continue + label = (w or "?").replace('"', "'").replace("\n", " ")[:18] + lines.append(f' "{label}（{n}）" : {n}') + if others_n > 0: + lines.append(f' "其余词合计" : {others_n}') + lines.append("```") + return "\n".join(lines) + + +def _scenario_summary_bullets(counter: Counter[str], n_texts: int, top_k: int = 5) -> list[str]: + if n_texts <= 0 or not counter: + return [] + ordered = counter.most_common() + lines: list[str] = [] + head = ordered[:top_k] + parts = [] + for label, cnt in head: + pct = 100.0 * cnt / n_texts + parts.append(f"「{label}」约 **{cnt}** 条（占有效文本 **{pct:.0f}%**）") + lines.append( + "用户自述的用途/场景（基于预设词组，**非语义分类**）：" + "；".join(parts) + "。" + ) + tail = [lbl for lbl, n in ordered[top_k:] if n > 0] + if tail: + lines.append(f"另有提及较少的场景标签：{'、'.join(tail)}。") + return lines + + +def _strategy_hints( + *, + cr1: float | None, + pst: dict[str, Any], + hits: Counter[str], + n_comments: int, + scen_counts: Counter[str], + scen_n_texts: int, +) -> list[str]: + """基于规则的「提示性」结论，均标注待验证。""" + hints: list[str] = [] + if cr1 is not None and cr1 >= 0.45: + hints.append( + "样本内品牌集中度较高（第一大品牌份额偏高），头部玩家占据显著曝光；新原料/解决方案宜明确差异化价值主张（**需线下渠道与招商信息交叉验证**）。" + ) + elif cr1 is not None and cr1 < 0.25: + hints.append( + "样本内品牌较分散，品类或关键词下竞争格局未固化，存在定位与叙事空间（**需扩大样本页数与关键词矩阵验证**）。" + ) + if pst.get("stdev") and pst.get("mean") and pst["mean"] > 0: + cv = pst["stdev"] / pst["mean"] + if cv > 0.35: + hints.append( + "价格离散度较高，同时存在偏低价与偏高价陈列，可分别对标「性价比带」与「品质/功能带」竞品（**终端到手价受促销影响，非成本结构**）。" + ) + if hits: + top = hits.most_common(3) + top_s = "、".join(w for w, _ in top) + hints.append( + f"评价文本中「{top_s}」等主题出现较多，可作为消费者沟通与产品卖点的假设输入（**非严格主题模型，建议人工抽样复核**）。" + ) + if n_comments < 5: + hints.append( + "有效评价样本偏少，消费者洞察部分仅作方向参考，正式结论建议加大 SKU 数或评论分页。" + ) + if scen_n_texts >= 5 and scen_counts: + top_lbl, top_n = scen_counts.most_common(1)[0] + share = top_n / scen_n_texts + if share >= 0.25: + hints.append( + f"用途/场景中「{top_lbl}」在约 {100 * share:.0f}% 的有效评价自述中出现，可作为沟通场景与卖点的优先假设（**词组规则，建议抽样核对原句**）。" + ) + if not hints: + hints.append( + "当前样本下自动规则未触发强信号；请结合业务目标人工解读对比矩阵与原始 CSV。" + ) + return hints + + +def _embed_chart(run_dir: Path, filename: str, caption: str = "") -> list[str]: + """若 ``report_assets/<filename>`` 存在则返回插图 Markdown 片段。""" + if not (run_dir / "report_assets" / filename).is_file(): + return [] + cap = (caption or "").strip() + out: list[str] = [] + if cap: + out.append(f"*{cap}*") + out.append("") + out.append(f"![](report_assets/{filename})") + out.append("") + return out + + +def _scenario_group_asset_slug(group: str, index: int) -> str: + """与 ``pipeline.reporting.charts`` 中场景分组图文件名规则一致（勿改格式）。""" + raw = (group or "").strip() + core = re.sub(r"[^\w\u4e00-\u9fff-]", "", raw)[:20] + if not core: + core = "group" + return f"i{index:02d}_{core}" + + +def _focus_scenario_combo_bar_filename(group: str, index: int) -> str: + """关注词 + 使用场景并排条形图（与 ``pipeline.reporting.charts.save_combo_focus_scenario_bar`` 同源）。""" + slug = _scenario_group_asset_slug(group, index) + return f"chart_focus_and_scenarios_bar__{slug}.png" + + +def _matrix_prices_sales_chart_filename(group: str, index: int) -> str: + """与 ``pipeline.reporting.charts.generate_report_charts`` 中 ``chart_matrix_prices_sales__*`` 一致。""" + slug = _scenario_group_asset_slug(group, index) + return f"chart_matrix_prices_sales__{slug}.png" + + +def _lines_4_reading_brand( + *, + cr1: float | None, + cr3: float | None, + top: str, + brand_rows_n: int, + n_structure: int, +) -> list[str]: + if cr1 is None or not (top or "").strip(): + return [] + lines = [ + "", + "**数据解读（规则摘要）**：", + "", + f"- 在含品牌字段的 **{brand_rows_n}** 条列表行（占本章结构样本 **{n_structure}** 行）中，" + f"「{_md_cell(top.strip(), 36)}」曝光约占 **{100 * cr1:.1f}%**（按行计，同一 SKU 多行会重复计）。", + ] + if cr3 is not None: + lines.append( + f"- 前三品牌合计约 **{100 * cr3:.1f}%**；若该比例偏高，说明搜索页品牌集中度高，" + "新品需搭配清晰的差异定位与资源投放，避免与头部在泛词下正面撞车。" + ) + lines.append("") + return lines + + +def _lines_4_reading_shop( + *, + cr1: float | None, + cr3: float | None, + top: str, + shop_rows_n: int, + n_structure: int, + unique_sku_basis: dict[str, Any] | None = None, +) -> list[str]: + if not shop_rows_n: + return [] + lines = [ + "", + "**数据解读（规则摘要）**：", + "", + f"- 含店铺名的列表行共 **{shop_rows_n}** 条（结构样本 **{n_structure}** 行），反映搜索曝光下的店铺格局。", + ] + if cr1 is not None and (top or "").strip(): + lines.append( + f"- 第一大店铺「{_md_cell(top.strip(), 40)}」约占 **{100 * cr1:.1f}%**（**按列表行计**，同一 SKU 多行重复曝光会重复计）；" + "该指标刻画的是**列表可见度**而非销量或全渠道市占。" + ) + if cr3 is not None: + lines.append( + f"- 前三店铺合计约 **{100 * cr3:.1f}%**；若集中度高，可考虑从店铺矩阵、旗舰店/专营店布局等角度拆解竞争。" + ) + if isinstance(unique_sku_basis, dict) and unique_sku_basis.get("n_unique_skus"): + u1 = unique_sku_basis.get("first_share") + u3 = unique_sku_basis.get("top_three_combined_share") + utop = (unique_sku_basis.get("top_label") or "").strip() + nuk = unique_sku_basis.get("n_unique_skus") + if isinstance(u1, (int, float)) and utop: + lines.append( + f"- **按去重 SKU 计**（列表内共 **{nuk}** 个 SKU，每个计 1 次；店铺取该 SKU 首次出现行）：" + f"第一大店铺「{_md_cell(utop, 40)}」约占 **{100 * float(u1):.1f}%**。" + "若与上行「按行计」差异大，通常因同一 SKU 在多页重复出现；**勿将行占比误称为「SKU 款数占比」或「市场份额」**。" + ) + if isinstance(u3, (int, float)): + lines.append(f"- 前三店铺合计（按去重 SKU）约 **{100 * float(u3):.1f}%**。") + lines.append("") + return lines + + +__all__ = [ + "_embed_chart", + "_focus_scenario_combo_bar_filename", + "_lines_4_reading_brand", + "_lines_4_reading_shop", + "_matrix_prices_sales_chart_filename", + "_mermaid_pie_focus_keywords", + "_scenario_group_asset_slug", + "_scenario_summary_bullets", + "_strategy_hints", +] diff --git a/backend/pipeline/competitor_report/run_context.py b/backend/pipeline/competitor_report/run_context.py new file mode 100644 index 0000000..3bd90cc --- /dev/null +++ b/backend/pipeline/competitor_report/run_context.py @@ -0,0 +1,91 @@ +"""运行目录解析、关键词推断、pc_search_raw 检索规模读取。""" +from __future__ import annotations + +import json +import re +from collections import Counter +from pathlib import Path +from typing import Any + + +def _run_batch_label(run_dir: Path) -> str: + name = run_dir.name + m = re.match(r"^(\d{8})_(\d{6})_", name) + if m: + return f"{m.group(1)} {m.group(2)}" + return name + + +def _resolve_existing_run_dir(raw: str | Path | None) -> Path | None: + if raw is None: + return None + s = str(raw).strip() + if not s: + return None + p = Path(s).expanduser() + if not p.is_absolute(): + p = (Path.cwd() / p).resolve() + else: + p = p.resolve() + return p + + +def _infer_keyword(run_dir: Path, meta: dict[str, Any] | None) -> str: + if meta: + k = str(meta.get("keyword") or "").strip() + if k: + return k + m = re.match(r"^\d{8}_\d{6}_(.+)$", run_dir.name) + if m: + return m.group(1).strip() + return "" + + +def _pc_search_result_count_from_raw( + run_dir: Path, +) -> tuple[int | None, str, list[int], int, int]: + """ + 从 ``pc_search_raw/*.json`` 读取 ``data.resultCount``（京东 PC 搜索接口返回的检索命中规模）。 + 多文件时取众数；返回 (众数值, data.listKeyWord 首见值, 出现过的不同取值升序, 解析到的样本文件数)。 + """ + raw_dir = run_dir / "pc_search_raw" + if not raw_dir.is_dir(): + return None, "", [], 0, 0 + counts: list[int] = [] + list_kw = "" + n_files = 0 + for p in sorted(raw_dir.glob("*.json")): + try: + obj = json.loads(p.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError, UnicodeError): + continue + n_files += 1 + if not isinstance(obj, dict): + continue + data = obj.get("data") + if not isinstance(data, dict): + continue + rc = data.get("resultCount") + val: int | None = None + if isinstance(rc, int) and not isinstance(rc, bool) and rc >= 0: + val = rc + elif isinstance(rc, str) and rc.strip().isdigit(): + val = int(rc.strip()) + if val is not None: + counts.append(val) + lk = data.get("listKeyWord") + if isinstance(lk, str) and lk.strip() and not list_kw: + list_kw = lk.strip() + if not counts: + return None, list_kw, [], n_files, 0 + consensus_rc, _freq = Counter(counts).most_common(1)[0] + uniques = sorted(set(counts)) + return consensus_rc, list_kw, uniques, n_files, len(counts) + + +__all__ = [ + "_infer_keyword", + "_pc_search_result_count_from_raw", + "_resolve_existing_run_dir", + "_run_batch_label", +] diff --git a/backend/pipeline/csv/__init__.py b/backend/pipeline/csv/__init__.py new file mode 100644 index 0000000..10d6545 --- /dev/null +++ b/backend/pipeline/csv/__init__.py @@ -0,0 +1,2 @@ +# -*- coding: utf-8 -*- +"""京东流水线 CSV 列规范（``schema``）与历史表头重写（``header_rewrite``）。""" diff --git a/backend/pipeline/csv/header_rewrite.py b/backend/pipeline/csv/header_rewrite.py new file mode 100644 index 0000000..67c6501 --- /dev/null +++ b/backend/pipeline/csv/header_rewrite.py @@ -0,0 +1,236 @@ +# -*- coding: utf-8 -*- +""" +将历史 JD 流水线 CSV 表头规范为 ``pipeline.csv.schema`` 中的纯中文表头（仅重命名与列序，不改单元格内容逻辑）。 + +用于已落盘的 ``pipeline_runs/...`` 目录；新跑批次由爬虫直接写出新表头。 +""" +from __future__ import annotations + +import csv +from pathlib import Path +from typing import Iterable + +from .schema import ( + COMMENT_CSV_COLUMNS, + COMMENT_ROW_DICT_KEYS, + DETAIL_CSV_COLUMNS, + JD_SEARCH_CSV_HEADERS, + JD_SEARCH_INTERNAL_KEYS, + MERGED_CSV_COLUMNS, + MERGED_LEAN_DETAIL_INTERNAL_KEYS, + LEAN_DETAIL_CSV_HEADERS, +) + + +def _search_fieldnames_canonical() -> list[str]: + return [JD_SEARCH_CSV_HEADERS[k] for k in JD_SEARCH_INTERNAL_KEYS] + + +def _merged_fieldnames_canonical() -> list[str]: + return list(MERGED_CSV_COLUMNS) + + +def _detail_fieldnames_canonical() -> list[str]: + return list(DETAIL_CSV_COLUMNS) + + +def _comment_fieldnames_canonical() -> list[str]: + return list(COMMENT_CSV_COLUMNS) + + +def build_legacy_header_map() -> dict[str, str]: + """ + 旧表头字符串 → 新表头（纯中文或与 schema 一致）。 + + 覆盖：带英文括号的搜索/合并表、英文 ``pipeline_keyword`` / ``detail_*``、 + 评价 CSV 的接口字段名、商详 ``skuId`` 等。 + """ + m: dict[str, str] = {} + + # --- 合并表 / pc_search 曾用的「括号英文」列名 --- + legacy_search_pairs: tuple[tuple[str, str], ...] = ( + ("主商品ID(wareId)", JD_SEARCH_CSV_HEADERS["item_id"]), + ("SKU(skuId)", JD_SEARCH_CSV_HEADERS["sku_id"]), + ("标题(wareName)", JD_SEARCH_CSV_HEADERS["title"]), + ( + "标价(jdPrice,jdPriceText,realPrice)", + JD_SEARCH_CSV_HEADERS["price"], + ), + ( + "券后到手价(couponPrice,subsidyPrice,finalPrice.estimatedPrice,priceShow)", + JD_SEARCH_CSV_HEADERS["coupon_price"], + ), + ( + "原价(oriPrice,originalPrice,marketPrice)", + JD_SEARCH_CSV_HEADERS["original_price"], + ), + ("卖点(sellingPoint)", JD_SEARCH_CSV_HEADERS["selling_point"]), + ("销量楼层(commentSalesFloor)", JD_SEARCH_CSV_HEADERS["comment_sales_floor"]), + ("销量展示(totalSales)", JD_SEARCH_CSV_HEADERS["total_sales"]), + ( + "榜单类文案(标签/腰带/标题数组中的榜、TOP 等)", + JD_SEARCH_CSV_HEADERS["hot_list_rank"], + ), + ("评价量(commentFuzzy)", JD_SEARCH_CSV_HEADERS["comment_count"]), + ("店铺名(shopName)", JD_SEARCH_CSV_HEADERS["shop_name"]), + ("店铺链接(shopUrl,shopId)", JD_SEARCH_CSV_HEADERS["shop_url"]), + ( + "店铺信息链接(shopInfoUrl,brandUrl)", + JD_SEARCH_CSV_HEADERS["shop_info_url"], + ), + ("地域(deliveryAddress,area,procity)", JD_SEARCH_CSV_HEADERS["location"]), + ( + "商品链接(toUrl,clickUrl,item.m.jd.com)", + JD_SEARCH_CSV_HEADERS["detail_url"], + ), + ("主图(imageurl,imageUrl)", JD_SEARCH_CSV_HEADERS["image"]), + ("秒杀(seckillInfo,secKill)", JD_SEARCH_CSV_HEADERS["seckill_info"]), + ( + "规格属性(propertyList,color,catid,shortName)", + JD_SEARCH_CSV_HEADERS["attributes"], + ), + ("类目(leafCategory,cid3Name,catid)", JD_SEARCH_CSV_HEADERS["leaf_category"]), + ("平台(platform)", JD_SEARCH_CSV_HEADERS["platform"]), + ("搜索词(keyword)", JD_SEARCH_CSV_HEADERS["keyword"]), + ("页码(page)", JD_SEARCH_CSV_HEADERS["page"]), + ) + for old, new in legacy_search_pairs: + m[old] = new + + # 合并表首列 + m["pipeline_keyword"] = "流水线关键词" + + # 商详块：历史合并表曾直接写英文内部键 + for ik, zh in zip(MERGED_LEAN_DETAIL_INTERNAL_KEYS, LEAN_DETAIL_CSV_HEADERS): + m[ik] = zh + m["comment_count"] = "评论条数" + m["comment_preview"] = "评价摘要" + + # --- comments_flat：接口字段 / 小写 sku --- + for api_k, zh in zip(COMMENT_ROW_DICT_KEYS, COMMENT_CSV_COLUMNS): + m[api_k] = zh + m["sku"] = "SKU" + + # --- detail_ware_export --- + m["skuId"] = "SKU" + + return m + + +LEGACY_HEADER_MAP: dict[str, str] = build_legacy_header_map() + + +def _normalize_field_key(k: str | None) -> str: + return (k or "").strip().lstrip("\ufeff") + + +def _row_with_canonical_keys( + row: dict[str, str], + legacy_map: dict[str, str], +) -> dict[str, str]: + """将一行从任意旧表头映射为 canonical 列名；同列多旧键时取非空优先。""" + out: dict[str, str] = {} + for raw_k, v in row.items(): + k = _normalize_field_key(raw_k) + if not k: + continue + nk = legacy_map.get(k, k) + sv = "" if v is None else str(v) + if nk not in out or (sv.strip() and not str(out.get(nk, "")).strip()): + out[nk] = sv + return out + + +def rewrite_csv_inplace( + path: Path, + fieldnames: list[str], + legacy_map: dict[str, str], + *, + dry_run: bool = False, +) -> tuple[bool, str]: + """ + 读 UTF-8 BOM CSV，按 ``fieldnames`` 写出（缺列填空）。 + + 返回 (是否执行写入, 说明)。 + """ + path = path.expanduser().resolve() + if not path.is_file(): + return False, f"跳过（不存在）: {path}" + + with path.open(encoding="utf-8-sig", newline="") as f: + rdr = csv.DictReader(f) + rows_in = list(rdr) + + if not rows_in: + return False, f"空文件: {path}" + + rows_out: list[dict[str, str]] = [] + extras: set[str] = set() + for row in rows_in: + canon = _row_with_canonical_keys(row, legacy_map) + for k in canon: + if k not in fieldnames: + extras.add(k) + rows_out.append(canon) + + fieldnames_out = list(fieldnames) + sorted(extras) + + if dry_run: + return True, f"[dry-run] 将写 {len(rows_out)} 行 -> {path.name}" + + with path.open("w", encoding="utf-8-sig", newline="") as f: + w = csv.DictWriter( + f, + fieldnames=fieldnames_out, + extrasaction="ignore", + ) + w.writeheader() + w.writerows( + [{fn: str(r.get(fn, "") or "") for fn in fieldnames_out} for r in rows_out] + ) + + return True, f"已写 {len(rows_out)} 行 -> {path}" + + +def rewrite_run_dir_csv_headers( + run_dir: Path, + *, + dry_run: bool = False, + only: Iterable[str] | None = None, +) -> list[str]: + """ + 处理单个 run 目录下四种 CSV（存在则处理）。 + + ``only`` 为文件名子集，例如 ``("keyword_pipeline_merged.csv",)``。 + """ + from pipeline.ingest import ( + FILE_COMMENTS_FLAT_CSV, + FILE_DETAIL_WARE_CSV, + FILE_MERGED_CSV, + FILE_PC_SEARCH_CSV, + ) + + run_dir = run_dir.expanduser().resolve() + lm = LEGACY_HEADER_MAP + only_set = {x.strip() for x in only} if only else None + + tasks: list[tuple[str, list[str]]] = [ + (FILE_MERGED_CSV, _merged_fieldnames_canonical()), + (FILE_PC_SEARCH_CSV, _search_fieldnames_canonical()), + (FILE_COMMENTS_FLAT_CSV, _comment_fieldnames_canonical()), + (FILE_DETAIL_WARE_CSV, _detail_fieldnames_canonical()), + ] + + messages: list[str] = [] + for fname, fieldnames in tasks: + if only_set is not None and fname not in only_set: + continue + ok, msg = rewrite_csv_inplace( + run_dir / fname, + fieldnames, + lm, + dry_run=dry_run, + ) + if ok or "空文件" in msg or "不存在" in msg: + messages.append(msg) + return messages diff --git a/backend/pipeline/csv/schema.py b/backend/pipeline/csv/schema.py new file mode 100644 index 0000000..a3c2569 --- /dev/null +++ b/backend/pipeline/csv/schema.py @@ -0,0 +1,268 @@ +""" +与 ``jd_pc_search`` 导出 CSV 列对齐的字段名映射（入库 / API / 导出共用）。 +内部键与爬虫侧 ``JD_ITEM_CSV_FIELDS`` / ``WARE_PARSED_CSV_FIELDNAMES`` 一致，便于对照源码。 +""" +from __future__ import annotations + +import re + + +def strip_buyer_ranking_line_prefix(value: str) -> str: + """去掉榜单列上的 ``榜单/曝光：`` 历史前缀，展示与入库一致。""" + s = (value or "").strip() + prefix = "榜单/曝光：" + if s.startswith(prefix): + return s[len(prefix) :].strip() + if s.startswith("榜单/曝光"): + return s[len("榜单/曝光") :].lstrip("：").strip() + return s + + +# --- 搜索导出 pc_search_export.csv（纯中文表头，与 jd_h5_search_requests.JD_EXPORT_COLUMN_HEADERS 一致）--- +JD_SEARCH_INTERNAL_KEYS: tuple[str, ...] = ( + "item_id", + "sku_id", + "title", + "price", + "coupon_price", + "original_price", + "selling_point", + "comment_sales_floor", + "total_sales", + "hot_list_rank", + "comment_count", + "shop_name", + "shop_url", + "shop_info_url", + "location", + "detail_url", + "image", + "seckill_info", + "attributes", + "leaf_category", + "platform", + "keyword", + "page", +) + +JD_SEARCH_CSV_HEADERS: dict[str, str] = { + "item_id": "主商品ID", + "sku_id": "SKU", + "title": "标题", + "price": "标价", + "coupon_price": "券后到手价", + "original_price": "原价", + "selling_point": "卖点", + "comment_sales_floor": "销量楼层", + "total_sales": "销量展示", + "hot_list_rank": "榜单类文案", + "comment_count": "评价量", + "shop_name": "店铺名", + "shop_url": "店铺链接", + "shop_info_url": "店铺信息链接", + "location": "地域", + "detail_url": "商品链接", + "image": "主图", + "seckill_info": "秒杀", + "attributes": "规格属性", + "leaf_category": "类目", + "platform": "平台", + "keyword": "搜索词", + "page": "页码", +} + +# CSV 表头 -> 模型属性名 +SEARCH_CSV_HEADER_TO_FIELD: dict[str, str] = { + h: k for k, h in JD_SEARCH_CSV_HEADERS.items() +} + +# lean 商详：ORM 内部键（英文 snake_case）；CSV 表头为中文（见 DETAIL_CSV_COLUMNS） +MERGED_LEAN_DETAIL_INTERNAL_KEYS: tuple[str, ...] = ( + "detail_brand", + "detail_price_final", + "detail_shop_name", + "detail_category_path", + "detail_product_attributes", + "detail_body_ingredients", + "buyer_ranking_line", + "buyer_promo_text", +) + +LEAN_DETAIL_EXPORT_FIELDNAMES: tuple[str, ...] = MERGED_LEAN_DETAIL_INTERNAL_KEYS + +LEAN_DETAIL_CSV_HEADERS: tuple[str, ...] = ( + "品牌", + "到手价", + "店铺名称", + "类目路径", + "商品参数", + "配料表", + "榜单排名", + "促销摘要", +) + +DETAIL_CSV_HEADER_TO_FIELD: dict[str, str] = dict( + zip(LEAN_DETAIL_CSV_HEADERS, MERGED_LEAN_DETAIL_INTERNAL_KEYS) +) + +# --- 商详 detail_ware_export.csv（lean：SKU + 上列；full 模式爬虫仍可能多列，入库只认 DETAIL_CSV_COLUMNS）--- +JD_DETAIL_MERGE_KEYS: tuple[str, ...] = MERGED_LEAN_DETAIL_INTERNAL_KEYS + +DETAIL_CSV_COLUMNS: tuple[str, ...] = ("SKU", *LEAN_DETAIL_CSV_HEADERS) + +DETAIL_CSV_TO_FIELD: dict[str, str] = { + "SKU": "sku_id", + **DETAIL_CSV_HEADER_TO_FIELD, +} + +# --- 评价 comments_flat.csv（表头中文；爬虫行字典仍用英文 API 键，写出时映射）--- +COMMENT_CSV_COLUMNS: tuple[str, ...] = ( + "SKU", + "评价ID", + "用户昵称", + "评价内容", + "评价时间", + "购买次数", + "晒图链接", + "评分", +) + +COMMENT_CSV_TO_FIELD: dict[str, str] = { + "SKU": "sku_id", + "评价ID": "comment_id", + "用户昵称": "user_nick_name", + "评价内容": "tag_comment_content", + "评价时间": "comment_date", + "购买次数": "buy_count_text", + "晒图链接": "large_pic_urls", + "评分": "comment_score", +} + +# 爬虫评价行 dict 键（与京东接口字段一致）→ CSV 中文表头 +COMMENT_ROW_DICT_KEYS: tuple[str, ...] = ( + "sku", + "commentId", + "userNickName", + "tagCommentContent", + "commentDate", + "buyCountText", + "largePicURLs", + "commentScore", +) + +# --- 合并宽表 keyword_pipeline_merged.csv（lean = 搜索块 + 商详块 + 评论块；改列请改对应块，勿在尾部堆列）--- + +MERGED_SEARCH_INTERNAL_KEYS: tuple[str, ...] = ( + "pipeline_keyword", + "sku_id", + "ware_id", + "title", + "price", + "coupon_price", + "original_price", + "selling_point", + "hot_list_rank", + "comment_fuzzy", + "comment_sales_floor", + "total_sales", + "shop_name", + "detail_url", + "image", + "attributes", + "leaf_category", + "keyword", + "page", +) + + +def _merged_search_csv_header_for_internal(k: str) -> str: + """整合表搜索块：内部键 → 中文 CSV 表头（与 PC 搜索导出列名对齐，合并表专有列单独处理）。""" + if k == "ware_id": + return JD_SEARCH_CSV_HEADERS["item_id"] + if k == "comment_fuzzy": + return "评价量" + return JD_SEARCH_CSV_HEADERS[k] + + +MERGED_SEARCH_CSV_COLUMNS: tuple[str, ...] = ( + "流水线关键词", + *( + _merged_search_csv_header_for_internal(k) + for k in MERGED_SEARCH_INTERNAL_KEYS[1:] + ), +) + +# 商详块：CSV 为中文表头；内部键见 MERGED_LEAN_DETAIL_INTERNAL_KEYS +MERGED_LEAN_DETAIL_KEYS: tuple[str, ...] = LEAN_DETAIL_CSV_HEADERS + +MERGED_COMMENT_CSV_COLUMNS: tuple[str, ...] = ( + "评论条数", + "评价摘要", +) + +MERGED_COMMENT_INTERNAL_KEYS: tuple[str, ...] = ( + "pipeline_comment_count", + "comment_preview", +) + +MERGED_CSV_COLUMNS: tuple[str, ...] = ( + *MERGED_SEARCH_CSV_COLUMNS, + *MERGED_LEAN_DETAIL_KEYS, + *MERGED_COMMENT_CSV_COLUMNS, +) + +MERGED_INTERNAL_KEYS: tuple[str, ...] = ( + *MERGED_SEARCH_INTERNAL_KEYS, + *MERGED_LEAN_DETAIL_INTERNAL_KEYS, + *MERGED_COMMENT_INTERNAL_KEYS, +) + +assert len(MERGED_CSV_COLUMNS) == len(MERGED_INTERNAL_KEYS) + +MERGED_CSV_TO_FIELD: dict[str, str] = dict(zip(MERGED_CSV_COLUMNS, MERGED_INTERNAL_KEYS)) + +MERGED_FIELD_TO_CSV_HEADER: dict[str, str] = { + internal: csv_h for csv_h, internal in MERGED_CSV_TO_FIELD.items() +} + + +def remap_merged_row_english_detail_keys_to_csv_headers(merged: dict[str, str]) -> None: + """整合表写入前：将 ``ware_flat`` / 抽取逻辑产生的英文商详与购买者内部键改为中文 CSV 列名（原地修改）。""" + for ik, zh in zip(MERGED_LEAN_DETAIL_INTERNAL_KEYS, LEAN_DETAIL_CSV_HEADERS): + if ik in merged: + merged[zh] = str(merged.get(ik) or "") + del merged[ik] + + +def infer_total_sales_from_sales_floor(cell: str) -> str: + """ + 从「销量楼层(commentSalesFloor)」列文案截取可作 ``销量展示(totalSales)`` 的片段（与列表接口未单独落 totalSales 列时的兜底一致）。 + """ + t = (cell or "").strip() + if not t: + return "" + m = re.search(r"已售\s*[\d,，.+]*\s*[万亿]?\s*\+?", t) + if m: + return m.group(0).strip() + m2 = re.search(r"已售\s*[\d,，.+\s万千亿]+", t) + return m2.group(0).strip() if m2 else "" + + +def merged_csv_effective_total_sales(row: dict[str, str]) -> str: + """合并表一行：优先已有 ``销量展示(totalSales)`` 列，否则从销量楼层推断。""" + h_ts = MERGED_FIELD_TO_CSV_HEADER["total_sales"] + h_fl = MERGED_FIELD_TO_CSV_HEADER["comment_sales_floor"] + direct = str(row.get(h_ts) or "").strip() + if direct: + return direct + return infer_total_sales_from_sales_floor(str(row.get(h_fl) or "")) + + +def search_csv_effective_total_sales(row: dict[str, str]) -> str: + """PC 搜索导出表一行：与 ``merged_csv_effective_total_sales`` 解析规则一致（中文表头）。""" + h_ts = JD_SEARCH_CSV_HEADERS["total_sales"] + h_fl = JD_SEARCH_CSV_HEADERS["comment_sales_floor"] + direct = str(row.get(h_ts) or "").strip() + if direct: + return direct + return infer_total_sales_from_sales_floor(str(row.get(h_fl) or "")) diff --git a/backend/pipeline/csv_schema.py b/backend/pipeline/csv_schema.py deleted file mode 100644 index 6a8c2db..0000000 --- a/backend/pipeline/csv_schema.py +++ /dev/null @@ -1,181 +0,0 @@ -""" -与 ``jd_pc_search`` 导出 CSV 列对齐的字段名映射（入库 / API / 导出共用）。 -内部键与爬虫侧 ``JD_ITEM_CSV_FIELDS`` / ``WARE_PARSED_CSV_FIELDNAMES`` 一致，便于对照源码。 -""" -from __future__ import annotations - -# --- 搜索导出 pc_search_export.csv（列名为中文，与 jd_h5_search_requests.JD_EXPORT_COLUMN_HEADERS 一致）--- -JD_SEARCH_INTERNAL_KEYS: tuple[str, ...] = ( - "item_id", - "sku_id", - "title", - "price", - "coupon_price", - "original_price", - "selling_point", - "comment_sales_floor", - "hot_list_rank", - "comment_count", - "shop_name", - "shop_url", - "shop_info_url", - "location", - "detail_url", - "image", - "seckill_info", - "attributes", - "leaf_category", - "platform", - "keyword", - "page", -) - -JD_SEARCH_CSV_HEADERS: dict[str, str] = { - "item_id": "主商品ID(wareId)", - "sku_id": "SKU(skuId)", - "title": "标题(wareName)", - "price": "标价(jdPrice,jdPriceText,realPrice)", - "coupon_price": "券后到手价(couponPrice,subsidyPrice,finalPrice.estimatedPrice,priceShow)", - "original_price": "原价(oriPrice,originalPrice,marketPrice)", - "selling_point": "卖点(sellingPoint)", - "comment_sales_floor": "销量楼层(commentSalesFloor)", - "hot_list_rank": "榜单类文案(标签/腰带/标题数组中的榜、TOP 等)", - "comment_count": "评价量(commentFuzzy)", - "shop_name": "店铺名(shopName)", - "shop_url": "店铺链接(shopUrl,shopId)", - "shop_info_url": "店铺信息链接(shopInfoUrl,brandUrl)", - "location": "地域(deliveryAddress,area,procity)", - "detail_url": "商品链接(toUrl,clickUrl,item.m.jd.com)", - "image": "主图(imageurl,imageUrl)", - "seckill_info": "秒杀(seckillInfo,secKill)", - "attributes": "规格属性(propertyList,color,catid,shortName)", - "leaf_category": "类目(leafCategory,cid3Name,catid)", - "platform": "平台(platform)", - "keyword": "搜索词(keyword)", - "page": "页码(page)", -} - -# CSV 表头 -> 模型属性名 -SEARCH_CSV_HEADER_TO_FIELD: dict[str, str] = { - h: k for k, h in JD_SEARCH_CSV_HEADERS.items() -} - -# lean 商详子集：合并宽表商详块、detail_ware_export（lean）、JdJobDetailRow 共用（CSV 列名与 ORM 一致） -LEAN_DETAIL_EXPORT_FIELDNAMES: tuple[str, ...] = ( - "detail_brand", - "detail_price_final", - "detail_shop_name", - "detail_category_path", - "detail_product_attributes", - "detail_body_ingredients", -) - -# --- 商详 detail_ware_export.csv（lean：skuId + 上列；full 模式爬虫仍可能多列，入库只认 DETAIL_CSV_COLUMNS）--- -JD_DETAIL_MERGE_KEYS: tuple[str, ...] = LEAN_DETAIL_EXPORT_FIELDNAMES - -DETAIL_CSV_COLUMNS: tuple[str, ...] = ("skuId", *JD_DETAIL_MERGE_KEYS) - -DETAIL_CSV_TO_FIELD: dict[str, str] = { - "skuId": "sku_id", - **{k: k for k in JD_DETAIL_MERGE_KEYS}, -} - -# --- 评价 comments_flat.csv --- -COMMENT_CSV_COLUMNS: tuple[str, ...] = ( - "sku", - "commentId", - "userNickName", - "tagCommentContent", - "commentDate", - "buyCountText", - "largePicURLs", - "commentScore", -) - -COMMENT_CSV_TO_FIELD: dict[str, str] = { - "sku": "sku_id", - "commentId": "comment_id", - "userNickName": "user_nick_name", - "tagCommentContent": "tag_comment_content", - "commentDate": "comment_date", - "buyCountText": "buy_count_text", - "largePicURLs": "large_pic_urls", - "commentScore": "comment_score", -} - -# --- 合并宽表 keyword_pipeline_merged.csv（lean = 搜索块 + 商详块 + 评论块；改列请改对应块，勿在尾部堆列）--- - -MERGED_SEARCH_CSV_COLUMNS: tuple[str, ...] = ( - "pipeline_keyword", - "SKU(skuId)", - "主商品ID(wareId)", - "标题(wareName)", - "标价(jdPrice,jdPriceText,realPrice)", - "券后到手价(couponPrice,subsidyPrice,finalPrice.estimatedPrice,priceShow)", - "原价(oriPrice,originalPrice,marketPrice)", - "卖点(sellingPoint)", - "榜单类文案(标签/腰带/标题数组中的榜、TOP 等)", - "评价量(commentFuzzy)", - "销量楼层(commentSalesFloor)", - "店铺名(shopName)", - "商品链接(toUrl,clickUrl,item.m.jd.com)", - "主图(imageurl,imageUrl)", - "规格属性(propertyList,color,catid,shortName)", - "类目(leafCategory,cid3Name,catid)", - "搜索词(keyword)", - "页码(page)", -) - -MERGED_SEARCH_INTERNAL_KEYS: tuple[str, ...] = ( - "pipeline_keyword", - "sku_id", - "ware_id", - "title", - "price", - "coupon_price", - "original_price", - "selling_point", - "hot_list_rank", - "comment_fuzzy", - "comment_sales_floor", - "shop_name", - "detail_url", - "image", - "attributes", - "leaf_category", - "keyword", - "page", -) - -# 商详块：列名与 ORM 属性同名；与 LEAN_DETAIL_EXPORT_FIELDNAMES / 流水线 lean 一致 -MERGED_LEAN_DETAIL_KEYS: tuple[str, ...] = LEAN_DETAIL_EXPORT_FIELDNAMES - -MERGED_COMMENT_CSV_COLUMNS: tuple[str, ...] = ( - "comment_count", - "comment_preview", -) - -MERGED_COMMENT_INTERNAL_KEYS: tuple[str, ...] = ( - "pipeline_comment_count", - "comment_preview", -) - -MERGED_CSV_COLUMNS: tuple[str, ...] = ( - *MERGED_SEARCH_CSV_COLUMNS, - *MERGED_LEAN_DETAIL_KEYS, - *MERGED_COMMENT_CSV_COLUMNS, -) - -MERGED_INTERNAL_KEYS: tuple[str, ...] = ( - *MERGED_SEARCH_INTERNAL_KEYS, - *MERGED_LEAN_DETAIL_KEYS, - *MERGED_COMMENT_INTERNAL_KEYS, -) - -assert len(MERGED_CSV_COLUMNS) == len(MERGED_INTERNAL_KEYS) - -MERGED_CSV_TO_FIELD: dict[str, str] = dict(zip(MERGED_CSV_COLUMNS, MERGED_INTERNAL_KEYS)) - -MERGED_FIELD_TO_CSV_HEADER: dict[str, str] = { - internal: csv_h for csv_h, internal in MERGED_CSV_TO_FIELD.items() -} diff --git a/backend/pipeline/dataset_api.py b/backend/pipeline/dataset_api.py new file mode 100644 index 0000000..72824e6 --- /dev/null +++ b/backend/pipeline/dataset_api.py @@ -0,0 +1,237 @@ +"""库内数据浏览 API：排序、价格、类目（§5 矩阵）与店铺（精确）筛选。""" +from __future__ import annotations + +from typing import Any + +from django.db.models import F, Q, QuerySet +from django.db.models.expressions import OrderBy +from rest_framework.request import Request + +SEARCH_SORT_FIELDS = frozenset( + { + "row_index", + "price", + "sku_id", + "title", + "leaf_category", + "matrix_group_label", + "total_sales", + "comment_count", + } +) +DETAIL_SORT_FIELDS = frozenset( + { + "row_index", + "price", + "sku_id", + "detail_category_path", + "detail_brand", + "matrix_group_label", + } +) +MERGED_SORT_FIELDS = frozenset( + { + "row_index", + "price", + "sku_id", + "title", + "leaf_category", + "detail_category_path", + "matrix_group_label", + "total_sales", + "comment_count", + } +) + + +def _parse_opt_float(val: str | None) -> float | None: + if val is None or not str(val).strip(): + return None + try: + return float(str(val).strip()) + except ValueError: + return None + + +def parse_sort_meta(request: Request) -> tuple[str, bool]: + sort = (request.query_params.get("sort") or "row_index").strip() + order = (request.query_params.get("order") or "asc").strip().lower() + desc = order == "desc" + return sort, desc + + +def price_bounds_from_request(request: Request) -> tuple[float | None, float | None]: + return ( + _parse_opt_float(request.query_params.get("price_min")), + _parse_opt_float(request.query_params.get("price_max")), + ) + + +def report_group_from_request(request: Request) -> str: + """与 §5 矩阵一致的类目名（如饼干、米）；查询参数 ``report_group``。""" + return (request.query_params.get("report_group") or "").strip() + + +def shop_from_request(request: Request) -> str: + """店铺名精确匹配（与摘要 ``shop_options`` 中某项一致）；查询参数 ``shop``。""" + return (request.query_params.get("shop") or "").strip() + + +def detail_category_q_from_request(request: Request) -> str: + return (request.query_params.get("detail_category_q") or "").strip() + + +def filter_echo( + *, + report_group: str, + shop: str, + price_min: float | None, + price_max: float | None, + detail_category_q: str, + sort: str, + desc: bool, +) -> dict[str, Any]: + return { + "report_group": report_group or None, + "shop": shop or None, + "price_min": price_min, + "price_max": price_max, + "detail_category_q": detail_category_q or None, + "sort": sort, + "order": "desc" if desc else "asc", + } + + +def apply_search_filters(qs: QuerySet, request: Request) -> QuerySet: + rg = report_group_from_request(request) + if rg: + qs = qs.filter(Q(matrix_group_label=rg) | Q(leaf_category=rg)) + sp = shop_from_request(request) + if sp: + qs = qs.filter(shop_name=sp) + pmin, pmax = price_bounds_from_request(request) + if pmin is not None: + qs = qs.filter(price_value__gte=pmin) + if pmax is not None: + qs = qs.filter(price_value__lte=pmax) + return qs + + +def apply_search_order(qs: QuerySet, sort: str, desc: bool) -> QuerySet: + sort = sort if sort in SEARCH_SORT_FIELDS else "row_index" + if sort == "price": + return qs.order_by( + OrderBy(F("price_value"), descending=desc, nulls_last=True), + "row_index", + ) + if sort == "total_sales": + return qs.order_by( + OrderBy(F("sales_sort_value"), descending=desc, nulls_last=True), + "row_index", + ) + if sort == "comment_count": + return qs.order_by( + OrderBy(F("comment_count_sort_value"), descending=desc, nulls_last=True), + "row_index", + ) + if sort == "row_index": + return qs.order_by(OrderBy(F("row_index"), descending=desc)) + field = { + "sku_id": "sku_id", + "title": "title", + "leaf_category": "leaf_category", + "matrix_group_label": "matrix_group_label", + }[sort] + return qs.order_by( + OrderBy(F(field), descending=desc, nulls_last=True), + "row_index", + ) + + +def apply_detail_filters(qs: QuerySet, request: Request) -> QuerySet: + rg = report_group_from_request(request) + if rg: + qs = qs.filter(matrix_group_label=rg) + sp = shop_from_request(request) + if sp: + qs = qs.filter(detail_shop_name=sp) + q = detail_category_q_from_request(request) + if q: + qs = qs.filter(detail_category_path__icontains=q) + pmin, pmax = price_bounds_from_request(request) + if pmin is not None: + qs = qs.filter(detail_price_value__gte=pmin) + if pmax is not None: + qs = qs.filter(detail_price_value__lte=pmax) + return qs + + +def apply_detail_order(qs: QuerySet, sort: str, desc: bool) -> QuerySet: + sort = sort if sort in DETAIL_SORT_FIELDS else "row_index" + if sort == "price": + return qs.order_by( + OrderBy(F("detail_price_value"), descending=desc, nulls_last=True), + "row_index", + ) + if sort == "row_index": + return qs.order_by(OrderBy(F("row_index"), descending=desc)) + field = { + "sku_id": "sku_id", + "detail_category_path": "detail_category_path", + "detail_brand": "detail_brand", + "matrix_group_label": "matrix_group_label", + }[sort] + return qs.order_by( + OrderBy(F(field), descending=desc, nulls_last=True), + "row_index", + ) + + +def apply_merged_filters(qs: QuerySet, request: Request) -> QuerySet: + rg = report_group_from_request(request) + if rg: + qs = qs.filter(matrix_group_label=rg) + sp = shop_from_request(request) + if sp: + qs = qs.filter(Q(shop_name=sp) | Q(detail_shop_name=sp)) + q = detail_category_q_from_request(request) + if q: + qs = qs.filter(detail_category_path__icontains=q) + pmin, pmax = price_bounds_from_request(request) + if pmin is not None: + qs = qs.filter(price_value__gte=pmin) + if pmax is not None: + qs = qs.filter(price_value__lte=pmax) + return qs + + +def apply_merged_order(qs: QuerySet, sort: str, desc: bool) -> QuerySet: + sort = sort if sort in MERGED_SORT_FIELDS else "row_index" + if sort == "price": + return qs.order_by( + OrderBy(F("price_value"), descending=desc, nulls_last=True), + "row_index", + ) + if sort == "total_sales": + return qs.order_by( + OrderBy(F("sales_sort_value"), descending=desc, nulls_last=True), + "row_index", + ) + if sort == "comment_count": + return qs.order_by( + OrderBy(F("comment_count_sort_value"), descending=desc, nulls_last=True), + "row_index", + ) + if sort == "row_index": + return qs.order_by(OrderBy(F("row_index"), descending=desc)) + field = { + "sku_id": "sku_id", + "title": "title", + "leaf_category": "leaf_category", + "detail_category_path": "detail_category_path", + "matrix_group_label": "matrix_group_label", + }[sort] + return qs.order_by( + OrderBy(F(field), descending=desc, nulls_last=True), + "row_index", + ) diff --git a/backend/pipeline/dataset_nonempty.py b/backend/pipeline/dataset_nonempty.py index 844c8dc..86120d1 100644 --- a/backend/pipeline/dataset_nonempty.py +++ b/backend/pipeline/dataset_nonempty.py @@ -1,7 +1,7 @@ """按任务扫描库内行，得到「全表至少一格非空」的列，供摘要 / 浏览 / 导出一致裁剪。""" from __future__ import annotations -from .csv_schema import ( +from .csv.schema import ( COMMENT_CSV_COLUMNS, COMMENT_CSV_TO_FIELD, DETAIL_CSV_COLUMNS, @@ -14,6 +14,8 @@ from .csv_schema import ( from .models import JdJobCommentRow, JdJobDetailRow, JdJobMergedRow, JdJobSearchRow, PipelineJob from .row_serialize import COMMENT_FIELDS_ORDER, DETAIL_FIELDS_ORDER +MATRIX_GROUP_COLUMN = {"key": "matrix_group_label", "label": "类目"} + def _is_nonempty(val) -> bool: if val is None: @@ -62,7 +64,13 @@ def nonempty_merged_fields_for_job(job: PipelineJob) -> list[str]: def search_columns_for_api(job: PipelineJob) -> list[dict[str, str]]: - return [{"key": k, "label": JD_SEARCH_CSV_HEADERS[k]} for k in nonempty_search_keys_for_job(job)] + cols = [ + {"key": k, "label": JD_SEARCH_CSV_HEADERS[k]} + for k in nonempty_search_keys_for_job(job) + ] + if JdJobSearchRow.objects.filter(job=job).exclude(matrix_group_label="").exists(): + cols.append(dict(MATRIX_GROUP_COLUMN)) + return cols def _detail_field_to_csv_col(field: str) -> str: @@ -73,10 +81,13 @@ def _detail_field_to_csv_col(field: str) -> str: def detail_columns_for_api(job: PipelineJob) -> list[dict[str, str]]: - return [ + cols = [ {"key": f, "label": _detail_field_to_csv_col(f)} for f in nonempty_detail_fields_for_job(job) ] + if JdJobDetailRow.objects.filter(job=job).exclude(matrix_group_label="").exists(): + cols.append(dict(MATRIX_GROUP_COLUMN)) + return cols def _comment_field_to_csv_col(field: str) -> str: @@ -94,21 +105,30 @@ def comment_columns_for_api(job: PipelineJob) -> list[dict[str, str]]: def merged_columns_for_api(job: PipelineJob) -> list[dict[str, str]]: - return [ + cols = [ {"key": k, "label": MERGED_FIELD_TO_CSV_HEADER[k]} for k in nonempty_merged_fields_for_job(job) ] + if JdJobMergedRow.objects.filter(job=job).exclude(matrix_group_label="").exists(): + cols.append(dict(MATRIX_GROUP_COLUMN)) + return cols def search_export_headers(job: PipelineJob) -> list[str]: keys = nonempty_search_keys_for_job(job) - return ["id", "row_index"] + [JD_SEARCH_CSV_HEADERS[k] for k in keys] + h = ["id", "row_index"] + [JD_SEARCH_CSV_HEADERS[k] for k in keys] + if JdJobSearchRow.objects.filter(job=job).exclude(matrix_group_label="").exists(): + h.append(MATRIX_GROUP_COLUMN["label"]) + return h def detail_export_headers(job: PipelineJob) -> list[str]: fields = set(nonempty_detail_fields_for_job(job)) cols = [c for c in DETAIL_CSV_COLUMNS if DETAIL_CSV_TO_FIELD[c] in fields] - return ["id", "row_index"] + cols + h = ["id", "row_index"] + cols + if JdJobDetailRow.objects.filter(job=job).exclude(matrix_group_label="").exists(): + h.append(MATRIX_GROUP_COLUMN["label"]) + return h def comment_export_headers(job: PipelineJob) -> list[str]: @@ -119,4 +139,7 @@ def comment_export_headers(job: PipelineJob) -> list[str]: def merged_export_headers(job: PipelineJob) -> list[str]: keys = nonempty_merged_fields_for_job(job) - return ["id", "row_index"] + [MERGED_FIELD_TO_CSV_HEADER[k] for k in keys] + h = ["id", "row_index"] + [MERGED_FIELD_TO_CSV_HEADER[k] for k in keys] + if JdJobMergedRow.objects.filter(job=job).exclude(matrix_group_label="").exists(): + h.append(MATRIX_GROUP_COLUMN["label"]) + return h diff --git a/backend/pipeline/demos/__init__.py b/backend/pipeline/demos/__init__.py new file mode 100644 index 0000000..6bb78d4 --- /dev/null +++ b/backend/pipeline/demos/__init__.py @@ -0,0 +1 @@ +"""本地调试脚本（不随 Django 路由加载）；见各 ``run_*`` 模块顶部用法。""" diff --git a/backend/pipeline/demos/chapter8_text_mining_probe.py b/backend/pipeline/demos/chapter8_text_mining_probe.py new file mode 100644 index 0000000..0b3ee29 --- /dev/null +++ b/backend/pipeline/demos/chapter8_text_mining_probe.py @@ -0,0 +1,803 @@ +""" +第八章「评论文本补充分析」独立脚本（**不修改**主报告核心逻辑）。 + +流程（按细类分组）：清洗（中文分词 + 停用词）→ **词云图（可选）** → 词频 / 关键词突出度 → 词对共现 → 主题归纳 +→ 规则化叙事小结 → 文末可选 **专用 LLM**（结构化 JSON + ``PROBE_TEXT_MINING_SYSTEM`` + ``_call_llm``；**非** ``COMMENT_GROUPS_SYSTEM``、**非**已废弃的星级子集预设口语短语情感条形图）。 + +依赖（请自行安装）:: + + pip install jieba scikit-learn numpy wordcloud matplotlib + +用法（在 ``backend`` 目录下）:: + + python -m pipeline.demos.chapter8_text_mining_probe --run-dir \"../data/JD/pipeline_runs/20260413_104252_低GI\" + python -m pipeline.demos.chapter8_text_mining_probe --run-dir \"...\" --out chapter8_probe.md + python -m pipeline.demos.chapter8_text_mining_probe --run-dir \"...\" --live-llm + python -m pipeline.demos.chapter8_text_mining_probe --run-dir \"...\" --live-llm --llm-chunked + +输出：默认写入 ``<run_dir>/chapter8_text_mining_probe.md``。 + +嵌入竞品报告：流水线默认开启（``get_default_report_config`` 中 ``chapter8_text_mining_probe``: true）；若任务显式关闭则为 false。开启时会生成本稿并调用 ``markdown_embed_body_for_competitor_report`` 写入 ``competitor_analysis.md`` 的 **第八章第二节（评论文本补充分析）**，替代原「关注词 + 场景」条图及对应两段大模型；**不再**嵌入原「评价正负面粗判」预设口语短语扇形图/条形图及同口径大模型块。 +""" +from __future__ import annotations + +import argparse +import json +import os +import re +import sys +from collections import Counter +from pathlib import Path +from typing import Any + +BACKEND_ROOT = Path(__file__).resolve().parents[2] +if str(BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(BACKEND_ROOT)) +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings") + +import django # noqa: E402 + +django.setup() + +JCR_ROOT = BACKEND_ROOT / "crawler_copy" / "jd_pc_search" +if str(JCR_ROOT) not in sys.path: + sys.path.insert(0, str(JCR_ROOT)) + +from pipeline.competitor_report import jd_report as jcr # noqa: E402 +import jd_keyword_pipeline as kpl # noqa: E402 + +from pipeline.csv.schema import MERGED_FIELD_TO_CSV_HEADER # noqa: E402 +from pipeline.llm.generate import _call_llm # noqa: E402 探针专用，不新增 generate 导出 + +# 导入失败时**不得** ``sys.exit``：本模块会被 ``runner`` 在 Web 请求中 import，退出会整进程 500。 +_PROBE_TEXT_MINING_DEPS_OK = False +_PROBE_TEXT_MINING_IMPORT_ERROR = "" +try: + import numpy as np # noqa: WPS433 + import jieba # noqa: WPS433 + from sklearn.decomposition import LatentDirichletAllocation # noqa: WPS433 + from sklearn.feature_extraction.text import ( # noqa: WPS433 + CountVectorizer, + TfidfVectorizer, + ) + _PROBE_TEXT_MINING_DEPS_OK = True +except ImportError as e: + np = None # type: ignore[assignment] + jieba = None # type: ignore[assignment] + LatentDirichletAllocation = None # type: ignore[assignment] + CountVectorizer = None # type: ignore[assignment] + TfidfVectorizer = None # type: ignore[assignment] + _PROBE_TEXT_MINING_IMPORT_ERROR = str(e) + +try: + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt # noqa: WPS433 + from wordcloud import WordCloud # noqa: WPS433 + + _WORDCLOUD_AVAILABLE = True +except ImportError: + plt = None # type: ignore[assignment] + WordCloud = None # type: ignore[assignment] + _WORDCLOUD_AVAILABLE = False + + +# 精简中文停用词（可换外部文件）；与业务无关，仅用于探针 +_STOP_BASIC: frozenset[str] = frozenset( + """ + 的 了 和 是 在 也 有 就 都 很 啊 还 吗 吧 呢 呀 哦 噢 哈 呵 与 及 或 等 为 被 让 从 到 把 而 又 对 中 这 那 其 一个 一些 没有 不是 可以 这样 我们 你们 他们 它们 它 会 要 能 去 来 做 用 给 自己 这个 那个 什么 怎么 如果 因为 所以 但是 而且 然后 还是 或者 还有 就是 只是 只是 已经 还是 + 非常 真的 比较 特别 感觉 觉得 认为 看到 收到 东西 商品 产品 卖家 买家 店铺 京东 物流 快递 包装 评价 评论 购买 买 卖 收到 天 次 个 款 种 条 块 + """.split() +) + +# --- 评论文本补充分析 · 专用 LLM（与正式报告 ``COMMENT_GROUPS_SYSTEM`` / 已废弃的预设口语短语情感口径均不同）--- +PROBE_TEXT_MINING_SYSTEM = """你是用户研究与文本挖掘方向的助手。 + +输入 JSON 为「第八章第二节评论文本补充分析」的**专用**结果（``schema_version``=1），**不是**正式竞品报告里的关注词规则统计、也不是已废弃的星级子集预设口语短语 Lexicon。其中的数字与词表来自**中文分词 + 统计工具**（词频、关键词突出度、共现、主题归纳），与业务侧子串计数**口径不同**。 + +每个 ``groups`` 元素含： +- ``probe_status``：``ok`` 表示该细类已完成分词与统计；``skipped`` 表示样本过少等未下钻。 +- ``word_freq_top`` / ``tfidf_top`` / ``cooccurrence_top``：统计型特征（开放词表）。 +- ``lda``：无监督自动归纳的主题词；**仅为探索**，同一词可出现在多主题，**禁止**当作严格品类或固定标签。 +- ``sample_text_snippets``：与正式报告管线同源的评价短摘录（若有），仅用于**对照语境**；若与关键词突出度焦点冲突，以**整句原文**为准，并在段末或「使用注意」中可点明「统计与语义可能不一致」。 + +**任务**：对 ``groups`` 中**每一项**输出对应 Markdown（**顺序与输入一致**）： +- 对 ``probe_status == "ok"``：以 ``#### `` + 与该条 ``group`` 字段**完全一致**的细类名作为小节标题（勿用 ``##`` 一级标题）；每段约 **100～260 字**。 +- 内容须包含：①用 **1～2 句**概括该细类评论**主要讨论焦点**（综合词频与关键词突出度，**不要罗列具体数字**）；② **1～2 句**说明共现词对**暗示**哪些维度常一起出现（**非因果**）；③若 ``lda.topics`` 非空，**1～2 句**说明主题粗分侧重点，并**明确**算法无监督、**不**与矩阵细类一一对应；④用 **1～2 句**体现 ``sample_text_snippets`` 中的**用户语气与关切**（以**转述**为主）；若必须引用原文，**全小节合计**仅 **一处**极短引号内容（**≤40 字**，**不要**输出 ``【细类…SKU…店铺…】`` 等长前缀）；若无可用摘录则写明；⑤ **使用场景（仅从评论推断）**：用 **0～2 句**概括**何时、何地、何人、如何搭配**等（如早餐、加餐、控糖人群、配牛奶等）——**只能**依据本细类 ``word_freq``/``tfidf``/``cooccurrence``/``lda`` 与摘录中**已出现或可合理概括**的信息；**禁止**套用正式报告「场景分组」或其它外部场景分类；若统计与摘录中**均无**场景线索，**一句**写明「评论中未体现清晰使用场景」即可。 +- 对 ``probe_status == "skipped"``：该小节仅 **一句**说明原因。 + +**禁止**：编造数据中未出现的品牌、价格、医学功效或疗效承诺；不要把 ``keyword`` 监测词写进「用户原话」；不要输出 Markdown 表格；不要声称本段与「正式报告第八章末」完全同源——本任务为**补充分析解读**。**禁止**把输入里的 ``sample_text_snippets`` **逐条罗列**、**多条整段复制**到输出（那不是归纳，是重复贴评论）。 + +全文末可另起一段 **「使用注意」**（简短）：点明开放词表统计与人工阅读差异、主题归纳局限、小样本细类不可靠。 + +总字数约 **800～4500 字**（细类多则偏长）。仅输出正文 Markdown，不要用代码围栏包裹全文。""" + +PROBE_TEXT_MINING_USER_PREFIX = ( + "请根据以下 JSON 撰写「评论文本补充分析」解读正文（Markdown）。\n\n" +) + + +def _is_noise_token(w: str) -> bool: + if len(w) < 2: + return True + if w in ("ldquo", "rdquo", "nbsp", "mdash"): + return True + if re.match(r"^[a-z]{1,8}$", w) and w not in ("gi",): + return True + return False + + +def _word_freq_from_cut_docs(cut_docs: list[str]) -> dict[str, int]: + c: Counter[str] = Counter() + for line in cut_docs: + for w in line.split(): + if _is_noise_token(w): + continue + c[w] += 1 + return dict(c) + + +def _font_path_chinese() -> str | None: + windir = os.environ.get("WINDIR", r"C:\Windows") + candidates = [ + Path(windir) / "Fonts" / "msyh.ttc", + Path(windir) / "Fonts" / "msyhbd.ttc", + Path(windir) / "Fonts" / "simhei.ttf", + Path(windir) / "Fonts" / "simsun.ttc", + Path("/usr/share/fonts/truetype/wqy/wqy-microhei.ttc"), + Path("/usr/share/fonts/truetype/noto/NotoSansCJK-Regular.ttc"), + ] + for p in candidates: + try: + if p.is_file(): + return str(p) + except OSError: + continue + return None + + +def _slug_safe(gname: str) -> str: + s = re.sub(r'[<>:"/\\|?*]', "_", (gname or "").strip()) + return s[:80] if len(s) > 80 else s + + +def _save_wordcloud_png( + freq: dict[str, int], + out_path: Path, + *, + font_path: str | None, +) -> str: + """写入 PNG；成功返回空串，失败返回错误说明。""" + if not _WORDCLOUD_AVAILABLE or WordCloud is None or plt is None: + return "wordcloud/matplotlib 未安装" + if not freq: + return "词频为空" + try: + out_path.parent.mkdir(parents=True, exist_ok=True) + wc = WordCloud( + font_path=font_path, + width=960, + height=540, + background_color="white", + max_words=220, + relative_scaling=0.35, + colormap="viridis", + prefer_horizontal=0.88, + min_font_size=10, + ).generate_from_frequencies(freq) + fig, ax = plt.subplots(figsize=(10.5, 6), dpi=120) + ax.imshow(wc, interpolation="bilinear") + ax.axis("off") + fig.tight_layout(pad=0) + fig.savefig(out_path, bbox_inches="tight", facecolor="white") + plt.close(fig) + except Exception as e: + return str(e) + return "" + + +def _load_run( + run_dir: Path, +) -> tuple[str, list[dict[str, str]], list[dict[str, str]]]: + run_dir = run_dir.resolve() + merged_path = run_dir / kpl.FILE_MERGED_CSV + if not merged_path.is_file(): + raise FileNotFoundError(f"缺少合并表: {merged_path}") + _, merged_rows = jcr._read_csv_rows(merged_path) + _, comment_rows = jcr._read_csv_rows(run_dir / kpl.FILE_COMMENTS_FLAT_CSV) + meta_path = run_dir / kpl.FILE_RUN_META_JSON + meta: dict[str, Any] | None = None + if meta_path.is_file(): + try: + meta = json.loads(meta_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + meta = None + kw = "" + if meta and str(meta.get("keyword") or "").strip(): + kw = str(meta.get("keyword")).strip() + return kw, merged_rows, comment_rows + + +def _cut_one(text: str) -> list[str]: + s = (text or "").strip() + if not s: + return [] + raw = jieba.lcut(s) + out: list[str] = [] + for w in raw: + w = w.strip() + if len(w) < 2: + continue + if w in _STOP_BASIC: + continue + if re.match(r"^[0-9\s\W_]+$", w): + continue + out.append(w) + return out + + +def _docs_cut(texts: list[str]) -> list[str]: + """每条评论 → 空格连接的分词串，供 sklearn。""" + rows: list[str] = [] + for t in texts: + toks = _cut_one(t) + if toks: + rows.append(" ".join(toks)) + return rows + + +def _term_freq_top(cut_docs: list[str], top_k: int) -> list[tuple[str, int]]: + c: Counter[str] = Counter() + for line in cut_docs: + for w in line.split(): + c[w] += 1 + return c.most_common(top_k) + + +def _tfidf_top(cut_docs: list[str], top_k: int) -> list[tuple[str, float]]: + if not cut_docs: + return [] + vec = TfidfVectorizer(max_features=min(2000, max(50, len(cut_docs) * 3))) + try: + X = vec.fit_transform(cut_docs) + except ValueError: + return [] + feats = np.array(vec.get_feature_names_out()) + scores = np.asarray(X.mean(axis=0)).ravel() + idx = np.argsort(-scores)[:top_k] + return [(str(feats[i]), float(scores[i])) for i in idx] + + +def _cooc_top_pairs( + cut_docs: list[str], + vocab_cap: int, + pair_top: int, +) -> list[tuple[str, str, int]]: + """同一条评论内无序词对共现（过滤低频词）。""" + term_freq = Counter() + for line in cut_docs: + for w in set(line.split()): + term_freq[w] += 1 + top_terms = {w for w, _ in term_freq.most_common(vocab_cap)} + pair_c: Counter[tuple[str, str]] = Counter() + for line in cut_docs: + toks = sorted({w for w in line.split() if w in top_terms}) + for i in range(len(toks)): + for j in range(i + 1, len(toks)): + a, b = toks[i], toks[j] + if a > b: + a, b = b, a + pair_c[(a, b)] += 1 + return [(a, b, n) for (a, b), n in pair_c.most_common(pair_top)] + + +def _lda_topics( + cut_docs: list[str], + n_topics: int, + n_top_words: int, +) -> tuple[list[list[str]], str]: + if len(cut_docs) < 4: + return [], "文本条数过少，跳过 LDA。" + n_topics = max(2, min(n_topics, len(cut_docs) // 2)) + try: + vec = CountVectorizer(max_df=0.95, min_df=2, max_features=800) + X = vec.fit_transform(cut_docs) + except ValueError as e: + return [], f"LDA 向量化失败：{e}" + if X.shape[0] < 3 or X.shape[1] < 3: + return [], "矩阵过稀疏，跳过 LDA。" + lda = LatentDirichletAllocation( + n_components=n_topics, + max_iter=30, + learning_method="batch", + random_state=42, + n_jobs=1, + ) + try: + lda.fit(X) + except Exception as e: + return [], f"LDA 拟合失败：{e}" + names = vec.get_feature_names_out() + topics: list[list[str]] = [] + for topic_idx, topic in enumerate(lda.components_): + top_ix = np.argsort(-topic)[:n_top_words] + topics.append([str(names[i]) for i in top_ix]) + return topics, "" + + +def _md_escape(s: str) -> str: + return (s or "").replace("|", "\\|").replace("\n", " ") + + +def _narrative_stub( + n_raw: int, + n_cut: int, + tf_top: list[tuple[str, int]], + tfidf_top: list[tuple[str, float]], + cooc: list[tuple[str, str, int]], + lda_topics: list[list[str]], + lda_note: str, +) -> str: + lines: list[str] = [ + f"本细类有效评论约 **{n_cut}** 条（原始非空 **{n_raw}** 条，经分词去停用后用于建模）。", + "", + "**词频 Top**：" + + ( + "、".join(f"「{w}」({n})" for w, n in tf_top[:12]) + if tf_top + else "（无）" + ) + + "。", + "", + "**关键词突出度 Top**（相对区分度）：" + + ( + "、".join(f"「{w}」({s:.3f})" for w, s in tfidf_top[:12]) + if tfidf_top + else "（无）" + ) + + "。", + "", + "**共现较强的词对**（同条评论内，供联想维度）：" + + ( + "；".join(f"「{a}」-「{b}」({c})" for a, b, c in cooc[:10]) + if cooc + else "（无）" + ) + + "。", + "", + ] + if lda_note: + lines.append(f"*主题归纳：{lda_note}*") + lines.append("") + elif lda_topics: + lines.append("**自动归纳的主题（无监督，仅作探索）**：") + for i, words in enumerate(lda_topics): + lines.append(f"- 主题 {i + 1}：{'、'.join(words)}") + lines.append("") + lines.append( + "> 以上为主题探索与统计摘要，**不等同**于业务结论；若与星级、规则词表冲突，以人工抽样为准。" + ) + return "\n".join(lines) + + +def _truncate_probe_payload(payload: dict[str, Any]) -> dict[str, Any]: + """压缩摘录长度，避免单次 JSON 顶满上下文。""" + out: dict[str, Any] = dict(payload) + groups: list[Any] = [] + for g in (out.get("groups") or []): + if not isinstance(g, dict): + groups.append(g) + continue + g2 = dict(g) + g2.pop("focus_hit_lines", None) + sn = g2.get("sample_text_snippets") + if isinstance(sn, list): + # 条数略减，降低模型「照抄罗列」倾向；仍以转述为主见系统提示 + g2["sample_text_snippets"] = [str(x)[:200] for x in sn[:6]] + groups.append(g2) + out["groups"] = groups + return out + + +def _merge_snippets_from_comment_groups( + probe_rows: list[dict[str, Any]], + *, + merged_rows: list[dict[str, str]], + comment_rows: list[dict[str, str]], +) -> None: + """把正式 ``build_comment_groups_llm_payload`` 中的摘录并入补充分析行（原地修改）。""" + sku_h = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + title_h = MERGED_FIELD_TO_CSV_HEADER["title"] + fb = jcr._consumer_feedback_by_matrix_group( + merged_rows=merged_rows, + comment_rows=comment_rows, + sku_header=sku_h, + ) + pl = jcr.build_comment_groups_llm_payload( + feedback_groups=fb, + merged_rows=merged_rows, + sku_header=sku_h, + title_h=title_h, + ) + by_g = {str(x.get("group")): x for x in pl if isinstance(x, dict)} + for row in probe_rows: + if row.get("probe_status") != "ok": + continue + gname = str(row.get("group") or "") + src = by_g.get(gname) + if not src: + continue + row["comment_flat_rows"] = src.get("comment_flat_rows") + sn = src.get("sample_text_snippets") + if isinstance(sn, list): + row["sample_text_snippets"] = [str(x)[:220] for x in sn[:8]] + + +def _run_probe_text_mining_llm( + payload: dict[str, Any], + *, + chunked: bool, +) -> str: + """补充分析专用：``PROBE_TEXT_MINING_SYSTEM`` + 结构化 JSON；可选按细类拆分调用。""" + if not payload.get("groups"): + return "> **补充分析 LLM 解读**：无分组数据，跳过。" + try: + if not chunked: + p = _truncate_probe_payload(payload) + raw = json.dumps(p, ensure_ascii=False) + if len(raw) > 88_000: + raw = ( + raw[:82_000] + + "\n\n…（JSON 过长已截断，仅依据可见字段撰写。）\n" + ) + return _call_llm( + PROBE_TEXT_MINING_SYSTEM, + PROBE_TEXT_MINING_USER_PREFIX + raw, + ).strip() + kw = str(payload.get("keyword") or "") + note = str(payload.get("probe_note") or "") + parts: list[str] = [] + for g in payload.get("groups") or []: + if not isinstance(g, dict): + continue + gname = str(g.get("group") or "?") + if g.get("probe_status") != "ok": + parts.append( + f"#### {gname}\n\n" + f"*（评论量不足，未做词云与主题归纳：{g.get('reason', '')}）*" + ) + continue + mini = { + "schema_version": payload.get("schema_version", 1), + "keyword": kw, + "probe_note": note, + "groups": [g], + } + raw = json.dumps(mini, ensure_ascii=False) + if len(raw) > 48_000: + raw = raw[:44_000] + "\n…\n" + parts.append( + _call_llm( + PROBE_TEXT_MINING_SYSTEM, + PROBE_TEXT_MINING_USER_PREFIX + raw, + ).strip() + ) + return "\n\n---\n\n".join(parts) + except Exception as e: + return f"> **补充分析 LLM 解读**调用失败：{e}" + + +def _ensure_probe_dependencies() -> None: + if not _PROBE_TEXT_MINING_DEPS_OK: + raise ImportError( + "第八章评论文本补充分析依赖未安装，请在 backend 环境下执行：" + "pip install jieba scikit-learn numpy wordcloud\n" + f"原始错误: {_PROBE_TEXT_MINING_IMPORT_ERROR}" + ) + + +def build_markdown( + run_dir: Path, + *, + min_texts: int, + lda_topics_n: int, + top_k_words: int, + cooc_vocab: int, + cooc_pairs: int, + live_llm: bool, + llm_chunked: bool, + wordcloud_enabled: bool, + wordcloud_max: int, +) -> str: + _ensure_probe_dependencies() + kw, merged, comments = _load_run(run_dir) + sku_h = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + groups = jcr._consumer_feedback_by_matrix_group( + merged_rows=merged, + comment_rows=comments, + sku_header=sku_h, + ) + + lines: list[str] = [ + "# 八、消费者反馈与用户画像（评论文本补充分析 · 实验稿）", + "", + f"- **运行目录**：`{run_dir}`", + f"- **监测词（run_meta）**：{kw or '—'}", + f"- **生成脚本**：`pipeline.demos.chapter8_text_mining_probe`", + "", + "## 8.0 说明", + "", + "本稿为**独立补充分析**，流程为：清洗评论 → 词云（可选）→ 词频与关键词 → 词对共现 → 主题归纳 → 文字小结；" + "文末可选用**专用提示词**由大模型解读（与已废弃的预设口语短语情感口径不同）。" + "**细类划分与 SKU 归因**与主报告一致；其余为中文分词与统计工具做的开放词表分析，**不替代**正式报告中的规则统计。", + "", + "---", + "", + ] + if wordcloud_enabled and not _WORDCLOUD_AVAILABLE: + lines.extend( + [ + "> **词云**：当前环境未安装 ``wordcloud`` / ``matplotlib``，已跳过出图。" + "请执行：``pip install wordcloud matplotlib``。", + "", + ] + ) + elif wordcloud_enabled and not _font_path_chinese(): + lines.extend( + [ + "> **词云字体**：未检测到常见中文字体路径，词云可能出现方框；" + "Windows 可确认 ``C:\\Windows\\Fonts\\msyh.ttc`` 是否存在。", + "", + ] + ) + + wc_n = 0 + probe_rows: list[dict[str, Any]] = [] + for gname, _cr_rows, texts in groups: + n_raw = len([t for t in texts if (t or "").strip()]) + if n_raw < min_texts: + probe_rows.append( + { + "group": gname, + "probe_status": "skipped", + "reason": f"有效文本 {n_raw} 条，低于 min_texts={min_texts}", + } + ) + lines.extend( + [ + f"## {gname}", + "", + f"*本细类有效评论仅 {n_raw} 条，低于分析所需最少条数（{min_texts} 条），故未生成词云与主题图。*", + "", + "---", + "", + ] + ) + continue + + cut_docs = _docs_cut(texts) + if len(cut_docs) < 2: + probe_rows.append( + { + "group": gname, + "probe_status": "skipped", + "reason": "分词后不足 2 条", + } + ) + lines.extend( + [ + f"## {gname}", + "", + "*分词后不足 2 条，跳过。*", + "", + "---", + "", + ] + ) + continue + + tf_top = _term_freq_top(cut_docs, top_k_words) + tfidf_top = _tfidf_top(cut_docs, top_k_words) + cooc = _cooc_top_pairs(cut_docs, cooc_vocab, cooc_pairs) + lda_t, lda_err = _lda_topics(cut_docs, lda_topics_n, 12) + if (lda_err or "").strip(): + lda_obj: dict[str, Any] = { + "status": "skipped", + "reason": lda_err.strip(), + } + else: + lda_obj = {"status": "ok", "topics": lda_t} + probe_rows.append( + { + "group": gname, + "probe_status": "ok", + "comment_text_units": n_raw, + "jieba_cut_document_count": len(cut_docs), + "word_freq_top": [ + {"term": w, "count": int(n)} for w, n in tf_top[:20] + ], + "tfidf_top": [ + {"term": w, "score": round(float(s), 4)} + for w, s in tfidf_top[:20] + ], + "cooccurrence_top": [ + {"term_a": a, "term_b": b, "joint_count": int(c)} + for a, b, c in cooc[:15] + ], + "lda": lda_obj, + } + ) + + lines.extend([f"## {gname}", ""]) + if ( + wordcloud_enabled + and _WORDCLOUD_AVAILABLE + and wc_n < wordcloud_max + ): + freq_wc = _word_freq_from_cut_docs(cut_docs) + fn = f"wordcloud_probe__{wc_n:02d}_{_slug_safe(gname)}.png" + img_path = run_dir.resolve() / "report_assets" / fn + err_wc = _save_wordcloud_png( + freq_wc, + img_path, + font_path=_font_path_chinese(), + ) + if not err_wc: + lines.append( + f"![词云（按词频权重）](report_assets/{fn})" + ) + lines.append("") + wc_n += 1 + else: + lines.append(f"> 词云未生成：{err_wc}") + lines.append("") + lines.append(_narrative_stub( + n_raw, + len(cut_docs), + tf_top, + tfidf_top, + cooc, + lda_t, + lda_err, + )) + lines.extend(["", "---", ""]) + + _merge_snippets_from_comment_groups( + probe_rows, + merged_rows=merged, + comment_rows=comments, + ) + llm_payload: dict[str, Any] = { + "schema_version": 1, + "keyword": kw, + "probe_note": ( + "中文分词 + 停用词；关键词突出度/共现/主题归纳为统计库；" + "与正式报告第八章第二节图表中的关注词计数等规则统计、已废弃的预设口语短语情感口径均不同；" + "评价短摘录合并自 build_comment_groups_llm_payload（与正式管线同源，不含关注词子串计数摘要），供语境对照；" + "「使用场景」若出现，须仅能从本 JSON 内统计与摘录推断，不接入正式场景分组。" + ), + "groups": probe_rows, + } + + lines.extend( + [ + "", + "---", + "", + "## 评论文本归纳（大模型 · 专用口径）", + "", + "> 输入为按细类整理后的词频、关键词、共现与主题等统计结果，以及少量原文摘录（供理解语境，**不是**要求逐条复述）；" + "归纳中的**使用场景**仅能从上述统计推断，**不等同**于正式的「场景分组」章节。", + "", + ] + ) + if live_llm: + lines.append(_run_probe_text_mining_llm(llm_payload, chunked=llm_chunked)) + else: + lines.append( + "> **补充分析 LLM 解读**：未启用。请使用 ``--live-llm``（需 ``AI_crawler`` 等可用）；" + "细类很多、单次 JSON 易超长时可加 ``--llm-chunked``（按细类多次调用后拼接）。" + ) + + lines.append("") + lines.append("*（完）*") + return "\n".join(lines) + + +def markdown_embed_body_for_competitor_report(full_probe_md: str) -> str: + """ + 将独立补充分析稿转为可嵌入 ``build_competitor_markdown`` 的 **第八章第二节正文**（不含 ``### 8.2`` 标题行；由 ``jd_report`` 统一加标题）： + 从 ``## 8.0 说明`` 起至文末，并把 ``## …`` 降为 ``#### …``，避免与宿主 ``## 八、`` 冲突。 + """ + lines = (full_probe_md or "").splitlines() + try: + start = next( + i + for i, ln in enumerate(lines) + if ln.strip() == "## 8.0 说明" or ln.strip().startswith("## 8.0 说明") + ) + except StopIteration: + return (full_probe_md or "").strip() + chunk = lines[start:] + out: list[str] = [] + for ln in chunk: + if ln.startswith("## ") and not ln.startswith("###"): + out.append("#### " + ln[3:]) + else: + out.append(ln) + while out and out[-1].strip() in ("*（完）*", ""): + out.pop() + while out and not out[-1].strip(): + out.pop() + return "\n".join(out).strip() + + +def main() -> None: + if not _PROBE_TEXT_MINING_DEPS_OK: + print( + "缺少依赖，请先安装：pip install jieba scikit-learn numpy wordcloud\n" + f"原始错误: {_PROBE_TEXT_MINING_IMPORT_ERROR}", + file=sys.stderr, + ) + sys.exit(1) + ap = argparse.ArgumentParser(description="第八章评论文本补充分析（独立脚本）") + ap.add_argument( + "--run-dir", + type=Path, + required=True, + help="pipeline_runs 下某批次目录（含 keyword_pipeline_merged.csv、comments_flat.csv）", + ) + ap.add_argument( + "--out", + type=Path, + default=None, + help="输出 Markdown 路径（默认 <run_dir>/chapter8_text_mining_probe.md）", + ) + ap.add_argument("--min-texts", type=int, default=8, help="细类最少评论条数才分析") + ap.add_argument("--lda-topics", type=int, default=4, help="自动主题归纳条数上限（会按样本量裁剪）") + ap.add_argument("--top-k-words", type=int, default=30, help="词频/关键词突出度展示长度") + ap.add_argument("--cooc-vocab", type=int, default=80, help="共现矩阵保留的高频词数") + ap.add_argument("--cooc-pairs", type=int, default=25, help="输出词对数量") + ap.add_argument( + "--live-llm", + action="store_true", + help="文末调用补充分析专用 LLM（PROBE_TEXT_MINING_SYSTEM + 结构化 JSON；需 AI_crawler 等）", + ) + ap.add_argument( + "--llm-chunked", + action="store_true", + help="按细类拆分多次调用同一补充分析提示词，防单次 JSON 过长", + ) + ap.add_argument( + "--no-wordcloud", + action="store_true", + help="不生成词云 PNG（默认生成，需 wordcloud+matplotlib）", + ) + ap.add_argument( + "--wordcloud-max", + type=int, + default=40, + help="最多为多少个细类各出一张词云（防文件过多）", + ) + args = ap.parse_args() + + md = build_markdown( + args.run_dir, + min_texts=args.min_texts, + lda_topics_n=args.lda_topics, + top_k_words=args.top_k_words, + cooc_vocab=args.cooc_vocab, + cooc_pairs=args.cooc_pairs, + live_llm=args.live_llm, + llm_chunked=args.llm_chunked, + wordcloud_enabled=not args.no_wordcloud, + wordcloud_max=max(0, args.wordcloud_max), + ) + out = args.out or (args.run_dir.resolve() / "chapter8_text_mining_probe.md") + out.write_text(md, encoding="utf-8") + print(f"已写入: {out}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/backend/pipeline/demos/dump_strategy_llm_input_md.py b/backend/pipeline/demos/dump_strategy_llm_input_md.py new file mode 100644 index 0000000..1bc775a --- /dev/null +++ b/backend/pipeline/demos/dump_strategy_llm_input_md.py @@ -0,0 +1,352 @@ +""" +导出「独立策略稿 · 大模型润色」一次调用与生产一致的完整入参（不请求网关）。 + +与 ``generate_strategy_draft_markdown_llm`` / ``resolve_strategy_draft_llm_input_snapshot`` +使用相同的截断阶梯与 ``payload`` 字段（含 ``strategy_decisions_substantive``）。 + +用法（在 backend 目录）:: + + # 按数据库任务（默认取最近成功任务；可指定 job-id） + python -m pipeline.demos.dump_strategy_llm_input_md [--job-id 12] [--matrix-index 0] + + # 仅磁盘 run_dir（无需 PipelineJob；job_id 写 0 进 JSON，仅影响底稿抬头占位） + python -m pipeline.demos.dump_strategy_llm_input_md --run-dir \"D:/.../pipeline_runs/某批次\" + + # 与线上一致：从文件载入当时提交的 strategy_decisions + python -m pipeline.demos.dump_strategy_llm_input_md --run-dir \"...\" --decisions-json decisions.json + + # 指定输出 + python -m pipeline.demos.dump_strategy_llm_input_md --run-dir \"...\" -o path/to/snap.md + + # 只打印摘要、不写文件 + python -m pipeline.demos.dump_strategy_llm_input_md --run-dir \"...\" --no-md +""" +from __future__ import annotations + +import argparse +import json +import os +import sys +from pathlib import Path +from typing import Any + +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings") + + +def _strategy_decisions_empty() -> dict[str, Any]: + return { + "product_role": "", + "stage_goal_type": "", + "time_horizon": "", + "success_criteria": "", + "non_goals": "", + "battlefield_one_line": "", + "positioning_choice": "", + "competitive_stance": "", + "pillar_product": "", + "pillar_price": "", + "pillar_channel": "", + "pillar_comm": "", + "audience_segment": "", + "competitor_reference": "", + "resource_notes": "", + "marketing_strategy": "", + "general_strategy": "", + "ack_risk_keywords": False, + "ack_risk_price": False, + "ack_risk_concentration": False, + } + + +def _merge_decisions(base: dict[str, Any], overlay: dict[str, Any] | None) -> dict[str, Any]: + out = dict(base) + if isinstance(overlay, dict): + out.update(overlay) + return out + + +def main() -> int: + import django + + django.setup() + + from django.utils import timezone + + from pipeline.jd.runner import build_competitor_brief_for_job + from pipeline.llm.generate_strategy import ( + STRATEGY_SYSTEM, + resolve_strategy_draft_llm_input_snapshot, + _min_strategy_completion_tokens, + ) + from pipeline.llm.llm_client import estimate_chat_input_tokens + from pipeline.models import JobStatus, PipelineJob + from pipeline.reporting.brief_strategy_scope import ( + filter_brief_for_strategy_matrix_group, + list_matrix_groups_for_api, + ) + from pipeline.reporting.report_matrix_group_evidence import ( + load_report_matrix_group_evidence_markdown, + ) + from pipeline.reporting.report_strategy_excerpt import load_report_strategy_excerpt + + p = argparse.ArgumentParser(description=__doc__) + src = p.add_mutually_exclusive_group() + src.add_argument("--job-id", type=int, default=None, help="PipelineJob 主键") + src.add_argument( + "--run-dir", + type=Path, + default=None, + help="运行目录（与 job.run_dir 相同结构；与 --job-id 二选一）", + ) + p.add_argument( + "--matrix-index", + type=int, + default=0, + help="矩阵分组下标；设为 -1 表示不收窄（全部分类）", + ) + p.add_argument( + "--no-scope", + action="store_true", + help="与 --matrix-index -1 相同：不收窄 brief、不抽细类报告节选", + ) + p.add_argument( + "--decisions-json", + type=Path, + default=None, + help="覆盖 strategy_decisions 的 JSON 对象文件（与线上一致时传入）", + ) + p.add_argument( + "--business-notes", + type=str, + default="", + help="与接口 business_notes 一致的业务备注", + ) + p.add_argument( + "--snapshot-job-id", + type=int, + default=None, + help="写入 payload.job_id（仅 --run-dir 时有效；默认 0）", + ) + p.add_argument( + "-o", + "--output", + type=Path, + default=None, + help="输出 .md 路径（默认：run_dir/strategy_draft_llm_input_snapshot.md 或 docs/planning/…）", + ) + p.add_argument( + "--no-md", + action="store_true", + help="不写入 Markdown，仅打印控制台摘要", + ) + args = p.parse_args() + + backend_dir = Path(__file__).resolve().parents[2] + repo_root = backend_dir.parent + default_docs_out = ( + repo_root / "docs" / "planning" / "策略生成-LLM全量输入快照.md" + ) + + run_dir_s: str + kw: str + rc: dict[str, Any] | None + job_id: int + + if args.run_dir is not None: + run_dir_p = args.run_dir.expanduser().resolve() + if not run_dir_p.is_dir(): + print(f"run_dir 不存在: {run_dir_p}", file=sys.stderr) + return 1 + rc_path = run_dir_p / "effective_report_config.json" + meta_path = run_dir_p / "run_meta.json" + if not rc_path.is_file() or not meta_path.is_file(): + print("缺少 effective_report_config.json 或 run_meta.json", file=sys.stderr) + return 1 + rc = json.loads(rc_path.read_text(encoding="utf-8")) + meta = json.loads(meta_path.read_text(encoding="utf-8")) + kw = (meta.get("keyword") or "").strip() + if not kw: + print("run_meta 无 keyword", file=sys.stderr) + return 1 + run_dir_s = str(run_dir_p) + job_id = int(args.snapshot_job_id) if args.snapshot_job_id is not None else 0 + else: + jid = args.job_id + if jid: + job = PipelineJob.objects.filter(pk=jid).first() + else: + job = ( + PipelineJob.objects.filter(status=JobStatus.SUCCESS) + .exclude(run_dir="") + .order_by("-id") + .first() + ) + if not job: + print("无可用任务：请指定 --job-id 或 --run-dir", file=sys.stderr) + return 1 + run_dir_s = job.run_dir + kw = job.keyword + rc = job.report_config if isinstance(job.report_config, dict) else None + job_id = job.id + + brief = build_competitor_brief_for_job( + run_dir_s, + kw, + report_config=rc, + ) + matrix_groups = list_matrix_groups_for_api(brief) + group_names = [g.get("group") for g in matrix_groups if isinstance(g, dict)] + scoped_label = "" + matrix_index: int | None = args.matrix_index + if args.no_scope: + matrix_index = -1 + if matrix_index is not None and matrix_index >= 0: + mg = brief.get("matrix_by_group") + if isinstance(mg, list) and matrix_index < len(mg): + scoped_label = (mg[matrix_index].get("group") or "").strip() + brief = filter_brief_for_strategy_matrix_group( + brief, matrix_group_index=matrix_index + ) + else: + print(f"matrix_index {matrix_index} 超出范围", file=sys.stderr) + return 1 + + gen_at = timezone.now().isoformat() + sd = _strategy_decisions_empty() + if args.decisions_json is not None: + dp = args.decisions_json.expanduser().resolve() + if not dp.is_file(): + print(f"decisions-json 不存在: {dp}", file=sys.stderr) + return 1 + try: + loaded = json.loads(dp.read_text(encoding="utf-8")) + except json.JSONDecodeError as e: + print(f"decisions-json 非合法 JSON: {e}", file=sys.stderr) + return 1 + if not isinstance(loaded, dict): + print("decisions-json 根须为 JSON 对象", file=sys.stderr) + return 1 + sd = _merge_decisions(sd, loaded) + + excerpt_raw, excerpt_src = load_report_strategy_excerpt(run_dir_s) + excerpt_raw = (excerpt_raw or "").strip() + + evidence_md = "" + evidence_src = "none" + if scoped_label: + evidence_md, evidence_src = load_report_matrix_group_evidence_markdown( + run_dir_s, + scoped_label, + ) + + payload, user_body, tier_note = resolve_strategy_draft_llm_input_snapshot( + job_id=job_id, + keyword=kw, + brief=brief, + business_notes=(args.business_notes or "").strip(), + generated_at_iso=gen_at, + strategy_decisions=sd, + report_strategy_excerpt=excerpt_raw or None, + report_matrix_group_evidence_md=evidence_md.strip() or None, + report_config=rc, + ) + + min_comp = _min_strategy_completion_tokens() + est_in = estimate_chat_input_tokens(STRATEGY_SYSTEM, user_body) + full_chars = len(STRATEGY_SYSTEM) + len(user_body) + rd = payload.get("rules_draft_markdown") + rd_len = len(rd) if isinstance(rd, str) else 0 + sb = payload.get("structured_brief") + sb_json_len = len(json.dumps(sb, ensure_ascii=False)) if sb else 0 + + print("run_dir:", run_dir_s) + print("job_id (payload):", job_id) + print("keyword:", kw) + print("tier:", tier_note) + print("MA_STRATEGY_MIN_COMPLETION_TOKENS:", min_comp) + print("strategy_decisions_substantive:", payload.get("strategy_decisions_substantive")) + print("matrix scope:", f"{matrix_index} → 「{scoped_label}」" if scoped_label else "未收窄") + print("report_strategy_excerpt:", excerpt_src, "raw chars:", len(excerpt_raw)) + print("report_matrix_group_evidence:", evidence_src, "chars in payload:", len(payload.get("report_matrix_group_evidence_md") or "")) + print("STRATEGY_SYSTEM chars:", len(STRATEGY_SYSTEM)) + print("user chars:", len(user_body)) + print("total chars:", full_chars) + print("estimate_chat_input_tokens:", est_in) + print("structured_brief JSON len:", sb_json_len) + print("rules_draft_markdown len (in payload):", rd_len) + + if args.no_md: + return 0 + + if args.output is not None: + out_path = args.output.expanduser().resolve() + elif args.run_dir is not None: + out_path = args.run_dir.expanduser().resolve() / "strategy_draft_llm_input_snapshot.md" + else: + out_path = default_docs_out + + out_path.parent.mkdir(parents=True, exist_ok=True) + lines: list[str] = [ + "# 独立策略稿 · 大模型一次调用的「全量输入」快照", + "", + "> **生成方式**：`pipeline.demos.dump_strategy_llm_input_md` 调用 " + "`resolve_strategy_draft_llm_input_snapshot`，与 ``generate_strategy_draft_markdown_llm`` " + "首档通过的 ``payload`` / ``user`` 一致（不请求网关）。", + "> **与线上一致**：将当时 POST 的 `strategy_decisions`、`business_notes`、`strategy_matrix_group_index` " + "与本脚本参数对齐即可复现。", + "", + "## 快照元数据", + "", + f"- **任务 ID（payload.job_id）**：{job_id}", + f"- **关键词**：{kw}", + f"- **run_dir**：`{run_dir_s}`", + f"- **矩阵分组**：{matrix_index if matrix_index is not None and matrix_index >= 0 else '未收窄（全部分类）'}{f' → 「{scoped_label}」' if scoped_label else ''}", + f"- **本任务可选细类（节选）**：{group_names[:20]}{'…' if len(group_names) > 20 else ''}", + f"- **选用档位**：{tier_note}", + f"- **strategy_decisions_substantive**：{payload.get('strategy_decisions_substantive')!r}", + f"- **第九章节选来源**：{excerpt_src}", + f"- **细类报告节选来源**：{evidence_src}", + f"- **MA_STRATEGY_MIN_COMPLETION_TOKENS**：{min_comp}", + f"- **System 字符数**：{len(STRATEGY_SYSTEM)}", + f"- **User 消息字符数**：{len(user_body)}", + f"- **合计约**：{full_chars} 字符", + f"- **estimate_chat_input_tokens（项目内启发式）**：{est_in}", + f"- **structured_brief 序列化长度**：{sb_json_len}", + f"- **rules_draft_markdown（payload 内）字符数**：{rd_len}", + "", + "---", + "", + "## 1. System 提示词（完整 `STRATEGY_SYSTEM`）", + "", + "```text", + STRATEGY_SYSTEM, + "```", + "", + "---", + "", + "## 2. User 消息（完整：`STRATEGY_USER_PREFIX` + JSON）", + "", + "以下为网关 **user** 角色一次发送的完整字符串（前缀 + 单行 JSON）。", + "", + "```text", + user_body, + "```", + "", + "---", + "", + "## 3. 同上 JSON 的排版版（便于人眼查看 `structured_brief` 结构）", + "", + "说明：若与第 2 节有任何不一致，以第 2 节（真实入参）为准。", + "", + "```json", + json.dumps(payload, ensure_ascii=False, indent=2), + "```", + "", + ] + out_path.write_text("\n".join(lines), encoding="utf-8") + print(f"Wrote {out_path} ({out_path.stat().st_size // 1024} KB)") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/pipeline/demos/run_price_groups_llm_demo.py b/backend/pipeline/demos/run_price_groups_llm_demo.py new file mode 100644 index 0000000..ba9a213 --- /dev/null +++ b/backend/pipeline/demos/run_price_groups_llm_demo.py @@ -0,0 +1,120 @@ +""" +细类价盘要点归纳：打印 ``generate_price_group_summaries_llm`` 输出（与报告 §6 后大模型段同源）。 + + cd backend + .venv\\Scripts\\python.exe -m pipeline.demos.run_price_groups_llm_demo --job 12 --live + .venv\\Scripts\\python.exe -m pipeline.demos.run_price_groups_llm_demo --merged "D:/path/keyword_pipeline_merged.csv" --live +""" +from __future__ import annotations + +import argparse +import json +import os +import sys +from pathlib import Path + +BACKEND_ROOT = Path(__file__).resolve().parents[2] +if str(BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(BACKEND_ROOT)) +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings") + +import django # noqa: E402 + +django.setup() + +JCR_ROOT = BACKEND_ROOT / "crawler_copy" / "jd_pc_search" +if str(JCR_ROOT) not in sys.path: + sys.path.insert(0, str(JCR_ROOT)) + +from pipeline.competitor_report import jd_report as jcr # noqa: E402 +import jd_keyword_pipeline as kpl # noqa: E402 + + +def main() -> None: + parser = argparse.ArgumentParser(description="价盘细类归纳 LLM demo") + parser.add_argument("--job", type=int, default=None, help="PipelineJob 主键，读 run_dir 下合并表") + parser.add_argument( + "--merged", + type=str, + default="", + help="keyword_pipeline_merged.csv 绝对或相对路径", + ) + parser.add_argument("--keyword", type=str, default="") + parser.add_argument( + "--live", + action="store_true", + help="调用真实大模型；否则只打印 payload 前两条摘要", + ) + parser.add_argument( + "--max-groups", + type=int, + default=0, + help="仅送前 N 个细类给模型（0 表示全部，大任务可设 5 试跑）", + ) + args = parser.parse_args() + + merged_rows: list[dict[str, str]] = [] + keyword = (args.keyword or "").strip() + + if args.merged: + mp = Path(args.merged).expanduser().resolve() + if not mp.is_file(): + print(f"合并表不存在: {mp}", file=sys.stderr) + sys.exit(1) + _, merged_rows = jcr._read_csv_rows(mp) + elif args.job is not None: + from pipeline.models import PipelineJob # noqa: WPS433 + + job = PipelineJob.objects.filter(pk=args.job).first() + if not job: + print(f"无此任务: {args.job}", file=sys.stderr) + sys.exit(1) + rd = (job.run_dir or "").strip() + if not rd: + print("任务无 run_dir", file=sys.stderr) + sys.exit(1) + run_dir = Path(rd).expanduser().resolve() + mp = run_dir / kpl.FILE_MERGED_CSV + if not mp.is_file(): + print(f"缺少合并表: {mp}", file=sys.stderr) + sys.exit(1) + _, merged_rows = jcr._read_csv_rows(mp) + if not keyword and (job.keyword or "").strip(): + keyword = str(job.keyword).strip() + else: + print("请指定 --job <id> 或 --merged <csv路径>", file=sys.stderr) + sys.exit(1) + + if not keyword: + keyword = "竞品监测" + + sku_h = "SKU(skuId)" + title_h = "标题(wareName)" + groups = jcr.build_price_groups_llm_payload( + merged_rows, title_h=title_h, sku_header=sku_h + ) + print(f"# payload: {len(groups)} 个细类, keyword={keyword}", file=sys.stderr) + if not groups: + print("build_price_groups_llm_payload 为空（合并表无行？）", file=sys.stderr) + sys.exit(1) + + if args.max_groups and args.max_groups > 0: + groups = groups[: args.max_groups] + print(f"# 截断为前 {len(groups)} 个细类", file=sys.stderr) + + if not args.live: + preview = json.dumps(groups[:2], ensure_ascii=False, indent=2) + print(preview[:6000]) + if len(preview) > 6000: + print("\n…") + print("\n加 --live 调用 generate_price_group_summaries_llm", file=sys.stderr) + return + + from pipeline.llm.generate import generate_price_group_summaries_llm # noqa: WPS433 + + out = generate_price_group_summaries_llm(groups, keyword=keyword) + print(out) + + +if __name__ == "__main__": + main() diff --git a/backend/pipeline/demos/run_report_llm_chapters_demo.py b/backend/pipeline/demos/run_report_llm_chapters_demo.py new file mode 100644 index 0000000..6ae68ae --- /dev/null +++ b/backend/pipeline/demos/run_report_llm_chapters_demo.py @@ -0,0 +1,369 @@ +""" +竞品报告中与大模型相关的块（与 ``pipeline.jd.runner.write_competitor_analysis_for_run_dir`` 同源）： + +- §5 后：``generate_matrix_group_summaries_llm`` +- §6 后：``generate_price_group_summaries_llm``、``generate_promo_group_summaries_llm`` +- （可选、默认关闭）``generate_comment_sentiment_analysis_llm``：按细类多次调用，写入报告 **8.3**（与探针 / 评论要点归纳并列） +- §8末细类评价：``generate_comment_group_summaries_llm`` +- §9 策略与机会：``generate_strategy_opportunities_llm``（``build_competitor_brief`` + 可选 ``chapter_llm_narratives`` 与各章归纳对齐） +- §8.5 类全文补充（独立长文）：``generate_competitor_report_markdown_llm`` + + cd backend + python -m pipeline.demos.run_report_llm_chapters_demo --run-dir "../data/JD/pipeline_runs/20260413_104252_低GI" + python -m pipeline.demos.run_report_llm_chapters_demo --run-dir "..." --live + python -m pipeline.demos.run_report_llm_chapters_demo --run-dir "..." --live --only matrix,price,promo +""" +from __future__ import annotations + +import argparse +import json +import os +import sys +import traceback +from pathlib import Path +from typing import Any, Callable + +BACKEND_ROOT = Path(__file__).resolve().parents[2] +if str(BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(BACKEND_ROOT)) +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings") + +import django # noqa: E402 + +django.setup() + +JCR_ROOT = BACKEND_ROOT / "crawler_copy" / "jd_pc_search" +if str(JCR_ROOT) not in sys.path: + sys.path.insert(0, str(JCR_ROOT)) + +from pipeline.competitor_report import jd_report as jcr # noqa: E402 +from pipeline.competitor_report.comment_sentiment import ( # noqa: E402 + build_comment_sentiment_llm_payload, +) +import jd_keyword_pipeline as kpl # noqa: E402 + +from pipeline.csv.schema import MERGED_FIELD_TO_CSV_HEADER # noqa: E402 +from pipeline.jd.runner import ( # noqa: E402 + get_default_report_config, + use_chunked_group_summaries_llm, +) + + +def _load_run( + run_dir: Path, +) -> tuple[ + str, + list[dict[str, str]], + list[dict[str, str]], + list[dict[str, str]], + dict[str, Any] | None, + dict[str, Any], +]: + run_dir = run_dir.resolve() + merged_path = run_dir / kpl.FILE_MERGED_CSV + if not merged_path.is_file(): + raise FileNotFoundError(f"缺少合并表: {merged_path}") + _, merged_rows = jcr._read_csv_rows(merged_path) + _, search_export_rows = jcr._read_csv_rows(run_dir / kpl.FILE_PC_SEARCH_CSV) + _, comment_rows = jcr._read_csv_rows(run_dir / kpl.FILE_COMMENTS_FLAT_CSV) + meta_path = run_dir / kpl.FILE_RUN_META_JSON + meta: dict[str, Any] | None = None + if meta_path.is_file(): + try: + meta = json.loads(meta_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + meta = None + eff_path = run_dir / "effective_report_config.json" + if eff_path.is_file(): + try: + eff_rc = json.loads(eff_path.read_text(encoding="utf-8")) + if not isinstance(eff_rc, dict): + eff_rc = get_default_report_config() + except json.JSONDecodeError: + eff_rc = get_default_report_config() + else: + eff_rc = get_default_report_config() + kw = "" + if meta and str(meta.get("keyword") or "").strip(): + kw = str(meta.get("keyword")).strip() + return kw, merged_rows, search_export_rows, comment_rows, meta, eff_rc + + +def _run_one( + name: str, + fn: Callable[[], str], + *, + live: bool, + preview_chars: int, +) -> None: + print(f"\n{'=' * 60}\n## {name}\n{'=' * 60}", flush=True) + if not live: + print("(dry-run：加 --live 将调用大模型)", flush=True) + return + try: + out = fn() + t = (out or "").strip() + print(f"ok，长度 {len(t)} 字符", flush=True) + if t: + head = t[:preview_chars] + print(head + ("…\n" if len(t) > preview_chars else "\n"), flush=True) + except Exception as e: + print(f"FAIL: {e}", flush=True) + traceback.print_exc() + + +def main() -> None: + parser = argparse.ArgumentParser(description="报告各块 LLM 串联试跑") + parser.add_argument( + "--run-dir", + type=str, + required=True, + help="流水线目录（含 keyword_pipeline_merged.csv）", + ) + parser.add_argument( + "--keyword", + type=str, + default="", + help="覆盖监测词（默认读 meta.keyword）", + ) + parser.add_argument("--live", action="store_true", help="真实调用大模型") + parser.add_argument( + "--only", + type=str, + default="", + help="逗号分隔子集：sentiment,matrix,price,promo,strategy_opp,comment_groups,report_supplement", + ) + parser.add_argument( + "--preview-chars", + type=int, + default=400, + help="--live 时每段打印前 N 字", + ) + args = parser.parse_args() + + run_dir = Path(args.run_dir).expanduser() + kw, merged, search_rows, comment_rows, meta, eff_rc = _load_run(run_dir) + keyword = (args.keyword or kw or "竞品监测").strip() + + only = {x.strip().lower() for x in args.only.split(",") if x.strip()} + all_names = { + "sentiment", + "matrix", + "price", + "promo", + "strategy_opp", + "comment_groups", + "report_supplement", + } + if not only: + only = all_names + + sku_h = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + title_h = MERGED_FIELD_TO_CSV_HEADER["title"] + + print( + f"# run_dir={run_dir}\n# keyword={keyword}\n" + f"# merged_rows={len(merged)} comments={len(comment_rows)} " + f"list={len(search_rows)}\n# only={only or all_names}", + flush=True, + ) + + from pipeline.llm.generate import ( # noqa: WPS433 + generate_comment_group_summaries_llm, + generate_comment_group_summaries_llm_chunked, + generate_comment_sentiment_analysis_llm, + generate_competitor_report_markdown_llm, + generate_matrix_group_summaries_llm, + generate_matrix_group_summaries_llm_chunked, + generate_price_group_summaries_llm, + generate_price_group_summaries_llm_chunked, + generate_promo_group_summaries_llm, + generate_promo_group_summaries_llm_chunked, + generate_strategy_opportunities_llm, + ) + + chunk_gr = use_chunked_group_summaries_llm(eff_rc) + + if "sentiment" in only: + comment_units, comment_scores = jcr._iter_comment_text_units_and_scores( + comment_rows, merged + ) + attr_units = jcr._comment_lines_with_product_context( + comment_rows, + merged, + sku_header=sku_h, + title_h=title_h, + ) + if len(attr_units) != len(comment_units): + attr_units = list(comment_units) + + def _sent() -> str: + pl = build_comment_sentiment_llm_payload( + comment_units, + scores=comment_scores, + attributed_texts=attr_units, + semantic_pool_max=40, + shuffle_seed=keyword, + ) + pl["keyword"] = keyword + return generate_comment_sentiment_analysis_llm(pl) + + _run_one( + "已弃用嵌入 · 评价情感 LLM 演示（llm_comment_sentiment）", + _sent, + live=args.live, + preview_chars=args.preview_chars, + ) + if not args.live: + print( + f" payload: comment_units={len(comment_units)} " + f"(需 >=2 条才会在生产流水线里调用)", + flush=True, + ) + + if "matrix" in only: + pl_mx = jcr.build_matrix_groups_llm_payload( + merged, sku_header=sku_h, title_h=title_h + ) + + def _mx() -> str: + if chunk_gr: + return generate_matrix_group_summaries_llm_chunked( + pl_mx, keyword=keyword + ) + return generate_matrix_group_summaries_llm(pl_mx, keyword=keyword) + + _run_one("§5 细类要点归纳（matrix）", _mx, live=args.live, preview_chars=args.preview_chars) + if not args.live: + print( + f" payload groups={len(pl_mx)} chunked_by_matrix={chunk_gr}", + flush=True, + ) + + if "price" in only: + pl_pr = jcr.build_price_groups_llm_payload( + merged, sku_header=sku_h, title_h=title_h + ) + + def _pr() -> str: + if chunk_gr: + return generate_price_group_summaries_llm_chunked( + pl_pr, keyword=keyword + ) + return generate_price_group_summaries_llm(pl_pr, keyword=keyword) + + _run_one("§6 细类价盘归纳（price）", _pr, live=args.live, preview_chars=args.preview_chars) + if not args.live: + print( + f" payload groups={len(pl_pr)} chunked_by_matrix={chunk_gr}", + flush=True, + ) + + if "promo" in only: + pl_po = jcr.build_promo_groups_llm_payload( + merged, sku_header=sku_h, title_h=title_h + ) + + def _po() -> str: + if chunk_gr: + return generate_promo_group_summaries_llm_chunked( + pl_po, keyword=keyword + ) + return generate_promo_group_summaries_llm(pl_po, keyword=keyword) + + _run_one( + "§6 细类促销与活动归纳（promo）", + _po, + live=args.live, + preview_chars=args.preview_chars, + ) + if not args.live: + print( + f" payload groups={len(pl_po)} chunked_by_matrix={chunk_gr}", + flush=True, + ) + + if "strategy_opp" in only: + brief = jcr.build_competitor_brief( + run_dir=run_dir, + keyword=keyword, + merged_rows=merged, + search_export_rows=search_rows, + comment_rows=comment_rows, + meta=meta, + report_config=eff_rc, + ) + + def _st() -> str: + return generate_strategy_opportunities_llm(brief, keyword=keyword) + + _run_one( + "§9 策略与机会（strategy_opp）", + _st, + live=args.live, + preview_chars=args.preview_chars, + ) + if not args.live: + print( + f" brief keys: {len(brief)} top-level fields", + flush=True, + ) + + if "comment_groups" in only: + fb = jcr._consumer_feedback_by_matrix_group( + merged_rows=merged, + comment_rows=comment_rows, + sku_header=sku_h, + ) + pl_cg = jcr.build_comment_groups_llm_payload( + feedback_groups=fb, + merged_rows=merged, + sku_header=sku_h, + title_h=title_h, + ) + + def _cg() -> str: + if chunk_gr: + return generate_comment_group_summaries_llm_chunked( + pl_cg, keyword=keyword + ) + return generate_comment_group_summaries_llm(pl_cg, keyword=keyword) + + _run_one( + "§8 细类评价要点（comment_groups）", + _cg, + live=args.live, + preview_chars=args.preview_chars, + ) + if not args.live: + print( + f" payload groups={len(pl_cg)} chunked_by_matrix={chunk_gr}", + flush=True, + ) + + if "report_supplement" in only: + brief = jcr.build_competitor_brief( + run_dir=run_dir, + keyword=keyword, + merged_rows=merged, + search_export_rows=search_rows, + comment_rows=comment_rows, + meta=meta, + report_config=eff_rc, + ) + + def _rp() -> str: + return generate_competitor_report_markdown_llm(brief, keyword) + + _run_one( + "§8.5 类报告补充（generate_competitor_report_markdown_llm）", + _rp, + live=args.live, + preview_chars=args.preview_chars, + ) + if not args.live: + print(" 使用 build_competitor_brief 全量摘要作为输入", flush=True) + + +if __name__ == "__main__": + main() diff --git a/backend/pipeline/demos/run_sentiment_one_matrix_group_demo.py b/backend/pipeline/demos/run_sentiment_one_matrix_group_demo.py new file mode 100644 index 0000000..37492bf --- /dev/null +++ b/backend/pipeline/demos/run_sentiment_one_matrix_group_demo.py @@ -0,0 +1,181 @@ +""" +仅针对**单个矩阵细类**试跑「8.3 评价正/负向主题」同源载荷与模型调用，**不**重写整份 competitor_analysis.md。 + + cd backend + python -m pipeline.demos.run_sentiment_one_matrix_group_demo --run-dir "../data/JD/pipeline_runs/20260413_104252_低GI" --group 饼干 + python -m pipeline.demos.run_sentiment_one_matrix_group_demo --run-dir "..." --group 饼干 --live + +加 ``--live`` 才会真实请求大模型；否则只打印该细类评价条数与是否找到分组。 +产出（仅 ``--live`` 且成功）：``run_dir/sentiment_8_3_one_group__{细类}.md`` 与同目录 ``sentiment_8_3_one_group__{细类}.json`` 元数据。 +""" +from __future__ import annotations + +import argparse +import json +import re +import sys +import traceback +from pathlib import Path +from typing import Any + +BACKEND_ROOT = Path(__file__).resolve().parents[2] +if str(BACKEND_ROOT) not in sys.path: + sys.path.insert(0, str(BACKEND_ROOT)) +import os + +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings") + +import django # noqa: E402 + +django.setup() + +JCR_ROOT = BACKEND_ROOT / "crawler_copy" / "jd_pc_search" +if str(JCR_ROOT) not in sys.path: + sys.path.insert(0, str(JCR_ROOT)) + +from pipeline.competitor_report import jd_report as jcr # noqa: E402 +import jd_keyword_pipeline as kpl # noqa: E402 + +from pipeline.csv.schema import MERGED_FIELD_TO_CSV_HEADER # noqa: E402 + + +def _safe_filename_part(s: str) -> str: + t = (s or "").strip() + if not t: + return "group" + return re.sub(r'[<>:"/\\|?*]', "_", t)[:80] + + +def main() -> None: + parser = argparse.ArgumentParser( + description="单细类评价正/负向主题 LLM（8.3 同源），不生成整报告", + ) + parser.add_argument("--run-dir", type=str, required=True, help="流水线目录") + parser.add_argument( + "--group", + type=str, + default="饼干", + help="矩阵细类名，须与第五章/feedback 分组名完全一致（默认：饼干）", + ) + parser.add_argument("--keyword", type=str, default="", help="覆盖监测词（默认读 run_meta.keyword）") + parser.add_argument( + "--live", + action="store_true", + help="真实调用大模型并写入 run_dir 下 md/json", + ) + args = parser.parse_args() + + run_dir = Path(args.run_dir).expanduser().resolve() + merged_path = run_dir / kpl.FILE_MERGED_CSV + if not merged_path.is_file(): + raise SystemExit(f"缺少合并表: {merged_path}") + + _, merged_rows = jcr._read_csv_rows(merged_path) + _, comment_rows = jcr._read_csv_rows(run_dir / kpl.FILE_COMMENTS_FLAT_CSV) + + meta_path = run_dir / kpl.FILE_RUN_META_JSON + meta_kw = "" + if meta_path.is_file(): + try: + meta = json.loads(meta_path.read_text(encoding="utf-8")) + meta_kw = str(meta.get("keyword") or "").strip() + except json.JSONDecodeError: + pass + keyword = (args.keyword or meta_kw or "监测词").strip() + + want = (args.group or "").strip() + feedback_groups = jcr._consumer_feedback_by_matrix_group( + merged_rows=merged_rows, + comment_rows=comment_rows, + sku_header=MERGED_FIELD_TO_CSV_HEADER["sku_id"], + ) + names = [g[0] for g in feedback_groups] + match: tuple[str, list[dict[str, str]], list[str]] | None = None + for item in feedback_groups: + if item[0] == want: + match = item + break + if match is None: + print(f"未找到细类「{want}」。当前分组：{names}", flush=True) + raise SystemExit(2) + + gname, cr_g, _tu = match + sub_merged = [ + r + for r in merged_rows + if jcr._competitor_matrix_group_key(r) == gname + ] + units, scores = jcr._iter_comment_text_units_and_scores(cr_g, sub_merged) + print( + f"run_dir={run_dir}\nkeyword={keyword!r}\ngroup={gname!r}\n" + f"comment_flat_rows={len(cr_g)} effective_text_units={len(units)}", + flush=True, + ) + if len(units) < 2: + print("有效评价正文不足 2 条，与生产流水线一致将跳过该细类。", flush=True) + raise SystemExit(3) + + if not args.live: + print("\n(dry-run：加 --live 将调用大模型并写入 sentiment_8_3_one_group__*.md)", flush=True) + return + + from pipeline.llm.generate import generate_comment_sentiment_analysis_llm # noqa: WPS433 + + sku_h = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + title_h = MERGED_FIELD_TO_CSV_HEADER["title"] + attr_units = jcr._comment_lines_with_product_context( + cr_g, + merged_rows, + sku_header=sku_h, + title_h=title_h, + ) + if len(attr_units) != len(units): + attr_units = list(units) + + try: + pl = jcr.build_comment_sentiment_llm_payload( + units, + scores=scores, + attributed_texts=attr_units, + max_samples_positive=16, + max_samples_negative=30, + max_samples_mixed=10, + max_chars_per_review=360, + semantic_pool_max=40, + shuffle_seed=f"{keyword}|{gname}", + ) + pl["keyword"] = keyword + pl["matrix_group_focus"] = gname + md_one = generate_comment_sentiment_analysis_llm(pl) + except Exception as e: + print(f"FAIL: {e}", flush=True) + traceback.print_exc() + raise SystemExit(1) from e + + body = f"#### {gname}\n\n{md_one.strip()}\n" + stem = _safe_filename_part(gname) + out_md = run_dir / f"sentiment_8_3_one_group__{stem}.md" + out_md.write_text( + "### 8.3 评价正/负向主题（按细类 · 大模型）· 单类试跑\n\n" + f"> keyword={keyword!r}，仅本节为脚本独立产出，未合并进 competitor_analysis.md。\n\n" + + body, + encoding="utf-8", + ) + rec: dict[str, Any] = { + "schema_version": 1, + "demo": "run_sentiment_one_matrix_group_demo", + "group": gname, + "keyword": keyword, + "ok": True, + "chars": len(md_one), + "markdown_file": out_md.name, + } + (run_dir / f"sentiment_8_3_one_group__{stem}.json").write_text( + json.dumps(rec, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + print(f"ok，已写 {out_md.name}（{len(md_one)} 字模型正文）", flush=True) + + +if __name__ == "__main__": + main() diff --git a/backend/pipeline/demos/test_regen_report_test_filename.py b/backend/pipeline/demos/test_regen_report_test_filename.py new file mode 100644 index 0000000..e7a3b1e --- /dev/null +++ b/backend/pipeline/demos/test_regen_report_test_filename.py @@ -0,0 +1,114 @@ +""" +在已有流水线目录上重生成报告，**将产出的 Markdown 以测试文件名落盘**，不长期覆盖 ``competitor_analysis.md``。 + +- 若目录下已有 ``competitor_analysis.md``：先复制为 ``competitor_analysis__pre_test_<时间戳>.md``，再重生成，再把**新生成**内容复制为 ``competitor_analysis__llmtest_<时间戳>.md``，最后把**原内容**写回 ``competitor_analysis.md``。 +- 若原本没有主报告：重生成后复制一份为 ``...__llmtest_...``，主文件即新生成结果。 + +用法（在 backend 下，已配置 Django 与 .env）:: + + python -m pipeline.demos.test_regen_report_test_filename + +可选环境变量：``MA_TEST_RUN_DIR`` = 绝对路径，默认自动选 ``data/JD/pipeline_runs`` 下第一个含 ``keyword_pipeline_merged.csv`` 的目录。 + +默认 **全量** 报告（``report_config=None``，与线上一致合并默认开关）。快测（只开矩阵 LLM 等）设环境变量 ``MA_TEST_REPORT_FAST=1``。 +""" +from __future__ import annotations + +import os +import shutil +import sys +from datetime import datetime +from pathlib import Path + +# backend 在 sys.path +_BACK = Path(__file__).resolve().parent.parent.parent +if str(_BACK) not in sys.path: + sys.path.insert(0, str(_BACK)) + +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings") +import django # noqa: E402 + +django.setup() + +from pipeline.jd.runner import regenerate_competitor_report # noqa: E402 + + +def _default_run_dir() -> Path: + custom = (os.environ.get("MA_TEST_RUN_DIR") or "").strip() + if custom: + return Path(custom).expanduser().resolve() + from django.conf import settings + + low = Path((settings.LOW_GI_PROJECT_ROOT or "").strip() or _BACK.parent) + pr = low / "data" / "JD" / "pipeline_runs" + if not pr.is_dir(): + raise FileNotFoundError(f"无 pipeline_runs: {pr}") + for d in sorted(pr.iterdir()): + if d.is_dir() and (d / "keyword_pipeline_merged.csv").is_file(): + return d.resolve() + raise FileNotFoundError(f"{pr} 下未找到含 keyword_pipeline_merged.csv 的目录") + + +def _keyword_from_run_dir(run_dir: Path) -> str: + meta = run_dir / "run_meta.json" + if not meta.is_file(): + return "" + import json + + try: + return str((json.loads(meta.read_text(encoding="utf-8")) or {}).get("keyword") or "").strip() + except json.JSONDecodeError: + return "" + + +def main() -> int: + run_dir = _default_run_dir() + kw = _keyword_from_run_dir(run_dir) + if not kw: + print("无法从 run_meta 读取 keyword，请设置环境变量或检查 run_dir", file=sys.stderr) + return 1 + if (os.environ.get("MA_TEST_REPORT_FAST") or "").strip().lower() in ( + "1", + "true", + "yes", + ): + report_config = { + "llm_comment_sentiment": False, + "llm_matrix_group_summaries": True, + "llm_comment_group_summaries": False, + "llm_price_group_summaries": False, + "llm_promo_group_summaries": False, + "llm_strategy_opportunities": False, + "chapter8_text_mining_probe": False, + "chapter8_text_mining_probe_live_llm": False, + } + else: + report_config = None + ts = datetime.now().strftime("%Y%m%d_%H%M%S") + main = run_dir / "competitor_analysis.md" + pre_bak = run_dir / f"competitor_analysis__pre_test_{ts}.md" + out_test = run_dir / f"competitor_analysis__llmtest_{ts}.md" + had_main = main.is_file() + if had_main: + shutil.copy2(main, pre_bak) + print("run_dir:", run_dir, file=sys.stderr) + print("keyword:", kw, file=sys.stderr) + print( + "regenerating (full config, may take long; set MA_TEST_REPORT_FAST=1 for quick) ...", + file=sys.stderr, + ) + regenerate_competitor_report(str(run_dir), kw, report_config=report_config) + if not main.is_file(): + print("未生成 competitor_analysis.md", file=sys.stderr) + return 2 + shutil.copy2(main, out_test) + if had_main and pre_bak.is_file(): + shutil.copy2(pre_bak, main) + print("test_output:", out_test) + if had_main: + print("restored:", main, "from", pre_bak, file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/pipeline/demos/try_openai_official_llm.py b/backend/pipeline/demos/try_openai_official_llm.py new file mode 100644 index 0000000..11a1904 --- /dev/null +++ b/backend/pipeline/demos/try_openai_official_llm.py @@ -0,0 +1,27 @@ +""" +试跑 OpenAI 官方「ChatGPT」文本适配器（不经过完整报告管线）。 + +准备：在 .env 中设置 MA_LLM_TEXT_PROVIDER=chatgpt 与 OPENAI_OFFICIAL_API_KEY（见 .env.example）。 + +用法（在 backend 目录下）： + + .venv\\Scripts\\python.exe -m pipeline.demos.try_openai_official_llm +""" +from __future__ import annotations + +import os +import sys + +if __name__ == "__main__": + if not (os.environ.get("OPENAI_OFFICIAL_API_KEY") or "").strip(): + print("未设置 OPENAI_OFFICIAL_API_KEY，退出。", file=sys.stderr) + sys.exit(1) + # 与生产一致时可在 .env 中设 MA_LLM_TEXT_PROVIDER=chatgpt；本脚本也强制用官方适配器 + from pipeline.llm.providers.adapters.openai_official_chatgpt import OpenAiOfficialChatGptTextLlm + + out = OpenAiOfficialChatGptTextLlm().complete_text( + "You reply in one short English sentence only.", + "What is 2+2?", + temperature=0.0, + ) + print(out) diff --git a/backend/pipeline/export_job.py b/backend/pipeline/export_job.py index 5f0be75..8aeea78 100644 --- a/backend/pipeline/export_job.py +++ b/backend/pipeline/export_job.py @@ -9,13 +9,14 @@ from typing import Any from django.db.models import QuerySet from openpyxl import Workbook -from .csv_schema import ( +from .csv.schema import ( COMMENT_CSV_TO_FIELD, DETAIL_CSV_TO_FIELD, JD_SEARCH_CSV_HEADERS, MERGED_FIELD_TO_CSV_HEADER, ) from .dataset_nonempty import ( + MATRIX_GROUP_COLUMN, comment_export_headers, detail_export_headers, merged_export_headers, @@ -34,20 +35,30 @@ from .row_serialize import ( ) -def _search_row_csv_dict(r: JdJobSearchRow, internal_keys: list[str]) -> dict[str, Any]: +def _search_row_csv_dict( + r: JdJobSearchRow, internal_keys: list[str], headers: list[str] +) -> dict[str, Any]: d = search_row_to_dict(r) out: dict[str, Any] = {"id": d["id"], "row_index": d["row_index"]} for k in internal_keys: out[JD_SEARCH_CSV_HEADERS[k]] = d.get(k, "") + zh = MATRIX_GROUP_COLUMN["label"] + if zh in headers: + out[zh] = d.get("matrix_group_label", "") return out -def _detail_row_csv_dict(r: JdJobDetailRow, csv_cols: list[str]) -> dict[str, Any]: +def _detail_row_csv_dict( + r: JdJobDetailRow, csv_cols: list[str], headers: list[str] +) -> dict[str, Any]: d = detail_row_to_dict(r) out: dict[str, Any] = {"id": d["id"], "row_index": d["row_index"]} for col in csv_cols: fn = DETAIL_CSV_TO_FIELD[col] out[col] = d.get(fn, "") + zh = MATRIX_GROUP_COLUMN["label"] + if zh in headers: + out[zh] = d.get("matrix_group_label", "") return out @@ -60,11 +71,16 @@ def _comment_row_csv_dict(r: JdJobCommentRow, csv_cols: list[str]) -> dict[str, return out -def _merged_row_csv_dict(r: JdJobMergedRow, internal_keys: list[str]) -> dict[str, Any]: +def _merged_row_csv_dict( + r: JdJobMergedRow, internal_keys: list[str], headers: list[str] +) -> dict[str, Any]: d = merged_row_to_dict(r) out: dict[str, Any] = {"id": d["id"], "row_index": d["row_index"]} for k in internal_keys: out[MERGED_FIELD_TO_CSV_HEADER[k]] = d.get(k, "") + zh = MATRIX_GROUP_COLUMN["label"] + if zh in headers: + out[zh] = d.get("matrix_group_label", "") return out @@ -98,20 +114,30 @@ def _prune_merged_dict(d: dict[str, Any], fields: list[str]) -> dict[str, Any]: def _rows_as_list_search(job: PipelineJob) -> list[dict[str, Any]]: keys = nonempty_search_keys_for_job(job) + extra_mg = JdJobSearchRow.objects.filter(job=job).exclude(matrix_group_label="").exists() qs = JdJobSearchRow.objects.filter(job=job) - return [ - _prune_search_dict(search_row_to_dict(obj), keys) - for obj in qs.order_by("row_index").iterator(chunk_size=400) - ] + out: list[dict[str, Any]] = [] + for obj in qs.order_by("row_index").iterator(chunk_size=400): + d = search_row_to_dict(obj) + row = _prune_search_dict(d, keys) + if extra_mg: + row["matrix_group_label"] = d.get("matrix_group_label", "") + out.append(row) + return out def _rows_as_list_detail(job: PipelineJob) -> list[dict[str, Any]]: fields = nonempty_detail_fields_for_job(job) + extra_mg = JdJobDetailRow.objects.filter(job=job).exclude(matrix_group_label="").exists() qs = JdJobDetailRow.objects.filter(job=job) - return [ - _prune_detail_dict(detail_row_to_dict(obj), fields) - for obj in qs.order_by("row_index").iterator(chunk_size=400) - ] + out: list[dict[str, Any]] = [] + for obj in qs.order_by("row_index").iterator(chunk_size=400): + d = detail_row_to_dict(obj) + row = _prune_detail_dict(d, fields) + if extra_mg: + row["matrix_group_label"] = d.get("matrix_group_label", "") + out.append(row) + return out def _rows_as_list_comment(job: PipelineJob) -> list[dict[str, Any]]: @@ -125,11 +151,16 @@ def _rows_as_list_comment(job: PipelineJob) -> list[dict[str, Any]]: def _rows_as_list_merged(job: PipelineJob) -> list[dict[str, Any]]: fields = nonempty_merged_fields_for_job(job) + extra_mg = JdJobMergedRow.objects.filter(job=job).exclude(matrix_group_label="").exists() qs = JdJobMergedRow.objects.filter(job=job) - return [ - _prune_merged_dict(merged_row_to_dict(obj), fields) - for obj in qs.order_by("row_index").iterator(chunk_size=400) - ] + out: list[dict[str, Any]] = [] + for obj in qs.order_by("row_index").iterator(chunk_size=400): + d = merged_row_to_dict(obj) + row = _prune_merged_dict(d, fields) + if extra_mg: + row["matrix_group_label"] = d.get("matrix_group_label", "") + out.append(row) + return out def build_json_bytes(*, job: PipelineJob, kind: str) -> tuple[bytes, str]: @@ -178,18 +209,21 @@ def _write_csv_from_qs( def build_csv_bytes(*, job: PipelineJob, kind: str) -> tuple[bytes, str]: if kind == "search": sk = nonempty_search_keys_for_job(job) + headers = search_export_headers(job) text = _write_csv_from_qs( qs=JdJobSearchRow.objects.filter(job=job), - headers=search_export_headers(job), - row_fn=lambda o, _sk=sk: _search_row_csv_dict(o, _sk), + headers=headers, + row_fn=lambda o, _sk=sk, _h=headers: _search_row_csv_dict(o, _sk, _h), ) name = f"job_{job.id}_search.csv" elif kind == "detail": - dcols = [c for c in detail_export_headers(job) if c not in ("id", "row_index")] + headers = detail_export_headers(job) + zh = MATRIX_GROUP_COLUMN["label"] + dcols = [c for c in headers if c not in ("id", "row_index", zh)] text = _write_csv_from_qs( qs=JdJobDetailRow.objects.filter(job=job), - headers=detail_export_headers(job), - row_fn=lambda o, _dc=dcols: _detail_row_csv_dict(o, _dc), + headers=headers, + row_fn=lambda o, _dc=dcols, _h=headers: _detail_row_csv_dict(o, _dc, _h), ) name = f"job_{job.id}_detail.csv" elif kind == "comments": @@ -202,22 +236,28 @@ def build_csv_bytes(*, job: PipelineJob, kind: str) -> tuple[bytes, str]: name = f"job_{job.id}_comments.csv" elif kind == "all": sk = nonempty_search_keys_for_job(job) - dcols = [c for c in detail_export_headers(job) if c not in ("id", "row_index")] + sheaders = search_export_headers(job) + dheaders = detail_export_headers(job) + zh = MATRIX_GROUP_COLUMN["label"] + dcols = [c for c in dheaders if c not in ("id", "row_index", zh)] ccols = [c for c in comment_export_headers(job) if c not in ("id", "row_index")] mk = nonempty_merged_fields_for_job(job) + mheaders = merged_export_headers(job) parts = [ "# search", _write_csv_from_qs( qs=JdJobSearchRow.objects.filter(job=job), - headers=search_export_headers(job), - row_fn=lambda o, _sk=sk: _search_row_csv_dict(o, _sk), + headers=sheaders, + row_fn=lambda o, _sk=sk, _h=sheaders: _search_row_csv_dict(o, _sk, _h), ), "", "# detail", _write_csv_from_qs( qs=JdJobDetailRow.objects.filter(job=job), - headers=detail_export_headers(job), - row_fn=lambda o, _dc=dcols: _detail_row_csv_dict(o, _dc), + headers=dheaders, + row_fn=lambda o, _dc=dcols, _h=dheaders: _detail_row_csv_dict( + o, _dc, _h + ), ), "", "# comments", @@ -230,18 +270,19 @@ def build_csv_bytes(*, job: PipelineJob, kind: str) -> tuple[bytes, str]: "# merged", _write_csv_from_qs( qs=JdJobMergedRow.objects.filter(job=job), - headers=merged_export_headers(job), - row_fn=lambda o, _mk=mk: _merged_row_csv_dict(o, _mk), + headers=mheaders, + row_fn=lambda o, _mk=mk, _h=mheaders: _merged_row_csv_dict(o, _mk, _h), ), ] text = "\n".join(parts) name = f"job_{job.id}_all.csv" elif kind == "merged": mk = nonempty_merged_fields_for_job(job) + headers = merged_export_headers(job) text = _write_csv_from_qs( qs=JdJobMergedRow.objects.filter(job=job), - headers=merged_export_headers(job), - row_fn=lambda o, _mk=mk: _merged_row_csv_dict(o, _mk), + headers=headers, + row_fn=lambda o, _mk=mk, _h=headers: _merged_row_csv_dict(o, _mk, _h), ) name = f"job_{job.id}_merged.csv" else: @@ -262,22 +303,25 @@ def build_xlsx_bytes(*, job: PipelineJob, kind: str) -> tuple[bytes, str]: ws = wb.active ws.title = "search"[:31] sk = nonempty_search_keys_for_job(job) + sheaders = search_export_headers(job) _append_sheet( ws, - search_export_headers(job), + sheaders, JdJobSearchRow.objects.filter(job=job), - lambda o, _sk=sk: _search_row_csv_dict(o, _sk), + lambda o, _sk=sk, _h=sheaders: _search_row_csv_dict(o, _sk, _h), ) name = f"job_{job.id}_search.xlsx" elif kind == "detail": ws = wb.active ws.title = "detail"[:31] - dcols = [c for c in detail_export_headers(job) if c not in ("id", "row_index")] + dheaders = detail_export_headers(job) + zh = MATRIX_GROUP_COLUMN["label"] + dcols = [c for c in dheaders if c not in ("id", "row_index", zh)] _append_sheet( ws, - detail_export_headers(job), + dheaders, JdJobDetailRow.objects.filter(job=job), - lambda o, _dc=dcols: _detail_row_csv_dict(o, _dc), + lambda o, _dc=dcols, _h=dheaders: _detail_row_csv_dict(o, _dc, _h), ) name = f"job_{job.id}_detail.xlsx" elif kind == "comments": @@ -293,23 +337,27 @@ def build_xlsx_bytes(*, job: PipelineJob, kind: str) -> tuple[bytes, str]: name = f"job_{job.id}_comments.xlsx" elif kind == "all": sk = nonempty_search_keys_for_job(job) - dcols = [c for c in detail_export_headers(job) if c not in ("id", "row_index")] + sheaders = search_export_headers(job) + dheaders = detail_export_headers(job) + zh = MATRIX_GROUP_COLUMN["label"] + dcols = [c for c in dheaders if c not in ("id", "row_index", zh)] ccols = [c for c in comment_export_headers(job) if c not in ("id", "row_index")] mk = nonempty_merged_fields_for_job(job) + mheaders = merged_export_headers(job) ws1 = wb.active ws1.title = "search"[:31] _append_sheet( ws1, - search_export_headers(job), + sheaders, JdJobSearchRow.objects.filter(job=job), - lambda o, _sk=sk: _search_row_csv_dict(o, _sk), + lambda o, _sk=sk, _h=sheaders: _search_row_csv_dict(o, _sk, _h), ) ws2 = wb.create_sheet("detail"[:31]) _append_sheet( ws2, - detail_export_headers(job), + dheaders, JdJobDetailRow.objects.filter(job=job), - lambda o, _dc=dcols: _detail_row_csv_dict(o, _dc), + lambda o, _dc=dcols, _h=dheaders: _detail_row_csv_dict(o, _dc, _h), ) ws3 = wb.create_sheet("comments"[:31]) _append_sheet( @@ -321,20 +369,21 @@ def build_xlsx_bytes(*, job: PipelineJob, kind: str) -> tuple[bytes, str]: ws4 = wb.create_sheet("merged"[:31]) _append_sheet( ws4, - merged_export_headers(job), + mheaders, JdJobMergedRow.objects.filter(job=job), - lambda o, _mk=mk: _merged_row_csv_dict(o, _mk), + lambda o, _mk=mk, _h=mheaders: _merged_row_csv_dict(o, _mk, _h), ) name = f"job_{job.id}_all.xlsx" elif kind == "merged": mk = nonempty_merged_fields_for_job(job) ws = wb.active ws.title = "merged"[:31] + mheaders = merged_export_headers(job) _append_sheet( ws, - merged_export_headers(job), + mheaders, JdJobMergedRow.objects.filter(job=job), - lambda o, _mk=mk: _merged_row_csv_dict(o, _mk), + lambda o, _mk=mk, _h=mheaders: _merged_row_csv_dict(o, _mk, _h), ) name = f"job_{job.id}_merged.xlsx" else: diff --git a/backend/pipeline/ingest.py b/backend/pipeline/ingest.py index 3de93cd..72ae2fe 100644 --- a/backend/pipeline/ingest.py +++ b/backend/pipeline/ingest.py @@ -10,10 +10,11 @@ import logging from pathlib import Path from typing import Any +from django.conf import settings from django.db import transaction from django.utils import timezone -from .csv_schema import ( +from .csv.schema import ( COMMENT_CSV_COLUMNS, COMMENT_CSV_TO_FIELD, DETAIL_CSV_COLUMNS, @@ -22,8 +23,13 @@ from .csv_schema import ( JD_SEARCH_INTERNAL_KEYS, MERGED_CSV_COLUMNS, MERGED_CSV_TO_FIELD, + MERGED_FIELD_TO_CSV_HEADER, SEARCH_CSV_HEADER_TO_FIELD, + merged_csv_effective_total_sales, + search_csv_effective_total_sales, + strip_buyer_ranking_line_prefix, ) +from pipeline.jd.matrix_group_label import matrix_group_label_from_detail_path from .models import ( JdJobCommentRow, JdJobDetailRow, @@ -33,6 +39,12 @@ from .models import ( JdProductSnapshot, PipelineJob, ) +from .price_parse import effective_list_price_value, float_price_from_cell +from .volume_parse import ( + comment_count_sort_value_from_cell, + comment_count_sort_value_from_merged, + sales_sort_value_from_search_cells, +) logger = logging.getLogger(__name__) @@ -41,9 +53,9 @@ FILE_PC_SEARCH_CSV = "pc_search_export.csv" FILE_DETAIL_WARE_CSV = "detail_ware_export.csv" FILE_COMMENTS_FLAT_CSV = "comments_flat.csv" -SKU_FIELD_MERGED = "SKU(skuId)" -WARE_FIELD = "主商品ID(wareId)" -TITLE_FIELD = "标题(wareName)" +SKU_FIELD_MERGED = MERGED_FIELD_TO_CSV_HEADER["sku_id"] +WARE_FIELD = MERGED_FIELD_TO_CSV_HEADER["ware_id"] +TITLE_FIELD = MERGED_FIELD_TO_CSV_HEADER["title"] BULK_CHUNK = 400 @@ -62,6 +74,11 @@ def _payload_as_json(row: dict[str, str]) -> dict[str, str]: return {str(k): str(v) if v is not None else "" for k, v in row.items()} +def _normalize_search_csv_total_sales(row: dict[str, str]) -> None: + h = JD_SEARCH_CSV_HEADERS["total_sales"] + row[h] = search_csv_effective_total_sales(row) + + def _search_row_kwargs(row: dict[str, str]) -> dict[str, str]: vals = {k: "" for k in JD_SEARCH_INTERNAL_KEYS} for csv_header, cell in row.items(): @@ -73,9 +90,12 @@ def _search_row_kwargs(row: dict[str, str]) -> dict[str, str]: def _detail_row_kwargs(row: dict[str, str]) -> dict[str, str]: - return { + kw = { DETAIL_CSV_TO_FIELD[col]: str(row.get(col) or "").strip() for col in DETAIL_CSV_COLUMNS } + if kw.get("buyer_ranking_line"): + kw["buyer_ranking_line"] = strip_buyer_ranking_line_prefix(kw["buyer_ranking_line"]) + return kw def _comment_row_kwargs(row: dict[str, str]) -> dict[str, str]: @@ -84,10 +104,19 @@ def _comment_row_kwargs(row: dict[str, str]) -> dict[str, str]: } +def _normalize_merged_csv_total_sales(row: dict[str, str]) -> None: + """列表未写 totalSales 列时，用销量楼层推断，保证入库与快照与报告计数一致。""" + h = MERGED_FIELD_TO_CSV_HEADER["total_sales"] + row[h] = merged_csv_effective_total_sales(row) + + def _merged_row_kwargs(row: dict[str, str]) -> dict[str, str]: - return { + kw = { MERGED_CSV_TO_FIELD[col]: str(row.get(col) or "").strip() for col in MERGED_CSV_COLUMNS } + if kw.get("buyer_ranking_line"): + kw["buyer_ranking_line"] = strip_buyer_ranking_line_prefix(kw["buyer_ranking_line"]) + return kw def _bulk_create_in_chunks(model, objects: list[Any]) -> None: @@ -95,10 +124,63 @@ def _bulk_create_in_chunks(model, objects: list[Any]) -> None: model.objects.bulk_create(objects[i : i + BULK_CHUNK]) +def _sync_search_rows_matrix_labels( + job: PipelineJob, merged_kw_list: list[tuple[int, dict[str, str]]] +) -> None: + """按 SKU 将合并表解析出的类目回填到搜索行（与 §5 矩阵**同一细类划分**）。""" + sku_to_mg: dict[str, str] = {} + for _, kw in merged_kw_list: + sk = (kw.get("sku_id") or "").strip() + if not sk: + continue + mg = matrix_group_label_from_detail_path(kw.get("detail_category_path") or "") + if mg: + sku_to_mg[sk] = mg + if not sku_to_mg: + return + chunk: list[JdJobSearchRow] = [] + for r in JdJobSearchRow.objects.filter(job=job).iterator(chunk_size=400): + sk = (r.sku_id or "").strip() + if sk and sk in sku_to_mg: + r.matrix_group_label = sku_to_mg[sk] + chunk.append(r) + if len(chunk) >= 400: + JdJobSearchRow.objects.bulk_update(chunk, ["matrix_group_label"]) + chunk.clear() + if chunk: + JdJobSearchRow.objects.bulk_update(chunk, ["matrix_group_label"]) + + def _run_dir(job: PipelineJob) -> Path: return Path(job.run_dir or "").expanduser().resolve() +def resolve_and_validate_run_dir(path_str: str) -> Path: + """ + 将用户输入解析为 ``LOW_GI_PROJECT_ROOT/data/JD`` 下的绝对路径，且须为已存在目录。 + + 相对路径相对 ``data/JD``（与创建任务时 ``pipeline_run_dir`` 语义一致）。 + """ + if not (path_str or "").strip(): + raise ValueError("run_dir 为空") + root = (settings.LOW_GI_PROJECT_ROOT or "").strip() + if not root: + raise ValueError("LOW_GI_PROJECT_ROOT 未配置") + project_data = Path(root).resolve() / "data" / "JD" + p = Path(path_str.strip()).expanduser() + if not p.is_absolute(): + p = project_data / p + p = p.resolve() + jd = project_data.resolve() + try: + p.relative_to(jd) + except ValueError as e: + raise ValueError(f"路径须位于京东数据目录下：{jd}") from e + if not p.is_dir(): + raise ValueError(f"目录不存在：{p}") + return p + + def ingest_job_dataset_rows(job: PipelineJob) -> dict[str, Any]: """ 删除该任务旧数据后，将 ``pc_search_export`` / ``detail_ware_export`` / ``comments_flat`` 全量写入数据库。 @@ -123,10 +205,31 @@ def ingest_job_dataset_rows(job: PipelineJob) -> dict[str, Any]: search_rows = _read_csv_rows(search_path) if not search_rows and search_path.is_file() is False: pass - s_objs: list[JdJobSearchRow] = [] + search_kw_list: list[tuple[int, dict[str, str]]] = [] for i, row in enumerate(search_rows): + _normalize_search_csv_total_sales(row) kw = _search_row_kwargs(row) - s_objs.append(JdJobSearchRow(job=job, row_index=i, **kw)) + search_kw_list.append((i, kw)) + s_objs: list[JdJobSearchRow] = [] + for i, kw in search_kw_list: + pv = effective_list_price_value( + kw.get("coupon_price"), kw.get("price"), kw.get("original_price") + ) + sv = sales_sort_value_from_search_cells( + kw.get("total_sales"), kw.get("comment_sales_floor") + ) + cv = comment_count_sort_value_from_cell(kw.get("comment_count")) + s_objs.append( + JdJobSearchRow( + job=job, + row_index=i, + matrix_group_label="", + price_value=pv, + sales_sort_value=sv, + comment_count_sort_value=cv, + **kw, + ) + ) _bulk_create_in_chunks(JdJobSearchRow, s_objs) stats["search_rows"] = len(s_objs) @@ -135,7 +238,17 @@ def ingest_job_dataset_rows(job: PipelineJob) -> dict[str, Any]: d_objs: list[JdJobDetailRow] = [] for i, row in enumerate(detail_rows): kw = _detail_row_kwargs(row) - d_objs.append(JdJobDetailRow(job=job, row_index=i, **kw)) + dpv = float_price_from_cell(kw.get("detail_price_final")) + mg = matrix_group_label_from_detail_path(kw.get("detail_category_path") or "") + d_objs.append( + JdJobDetailRow( + job=job, + row_index=i, + matrix_group_label=mg, + detail_price_value=dpv, + **kw, + ) + ) _bulk_create_in_chunks(JdJobDetailRow, d_objs) stats["detail_rows"] = len(d_objs) @@ -150,12 +263,37 @@ def ingest_job_dataset_rows(job: PipelineJob) -> dict[str, Any]: merged_path = run_dir / FILE_MERGED_CSV merged_rows = _read_csv_rows(merged_path) if merged_path.is_file() else [] - m_objs: list[JdJobMergedRow] = [] + merged_kw_list: list[tuple[int, dict[str, str]]] = [] for i, row in enumerate(merged_rows): + _normalize_merged_csv_total_sales(row) kw = _merged_row_kwargs(row) - m_objs.append(JdJobMergedRow(job=job, row_index=i, **kw)) + merged_kw_list.append((i, kw)) + m_objs: list[JdJobMergedRow] = [] + for i, kw in merged_kw_list: + mg = matrix_group_label_from_detail_path(kw.get("detail_category_path") or "") + pv = effective_list_price_value( + kw.get("coupon_price"), kw.get("price"), kw.get("original_price") + ) + msv = sales_sort_value_from_search_cells( + kw.get("total_sales"), kw.get("comment_sales_floor") + ) + mcv = comment_count_sort_value_from_merged( + kw.get("pipeline_comment_count") + ) + m_objs.append( + JdJobMergedRow( + job=job, + row_index=i, + matrix_group_label=mg, + price_value=pv, + sales_sort_value=msv, + comment_count_sort_value=mcv, + **kw, + ) + ) _bulk_create_in_chunks(JdJobMergedRow, m_objs) stats["merged_table_rows"] = len(m_objs) + _sync_search_rows_matrix_labels(job, merged_kw_list) return stats @@ -185,17 +323,22 @@ def ingest_job_merged_csv(job: PipelineJob) -> dict[str, Any]: sku = (row.get(SKU_FIELD_MERGED) or "").strip() if not sku: continue + _normalize_merged_csv_total_sales(row) + br_h = MERGED_FIELD_TO_CSV_HEADER["buyer_ranking_line"] + br = (row.get(br_h) or "").strip() + if br: + row[br_h] = strip_buyer_ranking_line_prefix(br) payload = _payload_as_json(row) title = (row.get(TITLE_FIELD) or "")[:2000] ware = (row.get(WARE_FIELD) or "").strip()[:64] - brand = (row.get("detail_brand") or "").strip()[:512] + brand = (row.get(MERGED_FIELD_TO_CSV_HEADER["detail_brand"]) or "").strip()[:512] price = ( - (row.get("detail_price_final") or "").strip() + (row.get(MERGED_FIELD_TO_CSV_HEADER["detail_price_final"]) or "").strip() or (row.get(JD_SEARCH_CSV_HEADERS["coupon_price"]) or "").strip() or (row.get(JD_SEARCH_CSV_HEADERS["price"]) or "").strip() )[:128] cat = ( - (row.get("detail_category_path") or "").strip() + (row.get(MERGED_FIELD_TO_CSV_HEADER["detail_category_path"]) or "").strip() or (row.get(JD_SEARCH_CSV_HEADERS["leaf_category"]) or "").strip() )[:2000] diff --git a/backend/pipeline/jd/__init__.py b/backend/pipeline/jd/__init__.py new file mode 100644 index 0000000..1185c36 --- /dev/null +++ b/backend/pipeline/jd/__init__.py @@ -0,0 +1,2 @@ +"""京东采集流水线编排：运行爬虫副本、购买者 CSV 导出、详情表再生、矩阵细类标签等。""" + diff --git a/backend/pipeline/jd/buyer_offer_export_csv.py b/backend/pipeline/jd/buyer_offer_export_csv.py new file mode 100644 index 0000000..5198898 --- /dev/null +++ b/backend/pipeline/jd/buyer_offer_export_csv.py @@ -0,0 +1,109 @@ +# -*- coding: utf-8 -*- +""" +将 ``detail_ware_export.csv`` 与 ``detail/ware_*_response.json`` 合并为一张表： +在原 lean 列后追加 ``buyer_ranking_line``、``buyer_promo_text``（促销相关摘要句，用稳定分隔符拼接）。 + +输出默认写入 ``<run_dir>/buyer_offer_profiles/buyer_offer_with_detail.csv``。 +""" +from __future__ import annotations + +import csv +import sys +from pathlib import Path + +# 与 pipeline.ingest / jd_keyword_pipeline 中文件名一致（避免 import ingest 触发 Django） +FILE_DETAIL_WARE_CSV = "detail_ware_export.csv" +# 与 jd_keyword_pipeline.DIR_BUYER_OFFER_PROFILES 一致 +DIR_BUYER_OFFER_PROFILES = "buyer_offer_profiles" +FILE_BUYER_OFFER_WITH_DETAIL_CSV = "buyer_offer_with_detail.csv" + +def _ensure_crawler_detail_path() -> None: + # 本文件位于 pipeline/jd/，向上两级为 backend/ + root = Path(__file__).resolve().parents[2] / "crawler_copy" / "jd_pc_search" + for sub in ("detail", ""): + p = root / sub if sub else root + s = str(p.resolve()) + if s not in sys.path: + sys.path.insert(0, s) + + +def export_buyer_offer_with_detail_csv( + run_dir: str | Path, + *, + promo_sep: str = " | ", +) -> Path: + """ + 读取 ``run_dir/detail_ware_export.csv`` 与 ``run_dir/detail/ware_{sku}_response.json``， + 写出 ``run_dir/buyer_offer_profiles/buyer_offer_with_detail.csv``。 + """ + _ensure_crawler_detail_path() + from jd_detail_buyer_extraction import ( # noqa: WPS433 + buyer_promo_text_from_profile, + buyer_ranking_line_from_profile, + extract_buyer_offer_profile_from_json_text, + ) + from jd_detail_ware_business_requests import ( # noqa: WPS433 + DETAIL_WARE_LEAN_CSV_FIELDNAMES, + ) + + run_dir = Path(run_dir).expanduser().resolve() + src = run_dir / FILE_DETAIL_WARE_CSV + if not src.is_file(): + raise FileNotFoundError(f"缺少详情汇总表: {src}") + detail_dir = run_dir / "detail" + if not detail_dir.is_dir(): + raise FileNotFoundError(f"缺少 detail 目录: {detail_dir}") + + out_dir = run_dir / DIR_BUYER_OFFER_PROFILES + out_dir.mkdir(parents=True, exist_ok=True) + out_path = out_dir / FILE_BUYER_OFFER_WITH_DETAIL_CSV + + fieldnames: list[str] = list(DETAIL_WARE_LEAN_CSV_FIELDNAMES) + + with src.open(encoding="utf-8-sig", newline="") as f: + reader = csv.DictReader(f) + rows_out: list[dict[str, str]] = [] + for row in reader: + sku = str(row.get("SKU") or row.get("skuId") or "").strip() + base = { + k: str(row.get(k) or "").strip() for k in DETAIL_WARE_LEAN_CSV_FIELDNAMES + } + rline = base.get("榜单排名") or base.get("buyer_ranking_line") or "" + ptext = base.get("促销摘要") or base.get("buyer_promo_text") or "" + if (not rline and not ptext) and sku: + jp = detail_dir / f"ware_{sku}_response.json" + if jp.is_file(): + text = jp.read_text(encoding="utf-8").strip() + if text: + prof = extract_buyer_offer_profile_from_json_text(text) + rline = buyer_ranking_line_from_profile(prof) + ptext = buyer_promo_text_from_profile(prof, sep=promo_sep) + base["榜单排名"] = rline + base["促销摘要"] = ptext + rows_out.append(base) + + with out_path.open("w", encoding="utf-8-sig", newline="") as f: + w = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore") + w.writeheader() + w.writerows(rows_out) + + return out_path + + +def main(argv: list[str] | None = None) -> None: + argv = argv if argv is not None else sys.argv[1:] + if len(argv) < 1: + print( + "用法: python -m pipeline.jd.buyer_offer_export_csv <run_dir>\n" + " 例: python -m pipeline.jd.buyer_offer_export_csv " + "data/JD/pipeline_runs/20260413_104252_低GI", + file=sys.stderr, + ) + sys.exit(2) + run_dir = argv[0] + path = export_buyer_offer_with_detail_csv(run_dir) + print(path) + + +if __name__ == "__main__": + main() diff --git a/backend/pipeline/detail_ware_regen.py b/backend/pipeline/jd/detail_ware_regen.py similarity index 74% rename from backend/pipeline/detail_ware_regen.py rename to backend/pipeline/jd/detail_ware_regen.py index 0aed5f3..c15f9de 100644 --- a/backend/pipeline/detail_ware_regen.py +++ b/backend/pipeline/jd/detail_ware_regen.py @@ -5,11 +5,12 @@ import csv import sys from pathlib import Path -from .ingest import FILE_DETAIL_WARE_CSV, FILE_MERGED_CSV, SKU_FIELD_MERGED +from ..csv.schema import MERGED_FIELD_TO_CSV_HEADER +from ..ingest import FILE_DETAIL_WARE_CSV, FILE_MERGED_CSV, SKU_FIELD_MERGED def _ensure_crawler_copy_path() -> None: - root = Path(__file__).resolve().parent.parent / "crawler_copy" / "jd_pc_search" + root = Path(__file__).resolve().parents[2] / "crawler_copy" / "jd_pc_search" for sub in ("detail", ""): p = root / sub if sub else root s = str(p.resolve()) @@ -19,6 +20,11 @@ def _ensure_crawler_copy_path() -> None: def regenerate_detail_ware_rows(run_dir: Path) -> list[dict[str, str]]: _ensure_crawler_copy_path() + from jd_detail_buyer_extraction import ( # noqa: WPS433 + buyer_promo_text_from_profile, + buyer_ranking_line_from_profile, + extract_buyer_offer_profile_from_json_text, + ) from jd_detail_ware_business_requests import ( # noqa: WPS433 DETAIL_WARE_LEAN_CSV_FIELDNAMES, detail_ware_lean_csv_row, @@ -43,7 +49,12 @@ def regenerate_detail_ware_rows(run_dir: Path) -> list[dict[str, str]]: if not jp.is_file(): continue text = jp.read_text(encoding="utf-8") - ing = (row.get("detail_body_ingredients") or "").strip() + ing = ( + row.get(MERGED_FIELD_TO_CSV_HEADER["detail_body_ingredients"]) + or row.get("detail_body_ingredients") + or "" + ).strip() + prof = extract_buyer_offer_profile_from_json_text(text) rows_out.append( detail_ware_lean_csv_row( sku, @@ -51,6 +62,8 @@ def regenerate_detail_ware_rows(run_dir: Path) -> list[dict[str, str]]: text, detail_body_ingredients=ing, detail_body_ingredients_source_url="", + buyer_ranking_line=buyer_ranking_line_from_profile(prof), + buyer_promo_text=buyer_promo_text_from_profile(prof), ) ) return rows_out diff --git a/backend/pipeline/jd/matrix_group_label.py b/backend/pipeline/jd/matrix_group_label.py new file mode 100644 index 0000000..dd825ae --- /dev/null +++ b/backend/pipeline/jd/matrix_group_label.py @@ -0,0 +1,63 @@ +""" +与 ``pipeline.competitor_report.matrix_group`` / 历史爬虫侧脚本中路径解析逻辑同源： +从商详 ``detail_category_path`` 解析 §5 竞品矩阵用的类目展示名（如饼干、米）。 +""" +from __future__ import annotations + +import re + + +def category_token_meaningless(seg: str) -> bool: + """纯数字类目 ID、空串或疑似内部编码的段，不宜直接作为矩阵分组展示名。""" + t = (seg or "").strip() + if not t: + return True + if t.isdigit(): + return True + if len(t) >= 14 and re.fullmatch(r"[A-Za-z0-9_\-]+", t): + return True + return False + + +def matrix_display_segment_from_parts(parts: list[str]) -> str | None: + """ + 与历史逻辑一致的主选段；若该段无意义则自右向左找第一段可读文本 + （避免「仅类目码」或中间段为数字 ID 时，把路径误解析成无意义的细类展示名）。 + """ + if not parts: + return None + if len(parts) >= 4: + preferred = parts[-2] + elif len(parts) >= 3: + preferred = parts[1] + elif len(parts) >= 2: + preferred = parts[1] + else: + preferred = parts[0] + order: list[str] = [] + if preferred: + order.append(preferred) + if len(parts) >= 2: + order.append(parts[-2]) + order.append(parts[-1]) + order.extend(reversed(parts)) + seen: set[str] = set() + for cand in order: + if not cand or cand in seen: + continue + seen.add(cand) + if not category_token_meaningless(cand): + return cand.strip() + return None + + +def matrix_group_label_from_detail_path(path: str) -> str: + """由 ``detail_category_path`` 文本解析细类展示名；空或无可读段则返回空串。""" + t = (path or "").strip() + if not t: + return "" + parts = [p.strip() for p in t.replace("＞", ">").split(">") if p.strip()] + if not parts: + return "" + key = matrix_display_segment_from_parts(parts) + return (key[:80] if key else "") diff --git a/backend/pipeline/jd/merged_regen.py b/backend/pipeline/jd/merged_regen.py new file mode 100644 index 0000000..0635354 --- /dev/null +++ b/backend/pipeline/jd/merged_regen.py @@ -0,0 +1,106 @@ +"""从 ``detail_ware_export.csv`` / ``detail/ware_*_response.json`` 补全并规范化 lean ``keyword_pipeline_merged.csv``（列序与 ``pipeline.csv.schema.MERGED_CSV_COLUMNS`` 一致）。""" +from __future__ import annotations + +import csv +import sys +from pathlib import Path + +from ..csv.schema import ( + MERGED_CSV_COLUMNS, + MERGED_FIELD_TO_CSV_HEADER, + MERGED_LEAN_DETAIL_INTERNAL_KEYS, + merged_csv_effective_total_sales, + strip_buyer_ranking_line_prefix, +) +from ..ingest import FILE_DETAIL_WARE_CSV, FILE_MERGED_CSV + +HOT_KEY = "榜单类文案" + + +def _ensure_crawler_detail_path() -> None: + root = Path(__file__).resolve().parents[2] / "crawler_copy" / "jd_pc_search" + for sub in ("detail", ""): + p = root / sub if sub else root + s = str(p.resolve()) + if s not in sys.path: + sys.path.insert(0, s) + + +def write_keyword_pipeline_merged_lean_csv(run_dir: Path) -> tuple[int, Path]: + """ + 读取已有 ``keyword_pipeline_merged.csv``（可缺列），按 lean 宽表列序重写： + - ``销量展示`` 列与入库一致（``merged_csv_effective_total_sales``） + - 商详块列优先与 ``detail_ware_export.csv`` 对齐；缺则尝试 ``detail/ware_{sku}_response.json`` + - 「榜单类文案」与「榜单排名」去掉 ``榜单/曝光：`` 前缀 + """ + _ensure_crawler_detail_path() + from jd_detail_buyer_extraction import ( # noqa: WPS433 + buyer_promo_text_from_profile, + buyer_ranking_line_from_profile, + extract_buyer_offer_profile_from_json_text, + ) + + run_dir = run_dir.expanduser().resolve() + merged_path = run_dir / FILE_MERGED_CSV + detail_path = run_dir / FILE_DETAIL_WARE_CSV + detail_dir = run_dir / "detail" + + if not merged_path.is_file(): + raise FileNotFoundError(f"缺少合并表: {merged_path}") + if not detail_dir.is_dir(): + raise FileNotFoundError(f"缺少 detail 目录: {detail_dir}") + + with merged_path.open(encoding="utf-8-sig", newline="") as f: + old_rows = list(csv.DictReader(f)) + + detail_by_sku: dict[str, dict[str, str]] = {} + if detail_path.is_file(): + with detail_path.open(encoding="utf-8-sig", newline="") as f: + for r in csv.DictReader(f): + sku = (r.get("SKU") or r.get("skuId") or "").strip() + if sku: + detail_by_sku[sku] = {k: str(r.get(k) or "").strip() for k in r} + + h_ts = MERGED_FIELD_TO_CSV_HEADER["total_sales"] + sku_h = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + br_h = MERGED_FIELD_TO_CSV_HEADER["buyer_ranking_line"] + pr_h = MERGED_FIELD_TO_CSV_HEADER["buyer_promo_text"] + rows_out: list[dict[str, str]] = [] + + for row in old_rows: + out = {col: str(row.get(col) or "").strip() for col in MERGED_CSV_COLUMNS} + out[h_ts] = merged_csv_effective_total_sales(out) + + if out.get(HOT_KEY): + out[HOT_KEY] = strip_buyer_ranking_line_prefix(out[HOT_KEY]) + + sku = (out.get(sku_h) or "").strip() + if sku and sku in detail_by_sku: + d = detail_by_sku[sku] + for ik in MERGED_LEAN_DETAIL_INTERNAL_KEYS: + ch = MERGED_FIELD_TO_CSV_HEADER[ik] + v = (d.get(ch) or d.get(ik) or "").strip() + if v: + out[ch] = v + elif sku: + jp = detail_dir / f"ware_{sku}_response.json" + if jp.is_file(): + text = jp.read_text(encoding="utf-8").strip() + if text: + prof = extract_buyer_offer_profile_from_json_text(text) + out[br_h] = buyer_ranking_line_from_profile(prof) + out[pr_h] = buyer_promo_text_from_profile(prof) + + out[br_h] = strip_buyer_ranking_line_prefix(out.get(br_h) or "") + rows_out.append(out) + + with merged_path.open("w", encoding="utf-8-sig", newline="") as f: + w = csv.DictWriter( + f, + fieldnames=list(MERGED_CSV_COLUMNS), + extrasaction="ignore", + ) + w.writeheader() + w.writerows(rows_out) + + return len(rows_out), merged_path diff --git a/backend/pipeline/jd/runner.py b/backend/pipeline/jd/runner.py new file mode 100644 index 0000000..21f3b24 --- /dev/null +++ b/backend/pipeline/jd/runner.py @@ -0,0 +1,940 @@ +""" +使用 ``crawler_copy/jd_pc_search`` 中的采集脚本执行流水线；竞品 Markdown 由 ``pipeline.competitor_report.jd_report`` 生成。 +依赖环境变量 ``LOW_GI_PROJECT_ROOT``（由 Django settings 从 ``market_assistant/.env`` 注入）。 +""" +from __future__ import annotations + +import json +import os +import sys +from pathlib import Path +from typing import Any + +from django.conf import settings + +from ..csv.schema import MERGED_FIELD_TO_CSV_HEADER +from ..models import PipelineJob +from ..serializers import REPORT_CONFIG_BOOL_KEYS + + +def merge_llm_supplement_with_rules_report(llm_md: str, rules_md: str) -> str: + """ + **以规则引擎全文为正文**（含第五章完整竞品矩阵、各章内嵌统计图与表格）。 + + 大模型稿作为 **8.5 小节**嵌入在 **第八章末、第九章策略** 之前，与第八章第二、三节等具体分析同卷连贯， + **不再**插在篇首「## 一、」之前。 + + 注：API「重新生成报告」已不再调用本函数，避免整篇 LLM 与矩阵/图表**数据含义**冲突；保留供脚本或将来显式开关复用。 + """ + body = (rules_md or "").strip() + sup = (llm_md or "").strip() + if not sup: + return body + if not body: + return sup + marker = "\n---\n\n## 九、策略与机会提示（假设清单，待验证）" + insert = ( + "\n\n---\n\n" + "### 8.5 大模型深度补充（与第二章至第八章第二节正文中的定量内容互补）\n\n" + "> **说明**：本段位于**第八章末**；**竞品矩阵、价盘表、统计图与第八章第二、三节等各节正文以相应章节为准**，" + "此处为跨小节语义整合，便于衔接第九章。\n\n" + f"{sup.strip()}\n" + "\n---\n\n## 九、策略与机会提示（假设清单，待验证）" + ) + if marker in body: + return body.replace(marker, insert, 1) + # 旧版报告标题或语言差异时的回退 + for alt in ( + "\n## 九、策略与机会提示（假设清单，待验证）", + "\n## 九、策略与机会提示", + ): + if alt in body and marker not in body: + return body.replace( + alt, + "\n\n---\n\n### 8.5 大模型深度补充（与第二章至第八章第二节正文中的定量内容互补）\n\n" + "> **说明**：位于第八章末；**矩阵与图表以正文为准**。\n\n" + f"{sup.strip()}\n" + + alt, + 1, + ) + app = "\n## 附录 A：数据留存说明" + if app in body: + tail = ( + "\n\n---\n\n### 8.5 大模型深度补充（与第二章至第八章第二节正文中的定量内容互补）\n\n" + f"{sup.strip()}\n" + ) + return body.replace(app, tail + app, 1) + return body + "\n\n---\n\n### 8.5 大模型深度补充\n\n" + sup.strip() + "\n" + + +def merge_llm_report_with_rules_charts(llm_md: str, rules_md: str) -> str: + """兼容旧名：等价于 ``merge_llm_supplement_with_rules_report``。""" + return merge_llm_supplement_with_rules_report(llm_md, rules_md) + + +def _flat_comment_texts(comment_rows: list[dict[str, str]]) -> list[str]: + """全部非空评价正文（与报告统计同源）。""" + out: list[str] = [] + for row in comment_rows: + t = (row.get("tagCommentContent") or "").strip() + if t: + out.append(t) + return out + + +def _safe_dir_segment_for_job(s: str, max_len: int = 48) -> str: + """与 ``jd_keyword_pipeline._safe_dir_segment`` 一致，避免多线程下改模块全局。""" + bad = '<>:"/\\|?*\n\r\t' + t = "".join("_" if c in bad else c for c in (s or "").strip())[:max_len] + t = t.strip(" .") or "run" + return t + + +def resolve_pipeline_run_directory_for_job(job: PipelineJob) -> Path: + """ + 在拉起子进程前固定本次 ``run_dir``（与 ``jd_keyword_pipeline._resolve_pipeline_run_dir`` **同一规则**）。 + 调用方负责 ``mkdir``。 + """ + root = (settings.LOW_GI_PROJECT_ROOT or "").strip() + if not root: + raise RuntimeError("LOW_GI_PROJECT_ROOT 未配置") + project_data = Path(root).resolve() / "data" / "JD" + prd = (job.pipeline_run_dir or "").strip() + kw = (job.keyword or "").strip() + if prd: + p = Path(prd).expanduser() + if not p.is_absolute(): + p = project_data / p + return p.resolve() + import time + + stamp = time.strftime("%Y%m%d_%H%M%S") + seg = _safe_dir_segment_for_job(kw) + return (project_data / "pipeline_runs" / f"{stamp}_{seg}").resolve() + + +def try_write_competitor_report_if_merged_exists( + run_dir: Path, + keyword: str, + *, + report_config: dict[str, Any] | None = None, +) -> None: + """若已有合并表则补写竞品 Markdown（用于子进程被 terminate 后的部分产物）。""" + _, kpl = _jd_crawler_modules() + base = Path(run_dir).resolve() + merged = base / kpl.FILE_MERGED_CSV + if not merged.is_file(): + return + try: + write_competitor_analysis_for_run_dir( + base, keyword, report_config=report_config + ) + except Exception: + pass + + +def _jd_crawler_modules(): + from pipeline.competitor_report import jd_report as jcr # noqa: WPS433 + + root = Path(settings.CRAWLER_JD_ROOT) + if not root.is_dir(): + raise FileNotFoundError(f"爬虫副本目录不存在: {root}") + root_s = str(root.resolve()) + if root_s not in sys.path: + sys.path.insert(0, root_s) + import jd_keyword_pipeline as kpl # noqa: WPS433 + + return jcr, kpl + + +def use_chunked_group_summaries_llm(report_config: dict[str, Any] | None) -> bool: + """ + 是否按矩阵细类**拆分**第五/六/八章等 group 归纳的 LLM 请求（默认开启）。 + + 关闭方式：``report_config`` 中 ``llm_group_summaries_chunk_by_matrix``: false， + 或环境变量 ``MA_LLM_GROUP_SUMMARIES_BULK=1``（恢复单次打包调用）。 + """ + rc = report_config if isinstance(report_config, dict) else {} + if os.environ.get("MA_LLM_GROUP_SUMMARIES_BULK", "").strip().lower() in ( + "1", + "true", + "yes", + ): + return False + ch = rc.get("llm_group_summaries_chunk_by_matrix") + if ch is None: + return True + return bool(ch) + + +def get_default_report_config() -> dict[str, Any]: + """与 ``pipeline.competitor_report.jd_report`` 模块常量一致的默认报告调参（供前端回填）。""" + jcr, _ = _jd_crawler_modules() + return { + "llm_comment_sentiment": True, + "llm_matrix_group_summaries": True, + "llm_comment_group_summaries": True, + "llm_price_group_summaries": True, + "llm_promo_group_summaries": True, + # 全任务第九章大模型长文已弃用：可执行策略由「策略制定」按矩阵细类生成（见 jd_report 第九章固定说明）。 + "llm_strategy_opportunities": False, + "llm_group_summaries_chunk_by_matrix": True, + "chapter8_text_mining_probe": True, + "chapter8_text_mining_probe_live_llm": True, + "chapter8_text_mining_probe_llm_chunked": True, + "chapter8_text_mining_probe_wordcloud": True, + "external_market_table_rows": [ + {"indicator": a, "value_and_scope": b, "source": c, "year": d} + for a, b, c, d in jcr.EXTERNAL_MARKET_TABLE_ROWS + ], + } + + +def merge_report_config_with_defaults( + report_config: dict[str, Any] | None, +) -> dict[str, Any]: + """ + 合并 ``get_default_report_config``：任务里常见仅含部分键；JSON ``null`` 的布尔开关视为未设置， + 否则 ``bool(None)`` 会把第五章矩阵/第六章促销等 LLM 归纳整段关掉。 + """ + eff_rc: dict[str, Any] = ( + dict(report_config) if isinstance(report_config, dict) else {} + ) + for _bk in REPORT_CONFIG_BOOL_KEYS: + if eff_rc.get(_bk) is None: + eff_rc.pop(_bk, None) + for _k, _v in get_default_report_config().items(): + if _k not in eff_rc: + eff_rc[_k] = _v + return eff_rc + + +def get_default_strategy_config() -> dict[str, Any]: + """策略生成页独立默认（与 ``report_config`` 无关），供前端回填与 PATCH 合并。""" + return { + # 默认走大模型润色；用户可在单次生成时勾选「仅规则稿」取消 + "use_llm_default": True, + } + + +def write_competitor_analysis_for_run_dir( + run_dir: Path, + keyword: str, + *, + report_config: dict[str, Any] | None = None, +) -> Path: + """ + 在已有流水线目录上读取 CSV / meta，写入 ``competitor_analysis.md``（不重新爬取）。 + """ + jcr, kpl = _jd_crawler_modules() + kw = (keyword or "").strip() + if not kw: + raise ValueError("keyword 不能为空") + + run_dir = Path(run_dir).resolve() + merged_path = run_dir / kpl.FILE_MERGED_CSV + if not merged_path.is_file(): + raise FileNotFoundError(f"缺少合并表，无法生成报告: {merged_path.name}") + + _, merged_rows = jcr._read_csv_rows(merged_path) + _, search_export_rows = jcr._read_csv_rows(run_dir / kpl.FILE_PC_SEARCH_CSV) + _, comment_rows = jcr._read_csv_rows(run_dir / kpl.FILE_COMMENTS_FLAT_CSV) + + meta_path = run_dir / kpl.FILE_RUN_META_JSON + meta: dict[str, Any] | None = None + if meta_path.is_file(): + try: + meta = json.loads(meta_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + meta = None + + eff_rc = merge_report_config_with_defaults(report_config) + all_tx = _flat_comment_texts(comment_rows) + suggest_path = run_dir / "keyword_suggest_llm.json" + suggest_record: dict[str, Any] = { + "schema_version": 3, + "total_comment_texts": len(all_tx), + } + skip_kw = os.environ.get("MA_SKIP_LLM_KEYWORD_SUGGEST", "").strip().lower() in ( + "1", + "true", + "yes", + ) + if not skip_kw: + try: + from ..llm.keyword_suggest import suggest_focus_keywords_from_all_comments + + brief_pre = jcr.build_competitor_brief( + run_dir=run_dir, + keyword=kw, + merged_rows=merged_rows, + search_export_rows=search_export_rows, + comment_rows=comment_rows, + meta=meta, + report_config=eff_rc, + ) + brief_slice = { + "keyword": brief_pre.get("keyword"), + "comment_focus_keywords": [], + "usage_scenarios": [], + "category_mix_top": (brief_pre.get("category_mix_top") or [])[:6], + "scope": brief_pre.get("scope"), + } + sug = suggest_focus_keywords_from_all_comments( + keyword=kw, + brief_slice=brief_slice, + all_comment_texts=all_tx, + ) + suggest_record.update(sug) + except Exception as e: + suggest_record["error"] = str(e) + suggest_record["suggested_focus_keywords"] = [] + else: + suggest_record["skipped"] = True + suggest_record["suggested_focus_keywords"] = [] + + suggest_record["suggested_scenario_groups"] = [] + suggest_record["scenario_note"] = "预设场景词组已废弃，不再写入 report_config。" + + suggest_path.write_text( + json.dumps(suggest_record, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + (run_dir / "effective_report_config.json").write_text( + json.dumps(eff_rc, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + + brief_final = jcr.build_competitor_brief( + run_dir=run_dir, + keyword=kw, + merged_rows=merged_rows, + search_export_rows=search_export_rows, + comment_rows=comment_rows, + meta=meta, + report_config=eff_rc, + ) + from ..reporting.charts import generate_report_charts + + generate_report_charts(run_dir, brief_final, report_config=eff_rc) + + llm_sentiment_md = "" + sentiment_llm_record: dict[str, Any] = { + "schema_version": 2, + "attempted": False, + "groups": [], + } + skip_sent = os.environ.get( + "MA_SKIP_LLM_COMMENT_SENTIMENT", "" + ).strip().lower() in ("1", "true", "yes") + env_on = os.environ.get("MA_ENABLE_LLM_COMMENT_SENTIMENT", "").strip().lower() in ( + "1", + "true", + "yes", + ) + want_sent = bool(eff_rc.get("llm_comment_sentiment")) or env_on + if want_sent and not skip_sent: + feedback_groups_sg = jcr._consumer_feedback_by_matrix_group( + merged_rows=merged_rows, + comment_rows=comment_rows, + sku_header=MERGED_FIELD_TO_CSV_HEADER["sku_id"], + ) + if not feedback_groups_sg: + sentiment_llm_record["skipped"] = "no_feedback_groups" + else: + sentiment_llm_record["attempted"] = True + from ..llm.generate import generate_comment_sentiment_analysis_llm + + parts_sent: list[str] = [] + grp_logs: list[dict[str, Any]] = [] + sku_h = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + title_h = MERGED_FIELD_TO_CSV_HEADER["title"] + for gname, cr_g, _tu in feedback_groups_sg: + sub_merged = [ + r + for r in merged_rows + if jcr._competitor_matrix_group_key(r) == gname + ] + units, scores = jcr._iter_comment_text_units_and_scores( + cr_g, sub_merged + ) + if len(units) < 2: + grp_logs.append( + { + "group": gname, + "skipped": "insufficient_comment_texts", + "n_texts": len(units), + } + ) + continue + try: + attr_units = jcr._comment_lines_with_product_context( + cr_g, + merged_rows, + sku_header=sku_h, + title_h=title_h, + ) + if len(attr_units) != len(units): + attr_units = list(units) + pl = jcr.build_comment_sentiment_llm_payload( + units, + scores=scores, + attributed_texts=attr_units, + max_samples_positive=16, + max_samples_negative=30, + max_samples_mixed=10, + max_chars_per_review=360, + semantic_pool_max=40, + shuffle_seed=f"{kw}|{gname}", + ) + pl["keyword"] = kw + pl["matrix_group_focus"] = gname + md_one = generate_comment_sentiment_analysis_llm(pl) + parts_sent.append(f"#### {gname}\n\n{md_one.strip()}\n") + grp_logs.append( + {"group": gname, "ok": True, "chars": len(md_one)} + ) + except Exception as e: + grp_logs.append({"group": gname, "ok": False, "error": str(e)}) + sentiment_llm_record["groups"] = grp_logs + llm_sentiment_md = "\n".join(parts_sent).strip() + sentiment_llm_record["chars"] = len(llm_sentiment_md) + if llm_sentiment_md: + sentiment_llm_record["ok"] = True + else: + sentiment_llm_record["ok"] = False + if grp_logs and all("skipped" in x for x in grp_logs): + sentiment_llm_record["skipped"] = "insufficient_comment_texts" + elif grp_logs and any("error" in x for x in grp_logs): + sentiment_llm_record["error"] = "all_groups_failed_or_skipped" + else: + sentiment_llm_record["error"] = "no_output" + elif skip_sent: + sentiment_llm_record["skipped"] = "MA_SKIP_LLM_COMMENT_SENTIMENT" + elif not want_sent: + sentiment_llm_record["skipped"] = "not_enabled" + + (run_dir / "comment_sentiment_llm.json").write_text( + json.dumps(sentiment_llm_record, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + + llm_matrix_md = "" + llm_price_md = "" + llm_promo_md = "" + llm_comment_gr_md = "" + llm_strategy_opp_md = "" + matrix_llm_rec: dict[str, Any] = {"schema_version": 1, "attempted": False} + price_llm_rec: dict[str, Any] = {"schema_version": 1, "attempted": False} + promo_llm_rec: dict[str, Any] = {"schema_version": 1, "attempted": False} + scenario_gr_llm_rec: dict[str, Any] = {"schema_version": 1, "attempted": False} + comment_gr_llm_rec: dict[str, Any] = {"schema_version": 1, "attempted": False} + strategy_opp_llm_rec: dict[str, Any] = {"schema_version": 1, "attempted": False} + sku_h = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + title_h = MERGED_FIELD_TO_CSV_HEADER["title"] + + def _env_on(name: str) -> bool: + return os.environ.get(name, "").strip().lower() in ("1", "true", "yes") + + skip_mx = _env_on("MA_SKIP_LLM_MATRIX_GROUP_SUMMARIES") + skip_pr = _env_on("MA_SKIP_LLM_PRICE_GROUP_SUMMARIES") + skip_po = _env_on("MA_SKIP_LLM_PROMO_GROUP_SUMMARIES") + skip_cg = _env_on("MA_SKIP_LLM_COMMENT_GROUP_SUMMARIES") + want_mx = bool(eff_rc.get("llm_matrix_group_summaries")) or _env_on( + "MA_ENABLE_LLM_MATRIX_GROUP_SUMMARIES" + ) + want_pr = bool(eff_rc.get("llm_price_group_summaries")) or _env_on( + "MA_ENABLE_LLM_PRICE_GROUP_SUMMARIES" + ) + want_po = bool(eff_rc.get("llm_promo_group_summaries")) or _env_on( + "MA_ENABLE_LLM_PROMO_GROUP_SUMMARIES" + ) + want_cg = bool(eff_rc.get("llm_comment_group_summaries")) or _env_on( + "MA_ENABLE_LLM_COMMENT_GROUP_SUMMARIES" + ) + skip_st = _env_on("MA_SKIP_LLM_STRATEGY_OPPORTUNITIES") + want_st = bool(eff_rc.get("llm_strategy_opportunities")) or _env_on( + "MA_ENABLE_LLM_STRATEGY_OPPORTUNITIES" + ) + + use_ch8_probe = bool(eff_rc.get("chapter8_text_mining_probe")) + chapter8_probe_embed_md = "" + ch8_probe_rec: dict[str, Any] = {"schema_version": 1, "attempted": False} + if use_ch8_probe: + ch8_probe_rec["attempted"] = True + try: + from ..demos.chapter8_text_mining_probe import ( + build_markdown as build_ch8_probe_full_md, + markdown_embed_body_for_competitor_report, + ) + + _rc = eff_rc + full_probe = build_ch8_probe_full_md( + run_dir, + min_texts=int(_rc.get("chapter8_probe_min_texts") or 8), + lda_topics_n=int(_rc.get("chapter8_probe_lda_topics") or 4), + top_k_words=int(_rc.get("chapter8_probe_top_k_words") or 30), + cooc_vocab=int(_rc.get("chapter8_probe_cooc_vocab") or 80), + cooc_pairs=int(_rc.get("chapter8_probe_cooc_pairs") or 25), + live_llm=bool(_rc.get("chapter8_text_mining_probe_live_llm", True)), + llm_chunked=bool( + _rc.get("chapter8_text_mining_probe_llm_chunked", True) + ), + wordcloud_enabled=bool( + _rc.get("chapter8_text_mining_probe_wordcloud", True) + ), + wordcloud_max=int(_rc.get("chapter8_probe_wordcloud_max") or 40), + ) + (run_dir / "chapter8_text_mining_probe.md").write_text( + full_probe, encoding="utf-8" + ) + chapter8_probe_embed_md = markdown_embed_body_for_competitor_report( + full_probe + ) + ch8_probe_rec["ok"] = True + ch8_probe_rec["chars_embed"] = len(chapter8_probe_embed_md) + except Exception as e: + ch8_probe_rec["ok"] = False + ch8_probe_rec["error"] = str(e) + + if use_ch8_probe and chapter8_probe_embed_md: + want_cg = False + + chunk_gr = use_chunked_group_summaries_llm(eff_rc) + + if want_mx and not skip_mx and merged_rows: + pl_mx = jcr.build_matrix_groups_llm_payload( + merged_rows, sku_header=sku_h, title_h=title_h + ) + if pl_mx: + matrix_llm_rec["attempted"] = True + try: + if chunk_gr: + from ..llm.generate import ( + generate_matrix_group_summaries_llm_chunked, + ) + + llm_matrix_md = generate_matrix_group_summaries_llm_chunked( + pl_mx, keyword=kw + ) + else: + from ..llm.generate import generate_matrix_group_summaries_llm + + llm_matrix_md = generate_matrix_group_summaries_llm( + pl_mx, keyword=kw + ) + matrix_llm_rec["ok"] = True + matrix_llm_rec["chars"] = len(llm_matrix_md) + matrix_llm_rec["chunked_by_matrix"] = chunk_gr + if chunk_gr: + matrix_llm_rec["chunk_count"] = len(pl_mx) + except Exception as e: + matrix_llm_rec["ok"] = False + matrix_llm_rec["error"] = str(e) + else: + matrix_llm_rec["skipped"] = "empty_matrix_groups_payload" + elif skip_mx: + matrix_llm_rec["skipped"] = "MA_SKIP_LLM_MATRIX_GROUP_SUMMARIES" + elif not want_mx: + matrix_llm_rec["skipped"] = "not_enabled" + + if want_pr and not skip_pr and merged_rows: + pl_pr = jcr.build_price_groups_llm_payload( + merged_rows, sku_header=sku_h, title_h=title_h + ) + if pl_pr: + price_llm_rec["attempted"] = True + try: + if chunk_gr: + from ..llm.generate import ( + generate_price_group_summaries_llm_chunked, + ) + + llm_price_md = generate_price_group_summaries_llm_chunked( + pl_pr, keyword=kw + ) + else: + from ..llm.generate import generate_price_group_summaries_llm + + llm_price_md = generate_price_group_summaries_llm( + pl_pr, keyword=kw + ) + price_llm_rec["ok"] = True + price_llm_rec["chars"] = len(llm_price_md) + price_llm_rec["chunked_by_matrix"] = chunk_gr + if chunk_gr: + price_llm_rec["chunk_count"] = len(pl_pr) + except Exception as e: + price_llm_rec["ok"] = False + price_llm_rec["error"] = str(e) + else: + price_llm_rec["skipped"] = "empty_price_payload" + elif skip_pr: + price_llm_rec["skipped"] = "MA_SKIP_LLM_PRICE_GROUP_SUMMARIES" + elif not want_pr: + price_llm_rec["skipped"] = "not_enabled" + + if want_po and not skip_po and merged_rows: + pl_po = jcr.build_promo_groups_llm_payload( + merged_rows, sku_header=sku_h, title_h=title_h + ) + if pl_po: + promo_llm_rec["attempted"] = True + try: + if chunk_gr: + from ..llm.generate import ( + generate_promo_group_summaries_llm_chunked, + ) + + llm_promo_md = generate_promo_group_summaries_llm_chunked( + pl_po, keyword=kw + ) + else: + from ..llm.generate import generate_promo_group_summaries_llm + + llm_promo_md = generate_promo_group_summaries_llm( + pl_po, keyword=kw + ) + promo_llm_rec["ok"] = True + promo_llm_rec["chars"] = len(llm_promo_md) + promo_llm_rec["chunked_by_matrix"] = chunk_gr + if chunk_gr: + promo_llm_rec["chunk_count"] = len(pl_po) + except Exception as e: + promo_llm_rec["ok"] = False + promo_llm_rec["error"] = str(e) + else: + promo_llm_rec["skipped"] = "empty_promo_payload" + elif skip_po: + promo_llm_rec["skipped"] = "MA_SKIP_LLM_PROMO_GROUP_SUMMARIES" + elif not want_po: + promo_llm_rec["skipped"] = "not_enabled" + + scenario_gr_llm_rec["skipped"] = "preset_scenario_summaries_removed" + + if want_cg and not skip_cg and merged_rows: + fb_cg = jcr._consumer_feedback_by_matrix_group( + merged_rows=merged_rows, + comment_rows=comment_rows, + sku_header=sku_h, + ) + pl_cg = jcr.build_comment_groups_llm_payload( + feedback_groups=fb_cg, + merged_rows=merged_rows, + sku_header=sku_h, + title_h=title_h, + ) + if pl_cg: + comment_gr_llm_rec["attempted"] = True + try: + if chunk_gr: + from ..llm.generate import ( + generate_comment_group_summaries_llm_chunked, + ) + + llm_comment_gr_md = generate_comment_group_summaries_llm_chunked( + pl_cg, keyword=kw + ) + else: + from ..llm.generate import generate_comment_group_summaries_llm + + llm_comment_gr_md = generate_comment_group_summaries_llm( + pl_cg, keyword=kw + ) + comment_gr_llm_rec["ok"] = True + comment_gr_llm_rec["chars"] = len(llm_comment_gr_md) + comment_gr_llm_rec["chunked_by_matrix"] = chunk_gr + if chunk_gr: + comment_gr_llm_rec["chunk_count"] = len(pl_cg) + except Exception as e: + comment_gr_llm_rec["ok"] = False + comment_gr_llm_rec["error"] = str(e) + else: + comment_gr_llm_rec["skipped"] = "empty_comment_groups_payload" + elif skip_cg: + comment_gr_llm_rec["skipped"] = "MA_SKIP_LLM_COMMENT_GROUP_SUMMARIES" + elif not want_cg: + comment_gr_llm_rec["skipped"] = "not_enabled" + + if want_st and not skip_st and isinstance(brief_final, dict) and brief_final: + strategy_opp_llm_rec["attempted"] = True + try: + from ..llm.generate import generate_strategy_opportunities_llm + + _strategy_narratives: dict[str, str] = {} + if (llm_matrix_md or "").strip(): + _strategy_narratives["sec5_matrix_group_summaries"] = llm_matrix_md + if (llm_price_md or "").strip(): + _strategy_narratives["sec6_price_group_summaries"] = llm_price_md + if (llm_promo_md or "").strip(): + _strategy_narratives["sec6_promo_group_summaries"] = llm_promo_md + if use_ch8_probe and (chapter8_probe_embed_md or "").strip(): + _strategy_narratives["sec8_3_text_mining_probe"] = ( + chapter8_probe_embed_md + ) + elif (llm_comment_gr_md or "").strip(): + _strategy_narratives["sec8_3_comment_focus_summaries"] = ( + llm_comment_gr_md + ) + + llm_strategy_opp_md = generate_strategy_opportunities_llm( + brief_final, + keyword=kw, + chapter_llm_narratives=_strategy_narratives or None, + ) + strategy_opp_llm_rec["ok"] = True + strategy_opp_llm_rec["chars"] = len(llm_strategy_opp_md) + strategy_opp_llm_rec["prior_chapter_narrative_keys"] = sorted( + _strategy_narratives.keys() + ) + if (llm_strategy_opp_md or "").strip(): + strategy_opp_llm_rec["markdown"] = llm_strategy_opp_md + except Exception as e: + strategy_opp_llm_rec["ok"] = False + strategy_opp_llm_rec["error"] = str(e) + elif skip_st: + strategy_opp_llm_rec["skipped"] = "MA_SKIP_LLM_STRATEGY_OPPORTUNITIES" + elif not want_st: + strategy_opp_llm_rec["skipped"] = "not_enabled" + + (run_dir / "matrix_groups_llm.json").write_text( + json.dumps(matrix_llm_rec, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + (run_dir / "price_groups_llm.json").write_text( + json.dumps(price_llm_rec, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + (run_dir / "promo_groups_llm.json").write_text( + json.dumps(promo_llm_rec, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + (run_dir / "comment_groups_llm.json").write_text( + json.dumps(comment_gr_llm_rec, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + (run_dir / "scenario_groups_llm.json").write_text( + json.dumps(scenario_gr_llm_rec, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + (run_dir / "strategy_opportunities_llm.json").write_text( + json.dumps(strategy_opp_llm_rec, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + if use_ch8_probe: + (run_dir / "chapter8_text_mining_probe.json").write_text( + json.dumps(ch8_probe_rec, ensure_ascii=False, indent=2), + encoding="utf-8", + ) + + md = jcr.build_competitor_markdown( + run_dir=run_dir, + keyword=kw, + merged_rows=merged_rows, + search_export_rows=search_export_rows, + comment_rows=comment_rows, + meta=meta, + report_config=eff_rc, + llm_sentiment_section_md=llm_sentiment_md or None, + llm_matrix_section_md=llm_matrix_md or None, + llm_price_groups_section_md=llm_price_md or None, + llm_promo_groups_section_md=llm_promo_md or None, + llm_scenario_groups_section_md=None, + llm_comment_groups_section_md=llm_comment_gr_md or None, + llm_strategy_opportunities_section_md=llm_strategy_opp_md or None, + chapter8_text_mining_probe_section_md=chapter8_probe_embed_md or None, + ) + + out_md = run_dir / "competitor_analysis.md" + out_md.write_text(md, encoding="utf-8") + return run_dir + + +def regenerate_competitor_report( + run_dir_str: str, + keyword: str, + *, + report_config: dict[str, Any] | None = None, +) -> Path: + """校验 ``run_dir`` 位于 ``LOW_GI_PROJECT_ROOT/data/JD`` 下后，重写竞品 Markdown。""" + low_root = (settings.LOW_GI_PROJECT_ROOT or "").strip() + if not low_root: + raise RuntimeError("LOW_GI_PROJECT_ROOT 未配置") + base = Path(run_dir_str).expanduser().resolve() + jd_root = (Path(low_root) / "data" / "JD").resolve() + try: + base.relative_to(jd_root) + except ValueError as e: + raise ValueError("run_dir 不在京东数据目录下") from e + return write_competitor_analysis_for_run_dir( + base, keyword, report_config=report_config + ) + + +def write_competitor_analysis_markdown(run_dir_str: str, markdown: str) -> Path: + """将已生成的 Markdown 正文写入 ``run_dir/competitor_analysis.md``（与规则重生成同路径）。""" + low_root = (settings.LOW_GI_PROJECT_ROOT or "").strip() + if not low_root: + raise RuntimeError("LOW_GI_PROJECT_ROOT 未配置") + base = Path(run_dir_str).expanduser().resolve() + jd_root = (Path(low_root) / "data" / "JD").resolve() + try: + base.relative_to(jd_root) + except ValueError as e: + raise ValueError("run_dir 不在京东数据目录下") from e + out = base / "competitor_analysis.md" + out.write_text(markdown or "", encoding="utf-8") + return out + + +def build_competitor_brief_for_job( + run_dir_str: str, + keyword: str, + *, + report_config: dict[str, Any] | None = None, +) -> dict[str, Any]: + """ + 读取 ``run_dir`` 下合并表 / 搜索导出 / 评价 / meta，返回与 Markdown 报告**同一套计数规则**的 **JSON 结构化摘要**（规则驱动）。 + ``run_dir`` 须位于 ``LOW_GI_PROJECT_ROOT/data/JD`` 下。 + """ + low_root = (settings.LOW_GI_PROJECT_ROOT or "").strip() + if not low_root: + raise RuntimeError("LOW_GI_PROJECT_ROOT 未配置") + base = Path(run_dir_str).expanduser().resolve() + jd_root = (Path(low_root) / "data" / "JD").resolve() + try: + base.relative_to(jd_root) + except ValueError as e: + raise ValueError("run_dir 不在京东数据目录下") from e + + jcr, kpl = _jd_crawler_modules() + kw = (keyword or "").strip() + if not kw: + raise ValueError("keyword 不能为空") + + merged_path = base / kpl.FILE_MERGED_CSV + if not merged_path.is_file(): + raise FileNotFoundError(f"缺少合并表，无法生成摘要: {merged_path.name}") + + _, merged_rows = jcr._read_csv_rows(merged_path) + _, search_export_rows = jcr._read_csv_rows(base / kpl.FILE_PC_SEARCH_CSV) + _, comment_rows = jcr._read_csv_rows(base / kpl.FILE_COMMENTS_FLAT_CSV) + + meta_path = base / kpl.FILE_RUN_META_JSON + meta: dict[str, Any] | None = None + if meta_path.is_file(): + try: + meta = json.loads(meta_path.read_text(encoding="utf-8")) + except json.JSONDecodeError: + meta = None + + eff: dict[str, Any] | None = None + if isinstance(report_config, dict): + eff = dict(report_config) + eff_path = base / "effective_report_config.json" + if eff_path.is_file(): + try: + loaded = json.loads(eff_path.read_text(encoding="utf-8")) + if isinstance(loaded, dict) and loaded: + eff = loaded + except json.JSONDecodeError: + pass + + eff_final = merge_report_config_with_defaults( + eff if isinstance(eff, dict) else None + ) + return jcr.build_competitor_brief( + run_dir=base, + keyword=kw, + merged_rows=merged_rows, + search_export_rows=search_export_rows, + comment_rows=comment_rows, + meta=meta, + report_config=eff_final, + ) + + +def run_jd_keyword_and_report( + keyword: str, + *, + max_skus: int | None = None, + page_start: int | None = None, + page_to: int | None = None, + pipeline_run_dir: str | None = None, + cookie_file_path: str | None = None, + pvid: str | None = None, + request_delay: str | None = None, + list_pages: str | None = None, + scenario_filter_enabled: bool | None = None, + report_config: dict[str, Any] | None = None, + cancel_check: Any | None = None, +) -> Path: + _, kpl = _jd_crawler_modules() + + kw = (keyword or "").strip() + if not kw: + raise ValueError("keyword 不能为空") + + backup: dict[str, Any] = {} + if cancel_check is not None: + backup["PIPELINE_CANCEL_CHECK"] = getattr(kpl, "PIPELINE_CANCEL_CHECK", None) + kpl.PIPELINE_CANCEL_CHECK = cancel_check + try: + if max_skus is not None: + backup["MAX_SKUS"] = kpl.MAX_SKUS + kpl.MAX_SKUS = max(1, int(max_skus)) + if page_start is not None: + backup["PAGE_START"] = kpl.PAGE_START + kpl.PAGE_START = max(1, int(page_start)) + if page_to is not None: + backup["PAGE_TO"] = kpl.PAGE_TO + kpl.PAGE_TO = max(1, int(page_to)) + + prd = (pipeline_run_dir or "").strip() + if prd: + backup["PIPELINE_RUN_DIR"] = kpl.PIPELINE_RUN_DIR + kpl.PIPELINE_RUN_DIR = prd + + cf = (cookie_file_path or "").strip() + if cf: + backup["PIPELINE_COOKIE_FILE"] = kpl.PIPELINE_COOKIE_FILE + kpl.PIPELINE_COOKIE_FILE = cf + + pv = (pvid or "").strip() + if pv: + backup["PVID"] = kpl.PVID + kpl.PVID = pv + + rd = (request_delay or "").strip() + if rd: + backup["REQUEST_DELAY"] = kpl.REQUEST_DELAY + kpl.REQUEST_DELAY = rd + + lp = (list_pages or "").strip() + if lp: + backup["LIST_PAGES"] = kpl.LIST_PAGES + kpl.LIST_PAGES = lp + + if scenario_filter_enabled is not None: + backup["SCENARIO_FILTER_ENABLED"] = kpl.SCENARIO_FILTER_ENABLED + kpl.SCENARIO_FILTER_ENABLED = bool(scenario_filter_enabled) + + run_dir = kpl.main(keyword=kw) + except kpl.PipelineCancelled as e: + run_dir_path = Path(e.run_dir).resolve() + merged = run_dir_path / kpl.FILE_MERGED_CSV + if merged.is_file(): + try: + write_competitor_analysis_for_run_dir( + run_dir_path, kw, report_config=report_config + ) + except Exception: + pass + raise + finally: + for name, val in backup.items(): + setattr(kpl, name, val) + + return write_competitor_analysis_for_run_dir( + Path(run_dir).resolve(), kw, report_config=report_config + ) diff --git a/backend/pipeline/jd_runner.py b/backend/pipeline/jd_runner.py deleted file mode 100644 index ad34844..0000000 --- a/backend/pipeline/jd_runner.py +++ /dev/null @@ -1,490 +0,0 @@ -""" -使用 ``crawler_copy/jd_pc_search`` 中的副本脚本执行流水线并生成竞品 Markdown。 -依赖环境变量 ``LOW_GI_PROJECT_ROOT``（由 Django settings 从 ``market_assistant/.env`` 注入）。 -""" -from __future__ import annotations - -import json -import os -import re -import sys -from pathlib import Path -from typing import Any - -from django.conf import settings - -from .models import PipelineJob - - -def merge_llm_supplement_with_rules_report(llm_md: str, rules_md: str) -> str: - """ - **以规则引擎全文为正文**（含 §5 完整竞品矩阵、各章内嵌统计图与表格）。 - - 大模型稿仅作为开篇「速读/策略补充」插入在「## 一、」之前，**不得**再用纯 LLM 稿 - 覆盖规则正文（否则会丢失矩阵与章节结构）。 - """ - body = (rules_md or "").strip() - sup = (llm_md or "").strip() - if not sup: - return body - if not body: - return sup - block = ( - "---\n\n" - "## 大模型速读与策略要点（补充）\n\n" - "> **说明**：以下由大模型依据结构化摘要生成，便于速览；**完整竞品对比矩阵、全部表格、" - "统计图与定量口径以正文各章（尤其 §5）为准**，请勿仅依据本段理解 SKU 明细。\n\n" - f"{sup}\n" - ) - marker = "\n---\n\n## 一、研究范围、数据来源与局限" - if marker in body: - return body.replace(marker, "\n" + block + marker, 1) - return block + "\n---\n\n" + body - - -def merge_llm_report_with_rules_charts(llm_md: str, rules_md: str) -> str: - """兼容旧名：等价于 ``merge_llm_supplement_with_rules_report``。""" - return merge_llm_supplement_with_rules_report(llm_md, rules_md) - - -def _flat_comment_texts(comment_rows: list[dict[str, str]]) -> list[str]: - """全部非空评价正文（与报告统计同源）。""" - out: list[str] = [] - for row in comment_rows: - t = (row.get("tagCommentContent") or "").strip() - if t: - out.append(t) - return out - - -def _safe_dir_segment_for_job(s: str, max_len: int = 48) -> str: - """与 ``jd_keyword_pipeline._safe_dir_segment`` 一致，避免多线程下改模块全局。""" - bad = '<>:"/\\|?*\n\r\t' - t = "".join("_" if c in bad else c for c in (s or "").strip())[:max_len] - t = t.strip(" .") or "run" - return t - - -def resolve_pipeline_run_directory_for_job(job: PipelineJob) -> Path: - """ - 在拉起子进程前固定本次 ``run_dir``（与 ``jd_keyword_pipeline._resolve_pipeline_run_dir`` 同口径）。 - 调用方负责 ``mkdir``。 - """ - root = (settings.LOW_GI_PROJECT_ROOT or "").strip() - if not root: - raise RuntimeError("LOW_GI_PROJECT_ROOT 未配置") - project_data = Path(root).resolve() / "data" / "JD" - prd = (job.pipeline_run_dir or "").strip() - kw = (job.keyword or "").strip() - if prd: - p = Path(prd).expanduser() - if not p.is_absolute(): - p = project_data / p - return p.resolve() - import time - - stamp = time.strftime("%Y%m%d_%H%M%S") - seg = _safe_dir_segment_for_job(kw) - return (project_data / "pipeline_runs" / f"{stamp}_{seg}").resolve() - - -def try_write_competitor_report_if_merged_exists( - run_dir: Path, - keyword: str, - *, - report_config: dict[str, Any] | None = None, -) -> None: - """若已有合并表则补写竞品 Markdown（用于子进程被 terminate 后的部分产物）。""" - _, kpl = _jd_crawler_modules() - base = Path(run_dir).resolve() - merged = base / kpl.FILE_MERGED_CSV - if not merged.is_file(): - return - try: - write_competitor_analysis_for_run_dir( - base, keyword, report_config=report_config - ) - except Exception: - pass - - -def _jd_crawler_modules(): - root = Path(settings.CRAWLER_JD_ROOT) - if not root.is_dir(): - raise FileNotFoundError(f"爬虫副本目录不存在: {root}") - root_s = str(root.resolve()) - if root_s not in sys.path: - sys.path.insert(0, root_s) - import jd_competitor_report as jcr # noqa: WPS433 - import jd_keyword_pipeline as kpl # noqa: WPS433 - - return jcr, kpl - - -def get_default_report_config() -> dict[str, Any]: - """与 ``jd_competitor_report`` 模块常量一致的默认报告调参（供前端回填）。""" - jcr, _ = _jd_crawler_modules() - return { - "llm_comment_sentiment": False, - "comment_focus_words": list(jcr.COMMENT_FOCUS_WORDS), - "comment_scenario_groups": [ - {"label": lbl, "triggers": list(trs)} - for lbl, trs in jcr.COMMENT_SCENARIO_GROUPS - ], - "external_market_table_rows": [ - {"indicator": a, "value_and_scope": b, "source": c, "year": d} - for a, b, c, d in jcr.EXTERNAL_MARKET_TABLE_ROWS - ], - } - - -def write_competitor_analysis_for_run_dir( - run_dir: Path, - keyword: str, - *, - report_config: dict[str, Any] | None = None, -) -> Path: - """ - 在已有流水线目录上读取 CSV / meta，写入 ``competitor_analysis.md``（不重新爬取）。 - """ - jcr, kpl = _jd_crawler_modules() - kw = (keyword or "").strip() - if not kw: - raise ValueError("keyword 不能为空") - - run_dir = Path(run_dir).resolve() - merged_path = run_dir / kpl.FILE_MERGED_CSV - if not merged_path.is_file(): - raise FileNotFoundError(f"缺少合并表，无法生成报告: {merged_path.name}") - - _, merged_rows = jcr._read_csv_rows(merged_path) - _, search_export_rows = jcr._read_csv_rows(run_dir / kpl.FILE_PC_SEARCH_CSV) - _, comment_rows = jcr._read_csv_rows(run_dir / kpl.FILE_COMMENTS_FLAT_CSV) - - meta_path = run_dir / kpl.FILE_RUN_META_JSON - meta: dict[str, Any] | None = None - if meta_path.is_file(): - try: - meta = json.loads(meta_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - meta = None - - eff_rc: dict[str, Any] = ( - dict(report_config) if isinstance(report_config, dict) else {} - ) - all_tx = _flat_comment_texts(comment_rows) - suggest_path = run_dir / "keyword_suggest_llm.json" - suggest_record: dict[str, Any] = { - "schema_version": 2, - "total_comment_texts": len(all_tx), - } - skip_kw = os.environ.get("MA_SKIP_LLM_KEYWORD_SUGGEST", "").strip().lower() in ( - "1", - "true", - "yes", - ) - if not skip_kw: - try: - from .llm_keyword_suggest import suggest_focus_keywords_from_all_comments - - brief_pre = jcr.build_competitor_brief( - run_dir=run_dir, - keyword=kw, - merged_rows=merged_rows, - search_export_rows=search_export_rows, - comment_rows=comment_rows, - meta=meta, - report_config=eff_rc, - ) - brief_slice = { - "keyword": brief_pre.get("keyword"), - "comment_focus_keywords": ( - brief_pre.get("comment_focus_keywords") or [] - )[:20], - "usage_scenarios": (brief_pre.get("usage_scenarios") or [])[:8], - "category_mix_top": (brief_pre.get("category_mix_top") or [])[:6], - "scope": brief_pre.get("scope"), - } - sug = suggest_focus_keywords_from_all_comments( - keyword=kw, - brief_slice=brief_slice, - all_comment_texts=all_tx, - ) - suggest_record.update(sug) - base_words = list(eff_rc.get("comment_focus_words") or []) - for w in sug.get("suggested_focus_keywords") or []: - if isinstance(w, str): - t = w.strip() - if t and t not in base_words: - base_words.append(t) - eff_rc["comment_focus_words"] = base_words[:80] - except Exception as e: - suggest_record["error"] = str(e) - suggest_record["suggested_focus_keywords"] = [] - else: - suggest_record["skipped"] = True - suggest_record["suggested_focus_keywords"] = [] - - suggest_path.write_text( - json.dumps(suggest_record, ensure_ascii=False, indent=2), - encoding="utf-8", - ) - (run_dir / "effective_report_config.json").write_text( - json.dumps(eff_rc, ensure_ascii=False, indent=2), - encoding="utf-8", - ) - - brief_final = jcr.build_competitor_brief( - run_dir=run_dir, - keyword=kw, - merged_rows=merged_rows, - search_export_rows=search_export_rows, - comment_rows=comment_rows, - meta=meta, - report_config=eff_rc, - ) - from .report_charts import generate_report_charts - - generate_report_charts(run_dir, brief_final) - - llm_sentiment_md = "" - sentiment_llm_record: dict[str, Any] = { - "schema_version": 1, - "attempted": False, - } - skip_sent = os.environ.get( - "MA_SKIP_LLM_COMMENT_SENTIMENT", "" - ).strip().lower() in ("1", "true", "yes") - env_on = os.environ.get("MA_ENABLE_LLM_COMMENT_SENTIMENT", "").strip().lower() in ( - "1", - "true", - "yes", - ) - want_sent = bool(eff_rc.get("llm_comment_sentiment")) or env_on - if want_sent and not skip_sent: - comment_units = jcr._iter_comment_text_units(comment_rows, merged_rows) - if len(comment_units) >= 2: - sentiment_llm_record["attempted"] = True - try: - from .llm_generate import generate_comment_sentiment_analysis_llm - - pl = jcr.build_comment_sentiment_llm_payload(comment_units) - pl["keyword"] = kw - llm_sentiment_md = generate_comment_sentiment_analysis_llm(pl) - sentiment_llm_record["ok"] = True - sentiment_llm_record["chars"] = len(llm_sentiment_md) - except Exception as e: - sentiment_llm_record["ok"] = False - sentiment_llm_record["error"] = str(e) - else: - sentiment_llm_record["skipped"] = "insufficient_comment_texts" - elif skip_sent: - sentiment_llm_record["skipped"] = "MA_SKIP_LLM_COMMENT_SENTIMENT" - elif not want_sent: - sentiment_llm_record["skipped"] = "not_enabled" - - (run_dir / "comment_sentiment_llm.json").write_text( - json.dumps(sentiment_llm_record, ensure_ascii=False, indent=2), - encoding="utf-8", - ) - - md = jcr.build_competitor_markdown( - run_dir=run_dir, - keyword=kw, - merged_rows=merged_rows, - search_export_rows=search_export_rows, - comment_rows=comment_rows, - meta=meta, - report_config=eff_rc, - llm_sentiment_section_md=llm_sentiment_md or None, - ) - out_md = run_dir / "competitor_analysis.md" - out_md.write_text(md, encoding="utf-8") - return run_dir - - -def regenerate_competitor_report( - run_dir_str: str, - keyword: str, - *, - report_config: dict[str, Any] | None = None, -) -> Path: - """校验 ``run_dir`` 位于 ``LOW_GI_PROJECT_ROOT/data/JD`` 下后，重写竞品 Markdown。""" - low_root = (settings.LOW_GI_PROJECT_ROOT or "").strip() - if not low_root: - raise RuntimeError("LOW_GI_PROJECT_ROOT 未配置") - base = Path(run_dir_str).expanduser().resolve() - jd_root = (Path(low_root) / "data" / "JD").resolve() - try: - base.relative_to(jd_root) - except ValueError as e: - raise ValueError("run_dir 不在京东数据目录下") from e - return write_competitor_analysis_for_run_dir( - base, keyword, report_config=report_config - ) - - -def write_competitor_analysis_markdown(run_dir_str: str, markdown: str) -> Path: - """将已生成的 Markdown 正文写入 ``run_dir/competitor_analysis.md``（与规则重生成同路径）。""" - low_root = (settings.LOW_GI_PROJECT_ROOT or "").strip() - if not low_root: - raise RuntimeError("LOW_GI_PROJECT_ROOT 未配置") - base = Path(run_dir_str).expanduser().resolve() - jd_root = (Path(low_root) / "data" / "JD").resolve() - try: - base.relative_to(jd_root) - except ValueError as e: - raise ValueError("run_dir 不在京东数据目录下") from e - out = base / "competitor_analysis.md" - out.write_text(markdown or "", encoding="utf-8") - return out - - -def build_competitor_brief_for_job( - run_dir_str: str, - keyword: str, - *, - report_config: dict[str, Any] | None = None, -) -> dict[str, Any]: - """ - 读取 ``run_dir`` 下合并表 / 搜索导出 / 评价 / meta，返回与 Markdown 报告同口径的 **JSON 结构化摘要**（规则驱动）。 - ``run_dir`` 须位于 ``LOW_GI_PROJECT_ROOT/data/JD`` 下。 - """ - low_root = (settings.LOW_GI_PROJECT_ROOT or "").strip() - if not low_root: - raise RuntimeError("LOW_GI_PROJECT_ROOT 未配置") - base = Path(run_dir_str).expanduser().resolve() - jd_root = (Path(low_root) / "data" / "JD").resolve() - try: - base.relative_to(jd_root) - except ValueError as e: - raise ValueError("run_dir 不在京东数据目录下") from e - - jcr, kpl = _jd_crawler_modules() - kw = (keyword or "").strip() - if not kw: - raise ValueError("keyword 不能为空") - - merged_path = base / kpl.FILE_MERGED_CSV - if not merged_path.is_file(): - raise FileNotFoundError(f"缺少合并表，无法生成摘要: {merged_path.name}") - - _, merged_rows = jcr._read_csv_rows(merged_path) - _, search_export_rows = jcr._read_csv_rows(base / kpl.FILE_PC_SEARCH_CSV) - _, comment_rows = jcr._read_csv_rows(base / kpl.FILE_COMMENTS_FLAT_CSV) - - meta_path = base / kpl.FILE_RUN_META_JSON - meta: dict[str, Any] | None = None - if meta_path.is_file(): - try: - meta = json.loads(meta_path.read_text(encoding="utf-8")) - except json.JSONDecodeError: - meta = None - - eff: dict[str, Any] | None = None - if isinstance(report_config, dict): - eff = dict(report_config) - eff_path = base / "effective_report_config.json" - if eff_path.is_file(): - try: - loaded = json.loads(eff_path.read_text(encoding="utf-8")) - if isinstance(loaded, dict) and loaded: - eff = loaded - except json.JSONDecodeError: - pass - - return jcr.build_competitor_brief( - run_dir=base, - keyword=kw, - merged_rows=merged_rows, - search_export_rows=search_export_rows, - comment_rows=comment_rows, - meta=meta, - report_config=eff, - ) - - -def run_jd_keyword_and_report( - keyword: str, - *, - max_skus: int | None = None, - page_start: int | None = None, - page_to: int | None = None, - pipeline_run_dir: str | None = None, - cookie_file_path: str | None = None, - pvid: str | None = None, - request_delay: str | None = None, - list_pages: str | None = None, - scenario_filter_enabled: bool | None = None, - report_config: dict[str, Any] | None = None, - cancel_check: Any | None = None, -) -> Path: - _, kpl = _jd_crawler_modules() - - kw = (keyword or "").strip() - if not kw: - raise ValueError("keyword 不能为空") - - backup: dict[str, Any] = {} - if cancel_check is not None: - backup["PIPELINE_CANCEL_CHECK"] = getattr(kpl, "PIPELINE_CANCEL_CHECK", None) - kpl.PIPELINE_CANCEL_CHECK = cancel_check - try: - if max_skus is not None: - backup["MAX_SKUS"] = kpl.MAX_SKUS - kpl.MAX_SKUS = max(1, int(max_skus)) - if page_start is not None: - backup["PAGE_START"] = kpl.PAGE_START - kpl.PAGE_START = max(1, int(page_start)) - if page_to is not None: - backup["PAGE_TO"] = kpl.PAGE_TO - kpl.PAGE_TO = max(1, int(page_to)) - - prd = (pipeline_run_dir or "").strip() - if prd: - backup["PIPELINE_RUN_DIR"] = kpl.PIPELINE_RUN_DIR - kpl.PIPELINE_RUN_DIR = prd - - cf = (cookie_file_path or "").strip() - if cf: - backup["PIPELINE_COOKIE_FILE"] = kpl.PIPELINE_COOKIE_FILE - kpl.PIPELINE_COOKIE_FILE = cf - - pv = (pvid or "").strip() - if pv: - backup["PVID"] = kpl.PVID - kpl.PVID = pv - - rd = (request_delay or "").strip() - if rd: - backup["REQUEST_DELAY"] = kpl.REQUEST_DELAY - kpl.REQUEST_DELAY = rd - - lp = (list_pages or "").strip() - if lp: - backup["LIST_PAGES"] = kpl.LIST_PAGES - kpl.LIST_PAGES = lp - - if scenario_filter_enabled is not None: - backup["SCENARIO_FILTER_ENABLED"] = kpl.SCENARIO_FILTER_ENABLED - kpl.SCENARIO_FILTER_ENABLED = bool(scenario_filter_enabled) - - run_dir = kpl.main(keyword=kw) - except kpl.PipelineCancelled as e: - run_dir_path = Path(e.run_dir).resolve() - merged = run_dir_path / kpl.FILE_MERGED_CSV - if merged.is_file(): - try: - write_competitor_analysis_for_run_dir( - run_dir_path, kw, report_config=report_config - ) - except Exception: - pass - raise - finally: - for name, val in backup.items(): - setattr(kpl, name, val) - - return write_competitor_analysis_for_run_dir( - Path(run_dir).resolve(), kw, report_config=report_config - ) diff --git a/backend/pipeline/llm/__init__.py b/backend/pipeline/llm/__init__.py new file mode 100644 index 0000000..7e58f56 --- /dev/null +++ b/backend/pipeline/llm/__init__.py @@ -0,0 +1 @@ +"""竞品报告相关的大模型调用（关键词建议、章节生成等）。""" diff --git a/backend/pipeline/llm/generate.py b/backend/pipeline/llm/generate.py new file mode 100644 index 0000000..ec9415a --- /dev/null +++ b/backend/pipeline/llm/generate.py @@ -0,0 +1,88 @@ +""" +竞品报告 / 策略稿的**大模型生成**：经 ``pipeline.llm`` → ``openai_gateway.chat_completion_text``（OpenAI 兼容）， +与配料多模态识别共用 `OPENAI_*` / `LLM_*` 环境配置。 + +实现已拆分为子模块（``llm_client``、``generate_*``），本模块保留对外符号以兼容 +``from pipeline.llm.generate import …`` 与测试中的 patch 路径。 +""" +from __future__ import annotations + +from .generate_competitor_full import ( + REPORT_SYSTEM, + REPORT_USER_PREFIX, + generate_competitor_report_markdown_llm, +) +from .generate_group_summaries import ( + COMMENT_GROUPS_SYSTEM, + COMMENT_GROUPS_USER_PREFIX, + MATRIX_GROUPS_SYSTEM, + MATRIX_GROUPS_USER_PREFIX, + PRICE_GROUPS_SYSTEM, + PRICE_GROUPS_USER_PREFIX, + PROMO_GROUPS_SYSTEM, + PROMO_GROUPS_USER_PREFIX, + _join_chunked_group_markdown, + generate_comment_group_summaries_llm, + generate_comment_group_summaries_llm_chunked, + generate_matrix_group_summaries_llm, + generate_matrix_group_summaries_llm_chunked, + generate_price_group_summaries_llm, + generate_price_group_summaries_llm_chunked, + generate_promo_group_summaries_llm, + generate_promo_group_summaries_llm_chunked, +) +from .generate_sections import ( + BRIDGE_SECTIONS_SYSTEM, + SENTIMENT_LLM_SYSTEM, + generate_comment_sentiment_analysis_llm, + generate_section_bridges_llm, + split_competitor_report_for_bridges, +) +from .generate_strategy import ( + STRATEGY_OPPORTUNITIES_SYSTEM, + STRATEGY_OPPORTUNITIES_USER_PREFIX, + STRATEGY_SYSTEM, + STRATEGY_USER_PREFIX, + generate_strategy_draft_markdown_llm, + generate_strategy_opportunities_llm, + resolve_strategy_draft_llm_input_snapshot, + strategy_decisions_substantive, +) +from .llm_client import call_llm as _call_llm + +__all__ = [ + "BRIDGE_SECTIONS_SYSTEM", + "COMMENT_GROUPS_SYSTEM", + "COMMENT_GROUPS_USER_PREFIX", + "MATRIX_GROUPS_SYSTEM", + "MATRIX_GROUPS_USER_PREFIX", + "PRICE_GROUPS_SYSTEM", + "PRICE_GROUPS_USER_PREFIX", + "PROMO_GROUPS_SYSTEM", + "PROMO_GROUPS_USER_PREFIX", + "REPORT_SYSTEM", + "REPORT_USER_PREFIX", + "SENTIMENT_LLM_SYSTEM", + "STRATEGY_OPPORTUNITIES_SYSTEM", + "STRATEGY_OPPORTUNITIES_USER_PREFIX", + "STRATEGY_SYSTEM", + "STRATEGY_USER_PREFIX", + "_call_llm", + "_join_chunked_group_markdown", + "generate_comment_group_summaries_llm", + "generate_comment_group_summaries_llm_chunked", + "generate_comment_sentiment_analysis_llm", + "generate_competitor_report_markdown_llm", + "generate_matrix_group_summaries_llm", + "generate_matrix_group_summaries_llm_chunked", + "generate_price_group_summaries_llm", + "generate_price_group_summaries_llm_chunked", + "generate_promo_group_summaries_llm", + "generate_promo_group_summaries_llm_chunked", + "generate_section_bridges_llm", + "generate_strategy_draft_markdown_llm", + "generate_strategy_opportunities_llm", + "resolve_strategy_draft_llm_input_snapshot", + "split_competitor_report_for_bridges", + "strategy_decisions_substantive", +] diff --git a/backend/pipeline/llm/generate_competitor_full.py b/backend/pipeline/llm/generate_competitor_full.py new file mode 100644 index 0000000..a916d65 --- /dev/null +++ b/backend/pipeline/llm/generate_competitor_full.py @@ -0,0 +1,62 @@ +"""竞品报告 8.5 节：整篇大模型补充（基于结构化 brief）。""" +from __future__ import annotations + +import json +from typing import Any + +from ..reporting.brief_compact import compact_brief_for_llm +from .llm_client import call_llm, estimate_chat_input_tokens, llm_context_window_size + +REPORT_SYSTEM = """你是业务与产品读者顾问。输入 JSON 含 `keyword`、`competitor_brief`（可能经裁剪）、 +`matrix_overview_for_llm`（按细分类目的 SKU 数与品牌样本）。 + +你的输出将**嵌入在规则报告第八章末**（作为「### 8.5 …」的正文，系统已加小节标题与说明），**紧接在** +消费者反馈（第八章第一至三节）**之后**、第九章策略**之前**。因此写的是**具体分析型补充**，不是篇首速读块。 + +所有数字、占比、条数、品牌名、价格区间等**必须严格来自输入 JSON**，禁止编造未在输入中出现的定量结论。 + +**硬性禁止**： +- 正文中**勿**写「第九章」「策略与机会」等与宿主文档已有标题**重复**的章名、小节名或起首套话；本段小节仅用 ``####`` 业务主题； +- **不要**使用「## 一」「## 八」等会打乱宿主文档的顶级章节号；请使用 ``####`` 或必要时 ``###`` 作为本段内小节标题； +- **不要**输出完整报告目录或复述「研究范围与方法」长章； +- **不要**撰写 Markdown 表格版「竞品对比矩阵」或罗列 SKU 明细——**正文已含矩阵**，此处只做分组级语义归纳； +- **不要使用** CR1、CR3 等集中度缩写作主表述；集中度请用「第一大店铺/品牌份额」「前三家合计份额」；输入中的英文字段名勿照抄进正文，请写成中文业务用语。 +- **店铺集中度（硬性）**：**禁止**编造「京东自营占比」「自营 SKU 超 X%」「POP 与自营比例」等**输入 JSON 未出现的**数字或店铺类型结论。仅当 `competitor_brief.concentration`、`list_shop_mix_top` 等字段中出现具体店铺名与份额时方可复述；若含 `unique_sku_basis`，须区分 **按列表行** 与 **按去重 SKU**，**禁止**将列表曝光写成「市场份额」或笼统「SKU 占比」。 + +**请输出**（仅输出将置于 8.5 小节 下的正文，不要自造「### 8.5」标题行）： +- **Markdown**，约 **800～1500 字**； +- 建议用 ``####`` 组织：**执行摘要级要点**、**竞争与价盘**、**用户声量与负向事由**（须归纳用户在抱怨什么类型的问题，而非只堆关键词）、**后续可验证动作（假设）**（不写第九章目录或重复策略章内容）； +- **归因与引语（硬性）**：`consumer_feedback_by_matrix_group` 等为**跨 SKU/跨店铺的关键词子串或条数统计**，**不能**单独据此推断「某一店铺某一单品」的结论。 + - **具体体验句式（含口感、包装等）**须以正文 **第八章** 中带 ``【细类｜SKU｜品名｜店铺】`` 前缀的抽样或文本挖掘归纳为准；本段**不要**新增无前缀、无店铺/品名/SKU 指向的「」引语。 + - 若写口感、包装、物流、价格等**聚合**维度，须**写明统计范围**；可结合 `matrix_overview_for_llm` 谈细类结构；**可一句**引导读者「见第八章按店铺/品名的评价归纳」。 + - 若第八章已归纳带店铺与 SKU 的负向主题，本段**只做执行摘要级收束**，勿重复编造新引文。 +- 语气专业、中文；某类信息在输入中缺失时**一句带过数据缺口**即可，**禁止**输出「本段未提供该项」等套话占位。""" + +REPORT_USER_PREFIX = """请根据以下 JSON 撰写上文所述 8.5 小节 嵌入段落（Markdown 正文，勿加 ### 8.5 标题）。\n\n""" + + +def generate_competitor_report_markdown_llm(brief: dict[str, Any], keyword: str) -> str: + ctx = llm_context_window_size() + buf = 256 + input_budget = ctx - buf - 256 + + caps = (88_000, 64_000, 48_000, 34_000, 24_000, 16_000, 11_000, 7_500) + user = "" + for max_chars in caps: + compact = compact_brief_for_llm(brief, max_chars=max_chars) + payload = { + "keyword": keyword, + "competitor_brief": compact, + "matrix_overview_for_llm": compact.get("matrix_overview_for_llm") or [], + } + raw = json.dumps(payload, ensure_ascii=False) + user = REPORT_USER_PREFIX + raw + if estimate_chat_input_tokens(REPORT_SYSTEM, user) < input_budget: + return call_llm(REPORT_SYSTEM, user) + + tail = "\n\n…（JSON 已截断以适配上下文；仅依据可见字段撰写，勿编造截断外数字。）\n" + room = max(0, int((input_budget - 800) / 0.55) - len(REPORT_SYSTEM) - len(REPORT_USER_PREFIX) - len(tail)) + if room < 2000: + room = 2000 + user = (REPORT_USER_PREFIX + raw[:room] + tail) if raw else (REPORT_USER_PREFIX + "{}" + tail) + return call_llm(REPORT_SYSTEM, user) diff --git a/backend/pipeline/llm/generate_group_summaries.py b/backend/pipeline/llm/generate_group_summaries.py new file mode 100644 index 0000000..c9662bc --- /dev/null +++ b/backend/pipeline/llm/generate_group_summaries.py @@ -0,0 +1,469 @@ +"""第五至第八章各细类归纳：矩阵/评论/场景/价盘/促销及分块调用。""" +from __future__ import annotations + +import json +from typing import Any + +from .llm_client import call_llm, estimate_chat_input_tokens, llm_context_window_size + +MATRIX_GROUPS_SYSTEM = """你是竞品分析顾问。输入为 JSON：``keyword`` 与 ``groups`` 数组。 +每个 group 含 ``group``（细分类目名）、``sku_count``、``price_stats``（该细类深入合并行可解析展示价的 min/max/median/mean/n，与 **第六章「按细类价盘」** 分位数表同源；无则 n=0 或缺字段）、 +``lines``（该细类下若干 SKU 的标题/卖点/配料**摘录**，均来自页面抓取拼接，可能截断）。 + +请**为每个细类**输出一小段 Markdown（全部 groups 都要写，顺序与输入一致）： +- 以 ``#### `` + 与该 group 字段**完全一致**的细类名作为小节标题（不要使用 ``##`` 一级标题）； +- 每段约 **100～200 字**中文：**主体**归纳该细类下**卖点表述共性**、**配料类型/宣称共性**（摘录中无配料则写「配料摘录较少」）；**品牌格局**可一句概括（仅依据摘录中可见品牌/系列，勿编造销量排名）； +- **价带/价位**：若 ``price_stats.n`` 为大于 0 的整数，**仅允许**用该对象里的数值写价带（如 min～max、中位数），且须与 ``price_stats`` **完全一致**，**禁止**写「价格带未明确」「未体现具体价位」「多为中端」等**与上述数值相矛盾**的表述；若 n=0 或无可信数值，**不要猜测价位**，可写一句「深入样本可解析数值价不足，价盘以 **第六章** 表格为准」； +- **禁止**输出 Markdown 表格、禁止逐条复述 SKU 明细表；勿编造功效、认证； +- 若 ``lines`` 很少，明确写「样本较少，归纳供启发」。 + +总输出约 **800～3500 字**（细类多则偏长）。仅输出正文 Markdown，不要用代码围栏包裹全文。""" + + +MATRIX_GROUPS_USER_PREFIX = ( + "请根据以下 JSON 撰写竞品报告第五章末「细类要点归纳」正文（Markdown）。\n\n" +) + + +def generate_matrix_group_summaries_llm( + groups: list[dict[str, Any]], *, keyword: str +) -> str: + trimmed: list[dict[str, Any]] = [] + for g in groups: + if not isinstance(g, dict): + continue + g2 = dict(g) + ln = g2.get("lines") + if isinstance(ln, list) and len(ln) > 22: + g2["lines"] = ln[:22] + trimmed.append(g2) + payload = {"keyword": keyword, "groups": trimmed} + raw = json.dumps(payload, ensure_ascii=False) + if len(raw) > 95_000: + for g2 in trimmed: + ln = g2.get("lines") + if isinstance(ln, list) and len(ln) > 12: + g2["lines"] = ln[:12] + raw = json.dumps({"keyword": keyword, "groups": trimmed}, ensure_ascii=False) + user = MATRIX_GROUPS_USER_PREFIX + raw + return call_llm(MATRIX_GROUPS_SYSTEM, user) + + +COMMENT_GROUPS_SYSTEM = """你是用户研究与品类顾问。输入为 JSON：``keyword`` 与 ``groups``。 +每个 group 含 ``group``（与 第五章矩阵一致的细分类目名）、``comment_flat_rows``、``effective_text_lines``（该细类下从评价中抽取的短文本单元）、``sample_text_snippets``（带 SKU/品名/店铺前缀的评价短摘录，已截断）。 +摘录行通常以 ``【细类：…｜SKU：…｜品名：…｜店铺：…】`` 开头：**品名/SKU/店铺**表示该句具体出自哪条链接；归纳时若引用原话，**须交代是「哪家店、哪条 SKU、哪款品名」上的反馈**，勿只写「有用户说口感差」而不指代产品。 +请**以整句语义**判断褒贬（如「软硬适中」「没那么甜」常为满意表述），不得仅凭片段词就写成负面结论。 + +请**为每个细类**输出一小段 Markdown（全部 groups 都要写，顺序与输入一致）： +- 以 ``#### `` + 与该 group 字段**完全一致**的细类名作为小节标题（不要使用 ``##`` 一级标题）； +- 每段约 **100～220 字**中文：归纳该细类下**消费者在讨论什么**（口感、价格、物流、功效疑虑等）；勿编造摘录中未出现的品牌、医学结论； +- **去重与可证（本章仅评论侧）**：本段**只**依据 ``sample_text_snippets`` 与 ``effective_text_lines`` 中的**原文**，**禁止**把 ``keyword``、品类常识或商品标题卖点套话写成「用户评价」；**禁止**各细类段首复用同一句总括（如「整体上满足了消费者对低 GI、高蛋白、便携性的需求」）；每段开头句式须**有变化**，并至少一句体现**该细类与相邻细类在讨论焦点上的差异**。**利益/诉求词**（低 GI、高蛋白、便携、代餐、控糖等）**仅当**在上述字段的**原文**中可子串命中或可明确同义（便携↔随身、小包装、单片、独立装等）时才写；若**未**出现「蛋白」「便携」「随身」「小包装」「单片」等，则**不得**写「高蛋白」「便携性」等；**禁止**为凑齐常见卖点组合而脑补未在输入中出现的词。 +- **禁止**输出 Markdown 表格、禁止逐条复述全部评价； +- 若 ``effective_text_lines`` 很少，明确写「样本较少，归纳供启发」。 + +总输出约 **600～3200 字**。仅输出正文 Markdown，不要用代码围栏包裹全文。""" + + +COMMENT_GROUPS_USER_PREFIX = ( + "请根据以下 JSON 撰写竞品报告第八章末「细类评论要点归纳」正文（Markdown）。\n\n" +) + + +def generate_comment_group_summaries_llm( + groups: list[dict[str, Any]], *, keyword: str +) -> str: + """ + 细类多、评价正文长时 JSON 易超上下文：按档位逐步缩短 ``effective_text_lines`` / + ``sample_text_snippets`` 直至估算 tokens 低于窗口（与 ``AI_crawler.chat_completion_text`` 预检一致）。 + """ + + def _compact_one( + g: dict[str, Any], + *, + eff_n: int, + eff_max: int, + sn_n: int, + sn_max: int, + ) -> dict[str, Any]: + g2: dict[str, Any] = { + "group": g.get("group"), + "comment_flat_rows": g.get("comment_flat_rows"), + } + el = g.get("effective_text_lines") + if isinstance(el, list): + g2["effective_text_lines"] = [str(x)[:eff_max] for x in el[:eff_n]] + else: + g2["effective_text_lines"] = [] + sn = g.get("sample_text_snippets") + if isinstance(sn, list): + g2["sample_text_snippets"] = [str(x)[:sn_max] for x in sn[:sn_n]] + else: + g2["sample_text_snippets"] = [] + return g2 + + ctx = llm_context_window_size() + budget = ctx - 512 - 256 + # 预留 ``max_tokens=8192`` 的完成空间；网关计输入 tokens 常高于本地粗估 + def _input_ok(system: str, user_p: str) -> bool: + est = estimate_chat_input_tokens(system, user_p) + return est < 15_500 + + levels: list[tuple[int, int, int, int]] = [ + (14, 260, 10, 200), + (12, 220, 8, 180), + (10, 180, 8, 160), + (8, 150, 6, 140), + (6, 120, 5, 120), + (5, 100, 4, 100), + (4, 80, 3, 80), + (3, 70, 3, 70), + (3, 50, 2, 60), + ] + user = "" + chosen = levels[-1] + for level in levels: + chosen = level + eff_n, eff_max, sn_n, sn_max = level + trimmed = [ + _compact_one(g, eff_n=eff_n, eff_max=eff_max, sn_n=sn_n, sn_max=sn_max) + for g in groups + if isinstance(g, dict) + ] + raw = json.dumps({"keyword": keyword, "groups": trimmed}, ensure_ascii=False) + if len(raw) > 48_000: + raw = raw[:44_000] + "\n…\n" + user = COMMENT_GROUPS_USER_PREFIX + raw + if _input_ok(COMMENT_GROUPS_SYSTEM, user): + break + else: + tail = "\n\n…（JSON 已截断以适配上下文；仅依据可见字段撰写。）\n" + eff_n, eff_max, sn_n, sn_max = chosen + trimmed = [ + _compact_one(g, eff_n=eff_n, eff_max=eff_max, sn_n=sn_n, sn_max=sn_max) + for g in groups + if isinstance(g, dict) + ] + raw = json.dumps({"keyword": keyword, "groups": trimmed}, ensure_ascii=False) + room = max( + 2000, + int((budget - 800) / 0.55) + - len(COMMENT_GROUPS_SYSTEM) + - len(COMMENT_GROUPS_USER_PREFIX) + - len(tail), + ) + user = COMMENT_GROUPS_USER_PREFIX + raw[: max(1500, room)] + tail + return call_llm(COMMENT_GROUPS_SYSTEM, user) + + +SCENARIO_GROUPS_SYSTEM = """你是用户研究与品类顾问。输入为 JSON：``keyword``、``scenario_lexicon``、``groups``。 +``scenario_lexicon`` 列出各场景标签及示例触发子串（与报告 **第八章第二节**（关注词与场景路径）右栏统计规则一致）。 +``groups`` 每项含 ``group``（与 第五章矩阵一致的细分类目名）、``effective_text_count``（有效评价文本条数）、 +``scenario_distribution``（各预设场景的 ``mention_rows`` 与 ``share_of_effective_texts``；**一条评价可计入多场景**；与 **第八章第二节** 图右栏同源）、 +``sample_text_snippets``（摘录行常含细类、SKU、品名、店铺等前缀的短引文，已截断）。 +统计为**子串命中**，不是语义主题模型。 + +请**为每个细类**输出一小段 Markdown（全部 groups 都要写，顺序与输入一致）： +- 以 ``#### `` + 与该条 ``group`` 字段**完全一致**的细类名作为小节标题； +- 每段约 **100～220 字**：归纳该细类用户**自述的使用场景/用途**结构（哪些场景标签相对突出、多场景叠加是否常见），可点到与其他细类的差异；**所有条数与占比须与 ``scenario_distribution``、``effective_text_count`` 一致**，禁止编造； +- 引用原话时须保留或复述摘录中的店铺/SKU/品名信息，勿虚构； +- **禁止** Markdown 表格、禁止复述全部摘录；若 ``effective_text_count`` 很小，写明「样本较少，归纳供启发」。 + +总输出约 **600～3200 字**。仅输出正文 Markdown，不要用代码围栏包裹全文。""" + + +SCENARIO_GROUPS_USER_PREFIX = ( + "请根据以下 JSON 撰写竞品报告 第八章第二节（右栏：使用场景）之后的「使用场景要点归纳」正文（Markdown）。\n\n" +) + + +def generate_scenario_group_summaries_llm( + payload: dict[str, Any], *, keyword: str +) -> str: + """与 ``generate_comment_group_summaries_llm`` 类似：细类多时长 JSON 按档压缩。""" + + def _compact_group( + g: dict[str, Any], + *, + dist_n: int, + sn_n: int, + sn_max: int, + ) -> dict[str, Any]: + g2: dict[str, Any] = { + "group": g.get("group"), + "effective_text_count": g.get("effective_text_count"), + } + dist = g.get("scenario_distribution") + if isinstance(dist, list): + g2["scenario_distribution"] = [] + for x in dist[:dist_n]: + if not isinstance(x, dict): + continue + g2["scenario_distribution"].append( + { + "scenario": x.get("scenario"), + "mention_rows": x.get("mention_rows"), + "share_of_effective_texts": x.get( + "share_of_effective_texts" + ), + } + ) + else: + g2["scenario_distribution"] = [] + sn = g.get("sample_text_snippets") + if isinstance(sn, list): + g2["sample_text_snippets"] = [ + str(x)[:sn_max] for x in sn[:sn_n] + ] + else: + g2["sample_text_snippets"] = [] + return g2 + + def _compact_lex(raw: Any, *, max_items: int, trig_n: int) -> list[dict[str, Any]]: + if not isinstance(raw, list): + return [] + out: list[dict[str, Any]] = [] + for item in raw[:max_items]: + if not isinstance(item, dict): + continue + tr = item.get("trigger_examples") + te = ( + [str(x)[:48] for x in tr[:trig_n]] + if isinstance(tr, list) + else [] + ) + out.append({"label": item.get("label"), "trigger_examples": te}) + return out + + groups_in = [g for g in (payload.get("groups") or []) if isinstance(g, dict)] + ctx = llm_context_window_size() + budget = ctx - 512 - 256 + + def _input_ok(system: str, user_p: str) -> bool: + est = estimate_chat_input_tokens(system, user_p) + return est < 15_500 + + levels: list[tuple[int, int, int, int, int]] = [ + (16, 14, 260, 10, 12), + (14, 12, 220, 8, 10), + (12, 10, 180, 8, 8), + (10, 8, 150, 6, 6), + (8, 6, 120, 5, 5), + (6, 5, 100, 4, 4), + (5, 4, 80, 3, 3), + ] + user = "" + chosen = levels[-1] + for level in levels: + chosen = level + dist_n, sn_n, sn_max, lex_n, trig_n = level + trimmed_g = [ + _compact_group(g, dist_n=dist_n, sn_n=sn_n, sn_max=sn_max) + for g in groups_in + ] + lex_c = _compact_lex( + payload.get("scenario_lexicon"), + max_items=lex_n, + trig_n=trig_n, + ) + body = { + "keyword": keyword, + "scenario_lexicon": lex_c, + "groups": trimmed_g, + } + raw = json.dumps(body, ensure_ascii=False) + if len(raw) > 48_000: + raw = raw[:44_000] + "\n…\n" + user = SCENARIO_GROUPS_USER_PREFIX + raw + if _input_ok(SCENARIO_GROUPS_SYSTEM, user): + break + else: + tail = "\n\n…（JSON 已截断以适配上下文；仅依据可见字段撰写。）\n" + dist_n, sn_n, sn_max, lex_n, trig_n = chosen + trimmed_g = [ + _compact_group(g, dist_n=dist_n, sn_n=sn_n, sn_max=sn_max) + for g in groups_in + ] + lex_c = _compact_lex( + payload.get("scenario_lexicon"), + max_items=lex_n, + trig_n=trig_n, + ) + raw = json.dumps( + { + "keyword": keyword, + "scenario_lexicon": lex_c, + "groups": trimmed_g, + }, + ensure_ascii=False, + ) + room = max( + 2000, + int((budget - 800) / 0.55) + - len(SCENARIO_GROUPS_SYSTEM) + - len(SCENARIO_GROUPS_USER_PREFIX) + - len(tail), + ) + user = SCENARIO_GROUPS_USER_PREFIX + raw[: max(1500, room)] + tail + return call_llm(SCENARIO_GROUPS_SYSTEM, user) + + +PRICE_GROUPS_SYSTEM = """你是定价与渠道顾问。输入为 JSON：``keyword`` 与 ``groups``。 +每个 group 含 ``group``（细分类目名，与 第五章矩阵、第六章「按细类价盘」小节一致）、``sku_count``、``price_stats``（该细类可解析展示价的 min/max/median/mean/n，与第六章各细类 Markdown 分位数表同源）、 +``listing_snippets``（若干「标题｜标价｜券后｜详情价」摘录，来自合并表字段，已截断）。 + +请**为每个细类**输出一小段 Markdown（全部 groups 都要写，顺序与输入一致）： +- 以 ``#### `` + 与该 group 字段**完全一致**的细类名作为小节标题； +- 每段约 **80～200 字**中文，**只写价盘与价差**：用 ``price_stats`` 概括价带/离散度（如 min～max、中位数、相对集中或拉得开）；用 ``listing_snippets`` 归纳**标价 vs 券后 vs 详情价**是否常一致、是否常见券后低于标价、价差幅度的大致印象；**可一句**联系标题里**显式出现的规格数字**（如克重、件数）解释**价高/价差大是否可能来自大规格或组合装**——仅当摘录里确有数字时写，勿展开成宣称解读； +- **硬性禁止**（本章不是卖点章）：不要列举或归纳「0 蔗糖 / 低 GI / 全麦 / 代餐 / 孕妇 / 控糖」等**营销宣称或场景关键词**；不要写配料、功效、品牌叙事、用户画像；这些若出现应留给报告 **第五章细类要点归纳**。 +- **禁止** Markdown 表格、禁止罗列全部 SKU；勿编造未出现的到手价、销量排名； +- 若 ``price_stats`` 中 n=0 或缺失，写「该细类无可解析数值价，从略」。 + +总输出约 **500～2800 字**。仅输出正文 Markdown，不要用代码围栏包裹全文。""" + + +PRICE_GROUPS_USER_PREFIX = ( + "请根据以下 JSON 撰写竞品报告第六章末「细类价盘要点归纳」正文（Markdown）。" + "本章只写**数值价带与标价/券后/详情价关系**，勿写卖点宣称关键词归纳。\n\n" +) + + +def generate_price_group_summaries_llm( + groups: list[dict[str, Any]], *, keyword: str +) -> str: + trimmed: list[dict[str, Any]] = [] + for g in groups: + if not isinstance(g, dict): + continue + g2 = dict(g) + sn = g2.get("listing_snippets") + if isinstance(sn, list) and len(sn) > 14: + g2["listing_snippets"] = sn[:14] + trimmed.append(g2) + payload = {"keyword": keyword, "groups": trimmed} + raw = json.dumps(payload, ensure_ascii=False) + if len(raw) > 95_000: + for g2 in trimmed: + sn = g2.get("listing_snippets") + if isinstance(sn, list) and len(sn) > 8: + g2["listing_snippets"] = sn[:8] + raw = json.dumps({"keyword": keyword, "groups": trimmed}, ensure_ascii=False) + user = PRICE_GROUPS_USER_PREFIX + raw + return call_llm(PRICE_GROUPS_SYSTEM, user) + + +PROMO_GROUPS_SYSTEM = """你是电商促销与价盘顾问。输入为 JSON：``keyword`` 与 ``groups``。 +每个 group 含 ``group``（细分类目名，与第五章矩阵、第六章一致）、``sku_count``、``rows_with_buyer_promo_text``（该细类合并表中「促销摘要」非空行数）、 +``promo_snippets``（若干条摘录：标题 + 促销摘要/榜单排名/榜单类文案等，已截断；**不含**列表卖点/腰带列）。 + +请**为每个细类**输出一小段 Markdown（全部 groups 都要写，顺序与输入一致）： +- 以 ``#### `` + 与该 group 字段**完全一致**的细类名作为小节标题； +- 每段约 **80～220 字**中文，**只写促销与活动形态**：如券后/满减/百亿补贴/新人包邮/限购提示/「到手价」展示方式、与 **第六章第一节** 规则统计可对照的**活动话术密度**印象；可一句点出**榜单曝光**是否常见、是否与价格带并存； +- **硬性禁止**：不要展开配料、功效、用户画像；不要复述第五章 的配料归纳；不要编造未在摘录中出现的具体金额或活动规则； +- 若该细类 ``rows_with_buyer_promo_text`` 为 0 且摘录几乎只有标题，写「该细类缺少购买者侧促销摘要，从略」。 + +总输出约 **500～2800 字**。仅输出正文 Markdown，不要用代码围栏包裹全文。""" + + +PROMO_GROUPS_USER_PREFIX = ( + "请根据以下 JSON 撰写竞品报告第六章「细类促销与活动要点归纳」正文（Markdown）。" + "依据 ``promo_snippets`` 中的促销摘要与榜单相关摘录，**不写**价带分位数（留给上一小节）。\n\n" +) + + +def generate_promo_group_summaries_llm( + groups: list[dict[str, Any]], *, keyword: str +) -> str: + trimmed: list[dict[str, Any]] = [] + for g in groups: + if not isinstance(g, dict): + continue + g2 = dict(g) + sn = g2.get("promo_snippets") + if isinstance(sn, list) and len(sn) > 14: + g2["promo_snippets"] = sn[:14] + trimmed.append(g2) + payload = {"keyword": keyword, "groups": trimmed} + raw = json.dumps(payload, ensure_ascii=False) + if len(raw) > 95_000: + for g2 in trimmed: + sn = g2.get("promo_snippets") + if isinstance(sn, list) and len(sn) > 8: + g2["promo_snippets"] = sn[:8] + raw = json.dumps({"keyword": keyword, "groups": trimmed}, ensure_ascii=False) + user = PROMO_GROUPS_USER_PREFIX + raw + return call_llm(PROMO_GROUPS_SYSTEM, user) + + +def _join_chunked_group_markdown(parts: list[str]) -> str: + """按细类多次调用 LLM 后的片段拼接（顺序与 ``groups`` 一致）。""" + return "\n\n".join(p.strip() for p in parts if (p or "").strip()) + + +def generate_matrix_group_summaries_llm_chunked( + groups: list[dict[str, Any]], *, keyword: str +) -> str: + """与 ``generate_matrix_group_summaries_llm`` 等价输出结构，但**每个矩阵细类单独**请求一次网关。""" + clean = [g for g in groups if isinstance(g, dict)] + if not clean: + return "" + parts = [ + generate_matrix_group_summaries_llm([g], keyword=keyword) for g in clean + ] + return _join_chunked_group_markdown(parts) + + +def generate_price_group_summaries_llm_chunked( + groups: list[dict[str, Any]], *, keyword: str +) -> str: + clean = [g for g in groups if isinstance(g, dict)] + if not clean: + return "" + parts = [ + generate_price_group_summaries_llm([g], keyword=keyword) for g in clean + ] + return _join_chunked_group_markdown(parts) + + +def generate_promo_group_summaries_llm_chunked( + groups: list[dict[str, Any]], *, keyword: str +) -> str: + clean = [g for g in groups if isinstance(g, dict)] + if not clean: + return "" + parts = [ + generate_promo_group_summaries_llm([g], keyword=keyword) for g in clean + ] + return _join_chunked_group_markdown(parts) + + +def generate_comment_group_summaries_llm_chunked( + groups: list[dict[str, Any]], *, keyword: str +) -> str: + clean = [g for g in groups if isinstance(g, dict)] + if not clean: + return "" + parts = [ + generate_comment_group_summaries_llm([g], keyword=keyword) for g in clean + ] + return _join_chunked_group_markdown(parts) + + +def generate_scenario_group_summaries_llm_chunked( + payload: dict[str, Any], *, keyword: str +) -> str: + """``scenario_lexicon`` 每轮原样附带，``groups`` 每次只含一个细类。""" + groups_in = [g for g in (payload.get("groups") or []) if isinstance(g, dict)] + if not groups_in: + return "" + lex = payload.get("scenario_lexicon") + base: dict[str, Any] = { + "scenario_lexicon": lex if isinstance(lex, list) else [], + } + parts = [ + generate_scenario_group_summaries_llm( + {**base, "groups": [g]}, + keyword=keyword, + ) + for g in groups_in + ] + return _join_chunked_group_markdown(parts) diff --git a/backend/pipeline/llm/generate_marketing_detail.py b/backend/pipeline/llm/generate_marketing_detail.py new file mode 100644 index 0000000..6ed6077 --- /dev/null +++ b/backend/pipeline/llm/generate_marketing_detail.py @@ -0,0 +1,210 @@ +"""策略稿 → 核心信息卡 → 营销内容多触点文案（两步 LLM，JSON 输出）。""" +from __future__ import annotations + +import json +from typing import Any + +from .llm_client import call_llm + +_MAX_STRATEGY_CHARS = 28_000 + +CORE_CARD_SYSTEM = """你是电商营销内容顾问。根据用户提供的「策略稿全文」与结构化决策、业务备注，输出**仅一段 UTF-8 JSON 对象**（不要 Markdown 代码围栏，不要前后说明文字）。 + +**硬性**： +- 事实、数字、功效、检测结论、销量、评价原文：**仅可**来自输入；**禁止**编造未出现的品牌名、数据、「用户说」引语。 +- 食品/健康相关：**禁止**治疗承诺与夸大疗效；无依据写「输入未体现」或「待法务确认」。 +- 句子短、可落地；兼顾**购买者决策**与**列表/商详/主图等多触点**上架可用性。 +- **读者第一眼须知道在卖什么**：禁止通篇只有「价值感」「信任」「体验」而**不出现可识别的品类/形态**（如饼干、燕麦、奶粉、饮料等）。若输入未给出具体 SKU 名，仍须写清**类目 + 形态/规格层级**（如「低 GI 方向早餐饼干（待业务定款）」），不得用「优质好物」「健康之选」等**无品类**的句子糊弄本条。 + +**JSON 键（须全部出现，值为字符串；无内容用空串）**： +- what_we_sell：**卖的是什么**（必填，建议 25～80 字）。写清**品类 + 主推形态/规格或适用场景**，让读者**不读策略稿**也能回答「你们在卖哪种货」。**仅可**综合策略稿、`strategy_decisions`（尤其 **pillar_product**、battlefield_one_line、audience_segment、marketing_strategy）、`business_notes` 与 `keyword` 监测语境中已出现的信息；若 `pillar_product` 非空须与之**不矛盾**。无具体商品名时须明确写「待业务补充主推 SKU/品名」，并保留类目词（可与关键词监测范围对读）。 +- one_liner_value：一句话价值主张（买家能得到什么） +- buyer_job_to_be_done：购买者的任务或情境（一句） +- key_pain_or_desire：核心痛点或欲望（与策略一致） +- why_this_product：为何要选这一款（相对同类，一句） +- proof_or_trust_angle：信任或证明角度（无依据写「输入未体现」） +- differentiation_vs_alternatives：与替代方案相比的差异（一句） +- price_value_framing：价位与价值感如何表述（与策略价位可对读；无则「待业务确认」） +- compliance_taboos：表述禁区摘要（来自业务备注或策略风险） +- open_points_for_business：待业务补充（无则空串） +""" + +DETAIL_PACK_SYSTEM = """你是京东场景营销内容写手。输入为已定稿的「核心信息卡」JSON 与关键词。请输出**仅一段 UTF-8 JSON 对象**（不要 Markdown 代码围栏）。 + +**硬性**： +- **仅可**依据核心信息卡展开；**禁止**新增数字、功效、认证、评价引语、竞品具体名（除非信息卡里已有）。 +- 购买者视角，短句；禁止输出 JSON 键名英文给最终读者（值全部为中文**多触点上架**可用文案）。 +- 不要泄露「核心信息卡」「策略稿」等内部词。 +- **更丰富≠编造**：可增加条数与段落，但**每一条**须能从信息卡对应字段找到方向；无依据处写「输入未体现」「待业务核对」，**禁止**为凑字数新增数字、销量、认证、评价引语、具体竞品名。 +- **每条 listing_titles、listing_subtitle、detail_headline、selling_bullets 的前两条**均须让读者能识别**在卖什么品类/什么货**（须与信息卡 **what_we_sell** 一致，可缩写但**禁止**偷换品类或只剩空洞形容词）。若信息卡 `what_we_sell` 已写品类，文案中**至少一处**直接出现该类目词或同义可识别表述。 +- **文生图/文生视频提示词**：须为**可直接复制**到常见文生图、文生视频模型的**中文**描述；**仅可**依据信息卡已有事实与品类，**禁止**在提示词里写「策略稿」「信息卡」「JSON」等元话语；**禁止**要求生成未授权的具体品牌 Logo、真实包装上的可辨认商标、带疗效承诺的贴片字。 +- **文生图须「有货、有卖点画面」**（硬性）： + - 从信息卡 **what_we_sell**、**one_liner_value**、**key_pain_or_desire**、**why_this_product**、**differentiation_vs_alternatives**、**price_value_framing** 中提炼 **1～3 条可画出来的卖点**，写入 ``text_to_image_prompt_main``；**场景图** ``text_to_image_prompt_scene`` 非空时须保留**至少 1 处**同款质地或品类辨识（非空场景图时）。 + - **禁止**整段只有「白底」「居中」「电商主图」「健康食品」等空壳，而**不出现具体货态**（形态、切片、包装类型、手持/摆放方式至少择一）。 + - **口感/质地类**（如松软、酥脆、绵密、有嚼劲）：**必须**写成**可见结构**，不能只写一次形容词了事。例：**松软**→「吐司切片横截面气孔细腻、边缘微翘显蓬松」「轻按后缓慢回弹」「手撕开可见柔软内里」；**酥脆**→「饼干断面层次清晰、碎屑自然」。信息卡未提质地则**不写**，勿编造。 + - **配料/品类视觉**（如全麦）：可写「麸皮颗粒隐约可见」「浅褐全麦外皮」等，**禁止**疗效字幕、血糖仪、前后对比治病画面。 + +**JSON 键（须全部出现）**： +- listing_titles：字符串数组，**6～9** 条商品短标题备选（每条约 30 字内；**每条须含可识别品类或品名线索**，禁止多条全是空洞套话；可有 2～3 条侧重不同角度：场景/质地/配料/人群） +- listing_subtitle：一条列表副文案（约 **60～90** 字内，信息不足则取下限） +- detail_headline：商品详情页首屏下 lead，**2～3 句**（**首句须点明卖的是什么货**，后接价值与差异；总长约 **80～160** 字） +- selling_bullets：字符串数组，**8～12** 条卖点（每条约 **40 字内**；须覆盖：品类形态、口感/质地（若信息卡有）、配料/健康表述（合规）、场景、信任点、与同类差异等**不同角度**，**禁止** 12 条重复同一句话换说法） +- spec_sidebar_lines：字符串数组，**0～5** 条参数区旁短句（可空数组） +- faq：对象数组，每项含 question、answer 字符串，**5～8** 组；答句不得超出信息卡承诺；可含「怎么保存」「适合谁」「和××区别」（××用泛称除非信息卡有品牌） +- detail_mid_story_paragraphs：字符串数组，**2～4 段**详情页**首屏之后**的中段叙事；每段 **70～150** 字；**仅**展开信息卡已有卖点与 `what_we_sell`，可分段讲「适合谁—怎么吃—为何值得」；**禁止**新数字、新功效、编造用户故事 +- usage_and_pairing_tips：字符串数组，**2～5** 条食用场景、保存提示、搭配建议（如早餐配牛奶）；信息卡未写保存条件则写「输入未体现具体保质期与保存要求，上架前请核对包装」类中性句，**禁止**编造保质期天数 +- short_graphic_post_variants：字符串数组，**3～5** 条短图文/种草贴变体；每条 **45～110** 字；须**首句或次句**点明品类；适合复制到站内动态；**禁止**销量名次、虚假好评引语 +- live_script_bullets：字符串数组，**4～7** 条直播或短视频**可照读要点**（每条约 **15～40** 字）；按顺序像口播提纲；**禁止**医疗承诺与未证实数据；可与 `live_or_short_hook` 呼应但勿逐句重复 +- traceability_note：**依据与边界**（必填，2～4 句）。用业务可读中文说明：本包与信息卡中**哪些承诺方向一致**、**哪些表述须业务或法务核对**、**输入未体现的不得对外宣称**；**禁止**新数字、新功效、新认证。 +- main_image_three_points：字符串数组，**恰好 3 条**，主图/首图用超短句（每条建议 6～14 字）；须与 **what_we_sell** 品类一致，可来自卖点压缩，禁止空泛口号 +- live_or_short_hook：一条直播或短视频开场钩句（≤40 字）；同一事实约束 +- customer_service_opening：一条客服首句/欢迎语建议（≤50 字）；同一事实约束 +- text_to_image_prompt_main：字符串，**主图/首图**文生图提示词（建议 **100～260** 字）。**必须**依次包含：① **具体货态**（与 **what_we_sell** 一致的品类+形态，如全麦吐司切片摞放、独立小包饼干）；② **至少一条质地/卖点的视觉化描写**（与信息卡一致，参见上文「松软→截面/按压/手撕」等）；③ **构图与背景**（如白底居中、轻微投影）；④ **光影**（柔和棚拍、写实）；⑤ **规避**（无 Logo、无疗效字、无竞品名）。**英文模型**可关键风格词括注英文。 +- text_to_image_prompt_scene：字符串，**场景/生活方式**备选图（建议 **80～200** 字）：早餐桌、手持、厨房台面等；**须含**与主图**同一品类**的清晰货态，并**至少一处**质地或食用情境（如蒸汽、刀切截面、蘸牛奶）。与主图完全重复则宁可缩短但保留情境差分。无合适场景时 ``""``。 +- text_to_video_prompt：字符串，文生视频提示词（建议 **100～260** 字），竖屏 9:16、**5～15 秒**。**须**含 **1 个能体现质地或卖点的镜头**（如慢镜撕开吐司见柔软内里、刀切截面特写、轻捏回弹），与信息卡卖点一致；另写开场与转场（推近/平移）。**禁止**疗效字幕、未授权标识；可「无对白」或「一句中性口播」。 +""" + +# 第二步 JSON 完整键表；模型漏键或旧落盘缺字段时由 ``normalize_detail_page_pack`` 补齐。 +_DETAIL_PAGE_PACK_DEFAULTS: dict[str, Any] = { + "listing_titles": [], + "listing_subtitle": "", + "detail_headline": "", + "selling_bullets": [], + "spec_sidebar_lines": [], + "faq": [], + "traceability_note": "", + "main_image_three_points": [], + "live_or_short_hook": "", + "customer_service_opening": "", + "text_to_image_prompt_main": "", + "text_to_image_prompt_scene": "", + "text_to_video_prompt": "", + "detail_mid_story_paragraphs": [], + "usage_and_pairing_tips": [], + "short_graphic_post_variants": [], + "live_script_bullets": [], +} + +_DETAIL_PAGE_PACK_LIST_KEYS: frozenset[str] = frozenset( + { + "listing_titles", + "selling_bullets", + "spec_sidebar_lines", + "main_image_three_points", + "faq", + "detail_mid_story_paragraphs", + "usage_and_pairing_tips", + "short_graphic_post_variants", + "live_script_bullets", + } +) + + +def normalize_detail_page_pack(data: dict[str, Any]) -> dict[str, Any]: + """保证 ``detail_page_pack`` 含全部约定键，避免模型漏输出或旧 JSON 缺字段。""" + out: dict[str, Any] = dict(data) + for key in _DETAIL_PAGE_PACK_DEFAULTS: + v = out.get(key) + if key in _DETAIL_PAGE_PACK_LIST_KEYS: + if isinstance(v, list): + continue + out[key] = [] + continue + if v is None: + out[key] = "" + elif not isinstance(v, str): + out[key] = str(v) + return out + + +def _truncate_strategy(md: str) -> tuple[str, bool]: + t = (md or "").strip() + if len(t) <= _MAX_STRATEGY_CHARS: + return t, False + return t[: _MAX_STRATEGY_CHARS].rstrip() + "\n\n…（策略正文已截断，以下同）\n", True + + +def _parse_llm_json(raw: str) -> dict[str, Any]: + s = (raw or "").strip() + if not s: + raise ValueError("大模型返回为空") + try: + out = json.loads(s) + except json.JSONDecodeError: + i = s.find("{") + j = s.rfind("}") + if i >= 0 and j > i: + out = json.loads(s[i : j + 1]) + else: + raise ValueError("大模型返回不是合法 JSON") from None + if not isinstance(out, dict): + raise ValueError("大模型 JSON 须为对象") + return out + + +def generate_core_info_card( + *, + keyword: str, + strategy_markdown: str, + strategy_decisions: dict[str, Any] | None, + business_notes: str, +) -> dict[str, Any]: + md, truncated = _truncate_strategy(strategy_markdown) + payload = { + "keyword": keyword, + "strategy_markdown": md, + "strategy_markdown_truncated": truncated, + "strategy_decisions": strategy_decisions or {}, + "business_notes": (business_notes or "").strip(), + } + user = ( + "请根据以下 JSON 输出核心信息卡（仅 JSON 对象）：\n" + + json.dumps(payload, ensure_ascii=False) + ) + raw = call_llm(CORE_CARD_SYSTEM, user) + return _parse_llm_json(raw) + + +def generate_detail_page_pack( + *, + keyword: str, + core_info_card: dict[str, Any], +) -> dict[str, Any]: + payload = { + "keyword": keyword, + "core_info_card": core_info_card, + } + user = ( + "请根据以下 JSON 输出营销内容多触点文案（**仅**一段 JSON 对象）。\n" + "**必填键名（缺一不可，勿省略）**：listing_titles, listing_subtitle, detail_headline, " + "selling_bullets, spec_sidebar_lines, faq, detail_mid_story_paragraphs, usage_and_pairing_tips, " + "short_graphic_post_variants, live_script_bullets, traceability_note, main_image_three_points, " + "live_or_short_hook, customer_service_opening, text_to_image_prompt_main, " + "text_to_image_prompt_scene, text_to_video_prompt。\n" + "**丰富度**：在遵守信息卡前提下尽量写满条数与段落；**禁止**为凑字编造。\n" + "**文生图/视频**：须让「货」和卖点**看得见**（如松软→截面气孔、手撕/按压回弹）；禁止整段只有白底健康食品而无具体形态与质地描写。\n" + "输入数据：\n" + + json.dumps(payload, ensure_ascii=False) + ) + raw = call_llm(DETAIL_PACK_SYSTEM, user) + return normalize_detail_page_pack(_parse_llm_json(raw)) + + +def generate_marketing_detail_pack( + *, + keyword: str, + strategy_markdown: str, + strategy_decisions: dict[str, Any] | None = None, + business_notes: str = "", +) -> dict[str, Any]: + core = generate_core_info_card( + keyword=keyword, + strategy_markdown=strategy_markdown, + strategy_decisions=strategy_decisions, + business_notes=business_notes, + ) + pack = generate_detail_page_pack(keyword=keyword, core_info_card=core) + return { + "core_info_card": core, + "detail_page_pack": pack, + } diff --git a/backend/pipeline/llm/generate_sections.py b/backend/pipeline/llm/generate_sections.py new file mode 100644 index 0000000..3d002ee --- /dev/null +++ b/backend/pipeline/llm/generate_sections.py @@ -0,0 +1,199 @@ +"""评价情感 LLM（可选）：默认**不**写入竞品报告；供独立调用或历史任务兼容。""" +from __future__ import annotations + +import json +import re +from typing import Any + +from ..reporting.brief_compact import compact_brief_for_llm +from .llm_client import call_llm + +SENTIMENT_LLM_SYSTEM = """你是电商/食品类用户研究助手。输入 JSON 含： + +- ``comment_sentiment_lexicon``：子串词表统计（与载荷内各计数字段**同一计数方式**；竞品报告**已不再**发布同口径扇形图/条形图；**仅作定量参考**；子串命中≠说话人态度）。 +- ``positive_lexeme_hits_top`` / ``negative_lexeme_hits_top``：短语级命中摘要（同源）。 +- ``sentiment_bucket_method``：``score_then_lexeme`` 表示**先按 1～5 星分桶**（无评分行再按关键词）；``keyword_substring_heuristic`` 表示**仅关键词**分桶；与 ``comment_sentiment_lexicon`` 内四象限计数一致。``sample_reviews_positive_biased`` / ``negative`` / ``mixed_tone`` 按该规则**机械归类**的抽样，**可能与整句真实褒贬不一致**（例如「软硬适中」曾被误归负向）。 +- **``sample_reviews_semantic_pool``**（若有）：本批评价经去重后的**随机/洗牌抽样**（来自全部有效条，不限于某一象限）。**归纳正/负向体验、引用「」短引文时，优先以此池与上述各列表中的原文为准，自行结合语境理解**：转折、对比（如「没那么甜」「软硬适中」）、先抑后扬/先扬后抑整句态度；**不得以子串是否命中负面词来断言该句为抱怨**。 + +每条样本通常以 ``【细类：…｜SKU：…｜品名：…｜店铺：…】`` 开头，表示 **第五章细类、SKU、品名、店铺**；写归纳与「」引文时须能还原「哪家店、哪条 SKU、哪款品名」，或保留前缀，**禁止**无指代地写「用户普遍…」。 + +**硬性要求**： +- **仅输出 Markdown 正文**（不要用 ``` 围栏包裹全文）； +- **不要编造**样本中未出现的具体事实、品牌、价格、医学功效； +- **定量数字**（条数、占比、lexicon 各字段）须与 ``comment_sentiment_lexicon`` **一致**，勿编造； +- **定性归纳**（满意点/抱怨点、引语是否算差评）：以**整句语义**为准；若某句在语义上为褒义或中性描述，**不得**放入「质地差、口感硬」等负向归因；若词表归类结果与句意冲突，**以句意为准**，并在「使用注意」点明「关键词归类**仅反映子串计数，不作态度判断**」。 +- **负向主题优先级（硬性）**：写「主要」「集中」「突出」类抱怨前，**必须对照** ``negative_lexeme_hits_top`` 各短语的 ``texts_matched``：若「口感硬/咬不动/发硬」等**预设短语命中为 0 或明显低于**其它维度（如分量、少、物流），**不得**把质地硬写成首要负向主题；若抽样原文与语义池里**反复出现**「分量少、太少、不够吃」等而预设短语未列出，仍须**单独归纳**（用户常用生活化表述，不必与预设表完全一致）。 +- 若某措辞**未**出现在任一抽样原文（含前缀后正文）中，**禁止**用引号写成直接引语。 +- **不要**只复述「某词出现 N 次」——若业务侧仍配图表则由图展示；你的价值是**语义归纳**。 + +**建议结构**（使用四级标题 ``####``）： +1. ``#### 正向体验主题``：3～6 条；概括满意点（口感、甜度、性价比等），**尽量**用「」引用 ``sample_reviews_semantic_pool`` 或其它样本中**语义确为正面**的短句（勿把对比褒义句当差评例子）。 +2. ``#### 负向评价主题归因``：**核心段落**。依据你读后判定为**确有不满**的句子，归纳 **4～8 个**问题维度（须覆盖**质地、分量/规格、价格、物流、包装**等中在原文中**实际出现**的类别，勿只写质地）。引文优先取自句意确为批评的原文（可来自任一档位键，不限于 ``sample_reviews_negative_biased``）；引文须含 ``【细类…｜…店铺…】`` 或同义店铺+品名/SKU。 +3. ``#### 混合评价中的典型张力``（可选）：同一评价里褒贬并存时，说明在争什么；若无则略写。 +4. ``#### 使用注意``：关键词子串统计的局限、``sample_reviews_semantic_pool`` 与词表归类的差异、抽样截断、非医学结论。 + +**篇幅**：若 JSON 含 ``matrix_group_focus``（单细类范围），本节总字数约 **500～1200 字**，勿再按全关键词池写「全行业泛化」；若**不含**该字段（全量池），总字数约 **700～1600 字**。简体中文，语气客观。""" + +# 嵌入报告 8.3 时外层为 ``#### {细类名}``；若内文仍用同级 ``#### 正向体验主题``， +# ``extract_level4_sections_by_group_title`` 会在第一个子 ``####`` 处截断，导致策略摘录/心得侧「同细类报告摘录」拿不到正文。 +_SENTIMENT_INNER_H4_TITLES: frozenset[str] = frozenset( + { + "正向体验主题", + "负向评价主题归因", + "混合评价中的典型张力", + "使用注意", + } +) + + +def demote_sentiment_inner_h4_to_h5_for_matrix_group(md: str) -> str: + """将情感归纳四个固定小节从 ``####`` 降为 ``#####``，以便嵌在 ``#### 细类`` 下仍能被按细类抽取。""" + out_lines: list[str] = [] + for line in (md or "").splitlines(): + m = re.match(r"^####\s+(.+)$", line) + if m: + title = m.group(1).strip() + if title in _SENTIMENT_INNER_H4_TITLES: + out_lines.append(f"##### {title}") + continue + out_lines.append(line) + return "\n".join(out_lines) + + +def generate_comment_sentiment_analysis_llm(payload: dict[str, Any]) -> str: + """基于 lexicon 统计 + 语义池与按词表归类的抽样，生成评价情感归纳段落（Markdown）；**默认不**嵌入竞品报告正文。""" + p = dict(payload) + scope_note = "" + mg = p.get("matrix_group_focus") + if isinstance(mg, str) and mg.strip(): + scope_note = ( + f"\n\n【范围】以下评价与统计**仅**来自细类「{mg.strip()}」；" + "正向/负向主题须贴合**该细类**语境，勿笼统写成「全关键词下用户普遍…」。\n" + ) + raw = json.dumps(p, ensure_ascii=False) + if len(raw) > 88_000: + for k, cap, maxlen in ( + ("sample_reviews_positive_biased", 6, 180), + ("sample_reviews_mixed_tone", 4, 180), + ("sample_reviews_negative_biased", 14, 200), + ("sample_reviews_semantic_pool", 30, 340), + ): + lst = p.get(k) + if isinstance(lst, list): + p[k] = [str(x)[:maxlen] for x in lst[:cap]] + raw = json.dumps(p, ensure_ascii=False) + if len(raw) > 88_000: + raw = raw[:82_000] + "\n\n…（输入过长已截断，请勿编造截断外内容）\n" + user = "请根据以下 JSON 按系统说明输出 Markdown：" + scope_note + "\n\n" + raw + out = call_llm(SENTIMENT_LLM_SYSTEM, user) + if isinstance(mg, str) and mg.strip(): + out = demote_sentiment_inner_h4_to_h5_for_matrix_group(out) + return out + + +def split_competitor_report_for_bridges( + md: str, *, max_excerpt: int = 1200 +) -> dict[str, dict[str, str]]: + """ + 按「## 一、」…「## 九、」切分规则报告；**只返回正文中实际出现的章**（略去未输出的章）。 + 每键含完整标题行与正文摘录（过长截断）。 + """ + pat = re.compile(r"^## ([一二三四五六七八九])、([^\n]*)$", re.MULTILINE) + matches = list(pat.finditer(md)) + out: dict[str, dict[str, str]] = {} + for i, m in enumerate(matches): + key = m.group(1) + rest = m.group(2) + title = f"## {key}、{rest}" + start = m.end() + end = matches[i + 1].start() if i + 1 < len(matches) else len(md) + body = md[start:end].strip() + exc = body[:max_excerpt] + if len(body) > max_excerpt: + exc += "\n\n…（本节摘录已截断）\n" + out[key] = {"title": title, "excerpt": exc} + return out + + +def _parse_llm_json_object(text: str) -> dict[str, Any]: + raw = (text or "").strip() + if not raw: + return {} + if raw.startswith("```"): + raw = re.sub(r"^```(?:json)?\s*", "", raw, flags=re.IGNORECASE) + raw = re.sub(r"\s*```\s*$", "", raw) + try: + obj = json.loads(raw) + return obj if isinstance(obj, dict) else {} + except json.JSONDecodeError: + pass + m = re.search(r"\{[\s\S]*\}", raw) + if m: + try: + obj = json.loads(m.group(0)) + return obj if isinstance(obj, dict) else {} + except json.JSONDecodeError: + pass + return {} + + +def _normalize_section_bridge_map(d: dict[str, Any]) -> dict[str, str]: + allowed = frozenset("一二三四五六七八九") + out: dict[str, str] = {} + for k, v in d.items(): + if not isinstance(k, str) or len(k) != 1 or k not in allowed: + continue + if isinstance(v, str) and v.strip(): + out[k] = v.strip() + return out + + +BRIDGE_SECTIONS_SYSTEM = """你是竞品监测报告的**章节衔接**撰稿助手。 + +**输入 JSON** 含： +- ``keyword``：监测词； +- ``competitor_brief``：与本报告一致的**结构化摘要**（已裁剪体积）； +- ``sections``：键为汉字「一」～「九」，每项含 ``title``（该章完整二级标题行）与 ``excerpt``（该章正文开头摘录，可能已截断）。 + +**任务**：为 **sections 中出现的每一键** 各写一段 **衔接性分析**（帮读者从摘要与摘录过渡到读该章表格/图），并与 ``competitor_brief`` 中的数字与结论一致。 + +**硬性要求**： +- **仅输出一个 UTF-8 JSON 对象**（不要用 markdown 代码围栏包裹整段输出）； +- 键必须为「一」「二」…「九」之一，且 **只对输入 sections 里存在的键** 给出字符串值；可省略无材料的键； +- 每个值为 **Markdown 片段**（约 3～10 句中文），**禁止**使用 ``## `` 开头的行（不要写新的二级章标题）；可使用 ``###`` / ``####`` 或加粗小标题； +- 所有**定量表述**须能在 ``competitor_brief`` 或对应 ``excerpt`` 中找到依据，**禁止编造** SKU 数、份额、价格； +- **店铺/自营相关（硬性）**：**禁止**编造「京东自营 SKU 占比」「自营占比超 X%」「POP/第三方占比」等**未在输入中出现的**具体比例或款数；若 ``competitor_brief.concentration.shops_from_list`` 有数据，写店铺集中度时须与之一致，并区分 **按列表行** 与 **按去重 SKU**（见 ``unique_sku_basis``），**禁止**将列表曝光写成「市场份额」或笼统「SKU 占比」。 +- 不要复述整章表格；不要写「详见下文矩阵」以外的空洞套话；可点出该章阅读重点（如价盘带、矩阵细类、评价规则局限等）。""" + + +def generate_section_bridges_llm( + *, + keyword: str, + brief: dict[str, Any], + sections: dict[str, dict[str, str]], +) -> dict[str, str]: + """一次 LLM 调用，返回各章衔接 Markdown 片段（键：一～九）。""" + if not sections: + return {} + compact = compact_brief_for_llm(brief, max_chars=100_000) + sec: dict[str, dict[str, str]] = { + k: {"title": v.get("title", ""), "excerpt": v.get("excerpt", "")} + for k, v in sections.items() + if isinstance(v, dict) + } + for max_exc in (1200, 900, 600, 400, 280): + for v in sec.values(): + ex = v.get("excerpt") or "" + if len(ex) > max_exc: + v["excerpt"] = ex[:max_exc] + "\n…\n" + payload = { + "keyword": keyword, + "competitor_brief": compact, + "sections": sec, + } + raw = json.dumps(payload, ensure_ascii=False) + if len(raw) <= 92_000: + break + user = "请严格按系统说明，**只输出一个 JSON 对象**（键为一～九，值为 Markdown 字符串）：\n\n" + raw + text = call_llm(BRIDGE_SECTIONS_SYSTEM, user) + return _normalize_section_bridge_map(_parse_llm_json_object(text)) diff --git a/backend/pipeline/llm/generate_strategy.py b/backend/pipeline/llm/generate_strategy.py new file mode 100644 index 0000000..c1b212f --- /dev/null +++ b/backend/pipeline/llm/generate_strategy.py @@ -0,0 +1,605 @@ +"""独立策略稿润色（`generate_strategy_draft_markdown_llm`）与可选的报告「策略与机会」块归纳（`generate_strategy_opportunities_llm`，默认产线关闭）。""" +from __future__ import annotations + +import json +import os +from typing import Any + +from ..reporting.brief_compact import compact_brief_for_llm +from ..reporting.strategy_draft import ( + build_strategy_draft_markdown, + report_uses_chapter8_text_mining_probe, +) +from .llm_client import call_llm, estimate_chat_input_tokens, llm_context_window_size + +# 与策略生成表单 POST 字段一致：任一则视为业务已提供「实质决策」，否则由模型基于数据推断草案。 +_STRATEGY_DECISION_SUBSTANTIVE_KEYS: tuple[str, ...] = ( + "product_role", + "stage_goal_type", + "battlefield_one_line", + "audience_segment", + "time_horizon", + "success_criteria", + "non_goals", + "positioning_choice", + "competitive_stance", + "pillar_product", + "pillar_price", + "pillar_channel", + "pillar_comm", + "marketing_strategy", + "general_strategy", + "competitor_reference", + "resource_notes", +) + + +def strategy_decisions_substantive(strategy_decisions: dict[str, Any] | None) -> bool: + """是否填写了至少一项策略表单文本字段（用于 LLM 是否「自动推断」全稿）。""" + if not isinstance(strategy_decisions, dict): + return False + for k in _STRATEGY_DECISION_SUBSTANTIVE_KEYS: + v = strategy_decisions.get(k) + if isinstance(v, str) and v.strip(): + return True + return False + + +def _omit_ch8_probe_wordchart_fields(compact: dict[str, Any]) -> None: + """ + 第八章文本挖掘（探针）为主时，去掉与**预设关注词/场景条形图**同源的统计字段， + 避免与报告 §8 文本挖掘主口径「两用数据」。 + + 仅影响传入大模型的 ``structured_brief``；``brief`` 全量仍可由规则稿使用。 + """ + for k in ( + "comment_focus_keywords", + "usage_scenarios", + "usage_scenarios_denominator", + "usage_scenarios_by_matrix_group", + "comment_sentiment_lexicon", + "strategy_hints", + ): + compact.pop(k, None) + cfb = compact.get("consumer_feedback_by_matrix_group") + if not isinstance(cfb, list): + return + slim: list[Any] = [] + for g in cfb: + if not isinstance(g, dict): + slim.append(g) + continue + slim.append( + { + k: v + for k, v in g.items() + if k not in ("focus_keyword_hits", "scenarios_top") + } + ) + compact["consumer_feedback_by_matrix_group"] = slim + + +STRATEGY_DATA_RULES = """**全局禁止编造（硬性）**：下列条款**同时**适用于 ① **独立策略稿**全文；② 若任务仍生成的**宿主报告内** ``####`` 策略归纳块（JSON 含 ``competitor_brief``）。**默认产线**下报告内第九章大模型长文已关闭，**独立策略稿不以该块为默认事实源**。 +- **事实与数字**：销量、GMV、占比、价带、条数、份额、券面额、满减/满折门槛、到手价、店铺/品牌计数与排名、SKU 数、接口返回量等，**仅可**来自**本次调用输入 JSON** 已给出的字段。**独立策略稿**侧为：`structured_brief`、`rules_draft_markdown` 内摘录、**可选** `report_strategy_excerpt`（**默认多为空**，见 ``load_report_strategy_excerpt``）、**可选** `report_matrix_group_evidence_md`（与同任务报告第五～第八章细类归纳同源）、`strategy_decisions`、`business_notes`。**报告内嵌策略块**侧为 ``competitor_brief``、可选 ``prior_chapter_llm_narratives``。**禁止**凭空新增、改口径或写成「已监测证实」而无字段支撑。 +- **主体与名称**：**禁止**引入上述输入中**未出现**的**具体**品牌名、店铺名、SKU 名、商品标题作为**事实陈述**；若 `strategy_decisions`/备注/brief/节选已含则可写；否则用「头部/同类竞品」等泛称或「待业务指定对标」。 +- **用户侧表述**：**禁止**虚构评价原文、访谈引语、带引号的「用户说…」；细则见下文「§2 针对痛点要怎么做」表**痛点简述**列。 +- **促销与活动**：**禁止**编造活动名、具体规则、补贴比例；细则见下文促销与第八章探针相关条款。 +- **策略动作与落地结果**：可写「建议」「假设」「待验证」的动作方向，**不得**编造「已执行」「已上线」「数据显示转化率/复购提升」等**无输入依据**的结果。 +- **信息不足**：须写「输入未体现」「待核对」「假设：」「待验证：」，**禁止**用确定语气掩盖缺失依据。 +- **与 §2.1「类目/细类」列一致（全文）**：除 §2.1 表格外，**摘要、一、三～八**凡写策略动作、阶段重点、资源分配、差异化或竞争应对，**优先**标明适用**类目/细类**；多细类策略冲突时**分条**写。**禁止**用「全站用户」「整体上一句」覆盖与 §2.1 已分行决策**矛盾**的表述。 + +**与竞品分析报告的分工（硬性）**： +- 宿主报告已含样本量、价带分布、词频/共现、矩阵、第八章文本挖掘等**统计分析**。策略稿**不得**重复展开同类内容：不重写词频表、细类评论条数罗列、统计方法说明、与报告图表逐条复述。 +- **允许**：用一两句**结论性**话概括用户侧/评论侧要点；必要时写「详见同任务《竞品分析报告》§× / 附录」。 +- **必须**把篇幅放在**策略**：§2「针对痛点要怎么做」、后文战术与节奏（**勿**与报告重复统计展开）。 + +**数据与口径（硬性，与宿主分析报告同源输入）**： +- **§2「针对痛点要怎么做」表（反捏造 + 分类目，硬性）**： + - **「类目/细类（本决策适用）」列**：须与 `structured_brief` 中类目混排、矩阵分组、§1.2 细类讨论或 `strategy_decisions` 已选战场**可对上**；**禁止**编造未出现的类目名。**多细类并存**（如饼干 vs 面包）时，**必须分行**分策，**禁止**用「全站用户」「整体策略」等**泛化**一句覆盖彼此冲突的动作。**若**类目或主推线尚不确定，该行可写「待业务定类」或「假设：优先××线」，并说明**分类决策依据或待补信息**；仍须避免与数据明显矛盾。 + - **「用户痛点（简述）」列**：**禁止**书写「用户反馈『……』」「评价称『……』」等**带引号的逐字原话**，除非该片段在 `structured_brief`、`strategy_hints`、`report_strategy_excerpt` 或 `business_notes` 中**已出现相同或明显包含**的文本；否则一律**不得**用引号假装引用。 + - 若输入仅有主题级信号（关注词、负向归因方向、价差行数等），痛点简述应写**可追溯归纳**，例如「与 brief 中 ×× 字段一致」「与报告第八章/节选已归纳的 ×× 主题一致」「监测摘要见 `strategy_hints` 第 n 条」，或写「**待原评论抽样核实**」——**禁止**把合理推测写成「用户已明确说……」的事实口吻。 + - **禁止**凭空发明痛点行（如「配料相似」「卖点雷同」「性价比一般」）作为**已监测结论**；此类表述仅当 `structured_brief`、节选或备注中**确有同类主题或措辞**时方可写入，否则不写或标为待验证假设。 +- **不得编造**销量、GMV、未在 `structured_brief` 与底稿中出现的占比或价格；底稿与摘要中的数字须保持一致。 +- **店铺集中度**仅可依据 `structured_brief.concentration` 与底稿，并区分**列表行**与**去重 SKU**；用「第一大……份额」「前三家合计」等中文，**不要用** CR1、CR3。 +- **禁止编造**「京东自营 SKU 占比」「自营超 X%」等摘要中未给出的定量句。 +- **矩阵**：若 `structured_brief` 含矩阵相关字段，须**呼应**细分类目与竞品矩阵结论，不得无故删光。 +- **第八章文本挖掘探针（当 JSON 中 `chapter8_text_mining_probe` 为真时）**： + - **禁止**将「关注词子串命中次数」「预设场景分组条数/占比」当作评论侧主论据。 + - 用户洞察、负向归因须与 **§8 文本挖掘** 及可选节选一致；促销与券价差须与 `price_promotion_signals`、报告**第六章**及 brief 已给字段一致（**默认**无宿主报告内长文策略节选时，**禁止**以「第九章已写」为凭据编造具体规则）；**禁止**编造满减门槛或补贴比例。 +- **可选 `report_strategy_excerpt`**：**默认多为空**。非空时战略方向与该节选不明显矛盾；**不得**把节选与 `structured_brief` 均未出现的数字当作事实。**为空时**以 `structured_brief`、`report_matrix_group_evidence_md`（若有）与底稿/表单为准，**禁止**编造「宿主报告策略章已断言的」具体结论或虚假背书。""" + +STRATEGY_SYSTEM = f"""你是市场策略顾问，根据**结构化监测摘要**与业务侧填写的**决策字段**，把「规则底稿」写成**短、可执行**的策略 Markdown **独立成稿**。 + +**输入**：`rules_draft_markdown`（规则骨架，**六主轴 + 品牌四线**结构，与 `docs/demo` 市场策略稿示例同构）、`structured_brief`、`strategy_decisions`、`business_notes`；**可选** `report_strategy_excerpt`（**默认多为空**，遗留或历史任务可能非空）；**可选** **`report_matrix_group_evidence_md`**（与所选细类对齐的宿主报告第五～八章归纳摘录——**主对齐源之一**）。 + +**与细类收窄及遗留节选（硬性）**： +- 当 JSON 含 `report_matrix_group_evidence_md` 且非空时： + - **定性主题**（用户讨论焦点、卖点/配料叙事、负向体验类型、场景与关注词归纳方向等）须与该节选及 `structured_brief` **方向一致**，**禁止**另写一套与节选**明显矛盾**的品类判断。 + - **数字、份额、价带、条数**仍以 **`structured_brief` 为准**；节选与 brief 数字冲突时**采纳 brief**，勿复述冲突数字句。 +- **`report_strategy_excerpt`**：**默认产线多为空**（报告内全任务大模型策略长文已弃用）。非空时仅作**弱参考**；写**收窄细类**策略时仍以 `structured_brief` + `report_matrix_group_evidence_md`（若有）为主，**不得**把全关键词池结论套成该细类已证实事实。**默认为空时禁止**写「报告第九章已归纳…」类虚假背书。 + +{STRATEGY_DATA_RULES} + +**规划核心 §1.1（硬性）：策略要写「怎么做」，不能只写「是什么」** +（与 `docs/planning/策略生成-框架确定.md` §1.1 一致；违反则成稿不合格。） +- **读者测试**：业务读者读完**摘要、一、三～八**任一大节后，应能回答至少一项：**谁（团队/渠道）在何触点、针对哪类用户或哪条痛点、采取什么动作、如何验收或待验证什么**。若某段只能回答「市场/品类/价带是什么样」而**没有**紧随或嵌入的「故本阶段须…」「优先…」类**动词句**，须改写或删并，**禁止**以形势描述段作为该节主体。 +- **背景上限**：**一、顾客是谁** 的 1.1 与 1.2 **禁止**扩写成第二份分析报告：合计**至多约五句**结论性背景（谁搜、关心什么、分细类一句）；价带分位、样本量拆解、词频/方法一句带过或写「详见同任务《竞品分析报告》」，**禁止**多段连续铺陈数据。 +- **摘要**：在「范围与样本」「用户侧」各**一句**可接受后，**阶段重点**必须是 **1～2 条完整执行句**，每条须含**可识别动作**（如统一商详第几屏表述、主图试点、规格命名、客服首句、跟价/不跟价说明等）之一，**禁止**单独使用「加强运营」「把握机会」「提升体验」「深化心智」等无主体、无触点、无痛点指向的套话。 +- **§2.1 表**：监测已支撑**多个**痛点或细类维度时，**至少两行**有实质内容（非空、非整格「待填」）；**策略动作**与**具体怎么做**两列须以**动词短语或短句**开头，**禁止**两列长期只有形容词、名词标签或泛化口号。 +- **§六～§八**：每一 numbered 小节（如 §6.2、§7.x、§8.x）须含**至少一条**可指回 §2.1 某一行的落地动作（可口头合并叙述）；**禁止**仅用「强化品牌/优化体验/夯实基础」等名词堆叠而无**谁做、在哪做、做哪一步**。 +- **反例（禁止作为节内主要篇幅）**：「当前品类呈现…」「市场整体…」「用户日益注重健康」等**纯判断句串**而无后续「因此我方本阶段…」；若保留背景，**一句**后必须接执行句。 + +**落实范围**：上文「全局禁止编造」适用于**摘要、一至十、附录**的每一句话与表格每一格；**不得**因章节不同而放宽。 + +**对外成稿与禁止技术泄露（硬性）**： +- 正文须为**可直接对业务或合作方阅读**的正式策略文档（对外前仍须按需脱敏）。**禁止**出现：反引号代码体、JSON 键名、英文字段名、内部数据结构名、源码或仓库路径、类文件名、「任务 ID」「工作台」「规则骨架」等系统痕迹；**禁止**照抄底稿中以 *成稿：*、*回答：*、*占位*、*骨架* 开头的**元说明句**，须改写为正式业务表述。 +- **禁止**在正文使用**写作指导式**套话（读者不应看到「作者须知」）：例如「须与 §2/§2.1 一致」「与 §2.1 类目列可对上」「为后文……埋伏笔」「回扣 §2」「承接 §2」「勿重复 §×」等——须直接写**实质策略内容**，勿解释章节之间如何对齐。 +- **一级标题**用文书式，例如 `# 「{{keyword}}」市场策略建议书（草案）`，其中 `keyword` 取自本消息 JSON 的同名字段；**勿**使用「草稿」「底稿」「归纳用」等对内用词。 +- **附录**中的采集范围等信息用**中文短句**（如「列表页约采集第 3～10 页」），**禁止** `page_start=` 等键值对或英文键名。 + +**业务决策未填写时的成稿义务（当 JSON 中 `strategy_decisions_substantive` 为 false 时）**： +- 视为未提交表单决策：须基于监测摘要、细类报告节选、底稿数据表及 `business_notes`（若有）**主动推断**一套连贯的**假设性**策略；在「策略范围与前提」写清推断前提（「假设：」「待业务确认：」），对阶段目标类型给出 **A～E 类选项及你从数据中归纳的推荐倾向**（不得只列选项而无立场）。 +- **§2.1 针对痛点要怎么做**须含**多行实质内容**，覆盖监测已支撑的主要细类与痛点，**禁止**整表留白或满篇 *（待填）*。 +- 仍须遵守全局禁止编造：数字、品牌、店铺、用户原话、活动规则仅可来自输入依据；无依据处用假设语气。 + +**决策边界（硬性）**： +- **当 `strategy_decisions_substantive` 为 true 时**：业务已在 `strategy_decisions` 中填写的项（角色、**本阶段策略目标类型**、时间、成功标准、战场一句话、定位勾选、竞争倾向、四柱、目标客群/对标/资源备注、**营销策略**与**总体策略**等）视为**已定决策**：成稿须**落实为具体执行句**，**不得**改写成相反结论或再要求用户「请选择」。**若「本阶段策略目标类型」在输入 JSON 中已给出非空文本**，「策略范围与前提」表中该列须**直接采用该表述**（可略作语序润色），**不得**改判为另一类阶段目标。 +- **当 `strategy_decisions_substantive` 为 true** 而部分表单项仍为空或占位：结合监测摘要与节选**补全为可执行表述**，与数据方向一致。 +- **当 `strategy_decisions_substantive` 为 false 时**：适用上文「业务决策未填写时的成稿义务」，**禁止**以「请先填表」类表述搪塞全篇。 +- **成稿阶段避免**：反复「请业务决策」；不确定时在 §2.1 用「类目/细类」+「假设：」「待业务确认：」**写清**，**禁止**只写泛化一句。 + +**输出结构与阅读顺序（须与 `rules_draft_markdown` 章节一致，勿另起目录）**： +**策略范围与前提（生成前先对齐）** → **摘要** → **一、顾客是谁**（含人群与路径、细类讨论、本品聚焦）→ **二、产品价值与用户痛点**（**仅 §2.1 针对痛点要怎么做** 表，**勿**再设独立「痛点表/价值对表/负向归因」子节）→ **三、为什么要买「这款产品」**（**仅 §3.1 购买者视角：为何要选这一款（依据与理由）**；**无 §3.2**，转化与价带应对已在 §2 表内则**勿重复**）→ **四、为什么要选「这个品牌」** → **五、与其它品牌有何不同** → **六、阶段目标与路径** → **七、品牌四线**（建设·打造·运营·体验）→ **八、战术支柱**（产品/定价/促销/渠道与传播）→ **九、风险、假设与待验证** → **十、下一步与节奏**（含业务备注）→ **附录**。 +可微调小节标题用语，**不得**删减上述逻辑块或把「诊断数据」与「落地动作」顺序颠倒；**禁止**私自恢复已删除的小节（如 §3.2）。 + +**语气**：面向业务读者，避免 CR1、心智等内部缩写；**勿在成稿中反复强调「对齐某报告第几章」**，以策略表述为主。 + +**策略表述硬性（痛点 → 怎么做，须覆盖全书，不得只写 §2～§8 部分章节）**： +- **总原则**：成稿**不是**第二份分析报告，也**不是**市场形势说明书。每条重要内容应能回答：**针对哪条用户痛点、在哪条类目/细类下**（与 **§2.1** 表对应）、**我们采取什么动作**、**在具体触点怎么做**（商详/主图/短视频/客服/规格/价格呈现等）、**如何验证**（若适用）。**「是什么」仅作每节不超过一两句的铺垫；「怎么做」须占可策略论述篇幅的主体。** +- **§2.1 针对痛点要怎么做**（若底稿已有表头）须**填写实质内容**；全稿**动作总锚**为 §2.1。若无表，须在 **§二** 或 **§八** 用等价分条写清「痛点—动作—落地—验证」。 + +**分节要求（与底稿章节一一对应，勿省略）**： +- **策略范围与前提**：回答「**这份策略是针对什么做的**」（监测任务、本品角色、战场、主推类目、**本阶段目标类型**、时间、成功标准）。与业务表单及备注对齐；`strategy_decisions_substantive` 为 false 时须写清假设前提与推荐目标类型（可含 A～E 选项），为 true 时未填项不得与已填决策矛盾。**禁止**与后文 §2.1、§六 自相矛盾。 +- **摘要**：除范围样本外，**阶段重点**须含 1～2 条**可执行动作**，指向优先痛点（非空泛「加强运营」）；须与上文「策略范围与前提」边界一致（**勿**在正文写「回扣 §2」「承接上文」等指导语）。 +- **一、顾客是谁**：**禁止**重复报告中的细类词频、分品类样本量展开、文本挖掘方法；用 **少量结论句**（谁搜、关心什么、决策场景）；**1.3 本品聚焦**须写清本期主攻人群/场景/细类（**勿**写「与 §2.1 可对上」类作者提示）。 +- **二**：**仅 §2.1** 一张表：「类目/细类（本决策适用）| 用户痛点（简述）| 策略动作 | 具体怎么做 | 如何验证」。须覆盖监测已支撑的主要维度（**按类目分行**，口感/质地分线、分量/规格、信任与价格等，依数据取舍）；**类目列 + 痛点简述列**遵守「§2 表」条款。**禁止**再写独立「痛点与证据表」「价值对表」「负向归因」子节（与 §二 重复的内容一律并入本表或删去）。 +- **三**：**仅 §3.1**，标题与底稿一致为**购买者视角：为何要选这一款（依据与理由）**。全文须站在**购买者**一侧：写其在浏览/比价时**为何值得把这一款放进购物车**（解决什么具体问题、相对同类获得感、价位是否可接受、信任点是什么），可用「用户/消费者」作主语。**先**保留或转述输入中已有**检索/样本与价带**（作买家决策背景，勿大段铺陈），**随后**用 1～2 句落到**购买动机**。**禁止**用运营/品牌单方口吻替代买家逻辑（如「适合××叙事切入」「策略上占位」「品类时机好」作为收尾而不说买家得到什么）。**禁止**以只适用于整个品类的宏观句作为**唯一或最后**结论；宏观背景若写，**必须**收束到「因此**买家**更愿为这一款付费」的可验证点（规格/配料/口感/价位等须与输入可对读）。可结合 brief 写价带锚点一句。**禁止**写 §3.2「转化障碍与应对」；若与购买相关的障碍与应对已在 §2.1 表内，§3.1 **勿再复述**。 +- **四**：品牌承诺与调性须能落到**可感知触点**（如商详第几屏、包装、客服首句），避免只有形容词。 +- **五**：§5.2 差异化、§5.3 竞争应对须写清**相对竞品多做什么/少做什么、具体一步动作**。 +- **六**：成功标准与 §6.2 路径须与 **§2.1** 动作**可对齐或合并叙述**；营销/总体策略句须为**动词导向**。 +- **七**：品牌四线**每一条**至少一句：**服务哪类痛点、本周/本阶段具体做哪一步**。 +- **八**：四支柱**每一支柱**须回扣 **痛点→动作→落地**（可与 §2.1 合并叙述，避免重复堆砌）。 +- **九**：在表单风险勾选之外，**每条风险**尽量带**应对动作或验证计划**（抽样、核对规则），勿只列风险标题。 +- **十**：下一步清单须为**可执行任务**（可含负责人/时间占位），与 §2.1 或 §六 优先级一致；可含「按类目核对主图/商详与 §2.1」类项。 + +**全书与 §2.1 类目列对齐（防泛化，与上条「全文一致」配套）**： +- **摘要**：阶段重点中的可执行动作**尽量**点明适用类目或主推线。 +- **四、五**：品牌承诺、差异化、竞争应对若因细类而异，**分款/分类目**写。 +- **六**：路径与成功标准若多类目并行，**分线**写 KPI 或写清主线/副线。 +- **七**：品牌四线每条宜**可指回** §2.1 某类目行；若四线共用全池，须一句交代**共用前提**。 +- **八**：四支柱下若产品/定价/促销策略因类目不同，**分子条**（如「饼干：…」「面包：…」），勿与 §2.1 矛盾。 +- **九**：可写「主推类目未定」「多线话术不一致」等风险及验证方式。 + +**口感/质地与细类（禁止「一词盖全站」）**： +- 监测中「**酥脆**」与「**松软**」等可能**同时**高频，通常对应**不同细类**（如饼干 vs 面包/糕点）或不同场景。成稿须**按主推细类或分产品线**表述：饼干线策略与酥脆/饱腹等对齐，面包/糕点线与松软/早餐等对齐；若多线并存须**分款分句**，**禁止**只写「要做松软」而忽略酥脆主导的细类，除非 `structured_brief`、表单或业务备注已明确**仅**推该线。 +- **产品策略句**须能指回：**本品是哪一类、解决哪条口感预期**，避免与数据里另一细类的主导词打架。 + +**促销：满减、满折、券（≠ 不管；≠ 编造）**： +- **必须**在 **§八.3 促销与活动策略**（及必要时 §七.3）写清：与 `price_promotion_signals`、报告第六章已归纳的**券、标价与到手价差、常见活动形态**如何承接（跟价节奏、规则透明、不与数据矛盾）；**禁止**因「没编出具体数字」就整节不写促销。 +- **禁止编造**输入中未出现的**具体**满减门槛、满额折扣、每满减金额；若摘要/报告未捕获某类机制，须明确写「**监测未捕获具体满减/满折规则，上架前须与运营及后台活动对齐后再对外宣称**」，并可列**待补信息**（如：是否参加跨店满减、店铺券类型）。 +- **区分**：「策略上跟券、保到手价透明」是成稿义务；「具体满 300 减 40」只能来自已有数据。 + +**输出**：仅 Markdown 正文（不要 ``` 围栏）；须收束各小节与全文，勿中途截断。""" + +STRATEGY_USER_PREFIX = ( + "请基于以下 JSON 输出最终策略稿（Markdown），正文须为对外可读正式文档，不得泄露 JSON 键名、字段名、源码路径或底稿中的编写提示语。\n" + "输出前自检：全文不得包含输入中未出现的具体数字、品牌/店铺名、用户引语与活动规则；不确定处须写「假设」「监测未体现」或「待业务核对」。\n" + "输出前自检（规划 §1.1）：摘要「阶段重点」是否为 1～2 条含动作+触点（或时间窗口）的执行句；第一章是否未写成长篇市场白皮书；§2.1 是否至少两行实质且「动作/怎么做」列为动词句；§六～§八 每节是否至少一条可落地的「谁在哪做什么」。若否，先改再输出。\n" + "若 JSON 中 `strategy_decisions_substantive` 为 false：你须基于监测摘要与细类报告节选**主动推断**完整策略草案（含 §2.1 多行实质内容），" + "在「策略范围与前提」标明假设前提，并对阶段目标给出 A～E 类型选项及**推荐倾向**；禁止全文停留在待填占位。\n" + "若 `strategy_decisions_substantive` 为 true：已填表单项视为已定须落实；空项结合数据补全，并与后文一致。\n\n" +) + + +def _build_strategy_draft_llm_payload_and_user( + *, + job_id: int, + keyword: str, + generated_at_iso: str, + strategy_decisions: dict[str, Any], + business_notes: str, + brief: dict[str, Any], + report_config: dict[str, Any] | None, + rules_md: str, + excerpt_raw: str, + group_evidence_raw: str, + compact_max: int, + excerpt_max: int, + rules_max: int | None, +) -> tuple[dict[str, Any], str]: + compact = compact_brief_for_llm(brief, max_chars=compact_max) + if report_uses_chapter8_text_mining_probe(report_config): + compact = dict(compact) + _omit_ch8_probe_wordchart_fields(compact) + ex = ( + _truncate_strategy_narrative(excerpt_raw, excerpt_max) if excerpt_raw else "" + ) + ev_max = min(24_000, max(3_000, excerpt_max + excerpt_max // 2)) + gm = ( + _truncate_strategy_narrative(group_evidence_raw, ev_max) + if group_evidence_raw + else "" + ) + if rules_max is None: + rd = rules_md + else: + rd = _truncate_rules_draft_md(rules_md, rules_max) + payload: dict[str, Any] = { + "job_id": job_id, + "keyword": keyword, + "generated_at_iso": generated_at_iso, + "strategy_decisions": strategy_decisions, + "strategy_decisions_substantive": strategy_decisions_substantive( + strategy_decisions + ), + "business_notes": business_notes, + "structured_brief": compact, + "rules_draft_markdown": rd, + "report_strategy_excerpt": ex, + "report_matrix_group_evidence_md": gm, + "chapter8_text_mining_probe": bool( + report_uses_chapter8_text_mining_probe(report_config) + ), + } + if report_uses_chapter8_text_mining_probe(report_config): + payload["structured_brief_omission_note"] = ( + "已启用第八章文本挖掘（探针为主）：structured_brief 已省略「关注词/场景子串计数」、按细类 feedback 中的 focus_keyword_hits/scenarios_top、" + "``strategy_hints`` 等；报告已**不再**输出 ``comment_sentiment_lexicon``（星级子集预设口语短语）及同口径图。**不得**再以这类子串计数、短语条形图或预设场景占比作为论据。" + "用户与评论侧须依报告 §8 文本挖掘归纳及 `report_matrix_group_evidence_md`；**促销、满减、券价差**须与报告第六章、`price_promotion_signals` 及 brief 已给字段一致；若 `report_strategy_excerpt` 非空则勿与其明显矛盾。**默认**节选为空，勿编造「报告策略长文已写明的」具体活动规则。" + ) + raw = json.dumps(payload, ensure_ascii=False) + if len(raw) > 500_000: + payload["rules_draft_markdown"] = _truncate_rules_draft_md(rd, 200_000) + raw = json.dumps(payload, ensure_ascii=False) + return payload, STRATEGY_USER_PREFIX + raw + + +def resolve_strategy_draft_llm_input_snapshot( + *, + job_id: int, + keyword: str, + brief: dict[str, Any], + business_notes: str, + generated_at_iso: str, + strategy_decisions: dict[str, Any], + report_strategy_excerpt: str | None = None, + report_matrix_group_evidence_md: str | None = None, + report_config: dict[str, Any] | None = None, +) -> tuple[dict[str, Any], str, str]: + """ + 复现 ``generate_strategy_draft_markdown_llm`` 在**首档通过** ``_strategy_prompt_ok_for_call`` 时的 + ``payload`` 与完整 ``user`` 字符串（不请求网关）。 + + 返回 ``(payload, user, tier_note)``；若所有档位均未通过，与生产一致仍返回最后一档兜底组装的 + ``(payload, user, tier_note)``（``tier_note`` 标明可能仍会由网关报错）。 + """ + rules_md = build_strategy_draft_markdown( + job_id=job_id, + keyword=keyword, + brief=brief, + business_notes=business_notes, + generated_at_iso=generated_at_iso, + strategy_decisions=strategy_decisions, + report_config=report_config, + for_llm_input=True, + ) + excerpt_raw = (report_strategy_excerpt or "").strip() + group_evidence_raw = (report_matrix_group_evidence_md or "").strip() + sys_prompt = STRATEGY_SYSTEM + min_comp = _min_strategy_completion_tokens() + min_comp_relaxed = max(256, min_comp // 2) + + for cap_brief, cap_excerpt, cap_rules in ( + (80_000, 24_000, None), + (64_000, 20_000, None), + (48_000, 17_000, None), + (36_000, 14_000, None), + (28_000, 11_000, None), + (22_000, 9_000, None), + (18_000, 7_000, None), + (14_000, 5_000, None), + (12_000, 4_000, 220_000), + (10_000, 3_500, 180_000), + (10_000, 3_000, 150_000), + (9_000, 2_500, 120_000), + (8_000, 2_000, 100_000), + (8_000, 2_000, 70_000), + ): + payload, user = _build_strategy_draft_llm_payload_and_user( + job_id=job_id, + keyword=keyword, + generated_at_iso=generated_at_iso, + strategy_decisions=strategy_decisions, + business_notes=business_notes, + brief=brief, + report_config=report_config, + rules_md=rules_md, + excerpt_raw=excerpt_raw, + group_evidence_raw=group_evidence_raw, + compact_max=cap_brief, + excerpt_max=cap_excerpt, + rules_max=cap_rules, + ) + if _strategy_prompt_ok_for_call( + sys_prompt, user, min_completion_tokens=min_comp + ): + rules_note = ( + "未截断" + if cap_rules is None + else f"rules_draft_markdown 截断上限 {cap_rules} 字" + ) + note = ( + "首档（标准 completion 阈值）：" + f"structured_brief max_chars={cap_brief}，" + f"report_strategy_excerpt / 节选侧 excerpt_max={cap_excerpt}，" + f"{rules_note}。" + ) + return payload, user, note + + for cap_brief, cap_excerpt, cap_rules in ( + (10_000, 2_000, 55_000), + (8_000, 1_500, 45_000), + (7_000, 1_200, 35_000), + ): + payload, user = _build_strategy_draft_llm_payload_and_user( + job_id=job_id, + keyword=keyword, + generated_at_iso=generated_at_iso, + strategy_decisions=strategy_decisions, + business_notes=business_notes, + brief=brief, + report_config=report_config, + rules_md=rules_md, + excerpt_raw=excerpt_raw, + group_evidence_raw=group_evidence_raw, + compact_max=cap_brief, + excerpt_max=cap_excerpt, + rules_max=cap_rules, + ) + if _strategy_prompt_ok_for_call( + sys_prompt, user, min_completion_tokens=min_comp_relaxed + ): + note = ( + "首档（relaxed completion 阈值）：" + f"structured_brief max_chars={cap_brief}，" + f"excerpt_max={cap_excerpt}，" + f"rules_draft 截断上限 {cap_rules}。" + ) + return payload, user, note + + payload, user = _build_strategy_draft_llm_payload_and_user( + job_id=job_id, + keyword=keyword, + generated_at_iso=generated_at_iso, + strategy_decisions=strategy_decisions, + business_notes=business_notes, + brief=brief, + report_config=report_config, + rules_md=rules_md, + excerpt_raw=excerpt_raw, + group_evidence_raw=group_evidence_raw, + compact_max=6_000, + excerpt_max=1_000, + rules_max=28_000, + ) + note = ( + "所有标准/relaxed 档位均未通过 ``_strategy_prompt_ok_for_call``，" + "与生产一致使用兜底档：structured_brief max_chars=6000，excerpt_max=1000，" + "rules_draft 截断上限 28000（网关仍可能报错）。" + ) + return payload, user, note + + +def generate_strategy_draft_markdown_llm( + *, + job_id: int, + keyword: str, + brief: dict[str, Any], + business_notes: str, + generated_at_iso: str, + strategy_decisions: dict[str, Any], + report_strategy_excerpt: str | None = None, + report_matrix_group_evidence_md: str | None = None, + report_config: dict[str, Any] | None = None, +) -> str: + """ + ``report_strategy_excerpt``：可选；由 ``load_report_strategy_excerpt`` 加载（见 ``reporting.report_strategy_excerpt``）。**默认产线**下多为空；非空多见于历史任务或曾显式开启报告内策略 LLM 的落盘。 + + ``report_matrix_group_evidence_md``：按所选矩阵细类从 ``competitor_analysis.md`` 抽取的第五～第八章大模型小节摘录（见 + ``reporting.report_matrix_group_evidence.load_report_matrix_group_evidence_markdown``）；用于与收窄后的 ``structured_brief`` 一并支撑策略叙事。 + """ + _payload, user, _tier = resolve_strategy_draft_llm_input_snapshot( + job_id=job_id, + keyword=keyword, + brief=brief, + business_notes=business_notes, + generated_at_iso=generated_at_iso, + strategy_decisions=strategy_decisions, + report_strategy_excerpt=report_strategy_excerpt, + report_matrix_group_evidence_md=report_matrix_group_evidence_md, + report_config=report_config, + ) + return call_llm(STRATEGY_SYSTEM, user) + + +STRATEGY_OPPORTUNITIES_SYSTEM = ( + STRATEGY_DATA_RULES + + """ + +你是 B 端市场与增长顾问。输入 JSON 含 ``keyword``、``competitor_brief``（与本任务规则报告同源的结构化摘要，可能经裁剪，并含 ``matrix_overview_for_llm``），以及可选 ``prior_chapter_llm_narratives``（本报告 **第五至第八章** 已生成的大模型归纳节选，与正文**同源**）。 + +请输出 **Markdown 正文**（不要用 ``` 围栏包裹），将**直接嵌入**宿主文档中**已存在章节标题之下**的小节，读者已知当前处于「策略与机会」相关章节。 + +**（与独立下载策略稿的关系）**：独立策略稿使用「摘要→一～十→附录」的六主轴结构；本节**不是**完整策略稿，仅输出下列 ``####`` 主题块，避免与宿主「九、策略与机会提示」等**已存在标题**字面重复。 + +**全局禁止编造**见上文 `STRATEGY_DATA_RULES` 段首；**本节每个 ``####`` 块**均须遵守（含不得虚构用户引语、未在 ``competitor_brief`` 出现的品牌名与数字）。 + +**与前文分析严格对齐（硬性，优先于自由发挥）**： +- **定性主题**（各细类讨论焦点、正负向体验、场景与关注词归纳、配料/卖点叙事、促销形态描述等）须与 ``prior_chapter_llm_narratives`` 中已出现的表述**方向一致**，**禁止**另写一套与节选**明显矛盾**的品类判断、品牌举例或用户痛点主题。 +- **定量与可核验事实**（价带分位数、店铺/品牌占比、条数、评论统计字段等）**以** ``competitor_brief`` **为准**；若节选与 brief 数字冲突，**采纳 brief**，且勿复述与数字冲突的节选句。 +- 若某键未出现在 ``prior_chapter_llm_narratives`` 或内容为空，则该维度**不得**编造与可能存在的报告其他章冲突的细节；仅依据 ``competitor_brief`` 或明确写「输入中未体现」。 +- **转化与体验**小节：正负向体验线索须**优先呼应** **第八章第二节 侧**节选（``sec8_3_comment_focus_summaries`` 或 ``sec8_3_text_mining_probe``，视何者存在；内部键名仍沿用 ``sec8_3_*``）；**禁止**将节选未提及的具体抱怨/品类问题写成**主要结论**；可写「假设：待结合业务验证」。 + +**标题与措辞（硬性）**： +- **禁止**在正文开头或任何位置重复宿主已有章名/小节名，包括但不限于：「第9章」「九、」「策略与机会提示」「策略与机会建议」「策略与机会」等（勿与报告固定章节标题撞车）；**不要**自造 ``##`` 一级标题； +- 小节标题**仅允许**使用业务主题式 ``####``（如下所列），从第一句起就进入实质内容。 + +**必须遵守**： +- **数字与事实**：价格分位数、集中度份额、条数、占比等**只能**来自 ``competitor_brief`` 中已有字段；**禁止编造**未出现的品牌销量、具体 GMV、未给出的到手价； +- **店铺类型占比（硬性）**：**禁止**编造「京东自营 SKU 占比」「自营款数占比超 X%」等表述，除非 ``competitor_brief`` 中 ``concentration.shops_from_list`` / ``list_shop_mix_top`` 等字段**已出现**对应店铺名与计数；若写第一大店铺份额，须与 ``shops_from_list`` 一致，并区分 **列表行** 与 **去重 SKU**（``unique_sku_basis``），**禁止**写成全渠道市占或模糊「SKU 占比」。 +- **语气**：分节给出**可操作的假设性建议**（定价区间思路、应对齐的差异化观测点、应规避的风险、促销与机制设计线索、转化与详情页/评价侧改进方向），每条建议用「假设：」「待验证：」等标明不确定性； +- **结构**：至少使用 ``####`` 组织以下主题（可合并子条，但须覆盖）：**定价与价带**、**差异化与应对齐的优势**、**风险与避免项**、**促销与活动机制**、**转化与体验**； +- **促销与活动机制（硬性）**：该节**必须优先依据** ``competitor_brief.price_promotion_signals``（券后/标价、价差等，若存在），并与 ``prior_chapter_llm_narratives.sec6_promo_group_summaries``（若有）**不矛盾**，给出**假设性**机制建议。**禁止**编造具体满减门槛、红包面额、补贴比例；**禁止**在输入中完全未出现任何列表侧价差或促销归纳信号时，仍写一大段具体「要做满减发红包」而无「输入中未捕获此类信号」的说明。 +- **转化与体验（硬性）**：须**同时**写清正向与负向；**禁止**使用「占比均超过 130 次」等**语义不通或混用次数/占比**的表述；数字表述须与 ``competitor_brief`` 一致。 +- **禁止**：不要写完整报告目录；不要复述「研究范围与方法」；不要使用 CR1/CR3 缩写（用「第一大……份额」「前三家合计」）；不要输出与输入矛盾的价带描述。 + +篇幅约 **900～3200 字**（数据丰富可偏长）。""" +) + + +STRATEGY_OPPORTUNITIES_USER_PREFIX = ( + "请根据以下 JSON 撰写策略归纳正文（Markdown）。" + "``competitor_brief`` 为结构化摘要；若含 ``prior_chapter_llm_narratives``，则为 第五至第八章 大模型归纳节选，须与策略正文对齐。" + "宿主报告已含「策略与机会」相关章节标题，**勿在输出中重复「九、」「策略与机会」类章名或小节名**。" + "输出前自检：不得编造 brief 与节选未出现的数字、品牌/店铺名、用户引语与活动规则；不确定须用「假设：」「待验证：」「输入未体现」。\n\n" +) + + +def _truncate_rules_draft_md(text: str, max_chars: int) -> str: + """规则策略底稿过长时截断，避免 JSON 与 completion 预算挤占输出。""" + s = (text or "").strip() + if not s: + return "" + if len(s) <= max_chars: + return s + return ( + s[: max_chars - 80].rstrip() + + "\n\n…（规则底稿已截断，请勿编造截断后内容。）\n" + ) + + +def _truncate_strategy_narrative(text: str, max_chars: int) -> str: + s = (text or "").strip() + if not s: + return "" + if len(s) <= max_chars: + return s + return ( + s[: max_chars - 80].rstrip() + + "\n\n…（前文各章归纳节选已截断；请勿编造截断后内容。）\n" + ) + + +def _strategy_prompt_fits_context(system: str, user: str) -> bool: + """若为 False，``chat_completion_text`` 会在发请求前因过长而抛错。""" + est = estimate_chat_input_tokens(system, user) + ctx = llm_context_window_size() + buf = 256 + return est < ctx - buf - 256 + + +def _strategy_completion_avail_tokens(system: str, user: str) -> int: + """ + 与 ``AI_crawler.chat_completion_text`` 中 ``avail = context_window - input_est - buf`` 一致， + 即本次调用实际可用于 **completion** 的上限（随后还会与 ``max_tokens`` 取 min）。 + 若该值过小，长文会在句中被截断（例如「转化与体验」末段不完整）。 + """ + est = estimate_chat_input_tokens(system, user) + ctx = llm_context_window_size() + buf = 256 + return ctx - est - buf + + +def _min_strategy_completion_tokens() -> int: + raw = (os.environ.get("MA_STRATEGY_MIN_COMPLETION_TOKENS") or "2048").strip() + try: + return max(256, int(raw)) + except ValueError: + return 2048 + + +def _strategy_prompt_ok_for_call(system: str, user: str, *, min_completion_tokens: int) -> bool: + return _strategy_prompt_fits_context( + system, user + ) and _strategy_completion_avail_tokens(system, user) >= min_completion_tokens + + +def generate_strategy_opportunities_llm( + brief: dict[str, Any], + *, + keyword: str, + chapter_llm_narratives: dict[str, str] | None = None, +) -> str: + """ + 基于 ``build_competitor_brief`` 全量摘要，生成策略与机会小节正文（不含章名，由宿主 Markdown 加标题）。 + + ``chapter_llm_narratives`` 为与本报告 第五至第八章 同源的大模型正文节选，键名稳定（见 runner 传入），用于与策略段严格对齐。 + """ + narr_in = { + k: v + for k, v in (chapter_llm_narratives or {}).items() + if isinstance(v, str) and v.strip() + } + sys_prompt = STRATEGY_OPPORTUNITIES_SYSTEM + + def _user_from_payload(p: dict[str, Any]) -> str: + return STRATEGY_OPPORTUNITIES_USER_PREFIX + json.dumps(p, ensure_ascii=False) + + min_comp = _min_strategy_completion_tokens() + min_comp_relaxed = max(256, min_comp // 2) + + for cap_brief, cap_narr in ( + (48_000, 2_800), + (42_000, 2_200), + (36_000, 1_700), + (30_000, 1_300), + (26_000, 950), + (22_000, 700), + (18_000, 500), + (16_000, 400), + (14_000, 320), + (12_000, 260), + (10_000, 200), + ): + compact = compact_brief_for_llm(brief, max_chars=cap_brief) + narratives = { + k: _truncate_strategy_narrative(v, cap_narr) for k, v in narr_in.items() + } + payload: dict[str, Any] = { + "keyword": keyword, + "competitor_brief": compact, + } + if narratives: + payload["prior_chapter_llm_narratives"] = narratives + user = _user_from_payload(payload) + if _strategy_prompt_ok_for_call(sys_prompt, user, min_completion_tokens=min_comp): + return call_llm(sys_prompt, user) + + for cap_brief in (40_000, 32_000, 26_000, 20_000, 16_000, 14_000, 12_000, 10_000): + compact = compact_brief_for_llm(brief, max_chars=cap_brief) + payload = {"keyword": keyword, "competitor_brief": compact} + user = _user_from_payload(payload) + if _strategy_prompt_ok_for_call(sys_prompt, user, min_completion_tokens=min_comp): + return call_llm(sys_prompt, user) + + for cap_brief in (14_000, 12_000, 10_000, 8_000): + compact = compact_brief_for_llm(brief, max_chars=cap_brief) + payload = {"keyword": keyword, "competitor_brief": compact} + user = _user_from_payload(payload) + if _strategy_prompt_ok_for_call(sys_prompt, user, min_completion_tokens=min_comp_relaxed): + return call_llm(sys_prompt, user) + + compact = compact_brief_for_llm(brief, max_chars=8_000) + payload = {"keyword": keyword, "competitor_brief": compact} + user = _user_from_payload(payload) + return call_llm(sys_prompt, user) diff --git a/backend/pipeline/llm_keyword_suggest.py b/backend/pipeline/llm/keyword_suggest.py similarity index 78% rename from backend/pipeline/llm_keyword_suggest.py rename to backend/pipeline/llm/keyword_suggest.py index c26574c..5dd697b 100644 --- a/backend/pipeline/llm_keyword_suggest.py +++ b/backend/pipeline/llm/keyword_suggest.py @@ -1,19 +1,17 @@ -"""在报告生成前：基于**全量**评价文本分块调用大模型，联想补充关注词（参与后续统计与报告）。""" +"""在报告生成前：基于评价正文调用大模型，联想**短语候选**（写入 keyword_suggest_llm.json；不再合并进预设词表）。""" from __future__ import annotations import json import re -import sys -from pathlib import Path from typing import Any -from django.conf import settings +from .llm_client import call_llm MAX_CHUNK_CHARS = 24_000 MAX_CHUNKS = 12 _CHUNK_SYSTEM = """你是电商评价挖掘助手。输入 JSON 含 keyword、excerpt_index、excerpts（一段用户评价正文合集）。 -任务：从 excerpts 中抽取值得纳入「关注词/卖点监测」的**中文短语**（2～12 字为主，可为词组）。 +任务：从 excerpts 中抽取**可人工选用的监测短语**（卖点、体验、规格等，**非**系统预设词表；2～12 字为主，可为词组）。 硬性规则： - 仅输出一段 JSON：{"phrases": ["短语1", ...]}，短语共 6～20 条。 @@ -22,25 +20,6 @@ _CHUNK_SYSTEM = """你是电商评价挖掘助手。输入 JSON 含 keyword、ex - 不要输出 JSON 以外的文字。""" -def _ensure_ai_crawler_path() -> None: - root = Path(settings.CRAWLER_JD_ROOT).resolve() - if not root.is_dir(): - raise FileNotFoundError(f"爬虫副本目录不存在: {root}") - rs = str(root) - if rs not in sys.path: - sys.path.insert(0, rs) - - -def _call_llm(system_prompt: str, user_prompt: str) -> str: - _ensure_ai_crawler_path() - import AI_crawler as ac # noqa: WPS433 - - return ac.chat_completion_text( - system_prompt=system_prompt, - user_prompt=user_prompt, - ) - - def _chunk_comment_texts(texts: list[str]) -> list[str]: """将全量评价划为若干段，控制单段字符量与最大段数。""" parts: list[str] = [] @@ -102,7 +81,6 @@ def suggest_focus_keywords_from_all_comments( if not all_comment_texts: return { "suggested_focus_keywords": [], - "suggested_scenario_hints": [], "rationale": "无评价正文可分析。", "chunks_processed": 0, "total_comment_texts": 0, @@ -123,7 +101,7 @@ def suggest_focus_keywords_from_all_comments( "excerpt_index": i + 1, "excerpts": ch, } - raw = _call_llm(_CHUNK_SYSTEM, json.dumps(payload, ensure_ascii=False)) + raw = call_llm(_CHUNK_SYSTEM, json.dumps(payload, ensure_ascii=False)) collected.extend(_parse_phrases_object(raw)) seen: set[str] = set() @@ -140,10 +118,9 @@ def suggest_focus_keywords_from_all_comments( out_kw = merged[:22] return { "suggested_focus_keywords": out_kw, - "suggested_scenario_hints": [], "rationale": ( f"基于全量 {len(all_comment_texts)} 条评价文本，分 {len(chunks)} 段调用模型抽取短语并去重；" - f"已排除与当前关注词统计表完全相同的词。" + "报告主文不以子串词表统计为主指标，本结果仅供业务人工参考。" ), "chunks_processed": len(chunks), "total_comment_texts": len(all_comment_texts), diff --git a/backend/pipeline/llm/llm_client.py b/backend/pipeline/llm/llm_client.py new file mode 100644 index 0000000..dbbeae6 --- /dev/null +++ b/backend/pipeline/llm/llm_client.py @@ -0,0 +1,30 @@ +"""竞品报告 LLM 调用：经 `providers` 工厂选择后端，并统一对输出做去围栏等归一化。""" +from __future__ import annotations + +from .providers.factory import get_text_llm +from .providers.shared.output_normalize import strip_outer_markdown_fence + + +def call_llm( + system_prompt: str, + user_prompt: str, + *, + temperature: float | None = None, +) -> str: + raw = get_text_llm().complete_text( + system_prompt, + user_prompt, + temperature=temperature, + ) + return strip_outer_markdown_fence(raw) + + +def estimate_chat_input_tokens(system_prompt: str, user_prompt: str) -> int: + """与当前所选文本后端的预检一致；默认与 ``AI_crawler`` 的保守估算同口径。""" + return get_text_llm().estimate_input_tokens(system_prompt, user_prompt) + + +def llm_context_window_size() -> int: + """与当前所选后端的上下文上限一致；默认与 ``AI_crawler.chat_completion_text`` 使用的环境变量一致。""" + return get_text_llm().context_window_tokens() + diff --git a/backend/pipeline/llm/providers/__init__.py b/backend/pipeline/llm/providers/__init__.py new file mode 100644 index 0000000..bac9c2c --- /dev/null +++ b/backend/pipeline/llm/providers/__init__.py @@ -0,0 +1,22 @@ +"""文本大模型调用的协议、适配器与工厂（与具体提示词/业务生成逻辑解耦）。""" + +from __future__ import annotations + +from .adapters import ( + CrawlerOpenAiCompatibleTextLlm, + DeepSeekTextLlm, + KimiMoonshotTextLlm, + OpenAiOfficialChatGptTextLlm, +) +from .factory import get_text_llm, reset_text_llm_client_for_tests +from .protocol import TextLlmClient + +__all__ = [ + "CrawlerOpenAiCompatibleTextLlm", + "DeepSeekTextLlm", + "KimiMoonshotTextLlm", + "OpenAiOfficialChatGptTextLlm", + "TextLlmClient", + "get_text_llm", + "reset_text_llm_client_for_tests", +] diff --git a/backend/pipeline/llm/providers/adapters/__init__.py b/backend/pipeline/llm/providers/adapters/__init__.py new file mode 100644 index 0000000..faa29d4 --- /dev/null +++ b/backend/pipeline/llm/providers/adapters/__init__.py @@ -0,0 +1,14 @@ +"""具体大模型通道实现：经统一协议暴露给 `factory`。""" +from __future__ import annotations + +from .crawler_openai_compatible import CrawlerOpenAiCompatibleTextLlm +from .deepseek_text import DeepSeekTextLlm +from .kimi_moonshot_text import KimiMoonshotTextLlm +from .openai_official_chatgpt import OpenAiOfficialChatGptTextLlm + +__all__ = [ + "CrawlerOpenAiCompatibleTextLlm", + "DeepSeekTextLlm", + "KimiMoonshotTextLlm", + "OpenAiOfficialChatGptTextLlm", +] diff --git a/backend/pipeline/llm/providers/adapters/crawler_openai_compatible.py b/backend/pipeline/llm/providers/adapters/crawler_openai_compatible.py new file mode 100644 index 0000000..2851236 --- /dev/null +++ b/backend/pipeline/llm/providers/adapters/crawler_openai_compatible.py @@ -0,0 +1,49 @@ +""" +经 ``pipeline.openai_gateway.text_chat.chat_completion_text`` 访问 OpenAI 兼容网关（与配料识别等共用环境变量，不再 import 爬虫目录）。 +""" +from __future__ import annotations + +import os + +from pipeline.openai_gateway import chat_completion_text + +from ..shared.token_heuristics import estimate_crawler_style_input_tokens + + +def _llm_context_window_size_from_env() -> int: + raw = ( + os.environ.get("LLM_CONTEXT_WINDOW") + or os.environ.get("OPENAI_CONTEXT_WINDOW") + or "32768" + ).strip() + try: + return max(4096, int(raw)) + except ValueError: + return 32768 + + +class CrawlerOpenAiCompatibleTextLlm: + """ + 文本任务默认后端：与历史 ``AI_crawler.chat_completion_text`` 行为一致，实现位于 ``pipeline.openai_gateway``。 + """ + + def complete_text( + self, + system_prompt: str, + user_prompt: str, + *, + temperature: float | None = None, + ) -> str: + kwargs: dict[str, object] = { + "system_prompt": system_prompt, + "user_prompt": user_prompt, + } + if temperature is not None: + kwargs["temperature"] = float(temperature) + return chat_completion_text(**kwargs) + + def estimate_input_tokens(self, system_prompt: str, user_prompt: str) -> int: + return estimate_crawler_style_input_tokens(system_prompt, user_prompt) + + def context_window_tokens(self) -> int: + return _llm_context_window_size_from_env() diff --git a/backend/pipeline/llm/providers/adapters/deepseek_text.py b/backend/pipeline/llm/providers/adapters/deepseek_text.py new file mode 100644 index 0000000..c7688de --- /dev/null +++ b/backend/pipeline/llm/providers/adapters/deepseek_text.py @@ -0,0 +1,172 @@ +""" +DeepSeek 官方 OpenAI 兼容 `chat/completions`（`https://api.deepseek.com`），**仅用于纯文本**（`call_llm` / 报告 / 策略）。 + +与 `OPENAI_*` / 配料多模 分离，独立 `DEEPSEEK_*` 凭据。 + +启用：`MA_LLM_TEXT_PROVIDER=deepseek`（或 `deep_seek`）。 + +**思考模式**（与官方「Thinking Mode」一致）：默认可通过 `DEEPSEEK_THINKING=0` 关闭。开启时在请求中携带 +`thinking.type=enabled` 与 `reasoning_effort`；不发送 `temperature`（官方在思考模式下忽略采样参数）。 +未指定 `DEEPSEEK_TEXT_MODEL` 时，开启思考默认用 `deepseek-v4-pro`，关闭时默认 `deepseek-chat`。 +详见 https://api-docs.deepseek.com/guides/thinking_mode +""" +from __future__ import annotations + +import os +from typing import Any + +import requests + +from pipeline.openai_gateway.chat_content import normalize_message_content +from pipeline.openai_gateway.estimate import ( + estimate_chat_input_tokens as estimate_crawler_style_input_tokens, +) + +_DEFAULT_BASE = "https://api.deepseek.com/v1" +_DEFAULT_MODEL_NO_THINK = "deepseek-chat" +_DEFAULT_MODEL_THINK = "deepseek-v4-pro" +# 常见 64k 级；若用长上下文/官方调整上限可改 DEEPSEEK_CONTEXT_WINDOW +_DEFAULT_CTX = 64_000 + +_BUF = 256 +_WANT_MAX = 8192 + + +def _read_timeout() -> tuple[float, float]: + read = 600 + raw = ( + os.environ.get("DEEPSEEK_TIMEOUT") + or os.environ.get("LLM_CHAT_TIMEOUT") + or os.environ.get("OPENAI_TIMEOUT") + or "" + ).strip() + if raw: + try: + read = max(60, int(raw)) + except ValueError: + pass + conn = 30.0 + raw_c = (os.environ.get("LLM_CHAT_CONNECT_TIMEOUT") or "").strip() + if raw_c: + try: + conn = max(5.0, float(raw_c)) + except ValueError: + pass + return (conn, float(read)) + + +def _thinking_enabled() -> bool: + v = (os.environ.get("DEEPSEEK_THINKING") or "1").strip().lower() + if v in ("0", "false", "off", "no", "disabled"): + return False + return True + + +def _reasoning_effort() -> str: + raw = (os.environ.get("DEEPSEEK_REASONING_EFFORT") or "high").strip().lower() + if raw in ("max", "high", "low", "medium", "xhigh"): + if raw in ("low", "medium"): + return "high" + if raw == "xhigh": + return "max" + return raw + return "high" + + +def _default_model() -> str: + return _DEFAULT_MODEL_THINK if _thinking_enabled() else _DEFAULT_MODEL_NO_THINK + + +def _resolve_deepseek_credentials() -> tuple[str, str, str]: + key = (os.environ.get("DEEPSEEK_API_KEY") or "").strip() + if not key: + raise ValueError( + "使用 deepseek 文本适配器需设置 DEEPSEEK_API_KEY，" + "与配料/视觉所用 OPENAI_API_KEY 分开配置。" + ) + base = (os.environ.get("DEEPSEEK_BASE_URL") or _DEFAULT_BASE).strip().rstrip("/") + model = (os.environ.get("DEEPSEEK_TEXT_MODEL") or os.environ.get("DEEPSEEK_MODEL") or "").strip() + if not model: + model = _default_model() + return key, base, model + + +def _context_window() -> int: + raw = (os.environ.get("DEEPSEEK_CONTEXT_WINDOW") or str(_DEFAULT_CTX)).strip() + try: + return max(4096, int(raw)) + except ValueError: + return _DEFAULT_CTX + + +def _default_temperature() -> float: + return 0.2 + + +class DeepSeekTextLlm: + """DeepSeek `chat/completions`；与 `KimiMoonshotTextLlm` 同形（max_tokens 预检）。""" + + def complete_text( + self, + system_prompt: str, + user_prompt: str, + *, + temperature: float | None = None, + ) -> str: + api_key, base, model = _resolve_deepseek_credentials() + think = _thinking_enabled() + body: dict[str, Any] = { + "model": model, + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_prompt}, + ], + "max_tokens": _WANT_MAX, + } + if think: + # 与 OpenAI 官方 Python SDK 合并 extra_body 后一致：思考模式不依赖 temperature + body["thinking"] = {"type": "enabled"} + body["reasoning_effort"] = _reasoning_effort() + else: + body["temperature"] = _default_temperature() if temperature is None else float(temperature) + est = estimate_crawler_style_input_tokens(system_prompt, user_prompt) + context_window = _context_window() + if est >= context_window - _BUF - 256: + raise ValueError( + f"提示词过长（估算输入约 {est} tokens，DEEPSEEK_CONTEXT_WINDOW={context_window}），" + "请缩小输入或调大 DEEPSEEK_TEXT_MODEL / DEEPSEEK_CONTEXT_WINDOW。" + ) + avail = context_window - est - _BUF + want = int(body.get("max_tokens") or _WANT_MAX) + body["max_tokens"] = max(256, min(want, max(avail, 256))) + r = requests.post( + f"{base}/chat/completions", + headers={ + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + }, + json=body, + timeout=_read_timeout(), + ) + try: + r.raise_for_status() + except requests.HTTPError as e: + snippet = "" + if e.response is not None: + snippet = (e.response.text or "")[:1200].replace("\r\n", "\n").replace("\n", " ") + if snippet: + raise requests.HTTPError( + f"{e!s} | body: {snippet}", + response=e.response, + request=e.request, + ) from e + raise + data = r.json() + msg = (data.get("choices") or [{}])[0].get("message") or {} + return normalize_message_content(msg.get("content")) + + def estimate_input_tokens(self, system_prompt: str, user_prompt: str) -> int: + return estimate_crawler_style_input_tokens(system_prompt, user_prompt) + + def context_window_tokens(self) -> int: + return _context_window() diff --git a/backend/pipeline/llm/providers/adapters/kimi_moonshot_text.py b/backend/pipeline/llm/providers/adapters/kimi_moonshot_text.py new file mode 100644 index 0000000..c583c4a --- /dev/null +++ b/backend/pipeline/llm/providers/adapters/kimi_moonshot_text.py @@ -0,0 +1,146 @@ +""" +月之暗面 Kimi（Moonshot）OpenAI 兼容 `chat/completions`，**仅用于纯文本**（`call_llm` / 报告 / 策略）。 + +与 `OPENAI_*` / `LLM_*` 分离，避免与自建网关（配料多模态等）混用同一套 Key。 + +启用：`MA_LLM_TEXT_PROVIDER=kimi`（或 `moonshot` / `kimi_moonshot`）。 + +环境变量：`KIMI_API_KEY`（必填）、`KIMI_BASE_URL`（默认 Moonshot 官方 v1）、`KIMI_TEXT_MODEL`、 +`KIMI_CONTEXT_WINDOW`、`KIMI_TIMEOUT` 等；见 `.env.example`。 +""" +from __future__ import annotations + +import os +from typing import Any + +import requests + +from pipeline.openai_gateway.chat_content import normalize_message_content +from pipeline.openai_gateway.estimate import ( + estimate_chat_input_tokens as estimate_crawler_style_input_tokens, +) + +_DEFAULT_BASE = "https://api.moonshot.cn/v1" +_DEFAULT_MODEL = "moonshot-v1-8k" +# 与常见 8k 窗口一致；若使用 moonshot-v1-128k 等请调大 KIMI_CONTEXT_WINDOW +_DEFAULT_CTX = 8192 + +_BUF = 256 +_WANT_MAX = 8192 + + +def _read_timeout() -> tuple[float, float]: + read = 600 + raw = ( + os.environ.get("KIMI_TIMEOUT") + or os.environ.get("LLM_CHAT_TIMEOUT") + or os.environ.get("OPENAI_TIMEOUT") + or "" + ).strip() + if raw: + try: + read = max(60, int(raw)) + except ValueError: + pass + conn = 30.0 + raw_c = (os.environ.get("LLM_CHAT_CONNECT_TIMEOUT") or "").strip() + if raw_c: + try: + conn = max(5.0, float(raw_c)) + except ValueError: + pass + return (conn, float(read)) + + +def _resolve_kimi_credentials() -> tuple[str, str, str]: + key = ( + (os.environ.get("KIMI_API_KEY") or os.environ.get("MOONSHOT_API_KEY") or "").strip() + ) + if not key: + raise ValueError( + "使用 kimi 文本适配器需设置 KIMI_API_KEY（或 MOONSHOT_API_KEY），" + "与配料/视觉所用 OPENAI_API_KEY 分开配置。" + ) + base = (os.environ.get("KIMI_BASE_URL") or _DEFAULT_BASE).strip().rstrip("/") + model = ( + os.environ.get("KIMI_TEXT_MODEL") + or os.environ.get("KIMI_MODEL") + or os.environ.get("MOONSHOT_MODEL") + or _DEFAULT_MODEL + ).strip() + return key, base, model + + +def _context_window() -> int: + raw = (os.environ.get("KIMI_CONTEXT_WINDOW") or str(_DEFAULT_CTX)).strip() + try: + return max(4096, int(raw)) + except ValueError: + return _DEFAULT_CTX + + +def _default_temperature() -> float: + return 0.2 + + +class KimiMoonshotTextLlm: + """Kimi OpenAI 兼容文本补全；行为与 `OpenAiOfficialChatGptTextLlm` 同形（max_tokens 预检）。""" + + def complete_text( + self, + system_prompt: str, + user_prompt: str, + *, + temperature: float | None = None, + ) -> str: + api_key, base, model = _resolve_kimi_credentials() + body: dict[str, Any] = { + "model": model, + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_prompt}, + ], + "temperature": _default_temperature() if temperature is None else float(temperature), + "max_tokens": _WANT_MAX, + } + est = estimate_crawler_style_input_tokens(system_prompt, user_prompt) + context_window = _context_window() + if est >= context_window - _BUF - 256: + raise ValueError( + f"提示词过长（估算输入约 {est} tokens，KIMI_CONTEXT_WINDOW={context_window}），" + "请缩小输入或换更大上下文的 KIMI_TEXT_MODEL / KIMI_CONTEXT_WINDOW。" + ) + avail = context_window - est - _BUF + want = int(body.get("max_tokens") or _WANT_MAX) + body["max_tokens"] = max(256, min(want, max(avail, 256))) + r = requests.post( + f"{base}/chat/completions", + headers={ + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + }, + json=body, + timeout=_read_timeout(), + ) + try: + r.raise_for_status() + except requests.HTTPError as e: + snippet = "" + if e.response is not None: + snippet = (e.response.text or "")[:1200].replace("\r\n", "\n").replace("\n", " ") + if snippet: + raise requests.HTTPError( + f"{e!s} | body: {snippet}", + response=e.response, + request=e.request, + ) from e + raise + data = r.json() + msg = (data.get("choices") or [{}])[0].get("message") or {} + return normalize_message_content(msg.get("content")) + + def estimate_input_tokens(self, system_prompt: str, user_prompt: str) -> int: + return estimate_crawler_style_input_tokens(system_prompt, user_prompt) + + def context_window_tokens(self) -> int: + return _context_window() diff --git a/backend/pipeline/llm/providers/adapters/openai_official_chatgpt.py b/backend/pipeline/llm/providers/adapters/openai_official_chatgpt.py new file mode 100644 index 0000000..47cea39 --- /dev/null +++ b/backend/pipeline/llm/providers/adapters/openai_official_chatgpt.py @@ -0,0 +1,135 @@ +""" +OpenAI 官方 `https://api.openai.com`（ChatGPT 系列）`chat/completions` 直连接口。 + +**凭据与网关与爬虫副本中的自建网关独立**，避免与 `OPENAI_BASE_URL` 指向的兼容网关共用时互相串环境。 +通过 `MA_LLM_TEXT_PROVIDER=openai_official`（或 `openai_chatgpt` / `chatgpt`）启用。 +""" +from __future__ import annotations + +import os +from typing import Any + +import requests + +from pipeline.openai_gateway.chat_content import normalize_message_content +from pipeline.openai_gateway.estimate import ( + estimate_chat_input_tokens as estimate_crawler_style_input_tokens, +) + +_DEFAULT_BASE = "https://api.openai.com/v1" +_DEFAULT_MODEL = "gpt-4o-mini" +# gpt-4o / 4.1 等常见上限；可按模型在 .env 中覆盖 +_DEFAULT_CTX = 128_000 + +_BUF = 256 +_WANT_MAX = 8192 + + +def _read_timeout() -> tuple[float, float]: + read = 600 + raw = (os.environ.get("OPENAI_OFFICIAL_TIMEOUT") or os.environ.get("LLM_CHAT_TIMEOUT") or os.environ.get("OPENAI_TIMEOUT") or "").strip() + if raw: + try: + read = max(60, int(raw)) + except ValueError: + pass + conn = 30.0 + raw_c = (os.environ.get("LLM_CHAT_CONNECT_TIMEOUT") or "").strip() + if raw_c: + try: + conn = max(5.0, float(raw_c)) + except ValueError: + pass + return (conn, float(read)) + + +def _resolve_credentials() -> tuple[str, str, str]: + key = (os.environ.get("OPENAI_OFFICIAL_API_KEY") or "").strip() + if not key: + msg = "使用 openai_official 适配器需设置环境变量 OPENAI_OFFICIAL_API_KEY（与自建网关/爬虫副本的 key 可分开）。" + raise ValueError(msg) + base = (os.environ.get("OPENAI_OFFICIAL_BASE_URL") or _DEFAULT_BASE).strip().rstrip("/") + model = ( + os.environ.get("OPENAI_OFFICIAL_TEXT_MODEL") + or os.environ.get("OPENAI_OFFICIAL_MODEL") + or _DEFAULT_MODEL + ).strip() + return key, base, model + + +def _context_window() -> int: + raw = (os.environ.get("OPENAI_OFFICIAL_CONTEXT_WINDOW") or str(_DEFAULT_CTX)).strip() + try: + return max(4096, int(raw)) + except ValueError: + return _DEFAULT_CTX + + +def _default_temperature() -> float: + return 0.2 + + +class OpenAiOfficialChatGptTextLlm: + """ + 直连 OpenAI 官方「Chat Completions」；请求体与 `AI_crawler.chat_completion_text` 同形， + 并在本地做与爬虫网关一致的 `max_tokens` 收紧，减少 400。 + """ + + def complete_text( + self, + system_prompt: str, + user_prompt: str, + *, + temperature: float | None = None, + ) -> str: + api_key, base, model = _resolve_credentials() + body: dict[str, Any] = { + "model": model, + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_prompt}, + ], + "temperature": _default_temperature() if temperature is None else float(temperature), + "max_tokens": _WANT_MAX, + } + est = estimate_crawler_style_input_tokens(system_prompt, user_prompt) + context_window = _context_window() + if est >= context_window - _BUF - 256: + raise ValueError( + f"提示词过长（估算输入约 {est} tokens，OPENAI_OFFICIAL_CONTEXT_WINDOW={context_window}），" + "请缩小输入或调大 OPENAI_OFFICIAL_CONTEXT_WINDOW。" + ) + avail = context_window - est - _BUF + want = int(body.get("max_tokens") or _WANT_MAX) + body["max_tokens"] = max(256, min(want, max(avail, 256))) + r = requests.post( + f"{base}/chat/completions", + headers={ + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + }, + json=body, + timeout=_read_timeout(), + ) + try: + r.raise_for_status() + except requests.HTTPError as e: + snippet = "" + if e.response is not None: + snippet = (e.response.text or "")[:1200].replace("\r\n", "\n").replace("\n", " ") + if snippet: + raise requests.HTTPError( + f"{e!s} | body: {snippet}", + response=e.response, + request=e.request, + ) from e + raise + data = r.json() + msg = (data.get("choices") or [{}])[0].get("message") or {} + return normalize_message_content(msg.get("content")) + + def estimate_input_tokens(self, system_prompt: str, user_prompt: str) -> int: + return estimate_crawler_style_input_tokens(system_prompt, user_prompt) + + def context_window_tokens(self) -> int: + return _context_window() diff --git a/backend/pipeline/llm/providers/factory.py b/backend/pipeline/llm/providers/factory.py new file mode 100644 index 0000000..bd14f19 --- /dev/null +++ b/backend/pipeline/llm/providers/factory.py @@ -0,0 +1,61 @@ +""" +根据 `MA_LLM_TEXT_PROVIDER` 选择文本大模型实现；未设置时与历史行为一致（经 `openai_gateway` 与自建兼容网关）。 +""" +from __future__ import annotations + +import os + +from .adapters.crawler_openai_compatible import CrawlerOpenAiCompatibleTextLlm +from .adapters.deepseek_text import DeepSeekTextLlm +from .adapters.kimi_moonshot_text import KimiMoonshotTextLlm +from .adapters.openai_official_chatgpt import OpenAiOfficialChatGptTextLlm +from .protocol import TextLlmClient + +# 模块级单例：避免重复构造；测试可用 `reset_text_llm_client_for_tests` 切换实现。 +_client: TextLlmClient | None = None + +# 与历史默认行为一致 +_DEFAULT_ID = "crawler_openai_compatible" +_ENV_KEY = "MA_LLM_TEXT_PROVIDER" + + +def _provider_id() -> str: + raw = (os.environ.get(_ENV_KEY) or _DEFAULT_ID).strip().lower() + return raw or _DEFAULT_ID + + +def _build_client(pid: str) -> TextLlmClient: + if pid in ( + "crawler_openai_compatible", + "crawler", + "default", + "openai_compatible", + ): + return CrawlerOpenAiCompatibleTextLlm() + if pid in ("openai_official", "openai_chatgpt", "chatgpt"): + return OpenAiOfficialChatGptTextLlm() + if pid in ("kimi", "moonshot", "kimi_moonshot", "moonshot_kimi"): + return KimiMoonshotTextLlm() + if pid in ("deepseek", "deep_seek"): + return DeepSeekTextLlm() + known = ( + "crawler_openai_compatible, crawler, default, openai_compatible, " + "openai_official, openai_chatgpt, chatgpt, " + "kimi, moonshot, kimi_moonshot, deepseek, deep_seek" + ) + raise ValueError( + f"不支持的 {_ENV_KEY}={pid!r}；已知取值：{known}。", + ) + + +def get_text_llm() -> TextLlmClient: + global _client + if _client is None: + _client = _build_client(_provider_id()) + return _client + + +def reset_text_llm_client_for_tests() -> None: + """供 pytest/集成测试在修改环境变量后清空缓存的客户端。""" + global _client + _client = None diff --git a/backend/pipeline/llm/providers/protocol.py b/backend/pipeline/llm/providers/protocol.py new file mode 100644 index 0000000..3060908 --- /dev/null +++ b/backend/pipeline/llm/providers/protocol.py @@ -0,0 +1,22 @@ +"""文本 LLM 客户端协议：业务侧只依赖本接口，具体网关由适配器 + 工厂选择。""" +from __future__ import annotations + +from typing import Protocol, runtime_checkable + + +@runtime_checkable +class TextLlmClient(Protocol): + def complete_text( + self, + system_prompt: str, + user_prompt: str, + *, + temperature: float | None = None, + ) -> str: + """一次 system + user 的纯文本补全，返回助理正文（无通用后处理，由 `llm_client.call_llm` 统一去围栏等）。""" + + def estimate_input_tokens(self, system_prompt: str, user_prompt: str) -> int: + """与当次后端的 `max_tokens` 预检/截断策略一致的输入侧 token 保守估算。""" + + def context_window_tokens(self) -> int: + """当前配置下的上下文 token 上限（与预检、策略模块档位一致）。""" diff --git a/backend/pipeline/llm/providers/shared/__init__.py b/backend/pipeline/llm/providers/shared/__init__.py new file mode 100644 index 0000000..9e132ea --- /dev/null +++ b/backend/pipeline/llm/providers/shared/__init__.py @@ -0,0 +1,12 @@ +"""与具体后端无关的轻量工具：去围栏、token 启发式、OpenAI 风格 message 正文解析。""" +from __future__ import annotations + +from .openai_message_content import normalize_message_content +from .output_normalize import strip_outer_markdown_fence +from .token_heuristics import estimate_crawler_style_input_tokens + +__all__ = [ + "estimate_crawler_style_input_tokens", + "normalize_message_content", + "strip_outer_markdown_fence", +] diff --git a/backend/pipeline/llm/providers/shared/openai_message_content.py b/backend/pipeline/llm/providers/shared/openai_message_content.py new file mode 100644 index 0000000..ad218a4 --- /dev/null +++ b/backend/pipeline/llm/providers/shared/openai_message_content.py @@ -0,0 +1,8 @@ +""" +解析 OpenAI `message.content`；实现位于 ``pipeline.openai_gateway.chat_content``，仅重导以保持与旧 import 路径兼容。 +""" +from __future__ import annotations + +from pipeline.openai_gateway.chat_content import normalize_message_content + +__all__ = ["normalize_message_content"] diff --git a/backend/pipeline/llm/providers/shared/output_normalize.py b/backend/pipeline/llm/providers/shared/output_normalize.py new file mode 100644 index 0000000..57931e3 --- /dev/null +++ b/backend/pipeline/llm/providers/shared/output_normalize.py @@ -0,0 +1,6 @@ +"""对模型输出去围栏等；与 ``openai_gateway.text_chat.strip_outer_markdown_fence`` 同义。""" +from __future__ import annotations + +from pipeline.openai_gateway.text_chat import strip_outer_markdown_fence + +__all__ = ["strip_outer_markdown_fence"] diff --git a/backend/pipeline/llm/providers/shared/token_heuristics.py b/backend/pipeline/llm/providers/shared/token_heuristics.py new file mode 100644 index 0000000..2ac539c --- /dev/null +++ b/backend/pipeline/llm/providers/shared/token_heuristics.py @@ -0,0 +1,8 @@ +""" +与 ``pipeline.openai_gateway.estimate.estimate_chat_input_tokens`` 同口径的保守估算。 +""" +from __future__ import annotations + +from pipeline.openai_gateway.estimate import estimate_chat_input_tokens as estimate_crawler_style_input_tokens + +__all__ = ["estimate_crawler_style_input_tokens"] diff --git a/backend/pipeline/llm_generate.py b/backend/pipeline/llm_generate.py deleted file mode 100644 index 09093f4..0000000 --- a/backend/pipeline/llm_generate.py +++ /dev/null @@ -1,150 +0,0 @@ -""" -竞品报告 / 策略稿的**大模型生成**：通过 ``crawler_copy/jd_pc_search/AI_crawler`` 的 -``chat_completion_text`` 调用，与配料识别共用网关与密钥。 -""" -from __future__ import annotations - -import json -import sys -from pathlib import Path -from typing import Any - -from django.conf import settings - -from .brief_compact import compact_brief_for_llm -from .strategy_draft import build_strategy_draft_markdown - - -def _ensure_ai_crawler_path() -> None: - root = Path(settings.CRAWLER_JD_ROOT).resolve() - if not root.is_dir(): - raise FileNotFoundError(f"爬虫副本目录不存在: {root}") - rs = str(root) - if rs not in sys.path: - sys.path.insert(0, rs) - - -def _call_llm(system_prompt: str, user_prompt: str) -> str: - _ensure_ai_crawler_path() - import AI_crawler as ac # noqa: WPS433 - - raw = ac.chat_completion_text( - system_prompt=system_prompt, - user_prompt=user_prompt, - ) - return ac.strip_outer_markdown_fence(raw) - - -REPORT_SYSTEM = """你撰写一段**短小的「速读与策略补充」**，插在完整规则报告**之前**供读者扫读。读者为业务与产品。 - -**输入**：JSON 含 `keyword`、`competitor_brief`（可能经裁剪）、`matrix_overview_for_llm`（按细分类目的 SKU 数与品牌样本）。 -所有数字、占比、条数、品牌名、价格区间等**必须严格来自输入 JSON**，禁止编造未在输入中出现的定量结论。 - -**硬性禁止**： -- **不要**输出完整报告目录或重复「研究范围与方法」等长章结构； -- **不要**撰写 Markdown 表格版「竞品对比矩阵」或罗列 SKU 明细——**正文报告已含完整矩阵**，此处仅可概括分组级结论（细类名、SKU 数、主要品牌来自 `matrix_overview_for_llm` / brief）； -- **不要**写「matrix_by_group 已省略」「仅保留代表性品牌」等免责声明，也不要引导读者认为明细缺失； -- **不要使用** CR1、CR3 等英文缩写；集中度请用「第一大品牌份额」「前三品牌合计份额」。 - -**请输出**（仅输出正文，不要前言后语）： -- 使用 **Markdown**，控制在约 **800～1500 字**； -- 建议小节标题（二级）：**执行摘要要点**、**竞争与价盘速读**、**用户声量与关注点**、**策略提示与数据边界**； -- 若有 `comment_sentiment_lexicon`，概括正/负向粗判与局限（非深度学习）； -- 语气专业、中文；缺失项写「本摘要未提供该项」而非猜测。""" - -REPORT_USER_PREFIX = """请根据以下 JSON 撰写完整竞品分析报告（Markdown 正文）。\n\n""" - - -def generate_competitor_report_markdown_llm(brief: dict[str, Any], keyword: str) -> str: - compact = compact_brief_for_llm(brief) - payload = { - "keyword": keyword, - "competitor_brief": compact, - "matrix_overview_for_llm": compact.get("matrix_overview_for_llm") or [], - } - user = REPORT_USER_PREFIX + json.dumps(payload, ensure_ascii=False) - return _call_llm(REPORT_SYSTEM, user) - - -SENTIMENT_LLM_SYSTEM = """你是电商/食品类用户研究助手。输入 JSON 含： -- ``comment_sentiment_lexicon``：关键词规则下的条数与短语命中（粗判，非深度学习）； -- ``sample_reviews_*``：按同一规则从评价中抽样的短文（已截断），**仅可依据这些原文与 lexicon 数字归纳**。 - -**硬性要求**： -- **仅输出 Markdown 正文**（不要用 ``` 围栏包裹全文）； -- **不要编造**样本中未出现的具体事实、品牌、价格、医学功效； -- 条数、占比等**定量表述须与** ``comment_sentiment_lexicon`` **一致**，勿与样本矛盾。 - -**建议结构**（使用四级标题 ``####``）： -1. ``#### 正向要点归纳``：3～6 条要点，概括满意点（口感、甜度、包装、物流、性价比等）； -2. ``#### 负向与风险点归纳``：3～6 条要点； -3. ``#### 使用注意``：1～2 句说明样本量、抽样局限、与关键词规则可能不一致之处。 - -总字数约 **400～900 字**，简体中文，语气客观。""" - - -def generate_comment_sentiment_analysis_llm(payload: dict[str, Any]) -> str: - """基于规则分桶抽样评价 + lexicon 统计，生成 §8.2 大模型解读段落（Markdown）。""" - p = dict(payload) - raw = json.dumps(p, ensure_ascii=False) - if len(raw) > 88_000: - for k in ( - "sample_reviews_positive_biased", - "sample_reviews_negative_biased", - "sample_reviews_mixed_tone", - ): - lst = p.get(k) - if isinstance(lst, list): - p[k] = [str(x)[:140] for x in lst[:8]] - raw = json.dumps(p, ensure_ascii=False) - if len(raw) > 88_000: - raw = raw[:82_000] + "\n\n…（输入过长已截断，请勿编造截断外内容）\n" - user = "请根据以下 JSON 按系统说明输出 Markdown：\n\n" + raw - return _call_llm(SENTIMENT_LLM_SYSTEM, user) - - -STRATEGY_SYSTEM = """你是市场策略顾问，根据**结构化监测摘要**与业务侧填写的**决策字段**，把「规则底稿」润色为可读的策略 Markdown。 - -**规则**： -- 输入 JSON 含 `rules_draft_markdown`（规则引擎生成的底稿，与同任务数据一致）、`structured_brief`（摘要子集）、`strategy_decisions`、`business_notes` 等； -- **不得编造**输入中不存在的销量、占比、价格数字；若底稿与摘要中有数字，须保持一致；表述集中度时用「第一大品牌份额」等中文，**不要用** CR1、CR3 缩写； -- 若 `structured_brief` 含 `matrix_overview_for_llm` 或矩阵相关字段，策略中应**呼应**细分类目分组与竞品矩阵结论，不得无故删光矩阵相关建议； -- 可调整段落衔接、标题层级、列表与表格呈现，使更易读；可补充「建议」「待业务确认」类表述，但不虚构竞品名称或数据； -- **仅输出** Markdown 正文（不要 ``` 围栏包裹全文）。""" - -STRATEGY_USER_PREFIX = """请基于以下 JSON 输出最终策略稿（Markdown）。\n\n""" - - -def generate_strategy_draft_markdown_llm( - *, - job_id: int, - keyword: str, - brief: dict[str, Any], - business_notes: str, - generated_at_iso: str, - strategy_decisions: dict[str, Any], -) -> str: - rules_md = build_strategy_draft_markdown( - job_id=job_id, - keyword=keyword, - brief=brief, - business_notes=business_notes, - generated_at_iso=generated_at_iso, - strategy_decisions=strategy_decisions, - ) - compact = compact_brief_for_llm(brief, max_chars=80_000) - payload = { - "job_id": job_id, - "keyword": keyword, - "generated_at_iso": generated_at_iso, - "strategy_decisions": strategy_decisions, - "business_notes": business_notes, - "structured_brief": compact, - "rules_draft_markdown": rules_md, - } - raw = json.dumps(payload, ensure_ascii=False) - if len(raw) > 500_000: - payload["rules_draft_markdown"] = rules_md[:200_000] + "\n\n…（底稿过长已截断，请勿编造截断后内容）\n" - raw = json.dumps(payload, ensure_ascii=False) - user = STRATEGY_USER_PREFIX + raw - return _call_llm(STRATEGY_SYSTEM, user) diff --git a/backend/pipeline/management/commands/ingest_pipeline_dataset.py b/backend/pipeline/management/commands/ingest_pipeline_dataset.py new file mode 100644 index 0000000..52d6516 --- /dev/null +++ b/backend/pipeline/management/commands/ingest_pipeline_dataset.py @@ -0,0 +1,128 @@ +# -*- coding: utf-8 -*- +""" +将磁盘上已存在的流水线批次目录（``data/JD/pipeline_runs/...``）导入数据库： + +- ``pc_search_export.csv`` / ``detail_ware_export.csv`` / ``comments_flat.csv`` +- ``keyword_pipeline_merged.csv``（合并宽表 + JdProduct / JdProductSnapshot） + +用法（在 ``backend`` 目录下）:: + + python manage.py ingest_pipeline_dataset --run-dir pipeline_runs/20260413_104252_低GI + +或绝对路径（仍须在 ``data/JD`` 下）:: + + python manage.py ingest_pipeline_dataset --run-dir "D:/.../data/JD/pipeline_runs/xxx" + +绑定已有 ``PipelineJob``:: + + python manage.py ingest_pipeline_dataset --job-id 12 --run-dir pipeline_runs/xxx + +新建任务并入库（关键词优先读 ``run_meta.json``）:: + + python manage.py ingest_pipeline_dataset --create --run-dir pipeline_runs/xxx --keyword 低GI +""" +from __future__ import annotations + +import json +from pathlib import Path + +from django.conf import settings +from django.core.management.base import BaseCommand, CommandError + +from pipeline.ingest import FILE_MERGED_CSV, ingest_job_full, resolve_and_validate_run_dir +from pipeline.models import JobStatus, PipelineJob + + +class Command(BaseCommand): + help = "将已有 pipeline run 目录下的 CSV 导入数据库（搜索/详情/评价/合并表与商品快照）。" + + def add_arguments(self, parser) -> None: + parser.add_argument( + "--run-dir", + type=str, + required=True, + help="相对 data/JD 的子路径，或位于 data/JD 下的绝对路径", + ) + parser.add_argument( + "--job-id", + type=int, + default=None, + help="绑定到已有 PipelineJob；未给则须配合 --create", + ) + parser.add_argument( + "--create", + action="store_true", + help="新建 PipelineJob（success）并写入 run_dir 后入库", + ) + parser.add_argument( + "--keyword", + type=str, + default="", + help="与 --create 合用；默认尝试从 run_meta.json 读取 keyword", + ) + + def handle(self, *args, **options): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + raise CommandError("请在 .env 中配置 LOW_GI_PROJECT_ROOT") + + raw = str(options["run_dir"] or "").strip() + try: + run_path = resolve_and_validate_run_dir(raw) + except ValueError as e: + raise CommandError(str(e)) from e + + merged = run_path / FILE_MERGED_CSV + if not merged.is_file(): + self.stdout.write( + self.style.WARNING( + f"缺少 {FILE_MERGED_CSV}，仍将尝试导入搜索/详情/评价（合并表与快照会跳过或报错）。" + ) + ) + + job_id = options.get("job_id") + create = bool(options.get("create")) + kw_in = (options.get("keyword") or "").strip() + + if job_id and create: + raise CommandError("请只使用 --job-id 或 --create 之一") + + if create: + meta_kw = "" + meta_path = run_path / "run_meta.json" + if meta_path.is_file(): + try: + meta = json.loads(meta_path.read_text(encoding="utf-8")) + if isinstance(meta, dict): + meta_kw = str(meta.get("keyword") or "").strip() + except (json.JSONDecodeError, OSError): + pass + keyword = kw_in or meta_kw or "imported" + job = PipelineJob.objects.create( + platform="jd", + keyword=keyword[:256], + status=JobStatus.SUCCESS, + run_dir=str(run_path), + ) + self.stdout.write(self.style.NOTICE(f"已创建任务 id={job.id} keyword={job.keyword!r}")) + elif job_id: + job = PipelineJob.objects.filter(pk=job_id).first() + if not job: + raise CommandError(f"找不到 PipelineJob id={job_id}") + job.run_dir = str(run_path) + job.save(update_fields=["run_dir", "updated_at"]) + self.stdout.write(self.style.NOTICE(f"已更新任务 id={job.id} 的 run_dir")) + else: + raise CommandError("请指定 --job-id 绑定已有任务，或使用 --create 新建任务") + + try: + stats = ingest_job_full(job) + except FileNotFoundError as e: + raise CommandError(str(e)) from e + + self.stdout.write(self.style.SUCCESS(json.dumps(stats, ensure_ascii=False, indent=2))) + self.stdout.write( + self.style.NOTICE( + f"完成。前端可打开任务 {job.id}，数据集接口：" + f"/api/pipeline/jobs/{job.id}/dataset/summary/ 等。" + ) + ) diff --git a/backend/pipeline/management/commands/refresh_jd_merged_total_sales.py b/backend/pipeline/management/commands/refresh_jd_merged_total_sales.py new file mode 100644 index 0000000..c7a127c --- /dev/null +++ b/backend/pipeline/management/commands/refresh_jd_merged_total_sales.py @@ -0,0 +1,87 @@ +# -*- coding: utf-8 -*- +""" +合并表 / 搜索导出表入库后若 ``total_sales`` 为空，可按与 ingest 相同规则从 ``comment_sales_floor`` 补全。 + + python manage.py refresh_jd_merged_total_sales + python manage.py refresh_jd_merged_total_sales --job-id 42 +""" +from __future__ import annotations + +from django.core.management.base import BaseCommand + +from pipeline.csv.schema import ( + JD_SEARCH_CSV_HEADERS, + MERGED_FIELD_TO_CSV_HEADER, + merged_csv_effective_total_sales, + search_csv_effective_total_sales, +) +from pipeline.models import JdJobMergedRow, JdJobSearchRow + + +class Command(BaseCommand): + help = "从销量楼层推断并回填 JdJobMergedRow / JdJobSearchRow 的 total_sales（与 ingest 一致）。" + + def add_arguments(self, parser) -> None: + parser.add_argument( + "--job-id", + type=int, + default=None, + help="仅处理该 PipelineJob；默认处理全部任务下的行", + ) + + def handle(self, *args, **options) -> None: + job_id = options.get("job_id") + n_merged = self._refresh_merged(job_id) + n_search = self._refresh_search(job_id) + self.stdout.write( + self.style.SUCCESS( + f"完成：JdJobMergedRow 更新约 {n_merged} 行，" + f"JdJobSearchRow 更新约 {n_search} 行" + ) + ) + + def _refresh_merged(self, job_id: int | None) -> int: + qs = JdJobMergedRow.objects.all().order_by("id") + if job_id is not None: + qs = qs.filter(job_id=job_id) + + h_ts = MERGED_FIELD_TO_CSV_HEADER["total_sales"] + h_fl = MERGED_FIELD_TO_CSV_HEADER["comment_sales_floor"] + updates: list[JdJobMergedRow] = [] + n_changed = 0 + for r in qs.iterator(chunk_size=800): + row = {h_ts: r.total_sales or "", h_fl: r.comment_sales_floor or ""} + eff = merged_csv_effective_total_sales(row) + if eff and eff != (r.total_sales or "").strip(): + r.total_sales = eff + updates.append(r) + n_changed += 1 + if len(updates) >= 500: + JdJobMergedRow.objects.bulk_update(updates, ["total_sales"]) + updates.clear() + if updates: + JdJobMergedRow.objects.bulk_update(updates, ["total_sales"]) + return n_changed + + def _refresh_search(self, job_id: int | None) -> int: + qs = JdJobSearchRow.objects.all().order_by("id") + if job_id is not None: + qs = qs.filter(job_id=job_id) + + h_ts = JD_SEARCH_CSV_HEADERS["total_sales"] + h_fl = JD_SEARCH_CSV_HEADERS["comment_sales_floor"] + updates: list[JdJobSearchRow] = [] + n_changed = 0 + for r in qs.iterator(chunk_size=800): + row = {h_ts: r.total_sales or "", h_fl: r.comment_sales_floor or ""} + eff = search_csv_effective_total_sales(row) + if eff and eff != (r.total_sales or "").strip(): + r.total_sales = eff + updates.append(r) + n_changed += 1 + if len(updates) >= 500: + JdJobSearchRow.objects.bulk_update(updates, ["total_sales"]) + updates.clear() + if updates: + JdJobSearchRow.objects.bulk_update(updates, ["total_sales"]) + return n_changed diff --git a/backend/pipeline/management/commands/regen_merged_csv.py b/backend/pipeline/management/commands/regen_merged_csv.py new file mode 100644 index 0000000..7124cd9 --- /dev/null +++ b/backend/pipeline/management/commands/regen_merged_csv.py @@ -0,0 +1,33 @@ +# -*- coding: utf-8 -*- +"""补全并规范化已有 run 目录下的 ``keyword_pipeline_merged.csv``（lean 列序，与 detail_ware 对齐）。""" +from __future__ import annotations + +from django.conf import settings +from django.core.management.base import BaseCommand, CommandError + +from pipeline.ingest import resolve_and_validate_run_dir +from pipeline.jd.merged_regen import write_keyword_pipeline_merged_lean_csv + + +class Command(BaseCommand): + help = "将 keyword_pipeline_merged.csv 重写为 lean 宽表列序，并刷新榜单/购买者摘要列。" + + def add_arguments(self, parser) -> None: + parser.add_argument( + "--run-dir", + type=str, + required=True, + help="相对 data/JD 的子路径，或位于 data/JD 下的绝对路径", + ) + + def handle(self, *args, **options): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + raise CommandError("请在 .env 中配置 LOW_GI_PROJECT_ROOT") + raw = str(options["run_dir"] or "").strip() + try: + run_path = resolve_and_validate_run_dir(raw) + except ValueError as e: + raise CommandError(str(e)) from e + + n, p = write_keyword_pipeline_merged_lean_csv(run_path) + self.stdout.write(self.style.SUCCESS(f"已写 {n} 行 -> {p}")) diff --git a/backend/pipeline/management/commands/rewrite_pipeline_csv_headers.py b/backend/pipeline/management/commands/rewrite_pipeline_csv_headers.py new file mode 100644 index 0000000..40a6fb6 --- /dev/null +++ b/backend/pipeline/management/commands/rewrite_pipeline_csv_headers.py @@ -0,0 +1,75 @@ +# -*- coding: utf-8 -*- +""" +将已有 run 目录下 CSV 表头重写为 ``pipeline.csv.schema`` 纯中文表头（仅重命名与列序，不修改业务逻辑）。 + +在 ``backend`` 目录:: + + python manage.py rewrite_pipeline_csv_headers --run-dir pipeline_runs/20260413_104252_低GI + python manage.py rewrite_pipeline_csv_headers --run-dir pipeline_runs/xxx --dry-run + +仅处理部分文件:: + + python manage.py rewrite_pipeline_csv_headers --run-dir pipeline_runs/xxx --file keyword_pipeline_merged.csv +""" +from __future__ import annotations + +from django.conf import settings +from django.core.management.base import BaseCommand, CommandError + +from pipeline.csv.header_rewrite import rewrite_run_dir_csv_headers +from pipeline.ingest import ( + FILE_COMMENTS_FLAT_CSV, + FILE_DETAIL_WARE_CSV, + FILE_MERGED_CSV, + FILE_PC_SEARCH_CSV, + resolve_and_validate_run_dir, +) + + +class Command(BaseCommand): + help = "将 pipeline run 目录内 CSV 表头规范为 pipeline.csv.schema 中的中文表头。" + + def add_arguments(self, parser) -> None: + parser.add_argument( + "--run-dir", + type=str, + required=True, + help="相对 data/JD 的子路径，或位于 data/JD 下的绝对路径", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="只打印将执行的操作，不写回文件", + ) + parser.add_argument( + "--file", + action="append", + dest="files", + metavar="NAME", + help=( + "只处理指定文件名，可多次传入。" + f"可选: {FILE_MERGED_CSV}, {FILE_PC_SEARCH_CSV}, " + f"{FILE_COMMENTS_FLAT_CSV}, {FILE_DETAIL_WARE_CSV}" + ), + ) + + def handle(self, *args, **options): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + raise CommandError("请在 .env 中配置 LOW_GI_PROJECT_ROOT") + + raw = str(options["run_dir"] or "").strip() + try: + run_path = resolve_and_validate_run_dir(raw) + except ValueError as e: + raise CommandError(str(e)) from e + + dry = bool(options["dry_run"]) + only = options.get("files") or None + + msgs = rewrite_run_dir_csv_headers(run_path, dry_run=dry, only=only) + for msg in msgs: + self.stdout.write(msg) + if dry: + self.stdout.write(self.style.WARNING("dry-run：未写入磁盘")) + else: + self.stdout.write(self.style.SUCCESS(f"完成: {run_path}")) diff --git a/backend/pipeline/management/commands/run_pipeline_job.py b/backend/pipeline/management/commands/run_pipeline_job.py index 7622a4e..45ca0d8 100644 --- a/backend/pipeline/management/commands/run_pipeline_job.py +++ b/backend/pipeline/management/commands/run_pipeline_job.py @@ -9,7 +9,7 @@ from django.conf import settings from django.core.management.base import BaseCommand, CommandError from pipeline.cookie_paste import normalize_browser_cookie_paste -from pipeline.jd_runner import run_jd_keyword_and_report +from pipeline.jd.runner import run_jd_keyword_and_report from pipeline.models import PipelineJob diff --git a/backend/pipeline/md_document_export.py b/backend/pipeline/md_document_export.py deleted file mode 100644 index 4396ed1..0000000 --- a/backend/pipeline/md_document_export.py +++ /dev/null @@ -1,250 +0,0 @@ -"""Markdown → Word（.docx）/ 简易 PDF；供任务报告与策略稿导出。""" -from __future__ import annotations - -import os -import re -from io import BytesIO -from pathlib import Path -from typing import Any -from xml.sax.saxutils import escape as xml_escape - - -def _strip_inline_md(s: str) -> str: - s = re.sub(r"\*\*(.+?)\*\*", r"\1", s) - s = re.sub(r"`([^`]+)`", r"\1", s) - return s - - -def _is_table_sep(line: str) -> bool: - t = line.strip() - if not t.startswith("|"): - return False - inner = t.strip("|").replace(" ", "") - return bool(inner) and all(p in ("", "---", ":---", "---:", ":---:") for p in t.split("|")) - - -_img_line = re.compile(r"^!\[([^\]]*)\]\(([^)]+)\)\s*$") - - -def markdown_to_docx_bytes(md: str, *, asset_root: Path | None = None) -> bytes: - from docx import Document - from docx.enum.text import WD_PARAGRAPH_ALIGNMENT - from docx.shared import Inches, Pt - - doc = Document() - try: - style = doc.styles["Normal"] - style.font.name = "Microsoft YaHei" - style.font.size = Pt(10.5) - except Exception: - pass - - lines = (md or "").replace("\r\n", "\n").split("\n") - i = 0 - in_fence = False - while i < len(lines): - raw = lines[i] - if raw.strip().startswith("```"): - in_fence = not in_fence - i += 1 - continue - if in_fence: - p = doc.add_paragraph(xml_escape(raw) or " ") - p.style = doc.styles["Normal"] - for run in p.runs: - run.font.name = "Consolas" - run.font.size = Pt(9) - i += 1 - continue - - line = raw.rstrip() - if not line.strip(): - doc.add_paragraph("") - i += 1 - continue - if line.startswith("# "): - doc.add_heading(_strip_inline_md(line[2:].strip()), level=0) - i += 1 - continue - if line.startswith("## "): - doc.add_heading(_strip_inline_md(line[3:].strip()), level=1) - i += 1 - continue - if line.startswith("### "): - doc.add_heading(_strip_inline_md(line[4:].strip()), level=2) - i += 1 - continue - if line.startswith("#### "): - doc.add_heading(_strip_inline_md(line[5:].strip()), level=3) - i += 1 - continue - mimg = _img_line.match(line.strip()) - if mimg and asset_root is not None: - rel = mimg.group(2).strip() - if not (rel.startswith("http://") or rel.startswith("https://")): - img_path = (asset_root / rel).resolve() - try: - img_path.relative_to(asset_root.resolve()) - except ValueError: - i += 1 - continue - if img_path.is_file(): - doc.add_picture(str(img_path), width=Inches(5.9)) - i += 1 - continue - if line.strip().startswith("|"): - rows: list[list[str]] = [] - while i < len(lines) and lines[i].strip().startswith("|"): - row_line = lines[i].strip() - if _is_table_sep(row_line): - i += 1 - continue - cells = [c.strip() for c in row_line.strip("|").split("|")] - rows.append([_strip_inline_md(c) for c in cells]) - i += 1 - if rows: - max_cols = max(len(r) for r in rows) - pad_rows = [r + [""] * (max_cols - len(r)) for r in rows] - tbl = doc.add_table(rows=len(pad_rows), cols=max_cols) - tbl.style = "Table Grid" - for ri, row in enumerate(pad_rows): - for ci, cell in enumerate(row): - tbl.rows[ri].cells[ci].text = cell - continue - - p = doc.add_paragraph() - p.alignment = WD_PARAGRAPH_ALIGNMENT.LEFT - text = _strip_inline_md(line) - p.add_run(text) - - bio = BytesIO() - doc.save(bio) - return bio.getvalue() - - -def _pdf_font_candidates() -> list[Path]: - raw = (os.environ.get("MA_PDF_FONT") or "").strip() - out: list[Path] = [] - if raw: - out.append(Path(raw)) - windir = os.environ.get("WINDIR", r"C:\Windows") - out.extend( - [ - Path(windir) / "Fonts" / "simhei.ttf", - Path(windir) / "Fonts" / "simsun.ttc", - Path(windir) / "Fonts" / "msyh.ttf", - ] - ) - return out - - -def markdown_to_pdf_bytes(md: str, *, asset_root: Path | None = None) -> bytes: - """简易纯文本流式 PDF；需本机 .ttf 中文字体或环境变量 MA_PDF_FONT。""" - from reportlab.lib.pagesizes import A4 - from reportlab.lib.styles import ParagraphStyle, getSampleStyleSheet - from reportlab.lib.units import cm - from reportlab.pdfbase import pdfmetrics - from reportlab.pdfbase.ttfonts import TTFont - from reportlab.platypus import Image as RLImage - from reportlab.platypus import Paragraph, SimpleDocTemplate, Spacer - - font_name = "MaExportCJK" - registered = False - for p in _pdf_font_candidates(): - if not p.is_file(): - continue - try: - if p.suffix.lower() == ".ttc": - try: - pdfmetrics.registerFont( - TTFont(font_name, str(p), subfontIndex=0) - ) - except TypeError: - pdfmetrics.registerFont(TTFont(font_name, str(p))) - else: - pdfmetrics.registerFont(TTFont(font_name, str(p))) - registered = True - break - except Exception: - continue - if not registered: - raise ValueError( - "未找到可用的中文字体文件。请在 Windows 上安装黑体/宋体，" - "或设置环境变量 MA_PDF_FONT 指向 .ttf 文件路径。" - ) - - styles = getSampleStyleSheet() - body = ParagraphStyle( - name="BodyCJK", - parent=styles["Normal"], - fontName=font_name, - fontSize=10, - leading=14, - ) - h1s = ParagraphStyle( - name="H1CJK", - parent=body, - fontSize=16, - leading=20, - spaceAfter=8, - ) - h2s = ParagraphStyle( - name="H2CJK", - parent=body, - fontSize=13, - leading=17, - spaceAfter=6, - ) - - story: list[Any] = [] - lines = (md or "").replace("\r\n", "\n").split("\n") - in_fence = False - for raw in lines: - if raw.strip().startswith("```"): - in_fence = not in_fence - continue - s = raw.rstrip() - if in_fence: - story.append(Paragraph(xml_escape(s or " "), body)) - story.append(Spacer(1, 0.1 * cm)) - continue - if not s.strip(): - story.append(Spacer(1, 0.15 * cm)) - continue - mimg = _img_line.match(s.strip()) - if mimg and asset_root is not None: - rel = mimg.group(2).strip() - if not (rel.startswith("http://") or rel.startswith("https://")): - img_path = (asset_root / rel).resolve() - try: - img_path.relative_to(asset_root.resolve()) - except ValueError: - continue - if img_path.is_file(): - story.append(RLImage(str(img_path), width=13 * cm)) - story.append(Spacer(1, 0.2 * cm)) - continue - plain = _strip_inline_md(s) - text = xml_escape(plain) - if s.startswith("# "): - story.append(Paragraph(xml_escape(plain[2:]), h1s)) - elif s.startswith("## "): - story.append(Paragraph(xml_escape(plain[3:]), h2s)) - elif s.startswith("### "): - story.append(Paragraph(xml_escape(plain[4:]), body)) - elif s.strip().startswith("|"): - story.append(Paragraph(text.replace("|", " │ "), body)) - else: - story.append(Paragraph(text, body)) - - buf = BytesIO() - doc = SimpleDocTemplate( - buf, - pagesize=A4, - leftMargin=2 * cm, - rightMargin=2 * cm, - topMargin=2 * cm, - bottomMargin=2 * cm, - ) - doc.build(story) - return buf.getvalue() diff --git a/backend/pipeline/migrations/0012_job_pause_checkpoint.py b/backend/pipeline/migrations/0012_job_pause_checkpoint.py new file mode 100644 index 0000000..e9248f5 --- /dev/null +++ b/backend/pipeline/migrations/0012_job_pause_checkpoint.py @@ -0,0 +1,65 @@ +# Generated manually for cookie pause / resume checkpoint + +import django.db.models.deletion +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ("pipeline", "0011_job_cancel_and_status"), + ] + + operations = [ + migrations.AddField( + model_name="pipelinejob", + name="resume_from_checkpoint", + field=models.BooleanField(db_index=True, default=False), + ), + migrations.AlterField( + model_name="pipelinejob", + name="status", + field=models.CharField( + choices=[ + ("pending", "待执行"), + ("running", "执行中"), + ("success", "成功"), + ("failed", "失败"), + ("cancelled", "已终止"), + ("paused", "已暂停（待换 Cookie 续跑）"), + ], + db_index=True, + default="pending", + max_length=16, + ), + ), + migrations.CreateModel( + name="PipelineJobCheckpoint", + fields=[ + ( + "id", + models.BigAutoField( + auto_created=True, + primary_key=True, + serialize=False, + verbose_name="ID", + ), + ), + ("phase", models.CharField(db_index=True, max_length=32)), + ("payload", models.JSONField(blank=True, default=dict)), + ("hint_zh", models.TextField(blank=True, default="")), + ("updated_at", models.DateTimeField(auto_now=True)), + ( + "job", + models.OneToOneField( + on_delete=django.db.models.deletion.CASCADE, + related_name="checkpoint_row", + to="pipeline.pipelinejob", + ), + ), + ], + options={ + "ordering": ["-updated_at"], + }, + ), + ] diff --git a/backend/pipeline/migrations/0013_rebuild_pipelinejobcheckpoint.py b/backend/pipeline/migrations/0013_rebuild_pipelinejobcheckpoint.py new file mode 100644 index 0000000..d15abc4 --- /dev/null +++ b/backend/pipeline/migrations/0013_rebuild_pipelinejobcheckpoint.py @@ -0,0 +1,34 @@ +# 0012 曾标记为已应用，但部分环境上 checkpoint 表仍为旧版 schema（stage/page_done 等）。 +# 与当前 PipelineJobCheckpoint（phase/payload/hint_zh）对齐：删表后按 0012 预期 DDL 重建。 + +from django.db import migrations + + +_REBUILD_SQL = """ +DROP TABLE IF EXISTS pipeline_pipelinejobcheckpoint; +CREATE TABLE "pipeline_pipelinejobcheckpoint" ( + "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT, + "phase" varchar(32) NOT NULL, + "payload" text NOT NULL CHECK ((JSON_VALID("payload") OR "payload" IS NULL)), + "hint_zh" text NOT NULL, + "updated_at" datetime NOT NULL, + "job_id" bigint NOT NULL UNIQUE REFERENCES "pipeline_pipelinejob" ("id") DEFERRABLE INITIALLY DEFERRED +); +CREATE INDEX "pipeline_pipelinejobcheckpoint_phase_12e50a62" ON "pipeline_pipelinejobcheckpoint" ("phase"); +""" + + +class Migration(migrations.Migration): + + dependencies = [ + ("pipeline", "0012_job_pause_checkpoint"), + ] + + operations = [ + migrations.SeparateDatabaseAndState( + state_operations=[], + database_operations=[ + migrations.RunSQL(_REBUILD_SQL, reverse_sql=migrations.RunSQL.noop), + ], + ), + ] diff --git a/backend/pipeline/migrations/0014_jdjobmergedrow_total_sales.py b/backend/pipeline/migrations/0014_jdjobmergedrow_total_sales.py new file mode 100644 index 0000000..2a9d9a5 --- /dev/null +++ b/backend/pipeline/migrations/0014_jdjobmergedrow_total_sales.py @@ -0,0 +1,18 @@ +# Generated manually for keyword_pipeline_merged.csv column 销量展示(totalSales). + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ("pipeline", "0013_rebuild_pipelinejobcheckpoint"), + ] + + operations = [ + migrations.AddField( + model_name="jdjobmergedrow", + name="total_sales", + field=models.TextField(blank=True, default=""), + ), + ] diff --git a/backend/pipeline/migrations/0015_jdjobsearchrow_total_sales.py b/backend/pipeline/migrations/0015_jdjobsearchrow_total_sales.py new file mode 100644 index 0000000..ef78c8f --- /dev/null +++ b/backend/pipeline/migrations/0015_jdjobsearchrow_total_sales.py @@ -0,0 +1,18 @@ +# Generated manually: align JdJobSearchRow with JD_SEARCH_INTERNAL_KEYS / pc_search_export. + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ("pipeline", "0014_jdjobmergedrow_total_sales"), + ] + + operations = [ + migrations.AddField( + model_name="jdjobsearchrow", + name="total_sales", + field=models.TextField(blank=True, default=""), + ), + ] diff --git a/backend/pipeline/migrations/0016_buyer_offer_csv_columns.py b/backend/pipeline/migrations/0016_buyer_offer_csv_columns.py new file mode 100644 index 0000000..fc0d34b --- /dev/null +++ b/backend/pipeline/migrations/0016_buyer_offer_csv_columns.py @@ -0,0 +1,33 @@ +# Generated by Django 5.2.1 on 2026-04-15 06:56 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ("pipeline", "0015_jdjobsearchrow_total_sales"), + ] + + operations = [ + migrations.AddField( + model_name="jdjobdetailrow", + name="buyer_ranking_line", + field=models.TextField(blank=True, default=""), + ), + migrations.AddField( + model_name="jdjobdetailrow", + name="buyer_promo_text", + field=models.TextField(blank=True, default=""), + ), + migrations.AddField( + model_name="jdjobmergedrow", + name="buyer_ranking_line", + field=models.TextField(blank=True, default=""), + ), + migrations.AddField( + model_name="jdjobmergedrow", + name="buyer_promo_text", + field=models.TextField(blank=True, default=""), + ), + ] diff --git a/backend/pipeline/migrations/0017_dataset_browse_filters.py b/backend/pipeline/migrations/0017_dataset_browse_filters.py new file mode 100644 index 0000000..e0356c9 --- /dev/null +++ b/backend/pipeline/migrations/0017_dataset_browse_filters.py @@ -0,0 +1,167 @@ +# Generated by Django 5.2.1 on 2026-04-16 01:49 + +import re + +import django.db.models.deletion +from django.db import migrations, models + + +def _float_price_from_cell(s): + t = (s or "").strip().replace(",", "").replace("，", "") + if not t: + return None + m = re.search(r"(\d+(?:\.\d+)?)", t) + if not m: + return None + try: + v = float(m.group(1)) + except ValueError: + return None + if 0 < v < 1_000_000: + return v + return None + + +def _effective_list_price_value(coupon, price, original): + for s in (coupon, price, original): + v = _float_price_from_cell(s) + if v is not None: + return v + return None + + +def backfill_dataset_browse_fields(apps, schema_editor): + JdLeafCategoryNorm = apps.get_model("pipeline", "JdLeafCategoryNorm") + JdJobSearchRow = apps.get_model("pipeline", "JdJobSearchRow") + JdJobMergedRow = apps.get_model("pipeline", "JdJobMergedRow") + JdJobDetailRow = apps.get_model("pipeline", "JdJobDetailRow") + + raw_labels = set( + JdJobSearchRow.objects.exclude(leaf_category="").values_list( + "leaf_category", flat=True + ) + ) + raw_labels |= set( + JdJobMergedRow.objects.exclude(leaf_category="").values_list( + "leaf_category", flat=True + ) + ) + labels = sorted({(x or "").strip()[:512] for x in raw_labels if (x or "").strip()}) + if labels: + JdLeafCategoryNorm.objects.bulk_create( + [JdLeafCategoryNorm(label=lbl) for lbl in labels], + ignore_conflicts=True, + ) + norm_by_label = {n.label: n.id for n in JdLeafCategoryNorm.objects.all()} + + chunk: list = [] + for r in JdJobSearchRow.objects.all().iterator(chunk_size=400): + lbl = (r.leaf_category or "").strip()[:512] + r.leaf_category_norm_id = norm_by_label.get(lbl) if lbl else None + r.price_value = _effective_list_price_value( + r.coupon_price, r.price, r.original_price + ) + chunk.append(r) + if len(chunk) >= 400: + JdJobSearchRow.objects.bulk_update( + chunk, ["leaf_category_norm_id", "price_value"] + ) + chunk.clear() + if chunk: + JdJobSearchRow.objects.bulk_update( + chunk, ["leaf_category_norm_id", "price_value"] + ) + + chunk = [] + for r in JdJobMergedRow.objects.all().iterator(chunk_size=400): + lbl = (r.leaf_category or "").strip()[:512] + r.leaf_category_norm_id = norm_by_label.get(lbl) if lbl else None + r.price_value = _effective_list_price_value( + r.coupon_price, r.price, r.original_price + ) + chunk.append(r) + if len(chunk) >= 400: + JdJobMergedRow.objects.bulk_update( + chunk, ["leaf_category_norm_id", "price_value"] + ) + chunk.clear() + if chunk: + JdJobMergedRow.objects.bulk_update( + chunk, ["leaf_category_norm_id", "price_value"] + ) + + chunk = [] + for r in JdJobDetailRow.objects.all().iterator(chunk_size=400): + r.detail_price_value = _float_price_from_cell(r.detail_price_final) + chunk.append(r) + if len(chunk) >= 400: + JdJobDetailRow.objects.bulk_update(chunk, ["detail_price_value"]) + chunk.clear() + if chunk: + JdJobDetailRow.objects.bulk_update(chunk, ["detail_price_value"]) + + +class Migration(migrations.Migration): + + dependencies = [ + ('pipeline', '0016_buyer_offer_csv_columns'), + ] + + operations = [ + migrations.CreateModel( + name='JdLeafCategoryNorm', + fields=[ + ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')), + ('label', models.CharField(db_index=True, max_length=512, unique=True)), + ], + options={ + 'ordering': ['label'], + }, + ), + migrations.AddField( + model_name='jdjobdetailrow', + name='detail_price_value', + field=models.FloatField(blank=True, db_index=True, null=True), + ), + migrations.AddField( + model_name='jdjobmergedrow', + name='price_value', + field=models.FloatField(blank=True, db_index=True, null=True), + ), + migrations.AddField( + model_name='jdjobsearchrow', + name='price_value', + field=models.FloatField(blank=True, db_index=True, null=True), + ), + migrations.AddIndex( + model_name='jdjobdetailrow', + index=models.Index(fields=['job', 'detail_price_value'], name='pipeline_jd_job_id_528890_idx'), + ), + migrations.AddField( + model_name='jdjobmergedrow', + name='leaf_category_norm', + field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='merged_rows', to='pipeline.jdleafcategorynorm'), + ), + migrations.AddField( + model_name='jdjobsearchrow', + name='leaf_category_norm', + field=models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='search_rows', to='pipeline.jdleafcategorynorm'), + ), + migrations.AddIndex( + model_name='jdjobmergedrow', + index=models.Index(fields=['job', 'leaf_category_norm'], name='pipeline_jd_job_id_fe7b92_idx'), + ), + migrations.AddIndex( + model_name='jdjobmergedrow', + index=models.Index(fields=['job', 'price_value'], name='pipeline_jd_job_id_49f026_idx'), + ), + migrations.AddIndex( + model_name='jdjobsearchrow', + index=models.Index(fields=['job', 'leaf_category_norm'], name='pipeline_jd_job_id_d77fae_idx'), + ), + migrations.AddIndex( + model_name='jdjobsearchrow', + index=models.Index(fields=['job', 'price_value'], name='pipeline_jd_job_id_b76707_idx'), + ), + migrations.RunPython(backfill_dataset_browse_fields, migrations.RunPython.noop), + ] diff --git a/backend/pipeline/migrations/0018_report_group_matrix_label.py b/backend/pipeline/migrations/0018_report_group_matrix_label.py new file mode 100644 index 0000000..892574c --- /dev/null +++ b/backend/pipeline/migrations/0018_report_group_matrix_label.py @@ -0,0 +1,92 @@ +# Generated by Django 5.2.1 on 2026-04-16 01:55 + +from django.db import migrations, models + + +def backfill_matrix_group_labels(apps, schema_editor): + from pipeline.jd.matrix_group_label import matrix_group_label_from_detail_path + + JdJobDetailRow = apps.get_model("pipeline", "JdJobDetailRow") + JdJobMergedRow = apps.get_model("pipeline", "JdJobMergedRow") + JdJobSearchRow = apps.get_model("pipeline", "JdJobSearchRow") + + chunk: list = [] + for r in JdJobDetailRow.objects.all().iterator(chunk_size=400): + r.matrix_group_label = matrix_group_label_from_detail_path( + r.detail_category_path or "" + ) + chunk.append(r) + if len(chunk) >= 400: + JdJobDetailRow.objects.bulk_update(chunk, ["matrix_group_label"]) + chunk.clear() + if chunk: + JdJobDetailRow.objects.bulk_update(chunk, ["matrix_group_label"]) + + chunk = [] + for r in JdJobMergedRow.objects.all().iterator(chunk_size=400): + r.matrix_group_label = matrix_group_label_from_detail_path( + r.detail_category_path or "" + ) + chunk.append(r) + if len(chunk) >= 400: + JdJobMergedRow.objects.bulk_update(chunk, ["matrix_group_label"]) + chunk.clear() + if chunk: + JdJobMergedRow.objects.bulk_update(chunk, ["matrix_group_label"]) + + sku_to_mg: dict[str, str] = {} + for r in JdJobMergedRow.objects.exclude(matrix_group_label="").iterator( + chunk_size=400 + ): + sk = str(r.sku_id or "").strip() + if sk: + sku_to_mg[sk] = r.matrix_group_label + + chunk = [] + for r in JdJobSearchRow.objects.all().iterator(chunk_size=400): + sk = str(r.sku_id or "").strip() + r.matrix_group_label = sku_to_mg.get(sk, "") + chunk.append(r) + if len(chunk) >= 400: + JdJobSearchRow.objects.bulk_update(chunk, ["matrix_group_label"]) + chunk.clear() + if chunk: + JdJobSearchRow.objects.bulk_update(chunk, ["matrix_group_label"]) + + +class Migration(migrations.Migration): + + dependencies = [ + ('pipeline', '0017_dataset_browse_filters'), + ] + + operations = [ + migrations.AddField( + model_name='jdjobdetailrow', + name='matrix_group_label', + field=models.CharField(blank=True, db_index=True, default='', help_text='与 §5 矩阵同源：由 detail_category_path 解析', max_length=80, verbose_name='报告细类'), + ), + migrations.AddField( + model_name='jdjobmergedrow', + name='matrix_group_label', + field=models.CharField(blank=True, db_index=True, default='', help_text='与 §5 矩阵同源：由 detail_category_path 解析', max_length=80, verbose_name='报告细类'), + ), + migrations.AddField( + model_name='jdjobsearchrow', + name='matrix_group_label', + field=models.CharField(blank=True, db_index=True, default='', help_text='与 §5 矩阵同源：由合并表商详路径解析；可按 SKU 从合并表回填', max_length=80, verbose_name='报告细类'), + ), + migrations.AddIndex( + model_name='jdjobdetailrow', + index=models.Index(fields=['job', 'matrix_group_label'], name='pipeline_jd_job_id_5595d3_idx'), + ), + migrations.AddIndex( + model_name='jdjobmergedrow', + index=models.Index(fields=['job', 'matrix_group_label'], name='pipeline_jd_job_id_163e3f_idx'), + ), + migrations.AddIndex( + model_name='jdjobsearchrow', + index=models.Index(fields=['job', 'matrix_group_label'], name='pipeline_jd_job_id_38fae5_idx'), + ), + migrations.RunPython(backfill_matrix_group_labels, migrations.RunPython.noop), + ] diff --git a/backend/pipeline/migrations/0019_matrix_label_verbose_category.py b/backend/pipeline/migrations/0019_matrix_label_verbose_category.py new file mode 100644 index 0000000..9471991 --- /dev/null +++ b/backend/pipeline/migrations/0019_matrix_label_verbose_category.py @@ -0,0 +1,28 @@ +# Generated by Django 5.2.1 on 2026-04-16 02:04 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('pipeline', '0018_report_group_matrix_label'), + ] + + operations = [ + migrations.AlterField( + model_name='jdjobdetailrow', + name='matrix_group_label', + field=models.CharField(blank=True, db_index=True, default='', help_text='与 §5 矩阵同源：由 detail_category_path 解析', max_length=80, verbose_name='类目'), + ), + migrations.AlterField( + model_name='jdjobmergedrow', + name='matrix_group_label', + field=models.CharField(blank=True, db_index=True, default='', help_text='与 §5 矩阵同源：由 detail_category_path 解析', max_length=80, verbose_name='类目'), + ), + migrations.AlterField( + model_name='jdjobsearchrow', + name='matrix_group_label', + field=models.CharField(blank=True, db_index=True, default='', help_text='与 §5 矩阵同源：由合并表商详路径解析；可按 SKU 从合并表回填', max_length=80, verbose_name='类目'), + ), + ] diff --git a/backend/pipeline/migrations/0020_search_merged_volume_sort.py b/backend/pipeline/migrations/0020_search_merged_volume_sort.py new file mode 100644 index 0000000..53cd993 --- /dev/null +++ b/backend/pipeline/migrations/0020_search_merged_volume_sort.py @@ -0,0 +1,99 @@ +# Generated by Django 5.2.1 on 2026-04-16 02:13 + +from django.db import migrations, models + + +def backfill_volume_sort_fields(apps, schema_editor): + from pipeline.volume_parse import ( + comment_count_sort_value_from_cell, + comment_count_sort_value_from_merged, + sales_sort_value_from_search_cells, + ) + + JdJobSearchRow = apps.get_model("pipeline", "JdJobSearchRow") + JdJobMergedRow = apps.get_model("pipeline", "JdJobMergedRow") + + chunk: list = [] + for r in JdJobSearchRow.objects.all().iterator(chunk_size=400): + r.sales_sort_value = sales_sort_value_from_search_cells( + r.total_sales or "", r.comment_sales_floor or "" + ) + r.comment_count_sort_value = comment_count_sort_value_from_cell( + r.comment_count or "" + ) + chunk.append(r) + if len(chunk) >= 400: + JdJobSearchRow.objects.bulk_update( + chunk, ["sales_sort_value", "comment_count_sort_value"] + ) + chunk.clear() + if chunk: + JdJobSearchRow.objects.bulk_update( + chunk, ["sales_sort_value", "comment_count_sort_value"] + ) + + chunk = [] + for r in JdJobMergedRow.objects.all().iterator(chunk_size=400): + r.sales_sort_value = sales_sort_value_from_search_cells( + r.total_sales or "", r.comment_sales_floor or "" + ) + r.comment_count_sort_value = comment_count_sort_value_from_merged( + r.pipeline_comment_count or "" + ) + chunk.append(r) + if len(chunk) >= 400: + JdJobMergedRow.objects.bulk_update( + chunk, ["sales_sort_value", "comment_count_sort_value"] + ) + chunk.clear() + if chunk: + JdJobMergedRow.objects.bulk_update( + chunk, ["sales_sort_value", "comment_count_sort_value"] + ) + + +class Migration(migrations.Migration): + + dependencies = [ + ('pipeline', '0019_matrix_label_verbose_category'), + ] + + operations = [ + migrations.AddField( + model_name='jdjobmergedrow', + name='comment_count_sort_value', + field=models.BigIntegerField(blank=True, db_index=True, help_text='从 pipeline_comment_count 解析，供排序', null=True), + ), + migrations.AddField( + model_name='jdjobmergedrow', + name='sales_sort_value', + field=models.BigIntegerField(blank=True, db_index=True, help_text='从 total_sales / 销量楼层解析，供排序', null=True), + ), + migrations.AddField( + model_name='jdjobsearchrow', + name='comment_count_sort_value', + field=models.BigIntegerField(blank=True, db_index=True, help_text='从评价量文案解析，供排序', null=True), + ), + migrations.AddField( + model_name='jdjobsearchrow', + name='sales_sort_value', + field=models.BigIntegerField(blank=True, db_index=True, help_text='从 total_sales / 销量楼层解析，供排序', null=True), + ), + migrations.AddIndex( + model_name='jdjobmergedrow', + index=models.Index(fields=['job', 'sales_sort_value'], name='pipeline_jd_job_id_b59c42_idx'), + ), + migrations.AddIndex( + model_name='jdjobmergedrow', + index=models.Index(fields=['job', 'comment_count_sort_value'], name='pipeline_jd_job_id_2c9d52_idx'), + ), + migrations.AddIndex( + model_name='jdjobsearchrow', + index=models.Index(fields=['job', 'sales_sort_value'], name='pipeline_jd_job_id_6798d3_idx'), + ), + migrations.AddIndex( + model_name='jdjobsearchrow', + index=models.Index(fields=['job', 'comment_count_sort_value'], name='pipeline_jd_job_id_f0aace_idx'), + ), + migrations.RunPython(backfill_volume_sort_fields, migrations.RunPython.noop), + ] diff --git a/backend/pipeline/migrations/0021_pipelinejob_strategy_config.py b/backend/pipeline/migrations/0021_pipelinejob_strategy_config.py new file mode 100644 index 0000000..7d30487 --- /dev/null +++ b/backend/pipeline/migrations/0021_pipelinejob_strategy_config.py @@ -0,0 +1,18 @@ +# Generated by Django 5.2.1 on 2026-04-17 08:40 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies = [ + ('pipeline', '0020_search_merged_volume_sort'), + ] + + operations = [ + migrations.AddField( + model_name='pipelinejob', + name='strategy_config', + field=models.JSONField(blank=True, default=dict), + ), + ] diff --git a/backend/pipeline/models.py b/backend/pipeline/models.py index 52b7796..b6b4d66 100644 --- a/backend/pipeline/models.py +++ b/backend/pipeline/models.py @@ -7,6 +7,7 @@ class JobStatus(models.TextChoices): SUCCESS = "success", "成功" FAILED = "failed", "失败" CANCELLED = "cancelled", "已终止" + PAUSED = "paused", "已暂停（待换 Cookie 续跑）" class PipelineJob(models.Model): @@ -36,6 +37,8 @@ class PipelineJob(models.Model): scenario_filter_enabled = models.BooleanField(null=True, blank=True) # 竞品报告 / competitor-brief：关注词、场景词组、外部市场表等（JSON，空对象=用爬虫脚本默认） report_config = models.JSONField(default=dict, blank=True) + # 策略生成页独立配置（与 report_config 无关），如默认是否使用大模型润色等 + strategy_config = models.JSONField(default=dict, blank=True) status = models.CharField( max_length=16, choices=JobStatus.choices, @@ -43,6 +46,7 @@ class PipelineJob(models.Model): db_index=True, ) cancellation_requested = models.BooleanField(default=False, db_index=True) + resume_from_checkpoint = models.BooleanField(default=False, db_index=True) run_dir = models.TextField(blank=True, default="") error_message = models.TextField(blank=True, default="") created_at = models.DateTimeField(auto_now_add=True) @@ -55,6 +59,26 @@ class PipelineJob(models.Model): return f"[{self.platform}] {self.keyword} ({self.status})" +class PipelineJobCheckpoint(models.Model): + """Cookie 暂停续跑等场景的断点元数据（与任务一对一）。""" + + job = models.OneToOneField( + PipelineJob, + on_delete=models.CASCADE, + related_name="checkpoint_row", + ) + phase = models.CharField(max_length=32, db_index=True) + payload = models.JSONField(default=dict, blank=True) + hint_zh = models.TextField(blank=True, default="") + updated_at = models.DateTimeField(auto_now=True) + + class Meta: + ordering = ["-updated_at"] + + def __str__(self) -> str: + return f"checkpoint job={self.job_id} phase={self.phase}" + + class JdProduct(models.Model): """京东 SKU 主档：同一 ``platform`` + ``sku_id`` 唯一，多次抓取时覆盖为最新一行合并表数据。""" @@ -120,6 +144,18 @@ class JdProductSnapshot(models.Model): return f"{self.product_id} @ job {self.job_id}" +class JdLeafCategoryNorm(models.Model): + """叶类目归一：与导出 ``leaf_category`` 原文一致（去空格），用于任务内类目筛选索引。""" + + label = models.CharField(max_length=512, unique=True, db_index=True) + + class Meta: + ordering = ["label"] + + def __str__(self) -> str: + return self.label[:80] + + class JdJobSearchRow(models.Model): """单次任务下 PC 搜索导出表一行，字段与 ``pc_search_export.csv`` 列一一对应（内部英文属性名）。""" @@ -137,6 +173,7 @@ class JdJobSearchRow(models.Model): original_price = models.TextField(blank=True, default="") selling_point = models.TextField(blank=True, default="") comment_sales_floor = models.TextField(blank=True, default="") + total_sales = models.TextField(blank=True, default="") hot_list_rank = models.TextField(blank=True, default="") comment_count = models.TextField(blank=True, default="") shop_name = models.TextField(blank=True, default="") @@ -148,6 +185,34 @@ class JdJobSearchRow(models.Model): seckill_info = models.TextField(blank=True, default="") attributes = models.TextField(blank=True, default="") leaf_category = models.TextField(blank=True, default="") + leaf_category_norm = models.ForeignKey( + JdLeafCategoryNorm, + null=True, + blank=True, + on_delete=models.SET_NULL, + related_name="search_rows", + ) + matrix_group_label = models.CharField( + max_length=80, + blank=True, + default="", + db_index=True, + verbose_name="类目", + help_text="与 §5 矩阵同源：由合并表商详路径解析；可按 SKU 从合并表回填", + ) + price_value = models.FloatField(null=True, blank=True, db_index=True) + sales_sort_value = models.BigIntegerField( + null=True, + blank=True, + db_index=True, + help_text="从 total_sales / 销量楼层解析，供排序", + ) + comment_count_sort_value = models.BigIntegerField( + null=True, + blank=True, + db_index=True, + help_text="从评价量文案解析，供排序", + ) platform = models.TextField(blank=True, default="") keyword = models.TextField(blank=True, default="") page = models.TextField(blank=True, default="") @@ -162,6 +227,11 @@ class JdJobSearchRow(models.Model): ] indexes = [ models.Index(fields=["job", "sku_id"]), + models.Index(fields=["job", "leaf_category_norm"]), + models.Index(fields=["job", "matrix_group_label"]), + models.Index(fields=["job", "price_value"]), + models.Index(fields=["job", "sales_sort_value"]), + models.Index(fields=["job", "comment_count_sort_value"]), ] def __str__(self) -> str: @@ -184,6 +254,17 @@ class JdJobDetailRow(models.Model): detail_category_path = models.TextField(blank=True, default="") detail_product_attributes = models.TextField(blank=True, default="") detail_body_ingredients = models.TextField(blank=True, default="") + buyer_ranking_line = models.TextField(blank=True, default="") + buyer_promo_text = models.TextField(blank=True, default="") + detail_price_value = models.FloatField(null=True, blank=True, db_index=True) + matrix_group_label = models.CharField( + max_length=80, + blank=True, + default="", + db_index=True, + verbose_name="类目", + help_text="与 §5 矩阵同源：由 detail_category_path 解析", + ) class Meta: ordering = ["row_index"] @@ -195,6 +276,8 @@ class JdJobDetailRow(models.Model): ] indexes = [ models.Index(fields=["job", "sku_id"]), + models.Index(fields=["job", "detail_price_value"]), + models.Index(fields=["job", "matrix_group_label"]), ] def __str__(self) -> str: @@ -255,11 +338,40 @@ class JdJobMergedRow(models.Model): hot_list_rank = models.TextField(blank=True, default="") comment_fuzzy = models.TextField(blank=True, default="") comment_sales_floor = models.TextField(blank=True, default="") + total_sales = models.TextField(blank=True, default="") shop_name = models.TextField(blank=True, default="") detail_url = models.TextField(blank=True, default="") image = models.TextField(blank=True, default="") attributes = models.TextField(blank=True, default="") leaf_category = models.TextField(blank=True, default="") + leaf_category_norm = models.ForeignKey( + JdLeafCategoryNorm, + null=True, + blank=True, + on_delete=models.SET_NULL, + related_name="merged_rows", + ) + matrix_group_label = models.CharField( + max_length=80, + blank=True, + default="", + db_index=True, + verbose_name="类目", + help_text="与 §5 矩阵同源：由 detail_category_path 解析", + ) + price_value = models.FloatField(null=True, blank=True, db_index=True) + sales_sort_value = models.BigIntegerField( + null=True, + blank=True, + db_index=True, + help_text="从 total_sales / 销量楼层解析，供排序", + ) + comment_count_sort_value = models.BigIntegerField( + null=True, + blank=True, + db_index=True, + help_text="从 pipeline_comment_count 解析，供排序", + ) keyword = models.TextField(blank=True, default="") page = models.TextField(blank=True, default="") detail_brand = models.TextField(blank=True, default="") @@ -268,6 +380,8 @@ class JdJobMergedRow(models.Model): detail_category_path = models.TextField(blank=True, default="") detail_product_attributes = models.TextField(blank=True, default="") detail_body_ingredients = models.TextField(blank=True, default="") + buyer_ranking_line = models.TextField(blank=True, default="") + buyer_promo_text = models.TextField(blank=True, default="") pipeline_comment_count = models.TextField(blank=True, default="") comment_preview = models.TextField(blank=True, default="") @@ -281,6 +395,11 @@ class JdJobMergedRow(models.Model): ] indexes = [ models.Index(fields=["job", "sku_id"]), + models.Index(fields=["job", "leaf_category_norm"]), + models.Index(fields=["job", "matrix_group_label"]), + models.Index(fields=["job", "price_value"]), + models.Index(fields=["job", "sales_sort_value"]), + models.Index(fields=["job", "comment_count_sort_value"]), ] def __str__(self) -> str: diff --git a/backend/pipeline/openai_gateway/__init__.py b/backend/pipeline/openai_gateway/__init__.py new file mode 100644 index 0000000..7083eae --- /dev/null +++ b/backend/pipeline/openai_gateway/__init__.py @@ -0,0 +1,44 @@ +""" +OpenAI 兼容网关（`chat/completions`）：纯文本、多模态配料、详情长图逆序等。 + +逻辑唯一在 ``pipeline.openai_gateway``；单图试跑在 ``python -m pipeline.openai_gateway``（见 ``__main__.py``），不再使用爬虫目录下的已删除脚本名。 +""" +from __future__ import annotations + +from .chat_content import normalize_message_content, strip_thinking_leaks_from_model_text +from .credentials import ( + _resolve_credentials, + resolve_credentials, + resolve_text_channel_credentials, + resolve_text_model_name, +) +from .ingredients_op import ( + REASON_NO_BODY_URLS, + REASON_NO_VISION_API, + extract_ingredients_from_body_image_urls_reversed, + extract_ingredients_from_body_image_urls_reversed_with_source, + extract_ingredients_from_image, + normalize_ingredients_text_for_csv, + parse_joined_image_urls, + sanitize_vision_ingredients_output, +) +from .text_chat import chat_completion_text, strip_outer_markdown_fence + +__all__ = [ + "REASON_NO_BODY_URLS", + "REASON_NO_VISION_API", + "_resolve_credentials", + "chat_completion_text", + "extract_ingredients_from_body_image_urls_reversed", + "extract_ingredients_from_body_image_urls_reversed_with_source", + "extract_ingredients_from_image", + "normalize_ingredients_text_for_csv", + "normalize_message_content", + "parse_joined_image_urls", + "resolve_credentials", + "resolve_text_channel_credentials", + "resolve_text_model_name", + "sanitize_vision_ingredients_output", + "strip_outer_markdown_fence", + "strip_thinking_leaks_from_model_text", +] diff --git a/backend/pipeline/openai_gateway/__main__.py b/backend/pipeline/openai_gateway/__main__.py new file mode 100644 index 0000000..d704d50 --- /dev/null +++ b/backend/pipeline/openai_gateway/__main__.py @@ -0,0 +1,109 @@ +# -*- coding: utf-8 -*- +""" +命令行单图配料表试跑（多模态）。实现位于本包，**非**对其它模块的再导出。 + +在 ``backend`` 目录下执行:: + + .venv\\Scripts\\python.exe -m pipeline.openai_gateway + +与 Django 一样从**仓库根** ``.env`` 读取 ``OPENAI_*`` / ``LLM_*``。 +""" +from __future__ import annotations + +import sys +from pathlib import Path +from typing import Any + +# 与 ingredients_defaults 同口径；可在此改图 URL/路径 +IMAGE_SOURCE = "https://img30.360buyimg.com/sku/jfs/t1/390444/8/13018/103574/6982e951Fc44d9d7b/00d62ee56189d75d.jpg.avif" + +_HERE = Path(__file__).resolve() +# openai_gateway -> pipeline -> backend -> 仓库根（.env） +_MA_ROOT = _HERE.parents[3] + + +def _load_market_assistant_dotenv() -> None: + try: + from dotenv import load_dotenv + except ImportError: + return + p = _MA_ROOT / ".env" + if p.is_file(): + load_dotenv(p) + + +def main() -> int: + _load_market_assistant_dotenv() + import requests + + from .ingredients_defaults import ( + IMAGE_REFERER, + MAX_TOKENS, + PROMPT_DEFAULT, + QWEN_OMNI_TEMPLATE, + TEMPERATURE, + USER_PROMPT, + ) + from .ingredients_op import _ingredient_extraction_acceptable + from . import extract_ingredients_from_image + + try: + if hasattr(sys.stdout, "reconfigure"): + sys.stdout.reconfigure(encoding="utf-8", errors="replace") + if hasattr(sys.stderr, "reconfigure"): + sys.stderr.reconfigure(encoding="utf-8", errors="replace") + except Exception: + pass + + src = (IMAGE_SOURCE or "").strip() + if not src: + print( + "[openai_gateway] 请在本文件 __main__.py 顶部设置 IMAGE_SOURCE（图片路径或 URL）。", + file=sys.stderr, + ) + return 2 + + prompt_use = (USER_PROMPT or "").strip() or None + extra: dict[str, Any] | None = None + if QWEN_OMNI_TEMPLATE: + extra = {"chat_template_kwargs": {"enable_thinking": False}} + + try: + text = extract_ingredients_from_image( + src, + user_prompt=prompt_use, + referer=(IMAGE_REFERER or "https://www.jd.com/").strip(), + temperature=float(TEMPERATURE), + max_tokens=int(MAX_TOKENS), + extra_json=extra, + prompt_default=PROMPT_DEFAULT, + ) + except ValueError as e: + print(f"[openai_gateway] {e}", file=sys.stderr) + return 2 + except requests.HTTPError as e: + err_body = "" + if e.response is not None and e.response.text: + err_body = e.response.text[:1500] + print(f"[openai_gateway] HTTP 错误: {e}\n{err_body}", file=sys.stderr) + return 1 + except Exception as e: + print(f"[openai_gateway] 失败: {e}", file=sys.stderr) + return 1 + + t = (text or "").strip() + if _ingredient_extraction_acceptable(t): + print(t) + return 0 + print( + "【未通过配料表校验】输出须同时包含包装配料表常见结构（如「配料/配料表/原料/食品添加剂」）" + "与含量或百分比等信息，或为「××（含量≥x%）」形态；纯食材/菜谱备料枚举不会采纳。" + "与 extract_ingredients_from_body_image_urls_reversed 流水线规则一致。" + ) + if t: + print(f"[openai_gateway] 模型原始输出（未采纳）: {t}", file=sys.stderr) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/backend/pipeline/openai_gateway/chat_content.py b/backend/pipeline/openai_gateway/chat_content.py new file mode 100644 index 0000000..86f175f --- /dev/null +++ b/backend/pipeline/openai_gateway/chat_content.py @@ -0,0 +1,63 @@ +"""解析 `chat/completions` 返回的 `message.content`（str 或多段），并剥掉泄漏到正文的「思考/推理」片段。""" +from __future__ import annotations + +import os +import re +from typing import Any + +# 部分网关/模型在仍开启「思考模式」时，会把推理混进 `content`；内部仍可思考，对下游/报告只返回正文。 +_F = re.IGNORECASE | re.DOTALL +_LT, _GT, _SL = (chr(60), chr(62), chr(47)) + + +def _pair(open_body: str, close_body: str) -> re.Pattern[str]: + o = _LT + open_body + _GT + c = _LT + _SL + close_body + _GT + return re.compile(re.escape(o) + r".*?" + re.escape(c), _F) + + +# 成对整段删尽（可多次出现）；`.*?` 非贪婪跨行 +_THINKING_SPAN_RES: list[re.Pattern[str]] = [ + _pair("redacted_thinking", "redacted_thinking"), + _pair("redacted_thinking", "think"), # 常见：以 </think> 收束 + _pair("think", "think"), +] + + +def strip_thinking_leaks_from_model_text(text: str) -> str: + """ + 从模型返回的「可见正文」中移除混在 `content` 里的思考/推理块（不关闭服务端思考，只净化下游看到的内容）。 + + 调试用：``MA_LLM_PRESERVE_THINKING_IN_OUTPUT=1`` 时不再剥离。 + """ + if not (text and text.strip()): + return text + if (os.environ.get("MA_LLM_PRESERVE_THINKING_IN_OUTPUT") or "").strip() in ( + "1", + "true", + "yes", + ): + return text + t = str(text) + for pat in _THINKING_SPAN_RES: + t = pat.sub("", t) + return t.strip() + + +def normalize_message_content(content: Any) -> str: + if content is None: + return "" + if isinstance(content, str): + return strip_thinking_leaks_from_model_text(content.strip()) + if isinstance(content, list): + parts: list[str] = [] + for item in content: + if isinstance(item, dict): + if item.get("type") == "text": + parts.append(str(item.get("text") or "")) + elif "text" in item: + parts.append(str(item.get("text") or "")) + elif isinstance(item, str): + parts.append(item) + return strip_thinking_leaks_from_model_text("".join(parts).strip()) + return strip_thinking_leaks_from_model_text(str(content).strip()) diff --git a/backend/pipeline/openai_gateway/constants.py b/backend/pipeline/openai_gateway/constants.py new file mode 100644 index 0000000..9c32b8b --- /dev/null +++ b/backend/pipeline/openai_gateway/constants.py @@ -0,0 +1,8 @@ +"""OpenAI 兼容调用的缺省常数（可仍由环境变量覆盖）。""" +from __future__ import annotations + +DEFAULT_MODEL = "Qwen/Qwen3-Omni-30B-A3B" +DEFAULT_USER_AGENT = ( + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " + "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" +) diff --git a/backend/pipeline/openai_gateway/credentials.py b/backend/pipeline/openai_gateway/credentials.py new file mode 100644 index 0000000..b2480cc --- /dev/null +++ b/backend/pipeline/openai_gateway/credentials.py @@ -0,0 +1,94 @@ +"""OpenAI 兼容网关：从环境或参数解析 API 凭证与模型名（文本 / 多模态共用）。""" +from __future__ import annotations + +import os + +from .constants import DEFAULT_MODEL + + +def _resolve_credentials( + api_key: str | None, + base_url: str | None, + model: str | None, +) -> tuple[str, str, str]: + key = ( + (api_key or "").strip() + or (os.environ.get("OPENAI_API_KEY") or os.environ.get("LLM_API_KEY") or "").strip() + ) + base = ( + (base_url or "").strip().rstrip("/") + or ( + os.environ.get("OPENAI_BASE_URL") or os.environ.get("LLM_BASE_URL") or "" + ).strip().rstrip("/") + ) + m = ( + (model or "").strip() + or ( + os.environ.get("OPENAI_VISION_MODEL") + or os.environ.get("LLM_MODEL") + or DEFAULT_MODEL + ).strip() + ) + if not key: + raise ValueError("请设置环境变量 OPENAI_API_KEY（或 LLM_API_KEY）") + if not base: + raise ValueError( + "请设置环境变量 OPENAI_BASE_URL（或 LLM_BASE_URL），例如 https://your-gateway.com/v1" + ) + return key, base, m + + +def resolve_text_channel_credentials( + api_key: str | None = None, + base_url: str | None = None, +) -> tuple[str, str]: + """ + 仅用于 ``text_chat.chat_completion_text`` 的 Key 与基址。 + + 可与商详多模/配料的 ``OPENAI_API_KEY``、``OPENAI_BASE_URL`` **分开**（例如同网关两枚 Key、或同 Key 不同路径）， + 任一侧未设 ``OPENAI_TEXT_*`` 时回退到 ``OPENAI_*`` / ``LLM_*``。 + + 别名：``LLM_TEXT_API_KEY``、``LLM_TEXT_BASE_URL``（与 ``OPENAI_TEXT_MODEL`` 等命名一致）。 + """ + key = (api_key or "").strip() or ( + (os.environ.get("OPENAI_TEXT_API_KEY") or os.environ.get("LLM_TEXT_API_KEY") or "").strip() + ) + if not key: + key = (os.environ.get("OPENAI_API_KEY") or os.environ.get("LLM_API_KEY") or "").strip() + base = (base_url or "").strip().rstrip("/") + if not base: + base = (os.environ.get("OPENAI_TEXT_BASE_URL") or os.environ.get("LLM_TEXT_BASE_URL") or "").strip().rstrip( + "/" + ) + if not base: + base = (os.environ.get("OPENAI_BASE_URL") or os.environ.get("LLM_BASE_URL") or "").strip().rstrip("/") + if not key: + raise ValueError("纯文本需设置 OPENAI_TEXT_API_KEY 或 OPENAI_API_KEY（或 LLM_* 别名）") + if not base: + raise ValueError("纯文本需设置 OPENAI_TEXT_BASE_URL 或 OPENAI_BASE_URL（或 LLM_* 别名）") + return key, base + + +def resolve_text_model_name(model: str | None = None) -> str: + m = (model or "").strip() + if m: + return m + for env in ( + "OPENAI_TEXT_MODEL", + "LLM_TEXT_MODEL", + "OPENAI_VISION_MODEL", + "LLM_MODEL", + ): + v = (os.environ.get(env) or "").strip() + if v: + return v + return DEFAULT_MODEL + + +def resolve_credentials( + api_key: str | None = None, + base_url: str | None = None, + model: str | None = None, +) -> tuple[str, str, str]: + """与历史 ``AI_crawler._resolve_credentials`` 行为一致，供业务显式预检（如多模态是否可用）。""" + return _resolve_credentials(api_key, base_url, model) diff --git a/backend/pipeline/openai_gateway/estimate.py b/backend/pipeline/openai_gateway/estimate.py new file mode 100644 index 0000000..842c3a9 --- /dev/null +++ b/backend/pipeline/openai_gateway/estimate.py @@ -0,0 +1,7 @@ +"""输入 token 保守估算（与历史 AI_crawler 同口径）。""" +from __future__ import annotations + + +def estimate_chat_input_tokens(system_prompt: str, user_prompt: str) -> int: + total_chars = len(system_prompt or "") + len(user_prompt or "") + return int(total_chars * 0.55) + 512 diff --git a/backend/pipeline/openai_gateway/ingredients_defaults.py b/backend/pipeline/openai_gateway/ingredients_defaults.py new file mode 100644 index 0000000..f7217a4 --- /dev/null +++ b/backend/pipeline/openai_gateway/ingredients_defaults.py @@ -0,0 +1,20 @@ +"""详情页配料多模态识别：提示词、温度与 Referer 等（与 `AI_crawler` 顶部运行配置同口径）。""" +from __future__ import annotations + +# 与 AI_crawler 中 PROMPT_DEFAULT 保持同步（脚本入口仍可在本地覆盖） +IMAGE_REFERER = "https://www.jd.com/" + +USER_PROMPT = "" +PROMPT_DEFAULT = ( + "请识别图片中的配料表，只输出配料列表本身，不要将菜品做法、步骤、用料等认为是配料表，不要误识别为食谱；用中文逗号或顿号分隔，" + "输出为连续一段文字，不要使用多行换行（避免与食谱、做法步骤混淆）。" + "每种原料名称只出现一次，禁止重复罗列同一添加剂（如磷酸三钾、磷酸三钠等勿循环抄写多遍）；" + "若图为表格中多行同类添加剂，可概括为「食品添加剂（按国家标准使用）」或合并为一句，勿展开成数百字重复。" + "【禁止猜测】必须严格依据图中清晰可见的印刷文字归纳；不得根据商品品类、常识或模糊字迹推测、补全、编造任何原料。" + "若本图无配料表、仅有产品信息/广告、文字被裁切、过小、模糊到无法逐字确认，或你只能「猜」出部分内容，则禁止输出配料列表：" + "请只输出且仅输出一句「无法识别图片中的配料表」（不要解释、不要道歉长文、不要列出疑似项）。" +) +TEMPERATURE = 0.0 +MAX_TOKENS = 2048 +# 部分 Qwen 网关需要关闭 thinking +QWEN_OMNI_TEMPLATE = False diff --git a/backend/crawler_copy/jd_pc_search/AI_crawler.py b/backend/pipeline/openai_gateway/ingredients_op.py similarity index 62% rename from backend/crawler_copy/jd_pc_search/AI_crawler.py rename to backend/pipeline/openai_gateway/ingredients_op.py index 1c4558f..8fdcaac 100644 --- a/backend/crawler_copy/jd_pc_search/AI_crawler.py +++ b/backend/pipeline/openai_gateway/ingredients_op.py @@ -1,113 +1,16 @@ -# -*- coding: utf-8 -*- -""" -从本地图片路径或图片 URL 调用 OpenAI 兼容多模态接口，提取配料表等；并提供**纯文本** ``chat/completions`` 供报告/策略等场景复用。 - -**密钥与网关仅通过环境变量配置**（勿写入代码）： - -- ``OPENAI_API_KEY``：API Key（必填） -- ``OPENAI_BASE_URL``：网关根地址，如 ``https://llm.example.com/v1``（必填，勿尾斜杠多余路径） -- ``OPENAI_VISION_MODEL``：视觉模型名（可选，默认 ``Qwen/Qwen3-Omni-30B-A3B``） - -**纯文本调用**（``chat_completion_text``）优先使用： - -- ``OPENAI_TEXT_MODEL`` 或 ``LLM_TEXT_MODEL``；未设置时依次回退到 ``OPENAI_VISION_MODEL``、``LLM_MODEL``、上述默认。 - -兼容别名（二选一即可）：``LLM_API_KEY``、``LLM_BASE_URL``、``LLM_MODEL``。 - -上述变量与 Django 共用 **一份** ``market_assistant/.env``（与本脚本所在 ``backend`` 的上三级目录下的 ``.env``；需 ``pip install python-dotenv``）。 - -**运行方式**：在下方「运行配置」里改好 ``IMAGE_SOURCE`` 等变量后，直接执行 ``python AI_crawler.py``，无需命令行参数。 -""" - +"""配料识别与详情长图逆序等逻辑。由 `AI_crawler` 抽离至 `pipeline.openai_gateway`。""" from __future__ import annotations import base64 -import os import re -import sys -from pathlib import Path from typing import Any import requests -_SCRIPT_DIR = Path(__file__).resolve().parent -# backend/crawler_copy/jd_pc_search -> parents[3] == market_assistant -_MA_ROOT = Path(__file__).resolve().parents[3] - - -def _load_market_assistant_dotenv() -> None: - """先于 LOW_GI_PROJECT_ROOT 解析加载 ``market_assistant/.env``（唯一配置源）。""" - try: - from dotenv import load_dotenv - except ImportError: - return - p = _MA_ROOT / ".env" - if p.is_file(): - load_dotenv(p) - - -_load_market_assistant_dotenv() - -from _low_gi_root import low_gi_project_root # noqa: E402 - -_PROJECT_ROOT = low_gi_project_root() - -# --------------------------------------------------------------------------- -# 运行配置（按需修改；启动时不要求命令行参数） -# --------------------------------------------------------------------------- -# 必填：本地图片路径，或 http(s) 图片链接（如京东主图 / 详情图） -IMAGE_SOURCE = "https://img30.360buyimg.com/sku/jfs/t1/390444/8/13018/103574/6982e951Fc44d9d7b/00d62ee56189d75d.jpg.avif" -# IMAGE_SOURCE = "https://img30.360buyimg.com/sku/jfs/t1/382894/31/7432/241977/694cf41aFa27be91e/00d63164ffeb8b46.jpg.avif" - -# 提示词：留空则使用 ``PROMPT_DEFAULT`` -USER_PROMPT = "" -PROMPT_DEFAULT = ( - "请识别图片中的配料表，只输出配料列表本身，不要将菜品做法、步骤、用料等认为是配料表，不要误识别为食谱；用中文逗号或顿号分隔，" - "输出为连续一段文字，不要使用多行换行（避免与食谱、做法步骤混淆）。" - "每种原料名称只出现一次，禁止重复罗列同一添加剂（如磷酸三钾、磷酸三钠等勿循环抄写多遍）；" - "若图为表格中多行同类添加剂，可概括为「食品添加剂（按国家标准使用）」或合并为一句，勿展开成数百字重复。" - "【禁止猜测】必须严格依据图中清晰可见的印刷文字归纳；不得根据商品品类、常识或模糊字迹推测、补全、编造任何原料。" - "若本图无配料表、仅有产品信息/广告、文字被裁切、过小、模糊到无法逐字确认，或你只能「猜」出部分内容，则禁止输出配料列表：" - "请只输出且仅输出一句「无法识别图片中的配料表」（不要解释、不要道歉长文、不要列出疑似项）。" -) - -# 拉取远程图时的 Referer（京东图床一般需类似商城域名） -IMAGE_REFERER = "https://www.jd.com/" - -TEMPERATURE = 0.0 -MAX_TOKENS = 2048 -# 部分 Qwen 网关需要关闭 thinking -QWEN_OMNI_TEMPLATE = False -# --------------------------------------------------------------------------- - -DEFAULT_MODEL = "Qwen/Qwen3-Omni-30B-A3B" -DEFAULT_USER_AGENT = ( - "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " - "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" -) - - -def _normalize_chat_content(content: Any) -> str: - """ - 兼容 OpenAI 兼容网关：``message.content`` 可能是 str，也可能是 - ``[{type:text, text:...}, ...]``；避免对 list 误用 ``.strip()`` 或得到怪异字符串。 - """ - if content is None: - return "" - if isinstance(content, str): - return content.strip() - if isinstance(content, list): - parts: list[str] = [] - for item in content: - if isinstance(item, dict): - if item.get("type") == "text": - parts.append(str(item.get("text") or "")) - elif "text" in item: - parts.append(str(item.get("text") or "")) - elif isinstance(item, str): - parts.append(item) - return "".join(parts).strip() - return str(content).strip() +from . import ingredients_defaults as _d +from .chat_content import normalize_message_content as _normalize_chat_content +from .constants import DEFAULT_USER_AGENT +from .credentials import _resolve_credentials def normalize_ingredients_text_for_csv(text: str) -> str: @@ -178,115 +81,6 @@ def sanitize_vision_ingredients_output(text: str) -> str: return t.strip() -def _resolve_credentials( - api_key: str | None, - base_url: str | None, - model: str | None, -) -> tuple[str, str, str]: - """凭证只从环境变量（及可选函数参数）读取，不在代码中写死。""" - key = ( - (api_key or "").strip() - or (os.environ.get("OPENAI_API_KEY") or os.environ.get("LLM_API_KEY") or "").strip() - ) - base = ( - (base_url or "").strip().rstrip("/") - or ( - os.environ.get("OPENAI_BASE_URL") or os.environ.get("LLM_BASE_URL") or "" - ).strip().rstrip("/") - ) - m = ( - (model or "").strip() - or ( - os.environ.get("OPENAI_VISION_MODEL") - or os.environ.get("LLM_MODEL") - or DEFAULT_MODEL - ).strip() - ) - if not key: - raise ValueError("请设置环境变量 OPENAI_API_KEY（或 LLM_API_KEY）") - if not base: - raise ValueError( - "请设置环境变量 OPENAI_BASE_URL（或 LLM_BASE_URL），例如 https://your-gateway.com/v1" - ) - return key, base, m - - -def resolve_text_model_name(model: str | None = None) -> str: - """ - 文本补全所用模型：显式 ``model`` 优先，否则读环境变量（见模块文档）。 - """ - m = (model or "").strip() - if m: - return m - for env in ( - "OPENAI_TEXT_MODEL", - "LLM_TEXT_MODEL", - "OPENAI_VISION_MODEL", - "LLM_MODEL", - ): - v = (os.environ.get(env) or "").strip() - if v: - return v - return DEFAULT_MODEL - - -def strip_outer_markdown_fence(text: str) -> str: - """若模型用 ``` / ```markdown 包裹全文，去掉最外层围栏。""" - t = (text or "").strip() - if not t.startswith("```"): - return t - lines = t.split("\n") - if lines and lines[0].strip().startswith("```"): - lines = lines[1:] - while lines and lines[-1].strip() == "```": - lines = lines[:-1] - return "\n".join(lines).strip() - - -def chat_completion_text( - *, - system_prompt: str, - user_prompt: str, - api_key: str | None = None, - base_url: str | None = None, - model: str | None = None, - temperature: float = 0.2, - max_tokens: int = 8192, - timeout: int = 300, - extra_json: dict[str, Any] | None = None, -) -> str: - """ - OpenAI 兼容网关的**纯文本**多轮占位为 system + user 各一条，与 ``extract_ingredients_from_image`` 共用凭证与端点。 - 返回助手消息正文（已 ``strip`` / 兼容 list 型 content）。 - """ - k, b, _ = _resolve_credentials(api_key, base_url, None) - m = resolve_text_model_name(model) - body: dict[str, Any] = { - "model": m, - "messages": [ - {"role": "system", "content": system_prompt}, - {"role": "user", "content": user_prompt}, - ], - "temperature": temperature, - "max_tokens": max_tokens, - } - if extra_json: - body.update(extra_json) - r = requests.post( - f"{b}/chat/completions", - headers={ - "Authorization": f"Bearer {k}", - "Content-Type": "application/json", - }, - json=body, - timeout=timeout, - ) - r.raise_for_status() - data = r.json() - msg = (data.get("choices") or [{}])[0].get("message") or {} - return _normalize_chat_content(msg.get("content")) - - def _mime_for_path(path: str) -> str: ext = path.lower().rsplit(".", 1)[-1] return { @@ -663,15 +457,15 @@ def extract_ingredients_from_body_image_urls_reversed_with_source( except ValueError: return REASON_NO_VISION_API, None - ref = (referer if referer is not None else IMAGE_REFERER) or "https://www.jd.com/" - temp = float(temperature) if temperature is not None else float(TEMPERATURE) - mt = int(max_tokens) if max_tokens is not None else int(MAX_TOKENS) + ref = (referer if referer is not None else _d.IMAGE_REFERER) or "https://www.jd.com/" + temp = float(temperature) if temperature is not None else float(_d.TEMPERATURE) + mt = int(max_tokens) if max_tokens is not None else int(_d.MAX_TOKENS) extra = extra_json - if extra is None and QWEN_OMNI_TEMPLATE: + if extra is None and _d.QWEN_OMNI_TEMPLATE: extra = {"chat_template_kwargs": {"enable_thinking": False}} - pu = user_prompt if user_prompt is not None else ((USER_PROMPT or "").strip() or None) - pd = prompt_default if prompt_default is not None else PROMPT_DEFAULT + pu = user_prompt if user_prompt is not None else ((_d.USER_PROMPT or "").strip() or None) + pd = prompt_default if prompt_default is not None else _d.PROMPT_DEFAULT n = len(urls) n_err = 0 @@ -739,66 +533,4 @@ def extract_ingredients_from_body_image_urls_reversed( max_tokens=max_tokens, extra_json=extra_json, ) - return text - - -def main() -> None: - try: - if hasattr(sys.stdout, "reconfigure"): - sys.stdout.reconfigure(encoding="utf-8", errors="replace") - if hasattr(sys.stderr, "reconfigure"): - sys.stderr.reconfigure(encoding="utf-8", errors="replace") - except Exception: - pass - - src = (IMAGE_SOURCE or "").strip() - if not src: - print( - "[AI_crawler] 请在文件顶部设置 IMAGE_SOURCE（图片路径或 URL）后重试。", - file=sys.stderr, - ) - sys.exit(2) - - prompt_use = (USER_PROMPT or "").strip() or None - extra = None - if QWEN_OMNI_TEMPLATE: - extra = {"chat_template_kwargs": {"enable_thinking": False}} - - try: - text = extract_ingredients_from_image( - src, - user_prompt=prompt_use, - referer=(IMAGE_REFERER or "https://www.jd.com/").strip(), - temperature=float(TEMPERATURE), - max_tokens=int(MAX_TOKENS), - extra_json=extra, - prompt_default=PROMPT_DEFAULT, - ) - except ValueError as e: - print(f"[AI_crawler] {e}", file=sys.stderr) - sys.exit(2) - except requests.HTTPError as e: - err_body = "" - if e.response is not None and e.response.text: - err_body = e.response.text[:1500] - print(f"[AI_crawler] HTTP 错误: {e}\n{err_body}", file=sys.stderr) - sys.exit(1) - except Exception as e: - print(f"[AI_crawler] 失败: {e}", file=sys.stderr) - sys.exit(1) - - t = (text or "").strip() - if _ingredient_extraction_acceptable(t): - print(t) - else: - print( - "【未通过配料表校验】输出须同时包含包装配料表常见结构（如「配料/配料表/原料/食品添加剂」）" - "与含量或百分比等信息，或为「××（含量≥x%）」形态；纯食材/菜谱备料枚举不会采纳。" - "与 extract_ingredients_from_body_image_urls_reversed 流水线规则一致。" - ) - if t: - print(f"[AI_crawler] 模型原始输出（未采纳）: {t}", file=sys.stderr) - - -if __name__ == "__main__": - main() + return text \ No newline at end of file diff --git a/backend/pipeline/openai_gateway/text_chat.py b/backend/pipeline/openai_gateway/text_chat.py new file mode 100644 index 0000000..e10c151 --- /dev/null +++ b/backend/pipeline/openai_gateway/text_chat.py @@ -0,0 +1,102 @@ +""" +OpenAI 兼容 `chat/completions` 纯文本（system + user）。 + +凭据与基址可经 ``OPENAI_TEXT_API_KEY`` / ``OPENAI_TEXT_BASE_URL`` 与多模/配料的 ``OPENAI_API_KEY`` / ``OPENAI_BASE_URL`` **分开**（可只覆写其中一项，另一项回退到 ``OPENAI_*``）。详见 ``credentials.resolve_text_channel_credentials``。 +""" +from __future__ import annotations + +import os +from typing import Any + +import requests + +from .chat_content import normalize_message_content +from .credentials import resolve_text_channel_credentials, resolve_text_model_name +from .estimate import estimate_chat_input_tokens +from .timeouts import chat_completion_read_timeout as _chat_completion_timeout + + +def strip_outer_markdown_fence(text: str) -> str: + """若模型用 ``` / ```markdown 包裹全文，去掉最外层围栏。""" + t = (text or "").strip() + if not t.startswith("```"): + return t + lines = t.split("\n") + if lines and lines[0].strip().startswith("```"): + lines = lines[1:] + while lines and lines[-1].strip() == "```": + lines = lines[:-1] + return "\n".join(lines).strip() + + +def chat_completion_text( + *, + system_prompt: str, + user_prompt: str, + api_key: str | None = None, + base_url: str | None = None, + model: str | None = None, + temperature: float = 0.2, + max_tokens: int = 8192, + timeout: int | tuple[float, float] | None = None, + extra_json: dict[str, Any] | None = None, +) -> str: + if timeout is None: + timeout = _chat_completion_timeout() + k, b = resolve_text_channel_credentials(api_key, base_url) + m = resolve_text_model_name(model) + body: dict[str, Any] = { + "model": m, + "messages": [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": user_prompt}, + ], + "temperature": temperature, + "max_tokens": max_tokens, + } + if extra_json: + body.update(extra_json) + ctx_raw = ( + os.environ.get("LLM_CONTEXT_WINDOW") + or os.environ.get("OPENAI_CONTEXT_WINDOW") + or "32768" + ).strip() + try: + context_window = max(4096, int(ctx_raw)) + except ValueError: + context_window = 32768 + buf = 256 + input_est = estimate_chat_input_tokens(system_prompt, user_prompt) + if input_est >= context_window - buf - 256: + raise ValueError( + f"提示词过长（估算输入约 {input_est} tokens，上下文上限 {context_window}），" + "请缩小报告/摘要输入或换更大上下文的模型；也可设置环境变量 LLM_CONTEXT_WINDOW。" + ) + avail = context_window - input_est - buf + want = int(body.get("max_tokens") or max_tokens) + body["max_tokens"] = max(256, min(want, max(avail, 256))) + r = requests.post( + f"{b}/chat/completions", + headers={ + "Authorization": f"Bearer {k}", + "Content-Type": "application/json", + }, + json=body, + timeout=timeout, + ) + try: + r.raise_for_status() + except requests.HTTPError as e: + snippet = "" + if e.response is not None: + snippet = (e.response.text or "")[:1200].replace("\r\n", "\n").replace("\n", " ") + if snippet: + raise requests.HTTPError( + f"{e!s} | body: {snippet}", + response=e.response, + request=e.request, + ) from e + raise + data = r.json() + msg = (data.get("choices") or [{}])[0].get("message") or {} + return normalize_message_content(msg.get("content")) diff --git a/backend/pipeline/openai_gateway/timeouts.py b/backend/pipeline/openai_gateway/timeouts.py new file mode 100644 index 0000000..11ef55e --- /dev/null +++ b/backend/pipeline/openai_gateway/timeouts.py @@ -0,0 +1,22 @@ +"""(连接, 读) 超时时长：供 ``chat/completions`` 与视觉请求使用。""" +from __future__ import annotations + +import os + + +def chat_completion_read_timeout() -> tuple[float, float]: + read = 600 + raw = (os.environ.get("LLM_CHAT_TIMEOUT") or os.environ.get("OPENAI_TIMEOUT") or "").strip() + if raw: + try: + read = max(60, int(raw)) + except ValueError: + pass + conn = 30.0 + raw_c = (os.environ.get("LLM_CHAT_CONNECT_TIMEOUT") or "").strip() + if raw_c: + try: + conn = max(5.0, float(raw_c)) + except ValueError: + pass + return (conn, float(read)) diff --git a/backend/pipeline/price_parse.py b/backend/pipeline/price_parse.py new file mode 100644 index 0000000..829adf4 --- /dev/null +++ b/backend/pipeline/price_parse.py @@ -0,0 +1,29 @@ +"""从爬虫导出单元格解析数值价格（供入库索引与数据集筛选）。""" +from __future__ import annotations + +import re + + +def float_price_from_cell(s: str | None) -> float | None: + t = (s or "").strip().replace(",", "").replace("，", "") + if not t: + return None + m = re.search(r"(\d+(?:\.\d+)?)", t) + if not m: + return None + try: + v = float(m.group(1)) + except ValueError: + return None + if 0 < v < 1_000_000: + return v + return None + + +def effective_list_price_value(coupon: str, price: str, original: str) -> float | None: + """优先券后价，其次标价，再次原价（与列表侧展示习惯一致）。""" + for s in (coupon, price, original): + v = float_price_from_cell(s) + if v is not None: + return v + return None diff --git a/backend/pipeline/report_charts.py b/backend/pipeline/report_charts.py deleted file mode 100644 index cabc2a6..0000000 --- a/backend/pipeline/report_charts.py +++ /dev/null @@ -1,378 +0,0 @@ -"""根据结构化 brief 生成报告用 PNG 统计图（matplotlib），写入 ``run_dir/report_assets/``。""" - -from __future__ import annotations - -import os -import re -from pathlib import Path -from typing import Any - - -def _setup_matplotlib_cjk() -> None: - import matplotlib - - matplotlib.use("Agg") - import matplotlib.pyplot as plt - from matplotlib import font_manager - - windir = os.environ.get("WINDIR", r"C:\Windows") - for name in ("simhei.ttf", "msyh.ttc", "simsun.ttc"): - fp = Path(windir) / "Fonts" / name - if fp.is_file(): - try: - font_manager.fontManager.addfont(str(fp)) - fam = font_manager.FontProperties(fname=str(fp)).get_name() - plt.rcParams["font.family"] = [fam] - break - except Exception: - continue - plt.rcParams["axes.unicode_minus"] = False - - -def _label_count_pairs( - items: Any, - *, - key_label: str = "label", - key_count: str = "count", - cap: int = 40, -) -> tuple[list[str], list[float]]: - labs: list[str] = [] - vals: list[float] = [] - if not isinstance(items, list): - return labs, vals - for item in items[:cap]: - if not isinstance(item, dict): - continue - lbl = str(item.get(key_label) or "").strip()[:48] - cnt = item.get(key_count) - if lbl and isinstance(cnt, (int, float)) and cnt > 0: - labs.append(lbl) - vals.append(float(cnt)) - return labs, vals - - -def _merge_labeled_counts_tail( - pairs: list[tuple[str, float]], *, max_items: int -) -> list[tuple[str, float]]: - if len(pairs) <= max_items: - return pairs - head = pairs[: max_items - 1] - rest = sum(c for _, c in pairs[max_items - 1 :]) - if rest > 0: - head.append(("其他", rest)) - return head - - -def _merge_tail_as_other( - labels: list[str], values: list[float], *, max_slices: int -) -> tuple[list[str], list[float]]: - pairs = [(l, v) for l, v in zip(labels, values) if v > 0] - if not pairs: - return [], [] - if len(pairs) <= max_slices: - return [p[0] for p in pairs], [p[1] for p in pairs] - head = pairs[: max_slices - 1] - rest = sum(v for _, v in pairs[max_slices - 1 :]) - labs = [p[0] for p in head] - vals = [p[1] for p in head] - if rest > 0: - labs.append("其他") - vals.append(rest) - return labs, vals - - -# 已不再写入报告正文的旧版图，避免 run_dir 里残留误导性 PNG -_OBSOLETE_REPORT_ASSETS: frozenset[str] = frozenset( - { - "chart_focus_keywords_bar.png", - "chart_usage_scenarios.png", - "chart_usage_scenarios_pie.png", - "chart_focus_keywords_pie.png", - } -) - - -def _cleanup_obsolete_report_assets(out_dir: Path) -> None: - """删除历史版本生成的、当前报告不再引用的插图文件。""" - if not out_dir.is_dir(): - return - for name in _OBSOLETE_REPORT_ASSETS: - fp = out_dir / name - if fp.is_file(): - try: - fp.unlink() - except OSError: - pass - for fp in out_dir.glob("chart_usage_scenarios_pie__*.png"): - try: - fp.unlink() - except OSError: - pass - - -def generate_report_charts(run_dir: Path, brief: dict[str, Any]) -> list[str]: - """生成扇形/条形 PNG。返回已写入的文件名列表（不含路径）。""" - _setup_matplotlib_cjk() - import matplotlib.pyplot as plt - - out_dir = Path(run_dir).resolve() / "report_assets" - out_dir.mkdir(parents=True, exist_ok=True) - _cleanup_obsolete_report_assets(out_dir) - created: list[str] = [] - - def save_bar_h( - labels: list[str], - values: list[float], - title: str, - fname: str, - xlabel: str = "", - ) -> None: - if not labels or not values or max(values) <= 0: - return - n = len(labels) - fig_h = max(3.2, min(14.0, 0.38 * n + 1.5)) - fig, ax = plt.subplots(figsize=(8.2, fig_h)) - y_pos = range(n) - ax.barh(list(y_pos), values, color="#2563eb", height=0.65) - ax.set_yticks(list(y_pos)) - ax.set_yticklabels(labels, fontsize=9) - ax.invert_yaxis() - ax.set_title(title, fontsize=12, pad=10) - if xlabel: - ax.set_xlabel(xlabel, fontsize=9) - fig.tight_layout() - path = out_dir / fname - fig.savefig(path, dpi=130, bbox_inches="tight") - plt.close(fig) - created.append(fname) - - def save_bar_h_share_of_text( - labels: list[str], - counts: list[float], - n_texts: int, - title: str, - fname: str, - ) -> None: - """ - 横轴 = count / n_texts * 100，与报告表格「占有效文本比例」一致（多标签下各柱比例可相加 >100%）。 - """ - if not labels or not counts or n_texts <= 0 or max(counts) <= 0: - return - pcts = [100.0 * c / n_texts for c in counts] - n_b = len(labels) - fig_h = max(3.2, min(14.0, 0.38 * n_b + 1.8)) - fig, ax = plt.subplots(figsize=(8.8, fig_h)) - y_pos = range(n_b) - bars = ax.barh(list(y_pos), pcts, color="#2563eb", height=0.65) - ax.set_yticks(list(y_pos)) - ax.set_yticklabels(labels, fontsize=9) - ax.invert_yaxis() - ax.set_title(title, fontsize=12, pad=10) - ax.set_xlabel("占有效评价文本比例（%）", fontsize=9) - xmax = max(pcts) * 1.12 + 4.0 - ax.set_xlim(0, max(xmax, max(pcts) + 10.0, 24.0)) - for bar, c, p in zip(bars, counts, pcts): - ax.text( - min(bar.get_width() + 0.6, ax.get_xlim()[1] * 0.97), - bar.get_y() + bar.get_height() / 2, - f"{int(c)}条 · {p:.1f}%", - va="center", - fontsize=8, - ) - fig.tight_layout() - path = out_dir / fname - fig.savefig(path, dpi=130, bbox_inches="tight") - plt.close(fig) - created.append(fname) - - def save_pie( - labels: list[str], - values: list[float], - title: str, - fname: str, - *, - max_slices: int = 8, - ) -> None: - labs, vals = _merge_tail_as_other(labels, values, max_slices=max_slices) - if not labs or not vals or sum(vals) <= 0: - return - fig, ax = plt.subplots(figsize=(7.2, 5.4)) - colors = plt.cm.Set3(range(len(labs))) - wedges, _t, autotexts = ax.pie( - vals, - labels=None, - autopct=lambda p: f"{p:.1f}%" if p >= 3.5 else "", - pctdistance=0.72, - colors=colors, - startangle=90, - ) - for t in autotexts: - t.set_fontsize(8) - ax.legend( - wedges, - labs, - loc="center left", - bbox_to_anchor=(1.02, 0.5), - fontsize=8, - frameon=False, - ) - ax.set_title(title, fontsize=12, pad=12) - fig.tight_layout() - path = out_dir / fname - fig.savefig(path, dpi=130, bbox_inches="tight") - plt.close(fig) - created.append(fname) - - mix = brief.get("category_mix_top") or [] - labs_m, vals_m = _label_count_pairs(mix) - save_pie( - labs_m, - vals_m, - "类目/可读名称分布（列表行占比）", - "chart_category_mix_pie.png", - ) - save_bar_h( - labs_m[:15], - vals_m[:15], - "类目分布（行数，Top）", - "chart_category_mix.png", - "行数", - ) - - brand_mix = brief.get("list_brand_mix_top") or [] - lb, vb = _label_count_pairs(brand_mix, key_label="label") - save_pie( - lb, - vb, - "品牌列表曝光占比", - "chart_brand_rows_pie.png", - ) - - shop_mix = brief.get("list_shop_mix_top") or [] - ls, vs = _label_count_pairs(shop_mix, key_label="label") - save_pie( - ls, - vs, - "店铺列表曝光占比", - "chart_shop_rows_pie.png", - ) - - def scenario_group_asset_slug(group: str, index: int) -> str: - """与 ``jd_competitor_report._scenario_group_asset_slug`` 保持一致。""" - raw = (group or "").strip() - core = re.sub(r"[^\w\u4e00-\u9fff-]", "", raw)[:20] - if not core: - core = "group" - return f"i{index:02d}_{core}" - - by_grp = brief.get("usage_scenarios_by_matrix_group") or [] - if isinstance(by_grp, list): - for item in by_grp: - if not isinstance(item, dict): - continue - slug = (item.get("chart_slug") or "").strip() - gname = str(item.get("group") or "").strip()[:24] - idx = item.get("matrix_group_index") - if not slug and gname != "" and isinstance(idx, int): - slug = scenario_group_asset_slug(gname, idx) - if not slug: - continue - scen_rows = item.get("scenarios") or [] - n_unit = int(item.get("effective_text_units") or 0) - gpairs: list[tuple[str, float]] = [] - if isinstance(scen_rows, list): - for r in scen_rows: - if not isinstance(r, dict): - continue - lb = str(r.get("scenario") or "").strip()[:48] - c = r.get("count") - if lb and isinstance(c, (int, float)) and c > 0: - gpairs.append((lb, float(c))) - gpairs = _merge_labeled_counts_tail(gpairs, max_items=14) - if gpairs and n_unit > 0: - gl = [p[0] for p in gpairs] - gv = [p[1] for p in gpairs] - title_base = f"「{gname}」· 场景/用途" if gname else "细类 · 场景/用途" - save_bar_h_share_of_text( - gl, - gv, - n_unit, - f"{title_base}（占有效评价文本比例）", - f"chart_usage_scenarios_bar__{slug}.png", - ) - - fb = brief.get("consumer_feedback_by_matrix_group") or [] - if isinstance(fb, list): - for item in fb: - if not isinstance(item, dict): - continue - slug = (item.get("chart_slug") or "").strip() - gname = str(item.get("group") or "").strip()[:24] - idx = item.get("matrix_group_index") - if not slug and gname != "" and isinstance(idx, int): - slug = scenario_group_asset_slug(gname, idx) - if not slug: - continue - hk = item.get("focus_keyword_hits") or [] - wl: list[str] = [] - vl: list[float] = [] - if isinstance(hk, list): - for row in hk[:20]: - if not isinstance(row, dict): - continue - w = str(row.get("word") or "").strip()[:32] - c = row.get("count") - if w and isinstance(c, (int, float)) and c > 0: - wl.append(w) - vl.append(float(c)) - wl = wl[:18] - vl = vl[:18] - if not wl: - continue - tkw = f"「{gname}」· 关注词命中次数" if gname else "细类 · 关注词命中次数" - save_bar_h(wl, vl, tkw, f"chart_focus_keywords_bar__{slug}.png", "命中次数") - - sent = brief.get("comment_sentiment_lexicon") or {} - if isinstance(sent, dict): - pie_labs = ["偏正向", "偏负向", "正负混合", "中性/空"] - pie_vals = [ - float(sent.get("positive_only") or 0), - float(sent.get("negative_only") or 0), - float(sent.get("mixed_positive_and_negative") or 0), - float(sent.get("neutral_or_empty") or 0), - ] - pl = [a for a, b in zip(pie_labs, pie_vals) if b > 0] - pv = [b for b in pie_vals if b > 0] - save_pie(pl, pv, "评价语气四象限占比", "chart_sentiment_overview_pie.png") - save_bar_h( - pl, - pv, - "评价正负面粗判（条数）", - "chart_sentiment.png", - "条数", - ) - - pos_h = sent.get("positive_tone_lexeme_hits") or [] - neg_h = sent.get("negative_tone_lexeme_hits") or [] - plx, pvx = _label_count_pairs( - pos_h, key_label="word", key_count="texts_matched", cap=16 - ) - save_bar_h( - plx, - pvx, - "正向/混合语境 · 正向口语短语命中条数", - "chart_positive_lexemes_bar.png", - "条数", - ) - nlx, nvx = _label_count_pairs( - neg_h, key_label="word", key_count="texts_matched", cap=16 - ) - save_bar_h( - nlx, - nvx, - "负向/混合语境 · 负向口语短语命中条数", - "chart_negative_lexemes_bar.png", - "条数", - ) - - return created diff --git a/backend/pipeline/reporting/__init__.py b/backend/pipeline/reporting/__init__.py new file mode 100644 index 0000000..5128178 --- /dev/null +++ b/backend/pipeline/reporting/__init__.py @@ -0,0 +1 @@ +"""规则报告与简报：统计图、Markdown/Office 导出、策略稿、简报 ZIP。""" diff --git a/backend/pipeline/brief_compact.py b/backend/pipeline/reporting/brief_compact.py similarity index 100% rename from backend/pipeline/brief_compact.py rename to backend/pipeline/reporting/brief_compact.py diff --git a/backend/pipeline/reporting/brief_concentration.py b/backend/pipeline/reporting/brief_concentration.py new file mode 100644 index 0000000..ab9258a --- /dev/null +++ b/backend/pipeline/reporting/brief_concentration.py @@ -0,0 +1,24 @@ +"""竞品简报「集中度」块：对外字段名面向非技术用户，并兼容旧键名。""" +from __future__ import annotations + +from typing import Any + + +def concentration_first_share(block: dict[str, Any] | None) -> Any: + """最大一家占全部相关行的比例（0～1）。新键 ``first_share``，旧键 ``cr1``。""" + if not block: + return None + v = block.get("first_share") + if v is not None: + return v + return block.get("cr1") + + +def concentration_top_three_share(block: dict[str, Any] | None) -> Any: + """前三名合计占全部相关行的比例（0～1）。新键 ``top_three_combined_share``，旧键 ``cr3``。""" + if not block: + return None + v = block.get("top_three_combined_share") + if v is not None: + return v + return block.get("cr3") diff --git a/backend/pipeline/brief_pack.py b/backend/pipeline/reporting/brief_pack.py similarity index 77% rename from backend/pipeline/brief_pack.py rename to backend/pipeline/reporting/brief_pack.py index daf4081..987e37e 100644 --- a/backend/pipeline/brief_pack.py +++ b/backend/pipeline/reporting/brief_pack.py @@ -7,6 +7,11 @@ import zipfile from pathlib import Path from typing import Any +from .brief_concentration import ( + concentration_first_share, + concentration_top_three_share, +) + def _pct(x: Any) -> str: if x is None: @@ -32,7 +37,7 @@ def markdown_summary_from_brief(brief: dict[str, Any]) -> str: lines: list[str] = [ "# 竞品要点摘录（机器整理）", "", - "> 与同批 **完整报告**、**结构化 JSON** 同源；规则汇总，定稿前请人工核对。", + "> 与同批 **完整报告**、**数据汇总**同源；规则汇总，定稿前请人工核对。", "", ] kw = brief.get("keyword") or "—" @@ -65,33 +70,46 @@ def markdown_summary_from_brief(brief: dict[str, Any]) -> str: if raw.get("result_count_consensus") is not None: lines.extend( [ - "## 列表侧检索规模（接口申报）", + "## 列表侧检索规模（平台展示）", "", - f"- **resultCount 共识值**：{_num(raw.get('result_count_consensus'))}", + f"- **检索结果条数（多份响应取一致值）**：{_num(raw.get('result_count_consensus'))}", "", ] ) conc = brief.get("concentration") or {} shops = conc.get("shops_from_list") or {} - if shops.get("cr1") is not None or shops.get("top_label"): + if concentration_first_share(shops) is not None or shops.get("top_label"): lines.extend( [ "## 店铺集中度（列表）", "", - f"- **第一大店铺份额**：{_pct(shops.get('cr1'))}（第一店铺：{shops.get('top_label') or '—'}）", - f"- **前三店铺合计份额**：{_pct(shops.get('cr3'))}", + f"- **第一大店铺份额（按列表行）**：{_pct(concentration_first_share(shops))}（第一店铺：{shops.get('top_label') or '—'}）", + f"- **前三店铺合计份额（按列表行）**：{_pct(concentration_top_three_share(shops))}", "", ] ) + usb = shops.get("unique_sku_basis") + if isinstance(usb, dict) and usb.get("n_unique_skus"): + u1 = usb.get("first_share") + u3 = usb.get("top_three_combined_share") + if u1 is not None: + lines.extend( + [ + f"- **按去重 SKU 计**：共 **{_num(usb.get('n_unique_skus'))}** 个 SKU；" + f"第一大店铺 **{_pct(u1)}**（「{usb.get('top_label') or '—'}」）", + f"- **前三店铺合计（按去重 SKU）**：{_pct(u3)}", + "", + ] + ) dbrand = conc.get("detail_brand_among_merged") or {} - if dbrand.get("cr1") is not None or dbrand.get("top_label"): + if concentration_first_share(dbrand) is not None or dbrand.get("top_label"): lines.extend( [ "## 品牌（深入样本）", "", - f"- **第一大品牌份额（深入样本）**：{_pct(dbrand.get('cr1'))}（头部：{dbrand.get('top_label') or '—'}）", - f"- **前三品牌合计份额**：{_pct(dbrand.get('cr3'))}", + f"- **第一大品牌份额（深入样本）**：{_pct(concentration_first_share(dbrand))}（头部：{dbrand.get('top_label') or '—'}）", + f"- **前三品牌合计份额**：{_pct(concentration_top_three_share(dbrand))}", "", ] ) @@ -103,7 +121,7 @@ def markdown_summary_from_brief(brief: dict[str, Any]) -> str: [ "## 价格（展示价统计）", "", - f"- **样本量 n**：{_num(pst.get('n'))}；**统计口径**：{src}", + f"- **样本量（条）**：{_num(pst.get('n'))}；**价格来源**：{src}", f"- **区间**：{_num(pst.get('min'))} ～ {_num(pst.get('max'))}；**中位数**：{_num(pst.get('median'))}", "", ] diff --git a/backend/pipeline/reporting/brief_strategy_scope.py b/backend/pipeline/reporting/brief_strategy_scope.py new file mode 100644 index 0000000..eefdcb3 --- /dev/null +++ b/backend/pipeline/reporting/brief_strategy_scope.py @@ -0,0 +1,304 @@ +"""按矩阵分组（细类）收窄 competitor brief，使策略生成输入仅含所选分组数据。""" +from __future__ import annotations + +import copy +from collections import Counter +from typing import Any + +from pipeline.competitor_report.csv_io import _collect_prices +from pipeline.competitor_report.price_stats import _price_stats_extended + + +def list_matrix_groups_for_api(brief: dict[str, Any]) -> list[dict[str, Any]]: + """供前端下拉：矩阵分组名称与索引（与 ``matrix_by_group`` 顺序一致）。""" + mg = brief.get("matrix_by_group") + if not isinstance(mg, list): + return [] + out: list[dict[str, Any]] = [] + for i, g in enumerate(mg): + if not isinstance(g, dict): + continue + name = (g.get("group") or "").strip() or "—" + skus = g.get("skus") if isinstance(g.get("skus"), list) else [] + out.append( + { + "index": i, + "group": name, + "sku_count": int(g.get("sku_count") or len(skus)), + } + ) + return out + + +def resolve_strategy_matrix_group_index( + brief: dict[str, Any], + *, + matrix_group_index: int | None = None, + matrix_group_label: str | None = None, +) -> tuple[int | None, str | None]: + """ + 解析请求中的矩阵分组。 + + 返回 ``(index, error)``： + - ``index is None`` 且 ``error is None``：未指定收窄（使用完整 brief）； + - ``index`` 为 ``int``：有效分组下标； + - ``error`` 非空：参数与 brief 不一致。 + """ + mg = brief.get("matrix_by_group") + if not isinstance(mg, list) or not mg: + if matrix_group_index is not None or ( + matrix_group_label and str(matrix_group_label).strip() + ): + return None, "当前任务 brief 中无 matrix_by_group，无法按细类收窄" + return None, None + + label = (matrix_group_label or "").strip() + has_idx = matrix_group_index is not None + has_lbl = bool(label) + + if not has_idx and not has_lbl: + return None, None + + if has_idx: + idx = int(matrix_group_index) + if idx < 0 or idx >= len(mg): + return None, f"strategy_matrix_group_index 须在 0～{len(mg) - 1} 之间" + + if has_lbl and not has_idx: + for i, g in enumerate(mg): + if not isinstance(g, dict): + continue + if (g.get("group") or "").strip() == label: + return i, None + return None, f"未找到矩阵分组「{label}」" + + assert has_idx + idx = int(matrix_group_index) + g0 = mg[idx] + gname = (g0.get("group") or "").strip() if isinstance(g0, dict) else "" + if has_lbl and gname != label: + return None, ( + f"strategy_matrix_group_index={idx} 对应分组「{gname}」，" + f"与 strategy_matrix_group「{label}」不一致" + ) + return idx, None + + +def _sku_rows_for_prices(skus: list[dict[str, Any]]) -> list[dict[str, str]]: + out: list[dict[str, str]] = [] + for s in skus: + if not isinstance(s, dict): + continue + out.append({k: str(v) if v is not None else "" for k, v in s.items()}) + return out + + +def _mix_top_rows( + skus: list[dict[str, Any]], key: str, *, top_n: int = 24 +) -> list[dict[str, Any]]: + c: Counter[str] = Counter() + for s in skus: + if not isinstance(s, dict): + continue + v = (s.get(key) or "").strip() or "（未标注）" + c[v] += 1 + rows: list[dict[str, Any]] = [] + for lab, cnt in c.most_common(top_n): + rows.append({"label": lab, "count": cnt}) + remainder = sum(cnt for _, cnt in c.most_common()[top_n:]) + if remainder > 0: + rows.append({"label": "（其余）", "count": remainder}) + return rows + + +def _category_mix_from_skus(skus: list[dict[str, Any]]) -> list[dict[str, Any]]: + c: Counter[str] = Counter() + for s in skus: + if not isinstance(s, dict): + continue + cat = (s.get("category") or "").strip() + if not cat: + continue + parts = [p.strip() for p in cat.split(">") if p.strip()] + leaf = parts[-1] if parts else cat + c[leaf] += 1 + return [{"label": k, "count": v} for k, v in c.most_common(24)] + + +def _concentration_brand_shop_from_skus(skus: list[dict[str, Any]]) -> dict[str, Any]: + """用矩阵内 SKU 粗算集中度（与全站列表口径不同，仅供分组内对照）。""" + brands = [ + (s.get("brand") or "").strip() + for s in skus + if isinstance(s, dict) and (s.get("brand") or "").strip() + ] + shops = [ + (s.get("shop") or "").strip() + for s in skus + if isinstance(s, dict) and (s.get("shop") or "").strip() + ] + + def _block(labels: list[str]) -> dict[str, Any]: + if not labels: + return { + "first_share": 0.0, + "top_three_combined_share": 0.0, + "top_label": "—", + "top_share_pct": "0%", + } + ct = Counter(labels) + n = len(labels) + top_lab, top_n = ct.most_common(1)[0] + top3 = sum(x for _, x in ct.most_common(3)) + return { + "first_share": top_n / n, + "top_three_combined_share": top3 / n, + "top_label": top_lab, + "top_share_pct": f"{100.0 * top_n / n:.1f}%", + } + + return { + "shops_from_list": _block(shops), + "list_brand_field": None, + "detail_brand_among_merged": _block(brands), + } + + +def filter_brief_for_strategy_matrix_group( + brief: dict[str, Any], + *, + matrix_group_index: int, +) -> dict[str, Any]: + """ + 深拷贝 ``brief``，仅保留 ``matrix_by_group[matrix_group_index]`` 及其对齐的 + ``consumer_feedback_by_matrix_group`` / ``usage_scenarios_by_matrix_group``， + 并重算与样本相关的价盘、类目与集中度（避免仍混入全关键词列表口径）。 + """ + b = copy.deepcopy(brief) + mg = b.get("matrix_by_group") + if not isinstance(mg, list) or matrix_group_index < 0 or matrix_group_index >= len( + mg + ): + return b + + chosen = mg[matrix_group_index] + if not isinstance(chosen, dict): + return b + + gname = (chosen.get("group") or "").strip() or "—" + skus = chosen.get("skus") if isinstance(chosen.get("skus"), list) else [] + n_skus = len(skus) + + b["matrix_by_group"] = [ + { + "group": gname, + "sku_count": int(chosen.get("sku_count") or n_skus), + "skus": skus, + } + ] + + def _pick_by_group( + items: Any, + ) -> list[dict[str, Any]]: + if not isinstance(items, list): + return [] + out: list[dict[str, Any]] = [] + for it in items: + if not isinstance(it, dict): + continue + if (it.get("group") or "").strip() == gname: + out.append(dict(it)) + break + gi = it.get("matrix_group_index") + if gi is not None and int(gi) == matrix_group_index: + out.append(dict(it)) + break + return out + + fb = b.get("consumer_feedback_by_matrix_group") + fb_one = _pick_by_group(fb) + b["consumer_feedback_by_matrix_group"] = fb_one + + ub = b.get("usage_scenarios_by_matrix_group") + b["usage_scenarios_by_matrix_group"] = _pick_by_group(ub) + + if fb_one: + f0 = fb_one[0] + b["comment_focus_keywords"] = list(f0.get("focus_keyword_hits") or []) + scenarios_top = f0.get("scenarios_top") or [] + b["usage_scenarios"] = list(scenarios_top) + denom = int(f0.get("effective_comment_text_units") or 0) + if denom <= 0: + denom = int(f0.get("comment_rows") or 0) + b["usage_scenarios_denominator"] = denom + else: + b["comment_focus_keywords"] = [] + b["usage_scenarios"] = [] + b["usage_scenarios_denominator"] = 0 + + rows_for_price = _sku_rows_for_prices( + [s for s in skus if isinstance(s, dict)] + ) + prices = _collect_prices(rows_for_price) + pst_merged = _price_stats_extended(prices) + b["price_stats_merged_sample"] = pst_merged + b["price_stats"] = dict(pst_merged) if pst_merged else {} + b["price_stats_source"] = "strategy_scope_matrix_group_skus" + b["price_stats_list_export"] = {} + + b["category_mix_top"] = _category_mix_from_skus( + [s for s in skus if isinstance(s, dict)] + ) + b["list_brand_mix_top"] = _mix_top_rows( + [s for s in skus if isinstance(s, dict)], "brand" + ) + b["list_shop_mix_top"] = _mix_top_rows( + [s for s in skus if isinstance(s, dict)], "shop" + ) + + b["concentration"] = _concentration_brand_shop_from_skus( + [s for s in skus if isinstance(s, dict)] + ) + + sc = b.get("scope") + if isinstance(sc, dict): + sc2 = dict(sc) + sc2["merged_sku_count"] = n_skus + if fb_one: + sc2["comment_flat_rows"] = int(fb_one[0].get("comment_rows") or 0) + b["scope"] = sc2 + + b["strategy_scope_applied"] = { + "matrix_group_index": matrix_group_index, + "group": gname, + "original_matrix_group_count": len(mg), + } + + notes = b.get("notes") + extra = ( + "策略生成已按矩阵分组收窄：下文统计与矩阵仅针对「" + + gname + + "」内 SKU；与全关键词搜索列表、全样本评价总量不同口径。" + ) + if isinstance(notes, list): + b["notes"] = [extra] + [n for n in notes if isinstance(n, str)] + else: + b["notes"] = [extra] + + b["price_promotion_signals"] = [] + b["strategy_hints"] = [] + + b["list_visibility_proxy"] = { + "total_rows": n_skus, + "unique_skus": n_skus, + "_strategy_scope_note": "矩阵所选分组内 SKU 数，非全关键词列表导出口径。", + } + + return b + + +__all__ = [ + "filter_brief_for_strategy_matrix_group", + "list_matrix_groups_for_api", + "resolve_strategy_matrix_group_index", +] diff --git a/backend/pipeline/reporting/charts.py b/backend/pipeline/reporting/charts.py new file mode 100644 index 0000000..561b2a3 --- /dev/null +++ b/backend/pipeline/reporting/charts.py @@ -0,0 +1,761 @@ +"""根据结构化 brief 生成报告用 PNG 统计图（matplotlib），写入 ``run_dir/report_assets/``。""" + +from __future__ import annotations + +import math +import os +import re +from pathlib import Path +from typing import Any + + +def _setup_matplotlib_cjk() -> None: + import matplotlib + + matplotlib.use("Agg") + import matplotlib.pyplot as plt + from matplotlib import font_manager + + windir = os.environ.get("WINDIR", r"C:\Windows") + for name in ("simhei.ttf", "msyh.ttc", "simsun.ttc"): + fp = Path(windir) / "Fonts" / name + if fp.is_file(): + try: + font_manager.fontManager.addfont(str(fp)) + fam = font_manager.FontProperties(fname=str(fp)).get_name() + plt.rcParams["font.family"] = [fam] + break + except Exception: + continue + plt.rcParams["axes.unicode_minus"] = False + + +def _label_count_pairs( + items: Any, + *, + key_label: str = "label", + key_count: str = "count", + cap: int = 40, +) -> tuple[list[str], list[float]]: + labs: list[str] = [] + vals: list[float] = [] + if not isinstance(items, list): + return labs, vals + for item in items[:cap]: + if not isinstance(item, dict): + continue + lbl = str(item.get(key_label) or "").strip()[:48] + cnt = item.get(key_count) + if lbl and isinstance(cnt, (int, float)) and cnt > 0: + labs.append(lbl) + vals.append(float(cnt)) + return labs, vals + + +def _merge_labeled_counts_tail( + pairs: list[tuple[str, float]], *, max_items: int +) -> list[tuple[str, float]]: + if len(pairs) <= max_items: + return pairs + head = pairs[: max_items - 1] + rest = sum(c for _, c in pairs[max_items - 1 :]) + if rest > 0: + head.append(("其他", rest)) + return head + + +def _float_price_from_cell(s: str) -> float | None: + t = (s or "").strip().replace(",", "").replace("，", "") + if not t: + return None + m = re.search(r"(\d+(?:\.\d+)?)", t) + if not m: + return None + try: + v = float(m.group(1)) + except ValueError: + return None + if 0 < v < 1_000_000: + return v + return None + + +def _cn_volume_int(s: str) -> int: + """ + 从搜索侧文案抽取非负整数（评价量/销量等）：支持「亿」「万」及纯数字； + 如 ``已售50万+ | good:99%好评`` → 500000。 + """ + t = (s or "").strip().replace(",", "").replace("，", "") + if not t: + return 0 + m = re.search(r"(\d+(?:\.\d+)?)\s*亿", t) + if m: + return int(round(float(m.group(1)) * 100_000_000)) + m = re.search(r"(\d+(?:\.\d+)?)\s*万", t) + if m: + return int(round(float(m.group(1)) * 10_000)) + m2 = re.search(r"(\d+)", t) + if m2: + return int(m2.group(1)) + return 0 + + +def _format_xaxis_int_cn(x: float, _pos: int | None) -> str: + """ + 横轴大整数刻度：用「万」「亿」表述，避免 matplotlib 默认 ``1e6`` 科学计数法。 + 用于销量、评价量、条数等非负计数。 + """ + if not math.isfinite(x): + return "" + if abs(x) < 1e-9: + return "0" + ax = abs(x) + sign = "-" if x < 0 else "" + if ax < 10_000: + return sign + str(int(round(ax))) + if ax < 100_000_000: + wan = ax / 10_000.0 + if wan >= 1000: + return sign + f"{wan:.0f}万" + if wan >= 100: + return sign + f"{wan:.0f}万" + if abs(wan - round(wan)) < 1e-6: + return sign + f"{int(round(wan))}万" + s = f"{wan:.1f}".rstrip("0").rstrip(".") + return sign + s + "万" + yi = ax / 100_000_000.0 + if abs(yi - round(yi)) < 1e-6: + return sign + f"{int(round(yi))}亿" + s = f"{yi:.2f}".rstrip("0").rstrip(".") + return sign + s + "亿" + + +def _merge_tail_as_other( + labels: list[str], values: list[float], *, max_slices: int +) -> tuple[list[str], list[float]]: + pairs = [(l, v) for l, v in zip(labels, values) if v > 0] + if not pairs: + return [], [] + if len(pairs) <= max_slices: + return [p[0] for p in pairs], [p[1] for p in pairs] + head = pairs[: max_slices - 1] + rest = sum(v for _, v in pairs[max_slices - 1 :]) + labs = [p[0] for p in head] + vals = [p[1] for p in head] + if rest > 0: + labs.append("其他") + vals.append(rest) + return labs, vals + + +# 已不再写入报告正文的旧版图，避免 run_dir 里残留误导性 PNG +# 横向条形图：统一柱厚、柱端数值字号（全文件条形图共用） +_BARH_HEIGHT = 0.6 +_BAR_VALUE_FONTSIZE = 8 + + +def _thin_barh_height(n: int) -> float: + """ + 横向条形图：类目条数 n 较少时降低 barh 的 height（与 y 轴跨度同量纲）。 + n=1 时若仍用 0.6 且 ylim 跨度仅 1，单条会占满大半幅、视觉上极粗。 + """ + if n <= 0: + return _BARH_HEIGHT + if n == 1: + return 0.30 + if n == 2: + return 0.44 + if n <= 5: + return 0.50 + if n <= 10: + return 0.54 + return _BARH_HEIGHT + + +def _set_barh_category_ylim(ax: Any, n: int) -> None: + """n 条类目横条时设置纵轴范围；n=1 时略放宽，使柱相对更细。""" + if n <= 0: + return + if n == 1: + ax.set_ylim(-1.0, 1.0) + else: + ax.set_ylim(-0.5, float(n) - 0.5) + + +def _fmt_bar_value(v: float, *, as_int: bool = False) -> str: + if as_int or (math.isfinite(v) and abs(v - round(v)) < 1e-6): + return str(int(round(v))) + s = f"{v:.2f}".rstrip("0").rstrip(".") + return s if s else "0" + + +def _annotate_barh_numeric( + ax: Any, + bars: Any, + values: list[float], + *, + as_int: bool = False, + x_pad_ratio: float = 0.02, +) -> None: + """在横向柱末端标注数值；调用前请已设置合适的 xlim。""" + if not bars or not values: + return + x1 = ax.get_xlim()[1] + if x1 <= 0: + return + pad = max(x1 * x_pad_ratio, 0.02 * max(values) if values else 0.1) + for bar, v in zip(bars, values): + if v is None or not math.isfinite(float(v)) or float(v) <= 0: + continue + w = bar.get_width() + ax.text( + w + pad, + bar.get_y() + bar.get_height() / 2, + _fmt_bar_value(float(v), as_int=as_int), + va="center", + fontsize=_BAR_VALUE_FONTSIZE, + ) + + +_OBSOLETE_REPORT_ASSETS: frozenset[str] = frozenset( + { + "chart_focus_keywords_bar.png", + "chart_usage_scenarios.png", + "chart_usage_scenarios_pie.png", + "chart_focus_keywords_pie.png", + "chart_comment_focus_global_bar.png", + "chart_usage_scenarios_global_bar.png", + "chart_sentiment_overview_pie.png", + "chart_sentiment.png", + "chart_positive_lexemes_bar.png", + "chart_negative_lexemes_bar.png", + } +) + + +def _cleanup_obsolete_report_assets(out_dir: Path) -> None: + """删除历史版本生成的、当前报告不再引用的插图文件。""" + if not out_dir.is_dir(): + return + for name in _OBSOLETE_REPORT_ASSETS: + fp = out_dir / name + if fp.is_file(): + try: + fp.unlink() + except OSError: + pass + for fp in out_dir.glob("chart_usage_scenarios_pie__*.png"): + try: + fp.unlink() + except OSError: + pass + for pat in ( + "chart_focus_keywords_bar__*.png", + "chart_usage_scenarios_bar__*.png", + ): + for fp in out_dir.glob(pat): + try: + fp.unlink() + except OSError: + pass + + +def generate_report_charts( + run_dir: Path, + brief: dict[str, Any], + *, + report_config: dict[str, Any] | None = None, +) -> list[str]: + """生成扇形/条形 PNG。返回已写入的文件名列表（不含路径）。 + + 若 ``report_config["chapter8_text_mining_probe"]`` 为真，**不**生成 ``chart_focus_and_scenarios_bar__*.png`` + （与竞品报告 §8.2 文本挖掘探针互斥，避免无效产出）。 + """ + _setup_matplotlib_cjk() + import matplotlib.pyplot as plt + from matplotlib.ticker import FuncFormatter + + out_dir = Path(run_dir).resolve() / "report_assets" + out_dir.mkdir(parents=True, exist_ok=True) + _cleanup_obsolete_report_assets(out_dir) + created: list[str] = [] + + def save_bar_h( + labels: list[str], + values: list[float], + title: str, + fname: str, + xlabel: str = "", + ) -> None: + if not labels or not values or max(values) <= 0: + return + n = len(labels) + fig_h = max(3.2, min(14.0, 0.38 * n + 1.5)) + fig, ax = plt.subplots(figsize=(8.2, fig_h)) + y_pos = range(n) + bh = _thin_barh_height(n) + bars = ax.barh( + list(y_pos), values, color="#2563eb", height=bh + ) + ax.set_yticks(list(y_pos)) + ax.set_yticklabels(labels, fontsize=9) + ax.invert_yaxis() + _set_barh_category_ylim(ax, n) + ax.set_title(title, fontsize=12, pad=10) + if xlabel: + ax.set_xlabel(xlabel, fontsize=9) + ax.tick_params(axis="y", left=True, right=False, labelleft=True, labelright=False) + vmax = max(values) + ax.set_xlim(0, vmax * 1.14 + max(0.08 * vmax, 0.5)) + _annotate_barh_numeric(ax, bars, list(values), as_int=True) + fig.tight_layout() + path = out_dir / fname + fig.savefig(path, dpi=130, bbox_inches="tight") + plt.close(fig) + created.append(fname) + + def save_bar_h_share_of_text( + labels: list[str], + counts: list[float], + n_texts: int, + title: str, + fname: str, + ) -> None: + """ + 横轴 = count / n_texts * 100，与报告表格「占有效文本比例」一致（多标签下各柱比例可相加 >100%）。 + """ + if not labels or not counts or n_texts <= 0 or max(counts) <= 0: + return + pcts = [100.0 * c / n_texts for c in counts] + n_b = len(labels) + fig_h = max(3.2, min(14.0, 0.38 * n_b + 1.8)) + fig, ax = plt.subplots(figsize=(8.8, fig_h)) + y_pos = range(n_b) + bh = _thin_barh_height(n_b) + bars = ax.barh(list(y_pos), pcts, color="#2563eb", height=bh) + ax.set_yticks(list(y_pos)) + ax.set_yticklabels(labels, fontsize=9) + ax.invert_yaxis() + _set_barh_category_ylim(ax, n_b) + ax.set_title(title, fontsize=12, pad=10) + ax.set_xlabel("占有效评价文本比例（%）", fontsize=9) + ax.tick_params(axis="y", left=True, right=False, labelleft=True, labelright=False) + xmax = max(pcts) * 1.12 + 4.0 + ax.set_xlim(0, max(xmax, max(pcts) + 10.0, 24.0)) + x1 = ax.get_xlim()[1] + pad = max(x1 * 0.015, 0.35) + for bar, c, p in zip(bars, counts, pcts): + ax.text( + min(bar.get_width() + pad, x1 * 0.985), + bar.get_y() + bar.get_height() / 2, + f"{int(c)}条 · {p:.1f}%", + va="center", + fontsize=_BAR_VALUE_FONTSIZE, + ) + fig.tight_layout() + path = out_dir / fname + fig.savefig(path, dpi=130, bbox_inches="tight") + plt.close(fig) + created.append(fname) + + def save_combo_focus_scenario_bar( + *, + gname: str, + slug: str, + wl: list[str], + vl: list[float], + gl: list[str], + gv: list[float], + n_texts: int, + ) -> None: + """左：关注词命中次数；右：场景占有效文本 %。两侧 **各自独立 Y 轴**（类目互不混用）。""" + has_l = bool(wl and vl and max(vl) > 0) + has_r = bool(gl and gv and n_texts > 0 and max(gv) > 0) + if not has_l and not has_r: + return + n_l = len(wl) if has_l else 0 + n_r = len(gl) if has_r else 0 + n_ref = max(n_l, n_r, 1) + fig_h = max(3.4, min(14.0, 0.38 * n_ref + 2.8)) + fig = plt.figure(figsize=(10.8, fig_h)) + ttl = (gname or "").strip()[:22] or "细类" + fig.suptitle( + f"「{ttl}」· 关注词与使用场景", + fontsize=11, + y=0.98, + ) + # 底对齐、高度按各自类目数比例分配，避免共用一个「拉伸后的」纵轴比例尺 + base_bottom = 0.10 + ax_w = 0.36 + x_gap = 0.06 + x_l = 0.07 + x_r = x_l + ax_w + x_gap + max_h = 0.72 + n_den = max(n_ref, 1) + h_l = max(0.26, max_h * (max(n_l, 1) / n_den)) if has_l else max(0.26, max_h * 0.35) + h_r = max(0.26, max_h * (max(n_r, 1) / n_den)) if has_r else max(0.26, max_h * 0.35) + ax_l = fig.add_axes([x_l, base_bottom, ax_w, h_l]) + ax_r = fig.add_axes([x_r, base_bottom, ax_w, h_r]) + if has_l: + y_pos = list(range(n_l)) + bh_l = _thin_barh_height(n_l) + bars_l = ax_l.barh( + y_pos, vl[:n_l], color="#2563eb", height=bh_l + ) + ax_l.set_yticks(y_pos) + ax_l.set_yticklabels(wl[:n_l], fontsize=8) + ax_l.invert_yaxis() + _set_barh_category_ylim(ax_l, n_l) + ax_l.set_xlabel("关注词子串命中次数", fontsize=9) + ax_l.set_title("关注词", fontsize=10, pad=6) + ax_l.tick_params(axis="y", left=True, right=False, labelleft=True, labelright=False) + vmax_l = max(vl[:n_l]) + ax_l.set_xlim(0, vmax_l * 1.14 + max(0.5, 0.08 * vmax_l)) + _annotate_barh_numeric( + ax_l, bars_l, list(vl[:n_l]), as_int=True + ) + else: + ax_l.text( + 0.5, + 0.5, + "本细类无关注词命中\n或无数文本", + ha="center", + va="center", + transform=ax_l.transAxes, + fontsize=10, + color="#64748b", + ) + ax_l.set_axis_off() + if has_r: + pcts = [100.0 * c / n_texts for c in gv[: len(gl)]] + n_b = len(gl) + y_pos = list(range(n_b)) + bh_r = _thin_barh_height(n_b) + bars = ax_r.barh(y_pos, pcts, color="#059669", height=bh_r) + ax_r.set_yticks(y_pos) + ax_r.set_yticklabels(gl[:n_b], fontsize=8) + ax_r.invert_yaxis() + _set_barh_category_ylim(ax_r, n_b) + ax_r.set_xlabel("占有效评价文本比例（%）", fontsize=9) + if pcts: + xmax = max(pcts) * 1.12 + 4.0 + ax_r.set_xlim(0, max(xmax, max(pcts) + 10.0, 24.0)) + else: + ax_r.set_xlim(0, 24.0) + x1r = ax_r.get_xlim()[1] + pad_r = max(x1r * 0.015, 0.35) + for bar, c, p in zip(bars, gv[:n_b], pcts): + ax_r.text( + min(bar.get_width() + pad_r, x1r * 0.985), + bar.get_y() + bar.get_height() / 2, + f"{int(c)}条 · {p:.1f}%", + va="center", + fontsize=_BAR_VALUE_FONTSIZE, + ) + ax_r.set_title("使用场景", fontsize=10, pad=6) + ax_r.yaxis.tick_left() + ax_r.yaxis.set_label_position("left") + ax_r.tick_params(axis="y", left=True, right=False, labelleft=True, labelright=False) + else: + ax_r.text( + 0.5, + 0.5, + "本细类无场景词命中\n或无数文本", + ha="center", + va="center", + transform=ax_r.transAxes, + fontsize=10, + color="#64748b", + ) + ax_r.set_axis_off() + path = out_dir / f"chart_focus_and_scenarios_bar__{slug}.png" + fig.savefig(path, dpi=130, bbox_inches="tight") + plt.close(fig) + created.append(path.name) + + def save_pie( + labels: list[str], + values: list[float], + title: str, + fname: str, + *, + max_slices: int = 8, + figsize: tuple[float, float] | None = None, + dpi: int = 120, + ) -> None: + labs, vals = _merge_tail_as_other(labels, values, max_slices=max_slices) + if not labs or not vals or sum(vals) <= 0: + return + fs = figsize if figsize is not None else (4.9, 3.65) + fig, ax = plt.subplots(figsize=fs) + colors = plt.cm.Set3(range(len(labs))) + wedges, _t, autotexts = ax.pie( + vals, + labels=None, + autopct=lambda p: f"{p:.1f}%" if p >= 3.5 else "", + pctdistance=0.72, + colors=colors, + startangle=90, + ) + for t in autotexts: + t.set_fontsize(8) + ax.legend( + wedges, + labs, + loc="center left", + bbox_to_anchor=(1.02, 0.5), + fontsize=8, + frameon=False, + ) + ax.set_title(title, fontsize=12, pad=12) + fig.tight_layout() + path = out_dir / fname + fig.savefig(path, dpi=dpi, bbox_inches="tight") + plt.close(fig) + created.append(fname) + + mix = brief.get("category_mix_top") or [] + labs_m, vals_m = _label_count_pairs(mix) + save_pie( + labs_m, + vals_m, + "细类分布（合并表 SKU）", + "chart_category_mix_pie.png", + ) + save_bar_h( + labs_m[:15], + vals_m[:15], + "细类分布（合并表 SKU 数，Top）", + "chart_category_mix.png", + "SKU 数", + ) + + brand_mix = brief.get("list_brand_mix_top") or [] + lb, vb = _label_count_pairs(brand_mix, key_label="label") + save_pie( + lb, + vb, + "品牌列表曝光占比", + "chart_brand_rows_pie.png", + ) + + shop_mix = brief.get("list_shop_mix_top") or [] + ls, vs = _label_count_pairs(shop_mix, key_label="label") + save_pie( + ls, + vs, + "店铺列表曝光占比", + "chart_shop_rows_pie.png", + ) + + def scenario_group_asset_slug(group: str, index: int) -> str: + """与 ``pipeline.competitor_report.jd_report`` / ``report_md_helpers._scenario_group_asset_slug`` 保持一致。""" + raw = (group or "").strip() + core = re.sub(r"[^\w\u4e00-\u9fff-]", "", raw)[:20] + if not core: + core = "group" + return f"i{index:02d}_{core}" + + _skip_focus_scenario_combo = bool( + isinstance(report_config, dict) + and report_config.get("chapter8_text_mining_probe") + ) + if _skip_focus_scenario_combo: + for fp in out_dir.glob("chart_focus_and_scenarios_bar__*.png"): + try: + fp.unlink() + except OSError: + pass + if not _skip_focus_scenario_combo: + scen_by_slug: dict[str, tuple[list[str], list[float], int]] = {} + by_grp = brief.get("usage_scenarios_by_matrix_group") or [] + if isinstance(by_grp, list): + for item in by_grp: + if not isinstance(item, dict): + continue + slug = (item.get("chart_slug") or "").strip() + gname = str(item.get("group") or "").strip()[:24] + idx = item.get("matrix_group_index") + if not slug and gname != "" and isinstance(idx, int): + slug = scenario_group_asset_slug(gname, idx) + if not slug: + continue + scen_rows = item.get("scenarios") or [] + n_unit = int(item.get("effective_text_units") or 0) + gpairs: list[tuple[str, float]] = [] + if isinstance(scen_rows, list): + for r in scen_rows: + if not isinstance(r, dict): + continue + lb = str(r.get("scenario") or "").strip()[:48] + c = r.get("count") + if lb and isinstance(c, (int, float)) and c > 0: + gpairs.append((lb, float(c))) + gpairs = _merge_labeled_counts_tail(gpairs, max_items=14) + if gpairs and n_unit > 0: + scen_by_slug[slug] = ( + [p[0] for p in gpairs], + [p[1] for p in gpairs], + n_unit, + ) + + fb = brief.get("consumer_feedback_by_matrix_group") or [] + if isinstance(fb, list): + for item in fb: + if not isinstance(item, dict): + continue + slug = (item.get("chart_slug") or "").strip() + gname = str(item.get("group") or "").strip()[:24] + idx = item.get("matrix_group_index") + if not slug and gname != "" and isinstance(idx, int): + slug = scenario_group_asset_slug(gname, idx) + if not slug: + continue + hk = item.get("focus_keyword_hits") or [] + wl: list[str] = [] + vl: list[float] = [] + if isinstance(hk, list): + for row in hk[:20]: + if not isinstance(row, dict): + continue + w = str(row.get("word") or "").strip()[:32] + c = row.get("count") + if w and isinstance(c, (int, float)) and c > 0: + wl.append(w) + vl.append(float(c)) + wl = wl[:18] + vl = vl[:18] + gl, gv, n_scen = scen_by_slug.get(slug, ([], [], 0)) + n_unit_fb = int(item.get("effective_comment_text_units") or 0) + n_texts = n_scen if n_scen > 0 else n_unit_fb + save_combo_focus_scenario_bar( + gname=gname, + slug=slug, + wl=wl, + vl=vl, + gl=gl, + gv=gv, + n_texts=n_texts, + ) + + matrix_groups = brief.get("matrix_by_group") or [] + if isinstance(matrix_groups, list): + for gi, block in enumerate(matrix_groups): + if not isinstance(block, dict): + continue + gname = str(block.get("group") or "").strip() + skus = block.get("skus") or [] + if not isinstance(skus, list) or not skus: + continue + slug = scenario_group_asset_slug(gname, gi) + rows_data: list[tuple[str, float | None, int]] = [] + for s in skus: + if not isinstance(s, dict): + continue + title = str(s.get("title") or "").strip() + sku = str(s.get("sku_id") or "").strip() + # 与 §5 矩阵「产品」列一致：纵轴优先品名，无标题时再退化为 SKU + if title: + label = title if len(title) <= 48 else title[:46] + "…" + elif sku: + label = sku if len(sku) <= 22 else sku[:20] + "…" + else: + label = "?" + p: float | None = None + for k in ( + "detail_price_final", + "list_price_show", + "coupon_or_detail_price", + ): + p = _float_price_from_cell(str(s.get(k) or "")) + if p is not None: + break + sales = _cn_volume_int(str(s.get("total_sales") or "")) + rows_data.append((label, p, sales)) + rows_data.sort(key=lambda x: x[0]) + if not rows_data: + continue + if not any( + (pr is not None and pr > 0) or sv > 0 + for _, pr, sv in rows_data + ): + continue + n = len(rows_data) + labels_mx = [x[0] for x in rows_data] + prices_mx = [x[1] for x in rows_data] + sales_mx = [x[2] for x in rows_data] + y_pos = list(range(n)) + fig_h = max(3.4, min(14.0, 0.38 * n + 2.4)) + fig, (ax_l, ax_r) = plt.subplots( + 1, 2, figsize=(10.6, fig_h), sharey=True + ) + bh_mx = _thin_barh_height(n) + price_w = [ + float(pr) + if pr is not None and pr > 0 and math.isfinite(pr) + else 0.0 + for pr in prices_mx + ] + bars_pl = ax_l.barh( + y_pos, price_w, height=bh_mx, color="#2563eb" + ) + ax_l.set_yticks(y_pos) + ax_l.set_yticklabels(labels_mx, fontsize=8) + ax_l.invert_yaxis() + _set_barh_category_ylim(ax_l, n) + ax_l.set_xlabel("展示价（元）", fontsize=9) + ax_l.set_title("展示价", fontsize=10, pad=8) + ax_l.tick_params(axis="y", left=True, right=False, labelleft=True, labelright=False) + pmax = max(price_w) if price_w else 0.0 + if pmax > 0: + ax_l.set_xlim(0, pmax * 1.12 + max(0.08 * pmax, 0.5)) + else: + ax_l.set_xlim(0, 1) + pad_p = max(ax_l.get_xlim()[1] * 0.012, 0.08) + for bar, pr in zip(bars_pl, prices_mx): + if pr is not None and pr > 0 and math.isfinite(pr): + ax_l.text( + bar.get_width() + pad_p, + bar.get_y() + bar.get_height() / 2, + _fmt_bar_value(float(pr), as_int=False), + va="center", + fontsize=_BAR_VALUE_FONTSIZE, + ) + sales_f = [float(s) for s in sales_mx] + bars_sr = ax_r.barh( + y_pos, sales_f, height=bh_mx, color="#059669" + ) + ax_r.set_xlabel("销量", fontsize=9) + ax_r.set_title("销量", fontsize=10, pad=8) + ax_r.xaxis.set_major_formatter( + FuncFormatter(_format_xaxis_int_cn) + ) + ax_r.tick_params(axis="y", left=False, labelleft=False) + smax = max(sales_f) if sales_f else 0.0 + if smax > 0: + ax_r.set_xlim(0, smax * 1.1 + max(0.04 * smax, smax * 0.02)) + else: + ax_r.set_xlim(0, 1) + pad_s = max(ax_r.get_xlim()[1] * 0.008, smax * 0.01 if smax else 0.1) + for bar, sv in zip(bars_sr, sales_mx): + if sv > 0: + ax_r.text( + bar.get_width() + pad_s, + bar.get_y() + bar.get_height() / 2, + _format_xaxis_int_cn(float(sv), None), + va="center", + fontsize=_BAR_VALUE_FONTSIZE, + ) + ttl = gname[:22] if gname else "细类" + fig.suptitle( + f"「{ttl}」· 竞品矩阵：价格与销量", + fontsize=11, + y=1.01, + ) + fig.tight_layout() + out_mx = out_dir / f"chart_matrix_prices_sales__{slug}.png" + fig.savefig(out_mx, dpi=130, bbox_inches="tight") + plt.close(fig) + created.append(out_mx.name) + + return created diff --git a/backend/pipeline/reporting/marketing_pack_persist.py b/backend/pipeline/reporting/marketing_pack_persist.py new file mode 100644 index 0000000..f68fc5d --- /dev/null +++ b/backend/pipeline/reporting/marketing_pack_persist.py @@ -0,0 +1,23 @@ +"""营销内容包等产物落盘（任务 run_dir，便于归档与审计）。""" +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + + +def persist_marketing_detail_pack_v1(run_dir: str | None, payload: dict[str, Any]) -> Path | None: + """ + 写入 ``{run_dir}/marketing/marketing_detail_pack_v1.json``。 + ``payload`` 建议与 API 响应体一致（含 schema_version、job_id、core_info_card 等）。 + """ + if not run_dir or not str(run_dir).strip(): + return None + root = Path(run_dir) + if not root.is_dir(): + return None + out_dir = root / "marketing" + out_dir.mkdir(parents=True, exist_ok=True) + path = out_dir / "marketing_detail_pack_v1.json" + path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8") + return path diff --git a/backend/pipeline/reporting/md_document_export.py b/backend/pipeline/reporting/md_document_export.py new file mode 100644 index 0000000..1e408da --- /dev/null +++ b/backend/pipeline/reporting/md_document_export.py @@ -0,0 +1,459 @@ +"""Markdown → Word（.docx）/ 简易 PDF；供任务报告与策略稿导出。""" +from __future__ import annotations + +import os +import re +from io import BytesIO +from pathlib import Path +from typing import Any +from xml.sax.saxutils import escape as xml_escape + + +def _strip_inline_md(s: str) -> str: + s = re.sub(r"\*\*(.+?)\*\*", r"\1", s) + s = re.sub(r"`([^`]+)`", r"\1", s) + return s + + +def _is_table_sep(line: str) -> bool: + t = line.strip() + if not t.startswith("|"): + return False + inner = t.strip("|").replace(" ", "") + return bool(inner) and all(p in ("", "---", ":---", "---:", ":---:") for p in t.split("|")) + + +_RE_HEADING = re.compile(r"^(#{1,6})\s+(.+)$") +_RE_UL = re.compile(r"^\s*[-*+]\s+(.+)$") +_RE_OL = re.compile(r"^\s*(\d+)\.\s+(.+)$") +_RE_BLOCKQUOTE = re.compile(r"^\s*>\s?(.*)$") +_RE_HR = re.compile(r"^\s*(?:[-*_]\s*){3,}\s*$") + +_img_line = re.compile(r"^!\[([^\]]*)\]\(([^)]+)\)\s*$") + + +def _match_heading(line: str) -> tuple[int, str] | None: + """返回 (docx level 0–8, 标题文本) 或 None。""" + m = _RE_HEADING.match(line.strip()) + if not m: + return None + depth = len(m.group(1)) + title = _strip_inline_md(m.group(2).strip()) + level = min(max(depth - 1, 0), 8) + return (level, title) + + +def markdown_to_docx_bytes(md: str, *, asset_root: Path | None = None) -> bytes: + from docx import Document + from docx.enum.text import WD_PARAGRAPH_ALIGNMENT + from docx.shared import Inches, Pt + + doc = Document() + try: + style = doc.styles["Normal"] + style.font.name = "Microsoft YaHei" + style.font.size = Pt(10.5) + except Exception: + pass + + def _add_list_bullet(text: str) -> None: + t = _strip_inline_md(text) + try: + doc.add_paragraph(t, style="List Bullet") + except KeyError: + doc.add_paragraph("• " + t) + + def _add_list_number(text: str) -> None: + t = _strip_inline_md(text) + try: + doc.add_paragraph(t, style="List Number") + except KeyError: + doc.add_paragraph(t) + + lines = (md or "").replace("\r\n", "\n").split("\n") + i = 0 + in_fence = False + while i < len(lines): + raw = lines[i] + if raw.strip().startswith("```"): + in_fence = not in_fence + i += 1 + continue + if in_fence: + p = doc.add_paragraph(xml_escape(raw) or " ") + p.style = doc.styles["Normal"] + for run in p.runs: + run.font.name = "Consolas" + run.font.size = Pt(9) + i += 1 + continue + + line = raw.rstrip() + if not line.strip(): + doc.add_paragraph("") + i += 1 + continue + + if _RE_HR.match(line): + doc.add_paragraph("") + i += 1 + continue + + hm = _match_heading(line) + if hm is not None: + doc.add_heading(hm[1], level=hm[0]) + i += 1 + continue + + mimg = _img_line.match(line.strip()) + if mimg and asset_root is not None: + rel = mimg.group(2).strip() + if not (rel.startswith("http://") or rel.startswith("https://")): + img_path = (asset_root / rel).resolve() + try: + img_path.relative_to(asset_root.resolve()) + except ValueError: + i += 1 + continue + if img_path.is_file(): + doc.add_picture(str(img_path), width=Inches(5.9)) + i += 1 + continue + + if line.strip().startswith("|"): + rows: list[list[str]] = [] + while i < len(lines) and lines[i].strip().startswith("|"): + row_line = lines[i].strip() + if _is_table_sep(row_line): + i += 1 + continue + cells = [c.strip() for c in row_line.strip("|").split("|")] + rows.append([_strip_inline_md(c) for c in cells]) + i += 1 + if rows: + max_cols = max(len(r) for r in rows) + pad_rows = [r + [""] * (max_cols - len(r)) for r in rows] + tbl = doc.add_table(rows=len(pad_rows), cols=max_cols) + tbl.style = "Table Grid" + for ri, row in enumerate(pad_rows): + for ci, cell in enumerate(row): + tbl.rows[ri].cells[ci].text = cell + continue + + mu = _RE_UL.match(line) + if mu: + _add_list_bullet(mu.group(1)) + i += 1 + continue + + mo = _RE_OL.match(line) + if mo: + _add_list_number(mo.group(2)) + i += 1 + continue + + mq = _RE_BLOCKQUOTE.match(line) + if mq: + inner = mq.group(1).strip() + if inner: + p = doc.add_paragraph() + p.paragraph_format.left_indent = Inches(0.25) + p.add_run(_strip_inline_md(inner)) + i += 1 + continue + + p = doc.add_paragraph() + p.alignment = WD_PARAGRAPH_ALIGNMENT.LEFT + text = _strip_inline_md(line) + p.add_run(text) + i += 1 + + bio = BytesIO() + doc.save(bio) + return bio.getvalue() + + +def _pdf_font_candidates() -> list[Path]: + raw = (os.environ.get("MA_PDF_FONT") or "").strip() + out: list[Path] = [] + if raw: + out.append(Path(raw)) + windir = os.environ.get("WINDIR", r"C:\Windows") + out.extend( + [ + Path(windir) / "Fonts" / "simhei.ttf", + Path(windir) / "Fonts" / "simsun.ttc", + Path(windir) / "Fonts" / "msyh.ttf", + ] + ) + # Linux / 容器常见中文字体（路径不存在则跳过） + out.extend( + [ + Path("/usr/share/fonts/truetype/wqy/wqy-microhei.ttc"), + Path("/usr/share/fonts/truetype/wqy/wqy-zenhei.ttc"), + Path("/usr/share/fonts/truetype/noto/NotoSansCJK-Regular.ttc"), + Path("/usr/share/fonts/truetype/noto/NotoSansCJKsc-Regular.otf"), + Path("/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc"), + ] + ) + return out + + +def _pdf_flowable_image(img_path: Path, *, max_w: float, max_h: float) -> Any: + """将插图缩放到不超过 max_w×max_h（ReportLab 单位，与 cm 一致），保持宽高比，避免矩阵长图撑爆版面。""" + from reportlab.lib.utils import ImageReader + from reportlab.platypus import Image as RLImage + + p = str(img_path) + try: + ir = ImageReader(p) + iw, ih = ir.getSize() + except Exception: + return RLImage(p, width=max_w * 0.9, height=max_h * 0.9) + if iw <= 0 or ih <= 0: + return RLImage(p, width=max_w * 0.9, height=max_h * 0.9) + w = float(max_w) + h = w * (float(ih) / float(iw)) + if h > float(max_h): + h = float(max_h) + w = h * (float(iw) / float(ih)) + return RLImage(p, width=w, height=h) + + +def markdown_to_pdf_bytes(md: str, *, asset_root: Path | None = None) -> bytes: + """简易 PDF；需本机 .ttf 中文字体或环境变量 MA_PDF_FONT。""" + from reportlab.lib import colors + from reportlab.lib.pagesizes import A4 + from reportlab.lib.styles import ParagraphStyle, getSampleStyleSheet + from reportlab.lib.units import cm + from reportlab.pdfbase import pdfmetrics + from reportlab.pdfbase.ttfonts import TTFont + from reportlab.platypus import Paragraph, SimpleDocTemplate, Spacer, Table, TableStyle + + font_name = "MaExportCJK" + registered = False + for p in _pdf_font_candidates(): + if not p.is_file(): + continue + try: + if p.suffix.lower() == ".ttc": + try: + pdfmetrics.registerFont( + TTFont(font_name, str(p), subfontIndex=0) + ) + except TypeError: + pdfmetrics.registerFont(TTFont(font_name, str(p))) + else: + pdfmetrics.registerFont(TTFont(font_name, str(p))) + registered = True + break + except Exception: + continue + if not registered: + raise ValueError( + "未找到可用的中文字体文件。请在 Windows 上安装黑体/宋体，" + "或设置环境变量 MA_PDF_FONT 指向 .ttf 文件路径。" + ) + + styles = getSampleStyleSheet() + body = ParagraphStyle( + name="BodyCJK", + parent=styles["Normal"], + fontName=font_name, + fontSize=10, + leading=14, + ) + h1s = ParagraphStyle( + name="H1CJK", + parent=body, + fontSize=16, + leading=20, + spaceAfter=8, + ) + h2s = ParagraphStyle( + name="H2CJK", + parent=body, + fontSize=13, + leading=17, + spaceAfter=6, + ) + h3s = ParagraphStyle( + name="H3CJK", + parent=body, + fontSize=12, + leading=16, + spaceAfter=5, + ) + h4s = ParagraphStyle( + name="H4CJK", + parent=body, + fontSize=11, + leading=15, + spaceAfter=4, + ) + h56s = ParagraphStyle( + name="H56CJK", + parent=body, + fontSize=10.5, + leading=14, + spaceAfter=3, + ) + quote_style = ParagraphStyle( + name="QuoteCJK", + parent=body, + leftIndent=14, + fontSize=9.5, + textColor=colors.HexColor("#444444"), + ) + bullet_body = ParagraphStyle( + name="BulletBodyCJK", + parent=body, + leftIndent=18, + bulletIndent=8, + firstLineIndent=0, + ) + + story: list[Any] = [] + lines = (md or "").replace("\r\n", "\n").split("\n") + i = 0 + in_fence = False + + def _para_cell(s: str, style: Any) -> Paragraph: + return Paragraph(xml_escape(_strip_inline_md(s)), style) + + while i < len(lines): + raw = lines[i] + if raw.strip().startswith("```"): + in_fence = not in_fence + i += 1 + continue + s = raw.rstrip() + if in_fence: + story.append(Paragraph(xml_escape(s or " "), body)) + story.append(Spacer(1, 0.1 * cm)) + i += 1 + continue + if not s.strip(): + story.append(Spacer(1, 0.15 * cm)) + i += 1 + continue + + if _RE_HR.match(s): + story.append(Spacer(1, 0.2 * cm)) + i += 1 + continue + + hm = _match_heading(s) + if hm is not None: + level, title = hm + title_esc = xml_escape(title) + if level == 0: + story.append(Paragraph(title_esc, h1s)) + elif level == 1: + story.append(Paragraph(title_esc, h2s)) + elif level == 2: + story.append(Paragraph(title_esc, h3s)) + elif level == 3: + story.append(Paragraph(title_esc, h4s)) + else: + story.append(Paragraph(title_esc, h56s)) + i += 1 + continue + + mimg = _img_line.match(s.strip()) + if mimg and asset_root is not None: + rel = mimg.group(2).strip() + if not (rel.startswith("http://") or rel.startswith("https://")): + img_path = (asset_root / rel).resolve() + try: + img_path.relative_to(asset_root.resolve()) + except ValueError: + i += 1 + continue + if img_path.is_file(): + story.append( + _pdf_flowable_image( + img_path, max_w=13 * cm, max_h=24 * cm + ) + ) + story.append(Spacer(1, 0.2 * cm)) + i += 1 + continue + + if s.strip().startswith("|"): + rows: list[list[str]] = [] + while i < len(lines) and lines[i].strip().startswith("|"): + row_line = lines[i].strip() + if _is_table_sep(row_line): + i += 1 + continue + cells = [c.strip() for c in row_line.strip("|").split("|")] + rows.append([_strip_inline_md(c) for c in cells]) + i += 1 + if rows: + max_cols = max(len(r) for r in rows) + pad_rows = [r + [""] * (max_cols - len(r)) for r in rows] + usable_w = 13 * cm + col_w = usable_w / float(max_cols) + data: list[list[Any]] = [] + for row in pad_rows: + data.append( + [_para_cell(c, body) for c in row] + ) + t = Table(data, colWidths=[col_w] * max_cols) + t.setStyle( + TableStyle( + [ + ("GRID", (0, 0), (-1, -1), 0.5, colors.grey), + ("VALIGN", (0, 0), (-1, -1), "TOP"), + ("LEFTPADDING", (0, 0), (-1, -1), 4), + ("RIGHTPADDING", (0, 0), (-1, -1), 4), + ("TOPPADDING", (0, 0), (-1, -1), 3), + ("BOTTOMPADDING", (0, 0), (-1, -1), 3), + ] + ) + ) + story.append(t) + story.append(Spacer(1, 0.15 * cm)) + continue + + mu = _RE_UL.match(s) + if mu: + txt = xml_escape(_strip_inline_md(mu.group(1))) + story.append(Paragraph(f"• {txt}", bullet_body)) + i += 1 + continue + + mo = _RE_OL.match(s) + if mo: + n, rest = mo.group(1), mo.group(2) + txt = xml_escape(_strip_inline_md(rest)) + story.append(Paragraph(f"{n}. {txt}", bullet_body)) + i += 1 + continue + + mq = _RE_BLOCKQUOTE.match(s) + if mq: + inner = mq.group(1).strip() + if inner: + story.append( + Paragraph(xml_escape(_strip_inline_md(inner)), quote_style) + ) + i += 1 + continue + + plain = _strip_inline_md(s) + story.append(Paragraph(xml_escape(plain), body)) + i += 1 + + buf = BytesIO() + doc = SimpleDocTemplate( + buf, + pagesize=A4, + leftMargin=2 * cm, + rightMargin=2 * cm, + topMargin=2 * cm, + bottomMargin=2 * cm, + ) + doc.build(story) + return buf.getvalue() diff --git a/backend/pipeline/reporting/report_matrix_group_evidence.py b/backend/pipeline/reporting/report_matrix_group_evidence.py new file mode 100644 index 0000000..aef47bc --- /dev/null +++ b/backend/pipeline/reporting/report_matrix_group_evidence.py @@ -0,0 +1,176 @@ +""" +从宿主报告 ``competitor_analysis.md`` 中按细类名抽取 **Markdown 四级标题** ``#### {细类名}`` +下的正文，用于策略生成时并入「与同细类对齐」的大模型归纳（第五～第八章各块）。 + +报告生成侧约定：矩阵/价盘/促销/评论/场景等 LLM 小节均以 ``#### `` + 与矩阵一致的细类名为小节标题 +（见 ``generate_group_summaries`` 系统提示）。 + +**第八章 8.3**（``generate_comment_sentiment_analysis_llm``）在每组 ``#### {细类}`` 下还会再嵌套 +``#### 正向体验主题`` 等四级标题（见 ``generate_sections.SENTIMENT_LLM_SYSTEM``）， +通用抽取在遇到下一行 ``####`` 时即结束，会把 8.3 正文误判为空；故 8.3 单独解析后再拼回。 +""" +from __future__ import annotations + +import re +from pathlib import Path +from typing import Literal + +# 与 ``jd_report.build_competitor_markdown`` 中 8.3 小节标题一致。 +_SENTIMENT_83_HEADING = "### 8.3 评价正/负向主题（按细类 · 大模型）" + +# 与 ``generate_sections.SENTIMENT_LLM_SYSTEM``「建议结构」四级标题一致；嵌套于 8.3 每组 ``#### 细类`` 之下。 +_SENTIMENT_LLM_INNER_LEVEL4 = frozenset( + { + "正向体验主题", + "负向评价主题归因", + "混合评价中的典型张力", + "使用注意", + } +) + + +def _md_splice_out_sentiment_83_section(md: str) -> str: + """去掉 8.3 整节，避免按细类抽取时把嵌套 ``####`` 误判为同级边界；其它章不变。""" + i = md.find(_SENTIMENT_83_HEADING) + if i < 0: + return md + tail = md[i + len(_SENTIMENT_83_HEADING) :] + m = re.search(r"^##\s+", tail, re.MULTILINE) + if not m: + return md[:i].rstrip() + "\n\n" + cut = i + len(_SENTIMENT_83_HEADING) + m.start() + return md[:i].rstrip() + "\n\n" + md[cut:].lstrip("\n") + + +def extract_sentiment_83_level4_body(md: str, group_title: str) -> str: + """ + 仅在 ``### 8.3 …`` 节内，抽取 ``#### {group_title}`` 下正文（**不含**该标题行）。 + + 允许正文内出现 ``#### 正向体验主题`` 等情感归纳子标题；遇下一 peer ``####``（另一细类）或 ``###``/``##`` 则结束。 + """ + title = (group_title or "").strip() + if not title or not (md or "").strip(): + return "" + + i = md.find(_SENTIMENT_83_HEADING) + if i < 0: + return "" + tail = md[i + len(_SENTIMENT_83_HEADING) :] + m_end = re.search(r"^##\s+", tail, re.MULTILINE) + chunk = tail[: m_end.start()] if m_end else tail + + lines = chunk.splitlines() + n = len(lines) + j = 0 + while j < n: + line = lines[j] + m4 = re.match(r"^####\s+(.+?)\s*$", line) + if m4 and m4.group(1).strip() == title: + j += 1 + body_lines: list[str] = [] + while j < n: + nxt = lines[j] + mpeer = re.match(r"^####\s+(.+?)\s*$", nxt) + if mpeer: + inner = mpeer.group(1).strip() + if inner in _SENTIMENT_LLM_INNER_LEVEL4: + body_lines.append(nxt) + j += 1 + continue + break + if re.match(r"^###\s", nxt) or re.match(r"^##\s", nxt): + break + body_lines.append(nxt) + j += 1 + return "\n".join(body_lines).strip() + j += 1 + return "" + + +def extract_level4_sections_by_group_title(md: str, group_title: str) -> list[str]: + """ + 返回全文内所有 ``#### {group_title}`` 小节正文（不含标题行），按出现顺序。 + 标题须与 ``group_title`` 去首尾空白后**完全一致**。 + """ + title = (group_title or "").strip() + if not title or not (md or "").strip(): + return [] + + lines = md.splitlines() + n = len(lines) + blocks: list[str] = [] + i = 0 + while i < n: + line = lines[i] + m = re.match(r"^####\s+(.+?)\s*$", line) + if m and m.group(1).strip() == title: + i += 1 + chunk: list[str] = [] + while i < n: + nxt = lines[i] + if re.match(r"^####\s", nxt): + break + if re.match(r"^###\s", nxt) or re.match(r"^##\s", nxt): + break + if re.match(r"^#\s", nxt) and not nxt.startswith("##"): + break + chunk.append(nxt) + i += 1 + body = "\n".join(chunk).strip() + if body: + blocks.append(body) + continue + i += 1 + return blocks + + +def load_report_matrix_group_evidence_markdown( + run_dir: Path | str, + group_title: str, + *, + max_chars: int = 28_000, +) -> tuple[str, Literal["competitor_analysis_md", "none"]]: + """ + 读取 ``run_dir/competitor_analysis.md``，抽取该细类在各章大模型小节下的归纳，拼接为一段 Markdown。 + + 若文件不存在或无任何匹配小节，返回 ``("", "none")``。 + """ + root = Path(run_dir) + path = root / "competitor_analysis.md" + cap = max(512, int(max_chars)) + if not path.is_file(): + return "", "none" + try: + full = path.read_text(encoding="utf-8") + except OSError: + return "", "none" + + without_83 = _md_splice_out_sentiment_83_section(full) + parts = extract_level4_sections_by_group_title(without_83, group_title) + s83 = extract_sentiment_83_level4_body(full, group_title) + if s83: + parts.append(s83) + if not parts: + return "", "none" + + intro = ( + f"> **说明**：以下为同任务《竞品分析报告》正文中、细类「**{group_title.strip()}**」下 " + "「#### …」小节的**大模型归纳**摘录（按正文出现顺序拼接），" + "覆盖矩阵/价盘/促销/评论与场景等块中**已生成**的段落，" + "并含 **§8.3 评价正/负向主题（按细类 · 大模型）** 内该细类段落（允许嵌套四级小标题）；" + "若某块未开 LLM 或未产出对应小节，则不会出现在此摘录中。\n\n" + ) + sep = "\n\n---\n\n" + body = intro + sep.join(parts) + if len(body) <= cap: + return body, "competitor_analysis_md" + tail = "\n\n…（已截断）\n" + room = max(400, cap - len(tail)) + return body[: room].rstrip() + tail, "competitor_analysis_md" + + +__all__ = [ + "extract_level4_sections_by_group_title", + "extract_sentiment_83_level4_body", + "load_report_matrix_group_evidence_markdown", +] diff --git a/backend/pipeline/reporting/report_strategy_excerpt.py b/backend/pipeline/reporting/report_strategy_excerpt.py new file mode 100644 index 0000000..3785a7c --- /dev/null +++ b/backend/pipeline/reporting/report_strategy_excerpt.py @@ -0,0 +1,78 @@ +""" +从任务 run 目录加载报告「九、策略与机会提示」下的正文片段，供策略稿 LLM **可选**参考（兼容 / 遗留）。 + +**默认产线**已弃用报告内第九章大模型长文，新任务节选多为空；见 ``jd_report._strategy_opportunities_reader_fixed_lines`` 与规划 +``docs/planning/strategy-marketing-content-alignment.md`` §2。 + +优先顺序： + +1. ``strategy_opportunities_llm.json`` 中的 ``markdown``（仅当非空；空壳 json 不回退 md，避免误载固定读者说明）； +2. 否则从 ``competitor_analysis.md`` 截取 ``## 九、策略与机会提示`` 至 ``## 附录`` 之前。 +""" +from __future__ import annotations + +import json +from pathlib import Path +from typing import Literal + +CHAPTER_NINE_HEADING = "## 九、策略与机会提示" + + +def extract_chapter_nine_strategy_markdown(full_md: str) -> str: + """ + 提取宿主报告中第九章策略块（含章节标题行），不含「附录」及之后内容。 + + 若未找到标题，返回空字符串。 + """ + t = full_md or "" + if not t.strip(): + return "" + key = CHAPTER_NINE_HEADING + i = t.find(key) + if i == -1: + return "" + chunk = t[i:] + j = chunk.find("\n## 附录") + if j != -1: + chunk = chunk[:j] + return chunk.rstrip() + + +def load_report_strategy_excerpt( + run_dir: Path | str, + *, + max_chars: int = 24_000, +) -> tuple[str, Literal["json_markdown", "competitor_analysis_md", "none"]]: + """ + 返回 ``(节选正文, 来源标签)``。节选可能为空（未生成第九章或未找到标题）。 + """ + root = Path(run_dir) + cap = max(512, int(max_chars)) + + json_path = root / "strategy_opportunities_llm.json" + if json_path.is_file(): + try: + data = json.loads(json_path.read_text(encoding="utf-8")) + except (json.JSONDecodeError, OSError): + data = None + if isinstance(data, dict): + md = (data.get("markdown") or "").strip() + if md: + s = md if len(md) <= cap else md[: cap - 80].rstrip() + "\n\n…（已截断）\n" + return s, "json_markdown" + # 已存在落盘记录但无大模型正文时，不再回退截取 competitor_analysis.md： + # 第九章现为固定读者说明，避免误入策略稿 ``report_strategy_excerpt``。 + return "", "none" + + md_path = root / "competitor_analysis.md" + if md_path.is_file(): + try: + full = md_path.read_text(encoding="utf-8") + except OSError: + return "", "none" + block = extract_chapter_nine_strategy_markdown(full) + if block.strip(): + s = block if len(block) <= cap else block[: cap - 80].rstrip() + "\n\n…（已截断）\n" + return s, "competitor_analysis_md" + + return "", "none" diff --git a/backend/pipeline/reporting/strategy_draft.py b/backend/pipeline/reporting/strategy_draft.py new file mode 100644 index 0000000..b8bfc8c --- /dev/null +++ b/backend/pipeline/reporting/strategy_draft.py @@ -0,0 +1,895 @@ +""" +市场策略 Markdown 草稿：**规则骨架**（占位 + 少量数据摘录），供业务与大模型成稿对齐。 + +- 决策在「策略生成」表单完成；未填项由大模型结合摘要与报告节选补全。 +- 骨架刻意短、可执行；避免与成稿重复的「假设 / 待验证」套话。 +""" +from __future__ import annotations + +import math +from typing import Any + +from .brief_concentration import ( + concentration_first_share, + concentration_top_three_share, +) + + +def _esc(s: Any) -> str: + t = "" if s is None else str(s).strip() + return t.replace("\r\n", "\n").replace("\r", "\n") + + +def _pct(x: Any) -> str: + if x is None: + return "—" + try: + v = float(x) + if math.isnan(v) or math.isinf(v): + return "—" + return f"{100 * v:.1f}%" + except (TypeError, ValueError): + return "—" + + +def _num(x: Any) -> str: + if x is None: + return "—" + if isinstance(x, bool): + return str(x) + if isinstance(x, int): + return str(x) + if isinstance(x, float): + if math.isnan(x) or math.isinf(x): + return "—" + if x == int(x): + return str(int(x)) + return f"{x:.2f}" + return str(x) + + +def _cr_narrative( + label: str, + cr1: Any, + cr3: Any, + top: Any, + *, + first_share_wording: tuple[str, str] | None = None, +) -> str | None: + """从集中度生成一句策略向描述，无数据则返回 None（正文避免英文缩写）。 + + ``first_share_wording`` 为 ``(第一大句前缀, 前三大句前缀)`` 时覆盖默认措辞（用于矩阵收窄口径， + 避免误写「列表行」）。 + """ + try: + c1 = float(cr1) if cr1 is not None else None + except (TypeError, ValueError): + c1 = None + if c1 is None and not (top or "").strip(): + return None + top_s = _esc(top) or "—" + if first_share_wording is not None: + w1, w3 = first_share_wording + elif "店铺" in label: + w1, w3 = "第一大店铺约占列表行的", "前三大店铺合计约占" + elif "品牌" in label: + w1, w3 = "第一大品牌约占", "前三大品牌合计约占" + else: + w1, w3 = "第一大主体约占", "前三大合计约占" + if c1 is not None: + if c1 >= 0.4: + tone = "偏高，头部资源集中" + elif c1 >= 0.25: + tone = "中等，存在可争夺空间" + else: + tone = "相对分散，差异化切入点可能更多" + return ( + f"- **{label}**：{w1} **{_pct(cr1)}**，{w3} **{_pct(cr3)}**；" + f"当前头部为「{top_s}」。*粗判：{tone}。*" + ) + return f"- **{label}**：头部为「{top_s}」（缺少占比时可结合列表与商详数据补全）。" + + +def _shop_unique_sku_basis_lines(shops: dict[str, Any]) -> list[str]: + """ + ``shops_from_list.unique_sku_basis`` 与竞品报告/摘要一致：按去重 SKU 的店铺集中度对照口径。 + """ + usb = shops.get("unique_sku_basis") if isinstance(shops, dict) else None + if not isinstance(usb, dict) or not usb.get("n_unique_skus"): + return [] + u1 = concentration_first_share(usb) + u3 = concentration_top_three_share(usb) + utop = _esc(usb.get("top_label") or "") + if u1 is None or not utop: + return [] + return [ + f"- **列表侧店铺（按去重 SKU）**：共 **{_num(usb.get('n_unique_skus'))}** 个去重 SKU；" + f"第一大店铺「{utop}」约占 **{_pct(u1)}**；前三合计 **{_pct(u3)}**。" + "*（与上行「按列表行」可能因同一 SKU 多行曝光而差异；非销量/市占。）*" + ] + + +def _goal_bullet(label: str, user_val: str, placeholder: str) -> str: + v = _esc(user_val).strip() + if v: + return f"- **{label}**：{v}" + return f"- **{label}**：*（{placeholder}）*" + + +def _pillar_cell(user_val: str) -> str: + v = _esc(user_val).strip() + return v if v else "*待填*" + + +def _pos_mark(choice: str, key: str) -> str: + return "[x]" if choice == key else "[ ]" + + +def _risk_line(checked: bool, text: str) -> str: + mark = "[x]" if checked else "[ ]" + return f"- {mark} {text}" + + +def filter_strategy_hints_for_ch8_probe(hints: Any) -> list[str]: + """ + 当报告以 **第八章文本挖掘** 为主呈现评论侧时，规则引擎的 ``strategy_hints`` 中仍可能含 + 「关注词出现较多」「预设场景占比」类句子（与 §8 主口径冲突）。此处剔除，避免进入策略底稿与 LLM。 + """ + if not isinstance(hints, list): + return [] + out: list[str] = [] + for h in hints: + s = _esc(h) if h is not None else "" + if not s.strip(): + continue + if "评价文本中「" in s and "等主题出现较多" in s: + continue + if "用途/场景中「" in s and "有效评价自述" in s: + continue + out.append(s) + return out if out else [ + "（与「关注词/预设场景条形图」相关的自动提示已省略；用户洞察请以报告 §8 文本挖掘及第五至第八章细类归纳为准；可执行策略以「策略制定」按细类生成为准。）" + ] + + +def report_uses_chapter8_text_mining_probe(report_config: dict[str, Any] | None) -> bool: + """ + 与任务 ``report_config`` 中 ``chapter8_text_mining_probe`` 一致；未显式设置时默认 ``True`` + （与 ``jd.runner.get_default_report_config`` 一致）。 + 用于 §1.2 文案分支及对 ``strategy_hints`` 的过滤：开启探针时与子串命中枚举相关的自动线索会被压掉； + 关闭时 §1.2 仍说明「简报不附带预设关注词/场景子串统计」，评论侧以报告第八章探针（若启用）与原文为准。 + """ + if not isinstance(report_config, dict): + return True + if "chapter8_text_mining_probe" in report_config: + return bool(report_config.get("chapter8_text_mining_probe")) + return True + + +def build_strategy_draft_markdown( + *, + job_id: int, + keyword: str, + brief: dict[str, Any], + business_notes: str = "", + generated_at_iso: str = "", + strategy_decisions: dict[str, Any] | None = None, + report_config: dict[str, Any] | None = None, + for_llm_input: bool = False, +) -> str: + """生成可下载的 Markdown：与「六主轴 + 品牌四线」示例稿同构的规则骨架，附录为数据速览。 + + ``for_llm_input=True`` 时供大模型归纳用：弱化源码/路径/任务 ID 等痕迹，减少对外成稿误复述。 + """ + use_ch8_probe = report_uses_chapter8_text_mining_probe(report_config) + d = strategy_decisions or {} + pos = _esc(d.get("positioning_choice") or "").strip() + kw = _esc(brief.get("keyword")) or _esc(keyword) or "—" + batch = _esc(brief.get("batch_label")) or "—" + if for_llm_input: + lines: list[str] = [ + f"# 「{kw}」", + "", + ] + if generated_at_iso: + lines.append( + f"> **生成时间**：{_esc(generated_at_iso)}（归纳用，**勿写入对外成稿正文**）" + ) + lines.append("") + lines.append( + "> **规划 §1.1（归纳用）**：成稿须以「怎么做」为主体，不以「是什么」铺陈代替策略；摘要「阶段重点」须 1～2 条执行句；第一章背景控制在少量结论句。**勿将本行写入对外正文。**" + ) + lines.append("") + else: + lines = [ + f"# 市场策略制定草稿 · 「{kw}」", + "", + "> **骨架说明**：本页为规则骨架；结构与 [`docs/demo`](docs/demo) 示例一致。**全局禁止编造**见 `generate_strategy.py` 中 `STRATEGY_DATA_RULES`。", + "", + ] + if generated_at_iso: + lines.append(f"> **生成时间**：{_esc(generated_at_iso)} · **任务 ID**：{job_id}") + lines.append("") + + scope = brief.get("scope") or {} + merged_n = scope.get("merged_sku_count") + comm_n = scope.get("comment_flat_rows") + + pr_role = _esc(d.get("product_role") or "").strip() + bf_line = _esc(d.get("battlefield_one_line") or "").strip() + aud = _esc(d.get("audience_segment") or "").strip() + th = _esc(d.get("time_horizon") or "").strip() + sc = _esc(d.get("success_criteria") or "").strip() + + def _scope_cell(val: str, placeholder: str) -> str: + return val if val else f"*（{placeholder}）*" + + scope_prelude = ( + [ + "*回答：**这份策略是针对什么、在什么边界里做的**——属「立项靶心」，不是执行摘要。表单已填则写成短句；未填保留占位，**勿**编造。*", + "*业务侧在动策略稿之前，应先对齐本节；阶段目标类型以业务内部规划口径为准（若有）。*", + "", + ] + if not for_llm_input + else [] + ) + _table_main_cat = ( + "*（待填：如饼干线 / 面包线 / 多线并行；未定写「待业务定类」）*" + if not for_llm_input + else "*（待确认）*" + ) + sgt = _esc(d.get("stage_goal_type") or "").strip() + _goal_type_placeholder = ( + "待确认：可填如拉新尝试、做销量与转化、守份额与复购、新品试水等业务阶段目标；与业务内部口径对齐" + if for_llm_input + else "成稿从业务侧阶段目标类型口径择一或组合，并与下列成功标准一致" + ) + _table_goal_type = _scope_cell(sgt, _goal_type_placeholder) + _summary_user_side = ( + "- **用户侧**：*（一两句结论即可：讨论焦点与负向主题；**按细类分句**归纳，**勿**混成「全站用户」一句；**勿**展开与报告重复的细类统计、词频。）*" + if not for_llm_input + else "- **用户侧**：—" + ) + _summary_stage = ( + "- **阶段重点**：*（须含 1～2 条**可执行动作**，回扣 §2 优先痛点；**尽量点明适用类目/主推线**；勿仅写「加强运营」。）*" + if not for_llm_input + else "- **阶段重点**：—" + ) + lines.extend( + [ + "## 策略范围与前提（生成前先对齐）", + "", + *scope_prelude, + "| 须明确项 | 填写或待确认 |", + "|----------|----------------|", + f"| **监测任务（数据同源）** | 关键词「{kw}」；批次 **{batch}**；与同任务《竞品分析报告》一致 |", + f"| **策略服务对象（本品角色）** | {_scope_cell(pr_role, '待填：新品 / 追赶 / 防守 / 拓品类 …')} |", + f"| **一句话战场** | {_scope_cell(bf_line, '在哪个需求场景、与谁抢同一批用户？')} |", + f"| **目标客群/场景** | {_scope_cell(aud, '可选')} |", + f"| **主推类目/细类** | {_table_main_cat} |", + f"| **本阶段策略目标类型** | {_table_goal_type} |", + f"| **时间范围** | {_scope_cell(th, '如本季度 / 未来 12 周')} |", + f"| **成功标准（可量化）** | {_scope_cell(sc, '搜索位次、转化、复购等')} |", + "", + "---", + "", + "## 摘要", + "", + f"- **范围与样本**：监测词「{kw}」；批次 **{batch}**；" + + ( + f"深入 SKU ≈ {_num(merged_n)}；评价条数 ≈ {_num(comm_n)}。" + if merged_n is not None or comm_n is not None + else "样本规模见附录。" + ), + _summary_user_side, + _summary_stage, + "", + "## 一、顾客是谁", + "", + "### 1.1 人群与决策路径", + "", + f"- **检索与货架语境**：{kw}；批次 {batch}。", + ] + ) + bf = _esc(d.get("battlefield_one_line") or "").strip() + if bf: + lines.append(f"- **一句话战场**：{bf}") + else: + lines.append("- **一句话战场**：*（在哪个需求场景、与谁抢同一批用户？）*") + lines.extend( + [ + ( + "- **典型路径**：搜索 → 列表比价 → 商详与配料 → 评价 → 下单/复购。" + if for_llm_input + else "- **典型路径**：*（成稿：搜索 → 列表比价 → 详情与配料 → 评价 → 下单/复购。）*" + ), + "", + *( + [] + if for_llm_input + else [ + "*成稿须与 §2 一致：写清「谁在什么任务下检索、决策」及**主攻类目/细类**（与 §2.1 类目列可对上），为后文「针对痛点怎么做」埋伏笔。*", + "", + ] + ), + "### 1.2 细类讨论焦点（评论文本分析）", + "", + ] + ) + if use_ch8_probe: + if not for_llm_input: + lines.extend( + [ + "*当前任务以**第八章评论侧文本挖掘**为主呈现时，此处**不**逐条罗列关注词子串命中次数。*", + "", + "- **饼干 / 糕点 / 面点等**：*（骨架占位；成稿**分细类**各一句归纳用户关心点，**勿**合并成模糊「全池」一句；**勿**复述 §8 词频与条数。）*", + "", + ] + ) + else: + if not for_llm_input: + lines.append( + "*简报中**不再**附带预设关注词/场景子串统计；评论侧请依据同任务《竞品分析报告》**第八章第二节**（文本挖掘探针，若已启用）及抽样原文撰写本节。*" + ) + lines.append("") + + mix = brief.get("category_mix_top") or [] + if mix: + lines.append("### 类目结构（摘录）") + lines.append("") + for item in mix[:6]: + if isinstance(item, dict): + lines.append(f"- {_esc(item.get('label'))}：{_num(item.get('count'))}") + lines.append("") + + lines.extend( + [ + ( + "### 1.3 本品聚焦" + if for_llm_input + else "### 1.3 本品聚焦（占位）" + ), + "", + *( + [] + if for_llm_input + else [ + "*成稿写清本期**主攻人群/场景/类目或主推细类**与 §2.1「类目/细类」列的对应关系。*", + "", + ] + ), + _goal_bullet("本品角色", str(d.get("product_role") or ""), "新品 / 追赶 / 防守 / 拓品类 …"), + _goal_bullet( + "本阶段策略目标类型", + str(d.get("stage_goal_type") or ""), + "与业务内部阶段目标口径对齐，或自填（如拉新尝试、做销量）", + ), + _goal_bullet( + "目标客群", + str(d.get("audience_segment") or ""), + "为谁、什么场景（可选）", + ), + _goal_bullet( + "主要对标", + str(d.get("competitor_reference") or ""), + "品牌或价位带参照（可选）", + ), + "", + ] + ) + + lines.extend( + [ + "## 二、产品价值与用户痛点", + "", + *( + [] + if for_llm_input + else [ + "*本节**仅**用下表写清**针对痛点要怎么做**（**类目** + 痛点简述 + 动作 + 落地 + 验证）。**不再**单设「痛点表 / 价值对表 / 负向归因」子节，避免与 §三、§八重复。*", + "", + "*「用户痛点（简述）」须与 `structured_brief` / 策略线索 / 报告节选**可核对**；**禁止**编造「用户反馈『……』」式引语，除非原句已出现在上述输入中。*", + "", + "*「类目/细类」列：写明本行决策**适用于哪一类**（如饼干/面包/全检索池）；多细类须**分行**，**禁止**用一句「全站」覆盖彼此冲突的策略；类目未定可写「待业务定类」并附分类假设。*", + "", + ] + ), + "### 2.1 针对痛点要怎么做", + "", + "| 类目/细类（本决策适用） | 用户痛点（简述） | 策略动作 | 具体怎么做（触点/话术/规格/渠道） | 如何验证 |", + "|--------------------------|------------------|----------|-----------------------------------|----------|", + "| *（如：饼干线 / 西式糕点 / 全池仅当可解释）* | *（口感/分量/价格信任等）* | *（动词句）* | *（可执行）* | *（指标或抽样）* |", + "| | | | | |", + "", + ] + ) + + raw_hints = brief.get("strategy_hints") or [] + hints = ( + filter_strategy_hints_for_ch8_probe(raw_hints) + if use_ch8_probe + else (list(raw_hints) if isinstance(raw_hints, list) else []) + ) + if hints: + lines.append( + "**监测摘要自动线索**" if for_llm_input else "**摘要自动线索（`strategy_hints`）**" + ) + lines.append("") + for h in hints: + lines.append(f"- {_esc(h)}") + lines.append("") + + pst = brief.get("price_stats") or {} + lines.extend( + [ + "## 三、为什么要买「这款产品」", + "", + "### 3.1 购买者视角：为何要选这一款（依据与理由）", + "", + ] + ) + raw = brief.get("pc_search_raw") or {} + if raw.get("result_count_consensus") is not None: + lines.append( + f"- **检索结果量级（需求侧参考，非销售额）**：{_num(raw.get('result_count_consensus'))}（站内匹配条数量级）" + if for_llm_input + else f"- **检索结果量级（需求侧参考，非销售额）**：{_num(raw.get('result_count_consensus'))}（列表 resultCount）" + ) + elif merged_n is not None: + lines.append(f"- **深入样本 SKU 数（监测范围）**：{_num(merged_n)}") + else: + lines.append( + "- **检索与样本尺度**：—" + if for_llm_input + else "- **检索与样本尺度**：*（成稿结合摘要与监测范围。）*" + ) + lines.append("") + if pst.get("n"): + src = _esc(brief.get("price_stats_source")) or "—" + src_disp = "本监测样本" if for_llm_input and src == "strategy_scope_matrix_group_skus" else src + lines.extend( + [ + f"- **价带摘录（支撑购买理由与价位锚点）**：来源 {src_disp}，n = {_num(pst.get('n'))}；" + f"区间 {_num(pst.get('min'))}～{_num(pst.get('max'))}；中位数 {_num(pst.get('median'))}。", + "", + ] + ) + else: + if for_llm_input: + lines.append("- **价带摘录**：监测摘要中暂无统计表，可结合同任务报告补一句与购买理由相关的价位锚点。") + else: + lines.append( + "*摘要中无价带统计，成稿可结合本批次价格数据在本节补一句价位锚点；**勿**重复 §2 已写的应对动作。*" + ) + lines.append("") + lines.append( + "- **购买理由（须站在购买者一侧写）**：用 1～2 句写清**买家为何愿意下单这一款**——解决什么顾虑、在货架上凭什么选它（获得感、可感知利益、价位是否值得等）；上列检索/价带仅作背景，勿喧宾夺主。" + "可用「用户/消费者」作主语，**禁止**用纯运营口吻（如「适合××叙事切入」「策略上占位」）代替购买动机；**禁止**以品类宏观句收尾而不落到本品可验证点。" + if for_llm_input + else "- **购买理由**：*（成稿：**购买者视角**——买家为何选这一款；承接上列依据与 §2 优先痛点；多细类则分句；**勿**只写品类风口或运营叙事；价带/规格动作已在 §2 表内则此处**勿再展开一遍**。）*" + ) + lines.append("") + + lines.extend( + [ + "## 四、为什么要选「这个品牌」", + "", + ( + "### 4.1 品牌承诺与调性" + if for_llm_input + else "### 4.1 品牌承诺与调性（占位）" + ), + "", + *( + [] + if for_llm_input + else [ + "*成稿：承诺与调性须能落到**触点**（商详/包装/客服首句等）上的**具体句子**；**若**多类目话术不同，按 §2.1 类目**分句**，勿仅形容词。*", + "", + ] + ), + ( + "- **一句话**：*（请写可落到商详/包装/客服等触点的承诺句）*" + if for_llm_input + else "- **一句话**：*（占位）*" + ), + ( + "- **调性**：透明、可验证、合规控糖叙事。" + if for_llm_input + else "- **调性**：透明、可验证、合规控糖叙事（成稿可细化）。" + ), + "", + "### 4.2 信任与证据", + "", + *( + ["- *（评价、配料、可核验表述边界。）*", ""] + if for_llm_input + else ["- *（成稿：评价、配料、可核验表述边界。）*", ""] + ), + *( + [] + if for_llm_input + else [ + "*本节写**用户为何信任、为何愿意选这个品牌**（承诺、证据、合规边界）；**价位阵地**（表单勾选的四类取向）见 **§8.2 定价策略**，勿混写。*", + "", + ] + ), + ] + ) + + conc = brief.get("concentration") or {} + shops = conc.get("shops_from_list") or {} + dbrand = conc.get("detail_brand_among_merged") or {} + scope_ap = brief.get("strategy_scope_applied") + scoped_matrix = isinstance(scope_ap, dict) and bool(scope_ap.get("group")) + gname_scoped = _esc(scope_ap.get("group")) if scoped_matrix else "" + lines.extend( + [ + "## 五、与其它品牌有何不同", + "", + "### 5.1 对比对象（摘录）", + "", + ] + ) + if scoped_matrix: + lines.append( + "*本任务已按矩阵细类收窄：**下列店铺/品牌占比均按该分组内「深入合并 SKU」条数统计**，" + "与全关键词 **PC 搜索列表行** 集中度**不是同一口径**；亦非销量或市占。*" + if not for_llm_input + else "*集中度：按所选矩阵分组内合并 SKU 条数；非全站列表行。*" + ) + lines.append("") + shop_label = ( + f"店铺分布（「{gname_scoped}」内样本 SKU）" + if scoped_matrix + else "列表侧店铺集中度" + ) + brand_label = ( + f"品牌分布（「{gname_scoped}」内样本 SKU）" + if scoped_matrix + else "深入样本内品牌集中度" + ) + scoped_wording: tuple[str, str] | None = ( + ("第一大店铺约占该分组样本 SKU 的", "前三大店铺合计约占") + if scoped_matrix + else None + ) + scoped_brand_wording: tuple[str, str] | None = ( + ("第一大品牌约占该分组样本 SKU 的", "前三大品牌合计约占") + if scoped_matrix + else None + ) + n_shop = _cr_narrative( + shop_label, + concentration_first_share(shops), + concentration_top_three_share(shops), + shops.get("top_label"), + first_share_wording=scoped_wording, + ) + n_brand = _cr_narrative( + brand_label, + concentration_first_share(dbrand), + concentration_top_three_share(dbrand), + dbrand.get("top_label"), + first_share_wording=scoped_brand_wording, + ) + if n_shop: + lines.append(n_shop) + for uline in _shop_unique_sku_basis_lines(shops): + lines.append(uline) + if n_brand: + lines.append(n_brand) + if not n_shop and not n_brand: + lines.append( + "- **竞争结构**：监测摘要未含集中度摘录。" + if for_llm_input + else "*本摘要未含集中度指标，请结合本批次竞争结构数据补全。*" + ) + lines.extend( + [ + "", + *( + [] + if for_llm_input + else [ + "", + "- **环境自测**：头部强势时是侧翼还是正面替代？格局分散时是否用细分场景切入？", + ] + ), + "", + ( + "### 5.2 差异化方向" + if for_llm_input + else "### 5.2 差异化方向（占位）" + ), + "", + *( + [] + if for_llm_input + else [ + "*成稿：相对竞品**多做什么/少做什么**，写**可执行的一步**；**若**差异因细类而异，**分类目**写（非空泛「更好」）。*", + "", + ] + ), + "| 差异点 | 说明 | 风险 |", + "|--------|------|------|", + ( + "| | | |" + if for_llm_input + else "| | *待填* | |" + ), + "", + ] + ) + lines.append("### 5.3 竞争应对") + lines.append("") + if not for_llm_input: + lines.append( + "*成稿：在表单倾向基础上，写清**跟价/不跟价时具体话术或机制**（一句即可）。*" + ) + lines.append("") + stance = _esc(d.get("competitive_stance") or "").strip() + stance_line = { + "flank": "- **本品倾向**：侧翼切入，避免与头部正面硬碰。", + "head_on": "- **本品倾向**：正面替代，对标头部主战场。", + "both": "- **本品倾向**：分层推进（部分场景侧翼、部分场景正面）。", + "undecided": ( + "- **本品倾向**：*（待确认）*" + if for_llm_input + else "- **本品倾向**：*（表单未选；成稿时据数据写清倾向）*" + ), + }.get(stance) + if stance_line: + lines.append(stance_line) + lines.append("") + + lines.extend( + [ + "## 六、阶段目标与路径", + "", + "### 6.1 本阶段定义", + "", + _goal_bullet("时间范围", str(d.get("time_horizon") or ""), "如：本季度 / 未来 12 周"), + _goal_bullet( + "成功标准（可量化）", + str(d.get("success_criteria") or ""), + "搜索位次、转化、复购等", + ), + _goal_bullet("非目标", str(d.get("non_goals") or ""), "明确不做什么（可选）"), + "", + "### 6.2 路径", + "", + *( + [] + if for_llm_input + else [ + "*成稿：路径须与 **§2.1 针对痛点要怎么做** 可对齐；营销/总体策略为**动词句**，回扣痛点；**多类目并行**时**分线**写目标或写清主线/副线。*", + "", + ] + ), + _goal_bullet( + "营销策略", + str(d.get("marketing_strategy") or ""), + "传播、活动、投放、内容主线（可选）", + ), + _goal_bullet( + "总体策略", + str(d.get("general_strategy") or ""), + "增长/品类/经营总原则（可选）", + ), + _goal_bullet( + "资源与预算备注", + str(d.get("resource_notes") or ""), + "人力、投放、产能等（可选）", + ), + "", + ] + ) + + pp = str(d.get("pillar_product") or "") + pr = str(d.get("pillar_price") or "") + pch = str(d.get("pillar_channel") or "") + pcm = str(d.get("pillar_comm") or "") + lines.extend( + [ + "## 七、品牌四线：建设 · 打造 · 运营 · 体验", + "", + *( + [] + if for_llm_input + else [ + "*（与表单「4P 策略支柱」对应：产品 / 定价 / 渠道 / 传播。）*", + "*成稿：**每条线**至少一句——服务哪类痛点、本阶段**具体做哪一步**；**尽量**与 §2.1「类目/细类」可对上，多类目则**分句**（勿四条同一泛化句）。*", + "", + ] + ), + "### 7.1 品牌建设", + "", + f"- {_pillar_cell(pp)}", + "", + "### 7.2 品牌打造", + "", + f"- {_pillar_cell(pr)}", + "", + "### 7.3 品牌运营", + "", + f"- {_pillar_cell(pch)}", + "", + "### 7.4 品牌体验", + "", + f"- {_pillar_cell(pcm)}", + "", + ] + ) + + if use_ch8_probe and not for_llm_input: + pst_sig = brief.get("price_promotion_signals") or {} + has_promo = isinstance(pst_sig, dict) and bool(pst_sig) + promo_hint = "*促销与活动线索：须与摘要 `price_promotion_signals` 及第六章/第九章已有归纳一致；无则勿编造具体满减门槛。*" + lines.extend( + [ + "", + promo_hint + if has_promo + else "*促销与价差：若摘要或价格信号有归纳则承接；无则勿编造。*", + "", + ] + ) + + lines.extend( + [ + "## 八、战术支柱", + "", + *( + [] + if for_llm_input + else [ + "*成稿：四支柱分别回扣 **痛点→动作→落地**（可与 §2.1 呼应，避免纯重复）；**若**产品/定价/促销因类目策略不同，**分细类**写子条，勿一条盖全站。*", + "", + ] + ), + "### 8.1 产品策略", + "", + f"- *（表单产品支柱：{_pillar_cell(pp)}）*", + "", + "### 8.2 定价策略", + "", + "**价位阵地取向（表单勾选；与监测价带可对读）**", + "", + f"- {_pos_mark(pos, 'top')} **贴顶**：中高位或头部价位带。", + f"- {_pos_mark(pos, 'mid')} **卡腰**：围绕中位数一带。", + f"- {_pos_mark(pos, 'entry')} **下探**：贴近区间下限。", + f"- {_pos_mark(pos, 'different')} **另起带**：规格/组合/服务差异化。", + "", + f"- *（表单价格支柱：{_pillar_cell(pr)}）*", + "", + "### 8.3 促销与活动策略", + "", + *( + [] + if for_llm_input + else [ + "*须写促销**原则**（券/到手价/跟价节奏）；**满减、满折、跨店**等：能引用的写清来源；监测未捕获具体门槛时写「待与运营/后台对齐」，**勿**整节留空，**勿**编造门槛数字。*", + "*与 `price_promotion_signals`、报告第六章一致；勿虚构活动。*", + "", + ] + ), + "### 8.4 渠道与传播", + "", + f"- *（渠道/传播：{_pillar_cell(pch)} / {_pillar_cell(pcm)}）*", + "", + ] + ) + + rk = bool(d.get("ack_risk_keywords")) + rp = bool(d.get("ack_risk_price")) + rc = bool(d.get("ack_risk_concentration")) + rk_kw = "评论侧归纳是否以偏概全？（需原评论抽样）" + lines.extend( + [ + "## 九、风险、假设与待验证", + "", + _risk_line(rk, rk_kw), + _risk_line(rp, "价格带是否含大促/异常挂价？（需核对清洗规则）"), + _risk_line(rc, "列表集中度与深入样本品牌是否不一致？（需解释渠道差异）"), + "", + *( + [] + if for_llm_input + else [ + "*成稿：每条风险尽量带**应对动作或验证计划**（抽样、核对规则），勿只列标题。*", + "", + "*业务备注见下节。*", + "", + ] + ), + "## 十、下一步与节奏", + "", + *( + [] + if for_llm_input + else [ + "*成稿：下列为**可执行任务**（可补负责人/时间）；与 §2.1 / §六 优先级一致；可含「按类目核对主图/商详与 §2.1 表」类项。*", + "", + ] + ), + "- [ ] 锁定主推款与对标；过法务与合规。", + "- [ ] 统一对外数据口径与话术。", + "- [ ] 下轮监测更新后迭代策略。", + "", + ] + ) + + notes = _esc(business_notes) + lines.extend( + [ + "### 业务约束与备注", + "", + (notes if notes else "*（未填写业务备注。）*"), + "", + "---", + "", + "## 附录：本任务关键数据一览", + "", + f"- **关键词**：{kw} · **批次**：{batch} · **摘要版本**：v{_num(brief.get('schema_version'))}", + ] + ) + meta = brief.get("meta") + meta_labels = { + "page_start": "起始页", + "page_to": "采集至页", + "max_skus_config": "SKU 上限", + "scenario_filter_enabled": "场景筛选", + } + if isinstance(meta, dict) and meta: + if for_llm_input: + bits_llm: list[str] = [] + ps, pt = meta.get("page_start"), meta.get("page_to") + if ps is not None and pt is not None: + bits_llm.append(f"列表页约第 {_esc(ps)}～{_esc(pt)} 页") + elif ps is not None: + bits_llm.append(f"列表自第 {_esc(ps)} 页起采集") + if meta.get("max_skus_config") is not None: + bits_llm.append(f"深入样本上限约 {_num(meta.get('max_skus_config'))} 个 SKU") + if meta.get("scenario_filter_enabled"): + bits_llm.append("已启用场景筛选") + if bits_llm: + lines.append(f"- **采集范围**：{'；'.join(bits_llm)}") + else: + bits = [] + for k in ("page_start", "page_to", "max_skus_config", "scenario_filter_enabled"): + if k in meta: + label = meta_labels.get(k, k) + bits.append(f"{label}={_esc(meta.get(k))}") + if bits: + lines.append(f"- **采集参数快照**：{'; '.join(bits)}") + raw = brief.get("pc_search_raw") or {} + if raw.get("result_count_consensus") is not None: + lines.append( + f"- **平台申报检索规模**：{_num(raw.get('result_count_consensus'))}" + if for_llm_input + else f"- **列表申报规模（resultCount）**：{_num(raw.get('result_count_consensus'))}" + ) + if for_llm_input: + lines.extend( + [ + "", + "*可与同任务《竞品分析报告》及本批次数据表对照核验。*", + "", + "---", + "", + ] + ) + else: + lines.extend( + [ + "", + "*同目录含本批次 CSV 与分析产出，可对照使用。*", + "", + "---", + "", + "*本稿由工作台「市场策略制定」生成；与同任务结构化分析数据一致。*", + "", + ] + ) + return "\n".join(lines) diff --git a/backend/pipeline/row_serialize.py b/backend/pipeline/row_serialize.py index d6ec855..3070276 100644 --- a/backend/pipeline/row_serialize.py +++ b/backend/pipeline/row_serialize.py @@ -3,7 +3,7 @@ from __future__ import annotations from typing import Any -from .csv_schema import ( +from .csv.schema import ( COMMENT_CSV_COLUMNS, COMMENT_CSV_TO_FIELD, DETAIL_CSV_COLUMNS, @@ -22,6 +22,7 @@ def search_row_to_dict(r: JdJobSearchRow) -> dict[str, Any]: out: dict[str, Any] = {"id": r.id, "row_index": r.row_index} for k in JD_SEARCH_INTERNAL_KEYS: out[k] = getattr(r, k) or "" + out["matrix_group_label"] = r.matrix_group_label or "" return out @@ -29,6 +30,7 @@ def detail_row_to_dict(r: JdJobDetailRow) -> dict[str, Any]: out: dict[str, Any] = {"id": r.id, "row_index": r.row_index} for k in DETAIL_FIELDS_ORDER: out[k] = getattr(r, k) or "" + out["matrix_group_label"] = r.matrix_group_label or "" return out @@ -43,4 +45,5 @@ def merged_row_to_dict(r: JdJobMergedRow) -> dict[str, Any]: out: dict[str, Any] = {"id": r.id, "row_index": r.row_index} for k in MERGED_FIELDS_ORDER: out[k] = getattr(r, k) or "" + out["matrix_group_label"] = r.matrix_group_label or "" return out diff --git a/backend/pipeline/serializers.py b/backend/pipeline/serializers.py index d7aa05f..4c7e189 100644 --- a/backend/pipeline/serializers.py +++ b/backend/pipeline/serializers.py @@ -5,36 +5,120 @@ from django.conf import settings from rest_framework import serializers from .cookie_paste import normalize_browser_cookie_paste -from .models import JdProduct, JdProductSnapshot, JobStatus, PipelineJob +from .models import ( + JdProduct, + JdProductSnapshot, + JobStatus, + PipelineJob, + PipelineJobCheckpoint, +) # 与 views._safe_file_for_job 中 mapping 一致，供前端展示「数据源是否就绪」 _REPORT_CONFIG_ALLOWED_KEYS = frozenset( { "llm_comment_sentiment", - "comment_focus_words", - "comment_scenario_groups", + "llm_matrix_group_summaries", + "llm_price_group_summaries", + "llm_promo_group_summaries", + "llm_strategy_opportunities", + "llm_comment_group_summaries", + "llm_group_summaries_chunk_by_matrix", + "chapter8_text_mining_probe", + "chapter8_text_mining_probe_live_llm", + "chapter8_text_mining_probe_llm_chunked", + "chapter8_text_mining_probe_wordcloud", + "chapter8_probe_min_texts", + "chapter8_probe_lda_topics", + "chapter8_probe_top_k_words", + "chapter8_probe_cooc_vocab", + "chapter8_probe_cooc_pairs", + "chapter8_probe_wordcloud_max", "external_market_table_rows", } ) +# 与 ``validate_report_config_body`` 中布尔校验一致；``null`` 视为未设置（须让默认 true/false 生效） +REPORT_CONFIG_BOOL_KEYS = frozenset( + { + "llm_comment_sentiment", + "llm_matrix_group_summaries", + "llm_price_group_summaries", + "llm_promo_group_summaries", + "llm_strategy_opportunities", + "llm_comment_group_summaries", + "llm_group_summaries_chunk_by_matrix", + "chapter8_text_mining_probe", + "chapter8_text_mining_probe_live_llm", + "chapter8_text_mining_probe_llm_chunked", + "chapter8_text_mining_probe_wordcloud", + } +) + def validate_report_config_body(value: dict) -> dict: if not isinstance(value, dict): raise serializers.ValidationError("须为 JSON 对象") + value = dict(value) + for _bk in REPORT_CONFIG_BOOL_KEYS: + if value.get(_bk) is None: + value.pop(_bk, None) + value.pop("llm_section_bridges", None) + # 已废弃字段：静默丢弃，兼容旧任务 JSON + value.pop("comment_focus_words", None) + value.pop("comment_scenario_groups", None) + value.pop("llm_scenario_group_summaries", None) extra = set(value.keys()) - _REPORT_CONFIG_ALLOWED_KEYS if extra: raise serializers.ValidationError( f"未知字段：{', '.join(sorted(extra))}" ) - if "llm_comment_sentiment" in value and value["llm_comment_sentiment"] is not None: - if not isinstance(value["llm_comment_sentiment"], bool): - raise serializers.ValidationError("llm_comment_sentiment 须为 true 或 false") + for k in REPORT_CONFIG_BOOL_KEYS: + if k in value and not isinstance(value[k], bool): + raise serializers.ValidationError(f"{k} 须为 true 或 false") + for k in ( + "chapter8_probe_min_texts", + "chapter8_probe_lda_topics", + "chapter8_probe_top_k_words", + "chapter8_probe_cooc_vocab", + "chapter8_probe_cooc_pairs", + "chapter8_probe_wordcloud_max", + ): + if k in value and value[k] is not None: + if not isinstance(value[k], int): + raise serializers.ValidationError(f"{k} 须为整数") raw = json.dumps(value, ensure_ascii=False) if len(raw) > 120_000: raise serializers.ValidationError("报告配置体积过大") return value +_STRATEGY_CONFIG_ALLOWED_KEYS = frozenset({"use_llm_default"}) +_DEFAULT_STRATEGY_CONFIG: dict[str, bool] = {"use_llm_default": True} + + +def validate_strategy_config_body(value: dict) -> dict: + """策略生成页独立 JSON（与 ``report_config`` 无关）。""" + if not isinstance(value, dict): + raise serializers.ValidationError("须为 JSON 对象") + value = dict(value) + extra = set(value.keys()) - _STRATEGY_CONFIG_ALLOWED_KEYS + if extra: + raise serializers.ValidationError( + f"未知字段：{', '.join(sorted(extra))}" + ) + if "use_llm_default" in value and value["use_llm_default"] is not None: + if not isinstance(value["use_llm_default"], bool): + raise serializers.ValidationError("use_llm_default 须为 true 或 false") + merged = { + **_DEFAULT_STRATEGY_CONFIG, + **{k: value[k] for k in _STRATEGY_CONFIG_ALLOWED_KEYS if k in value}, + } + raw = json.dumps(merged, ensure_ascii=False) + if len(raw) > 32_000: + raise serializers.ValidationError("策略配置体积过大") + return merged + + _ARTIFACT_FILES: tuple[tuple[str, str], ...] = ( ("merged", "keyword_pipeline_merged.csv"), ("pc_search", "pc_search_export.csv"), @@ -49,6 +133,7 @@ class PipelineJobSerializer(serializers.ModelSerializer): inline_cookie_used = serializers.SerializerMethodField() analysis_artifacts = serializers.SerializerMethodField() + checkpoint = serializers.SerializerMethodField() class Meta: model = PipelineJob @@ -67,8 +152,11 @@ class PipelineJobSerializer(serializers.ModelSerializer): "list_pages", "scenario_filter_enabled", "report_config", + "strategy_config", "status", "cancellation_requested", + "resume_from_checkpoint", + "checkpoint", "run_dir", "error_message", "analysis_artifacts", @@ -79,22 +167,39 @@ class PipelineJobSerializer(serializers.ModelSerializer): "id", "inline_cookie_used", "analysis_artifacts", + "checkpoint", "status", "cancellation_requested", + "resume_from_checkpoint", "run_dir", "error_message", "created_at", "updated_at", "report_config", + "strategy_config", ] def get_inline_cookie_used(self, obj: PipelineJob) -> bool: return bool((obj.cookie_text or "").strip()) + def get_checkpoint(self, obj: PipelineJob) -> dict | None: + try: + c = obj.checkpoint_row + except PipelineJobCheckpoint.DoesNotExist: + return None + return { + "phase": c.phase, + "payload": c.payload, + "hint_zh": c.hint_zh, + "updated_at": c.updated_at, + } + def get_analysis_artifacts(self, obj: PipelineJob) -> dict[str, bool] | None: - if obj.status not in (JobStatus.SUCCESS, JobStatus.CANCELLED) or not ( - obj.run_dir or "" - ).strip(): + if obj.status not in ( + JobStatus.SUCCESS, + JobStatus.CANCELLED, + JobStatus.PAUSED, + ) or not (obj.run_dir or "").strip(): return None try: base = Path(obj.run_dir).expanduser().resolve() @@ -181,6 +286,17 @@ def _jd_data_root() -> Path: return (Path(root) / "data" / "JD").resolve() +class JobResumeRequestSerializer(serializers.Serializer): + """从断点续跑时可选更新 Cookie。""" + + cookie_text = serializers.CharField( + required=False, + allow_blank=True, + default="", + max_length=500_000, + ) + + class CreatePipelineJobSerializer(serializers.Serializer): keyword = serializers.CharField(max_length=256, trim_whitespace=True) platform = serializers.ChoiceField(choices=["jd"], default="jd") @@ -262,6 +378,15 @@ class JobReportConfigPatchSerializer(serializers.Serializer): return validate_report_config_body(value) +class JobStrategyConfigPatchSerializer(serializers.Serializer): + strategy_config = serializers.JSONField() + + def validate_strategy_config(self, value): + if not isinstance(value, dict): + raise serializers.ValidationError("须为 JSON 对象") + return validate_strategy_config_body(value) + + class RegenerateReportRequestSerializer(serializers.Serializer): """重新生成竞品报告：规则引擎或大模型（与 ``AI_crawler.chat_completion_text`` 同一网关）。""" @@ -285,6 +410,13 @@ class StrategyDraftRequestSerializer(serializers.Serializer): product_role = serializers.CharField( required=False, allow_blank=True, default="", max_length=500, trim_whitespace=False ) + stage_goal_type = serializers.CharField( + required=False, + allow_blank=True, + default="", + max_length=1000, + trim_whitespace=False, + ) time_horizon = serializers.CharField( required=False, allow_blank=True, default="", max_length=200, trim_whitespace=False ) @@ -319,6 +451,21 @@ class StrategyDraftRequestSerializer(serializers.Serializer): pillar_comm = serializers.CharField( required=False, allow_blank=True, default="", max_length=800, trim_whitespace=False ) + audience_segment = serializers.CharField( + required=False, allow_blank=True, default="", max_length=500, trim_whitespace=False + ) + competitor_reference = serializers.CharField( + required=False, allow_blank=True, default="", max_length=800, trim_whitespace=False + ) + resource_notes = serializers.CharField( + required=False, allow_blank=True, default="", max_length=1000, trim_whitespace=False + ) + marketing_strategy = serializers.CharField( + required=False, allow_blank=True, default="", max_length=2000, trim_whitespace=False + ) + general_strategy = serializers.CharField( + required=False, allow_blank=True, default="", max_length=2000, trim_whitespace=False + ) ack_risk_keywords = serializers.BooleanField(required=False, default=False) ack_risk_price = serializers.BooleanField(required=False, default=False) ack_risk_concentration = serializers.BooleanField(required=False, default=False) @@ -327,3 +474,42 @@ class StrategyDraftRequestSerializer(serializers.Serializer): default="rules", required=False, ) + strategy_matrix_group_index = serializers.IntegerField( + required=False, + allow_null=True, + min_value=0, + ) + strategy_matrix_group = serializers.CharField( + required=False, + allow_blank=True, + default="", + max_length=200, + trim_whitespace=True, + ) + + +class MarketingDetailPackRequestSerializer(serializers.Serializer): + """策略稿正文 → 核心信息卡 → 营销内容（多触点文案，两步 LLM）。""" + + strategy_markdown = serializers.CharField( + required=True, + allow_blank=False, + min_length=20, + max_length=600_000, + trim_whitespace=False, + ) + business_notes = serializers.CharField( + required=False, + allow_blank=True, + default="", + max_length=20_000, + trim_whitespace=False, + ) + strategy_decisions = serializers.JSONField(required=False, allow_null=True) + + def validate_strategy_decisions(self, value): + if value is None: + return {} + if not isinstance(value, dict): + raise serializers.ValidationError("须为 JSON 对象") + return value diff --git a/backend/pipeline/strategy_draft.py b/backend/pipeline/strategy_draft.py deleted file mode 100644 index 09e65c5..0000000 --- a/backend/pipeline/strategy_draft.py +++ /dev/null @@ -1,363 +0,0 @@ -""" -市场策略 Markdown 草稿：侧重**策略制定框架**（目标、战场、定位、支柱、行动）， -基于同任务结构化摘要与可选业务备注规则生成；附录为关键数据速览。 - -后续可接 LLM 润色；当前无模型调用，便于验收与追溯。 -""" -from __future__ import annotations - -import math -from typing import Any - - -def _esc(s: Any) -> str: - t = "" if s is None else str(s).strip() - return t.replace("\r\n", "\n").replace("\r", "\n") - - -def _pct(x: Any) -> str: - if x is None: - return "—" - try: - v = float(x) - if math.isnan(v) or math.isinf(v): - return "—" - return f"{100 * v:.1f}%" - except (TypeError, ValueError): - return "—" - - -def _num(x: Any) -> str: - if x is None: - return "—" - if isinstance(x, bool): - return str(x) - if isinstance(x, int): - return str(x) - if isinstance(x, float): - if math.isnan(x) or math.isinf(x): - return "—" - if x == int(x): - return str(int(x)) - return f"{x:.2f}" - return str(x) - - -def _cr_narrative(label: str, cr1: Any, cr3: Any, top: Any) -> str | None: - """从集中度生成一句策略向描述，无数据则返回 None。""" - try: - c1 = float(cr1) if cr1 is not None else None - except (TypeError, ValueError): - c1 = None - if c1 is None and not (top or "").strip(): - return None - top_s = _esc(top) or "—" - if c1 is not None: - if c1 >= 0.4: - tone = "偏高，头部资源集中" - elif c1 >= 0.25: - tone = "中等，存在可争夺空间" - else: - tone = "相对分散，差异化切入点可能更多" - return f"- **{label}**：第一大品牌/店份额 ≈ {_pct(cr1)}，前三合计份额 ≈ {_pct(cr3)}；头部为「{top_s}」。*粗判：{tone}。*" - return f"- **{label}**：头部标签「{top_s}」（缺少份额指标时可结合列表/商详数据补全）。" - - -def _goal_bullet(label: str, user_val: str, placeholder: str) -> str: - v = _esc(user_val).strip() - if v: - return f"- **{label}**：{v}" - return f"- **{label}**：*（{placeholder}）*" - - -def _pillar_cell(user_val: str) -> str: - v = _esc(user_val).strip() - return v if v else "*待填*" - - -def _pos_mark(choice: str, key: str) -> str: - return "[x]" if choice == key else "[ ]" - - -def _risk_line(checked: bool, text: str) -> str: - mark = "[x]" if checked else "[ ]" - return f"- {mark} {text}" - - -def build_strategy_draft_markdown( - *, - job_id: int, - keyword: str, - brief: dict[str, Any], - business_notes: str = "", - generated_at_iso: str = "", - strategy_decisions: dict[str, Any] | None = None, -) -> str: - """生成可下载的 Markdown：策略框架为主，附录为数据速览。""" - d = strategy_decisions or {} - pos = _esc(d.get("positioning_choice") or "").strip() - kw = _esc(brief.get("keyword")) or _esc(keyword) or "—" - lines: list[str] = [ - f"# 市场策略制定草稿 · 「{kw}」", - "", - "> 本稿用于**辅助制定市场策略**；由规则根据本批次结构化摘要与业务备注生成，**非大模型自由发挥**，定稿前请业务修订。", - "", - ] - if generated_at_iso: - lines.append(f"> **生成时间**：{_esc(generated_at_iso)} · **任务 ID**：{job_id}") - lines.append("") - - lines.extend( - [ - "---", - "", - "## 一、战略背景与目标（请业务补全）", - "", - _goal_bullet("本品角色", str(d.get("product_role") or ""), "新品 / 追赶 / 防守 / 拓品类 …"), - _goal_bullet("时间范围", str(d.get("time_horizon") or ""), "如：本季度 / 未来 12 周"), - _goal_bullet( - "成功标准（可量化）", - str(d.get("success_criteria") or ""), - "如：搜索位次、转化率、声量、复购 …", - ), - _goal_bullet("非目标（明确不做什么）", str(d.get("non_goals") or ""), "可选"), - "", - ] - ) - - scope = brief.get("scope") or {} - merged_n = scope.get("merged_sku_count") - comm_n = scope.get("comment_flat_rows") - lines.extend( - [ - "## 二、战场界定（监测语境）", - "", - f"- **监测关键词 / 货架语境**：{kw}", - f"- **批次**：{_esc(brief.get('batch_label')) or '—'}", - ] - ) - if merged_n is not None or comm_n is not None: - lines.append( - f"- **深入样本规模**：深入 SKU ≈ {_num(merged_n)}；评价扁平条数 ≈ {_num(comm_n)}。" - "*策略含义：样本越大，以下「假设」越需抽样复核原评论。*" - ) - bf = _esc(d.get("battlefield_one_line") or "").strip() - if bf: - lines.append(f"- **一句话战场**：{bf}") - else: - lines.append( - "- **一句话战场**：*（请用业务语言写：我们在哪个需求场景、与谁抢同一批用户？）*" - ) - lines.append("") - - conc = brief.get("concentration") or {} - shops = conc.get("shops_from_list") or {} - dbrand = conc.get("detail_brand_among_merged") or {} - lines.extend(["## 三、竞争格局 → 策略含义", ""]) - n_shop = _cr_narrative("列表侧店铺集中度", shops.get("cr1"), shops.get("cr3"), shops.get("top_label")) - n_brand = _cr_narrative("深入样本内品牌集中度", dbrand.get("cr1"), dbrand.get("cr3"), dbrand.get("top_label")) - if n_shop: - lines.append(n_shop) - if n_brand: - lines.append(n_brand) - if not n_shop and not n_brand: - lines.append("*本摘要未含集中度指标，请结合本批次竞争结构数据补全后再写判断。*") - lines.extend( - [ - "", - "**可下判断的提问（自测）**", - "", - "- 若头部已占稳心智，本品是**侧翼**还是**正面替代**？", - "- 店铺/品牌分散时，是否适合用**细分场景**或**内容教育**切入？", - "", - ] - ) - stance = _esc(d.get("competitive_stance") or "").strip() - stance_line = { - "flank": "- **本品倾向**：倾向**侧翼切入**，避免与头部正面硬碰。", - "head_on": "- **本品倾向**：倾向**正面替代**，对标头部主战场。", - "both": "- **本品倾向**：计划**分层推进**（部分场景侧翼、部分场景正面）。", - "undecided": "- **本品倾向**：**尚未拍板**，需在会议中对齐后再定主战场叙事。", - }.get(stance) - if stance_line: - lines.append(stance_line) - lines.append("") - - mix = brief.get("category_mix_top") or [] - if mix: - lines.append("### 类目结构提示（Top）") - lines.append("") - lines.append("*以下仅作「货架长什么样」的速记。*") - for item in mix[:6]: - if isinstance(item, dict): - lines.append(f"- {_esc(item.get('label'))}：{_num(item.get('count'))}") - lines.append("") - - pst = brief.get("price_stats") or {} - lines.extend(["## 四、价格带与定位选项（启发式）", ""]) - if pst.get("n"): - src = _esc(brief.get("price_stats_source")) or "—" - lines.extend( - [ - f"- **统计口径**：{src}，有效价样本 n = {_num(pst.get('n'))}。", - f"- **展示价区间**：{_num(pst.get('min'))} ～ {_num(pst.get('max'))}；**中位数** {_num(pst.get('median'))}。", - "", - "**定位选项（请勾一条或改写，并写明理由）**", - "", - f"- {_pos_mark(pos, 'top')} **贴顶**：对标中高位或头部价位带，强调品质/成分/背书。", - f"- {_pos_mark(pos, 'mid')} **卡腰**：围绕中位数一带，强调性价比与场景匹配。", - f"- {_pos_mark(pos, 'entry')} **下探**：贴近区间下限，强调入门与拉新（注意毛利与品牌调性）。", - f"- {_pos_mark(pos, 'different')} **另起带**：刻意避开主价格带，用规格/组合/服务差异化。", - "", - ] - ) - else: - lines.append("*摘要中无价带统计，请结合本批次价格相关数据补全后再填上表。*") - lines.append("") - lines.extend( - [ - "**定位选项（请勾一条或改写，并写明理由）**", - "", - f"- {_pos_mark(pos, 'top')} **贴顶**：对标中高位或头部价位带，强调品质/成分/背书。", - f"- {_pos_mark(pos, 'mid')} **卡腰**：围绕中位数一带，强调性价比与场景匹配。", - f"- {_pos_mark(pos, 'entry')} **下探**：贴近区间下限，强调入门与拉新（注意毛利与品牌调性）。", - f"- {_pos_mark(pos, 'different')} **另起带**：刻意避开主价格带，用规格/组合/服务差异化。", - "", - ] - ) - - ckw = brief.get("comment_focus_keywords") or [] - usc = brief.get("usage_scenarios") or [] - lines.extend(["## 五、用户需求与场景 — 可写成策略的假设", ""]) - lines.append( - "*下列由关注词/场景**计数**转化而来，是「待验证假设」而非结论；请结合评价原文抽样修订。*" - ) - lines.append("") - if ckw: - for item in ckw[:8]: - if isinstance(item, dict): - w = _esc(item.get("word")) - c = _num(item.get("count")) - lines.append( - f"- **假设**：用户决策中「{w}」被频繁提及（约 {c} 次统计命中）—— " - f"*可追问：本品故事是否正面回应？传播关键词是否覆盖？*" - ) - if usc: - for item in usc[:6]: - if isinstance(item, dict): - sc = _esc(item.get("scenario")) - cn = _num(item.get("count")) - sh = _pct(item.get("share_of_text_units")) - lines.append( - f"- **场景命题**：「{sc}」在预设场景中约 {cn} 条、约占 {sh} 文本单元—— " - f"*可追问：主图/详情/客服话术是否对齐该场景？*" - ) - if not ckw and not usc: - lines.append("*摘要中无关注词/场景组结果，请补全评论侧分析后再写本节。*") - lines.append("") - - hints = brief.get("strategy_hints") or [] - lines.extend( - [ - "## 六、机会方向与策略支柱（草案）", - "", - "### 规则引擎提示（来自摘要 `strategy_hints`）", - "", - ] - ) - if hints: - for h in hints: - lines.append(f"- {_esc(h)}") - else: - lines.append("*（当前无自动线索，请结合本批次结论手写 3～5 条机会）*") - pp = str(d.get("pillar_product") or "") - pr = str(d.get("pillar_price") or "") - pch = str(d.get("pillar_channel") or "") - pcm = str(d.get("pillar_comm") or "") - lines.extend( - [ - "", - "### 策略支柱 — 请业务逐项填空", - "", - "| 支柱 | 本品打算怎么做 | 与头部差异 | 证据 / 出处 |", - "|------|----------------|------------|-------------|", - f"| 产品 | {_pillar_cell(pp)} | *待填* | *§* |", - f"| 价格 | {_pillar_cell(pr)} | *待填* | *§* |", - f"| 渠道/触点 | {_pillar_cell(pch)} | *待填* | *§* |", - f"| 传播与内容 | {_pillar_cell(pcm)} | *待填* | *§* |", - "", - ] - ) - - rk = bool(d.get("ack_risk_keywords")) - rp = bool(d.get("ack_risk_price")) - rc = bool(d.get("ack_risk_concentration")) - lines.extend( - [ - "## 七、风险与待证伪", - "", - _risk_line(rk, "关注词/场景是否**以偏概全**？（需原评论抽样）"), - _risk_line(rp, "价格带是否含大促/异常挂价？（需核对清洗口径）"), - _risk_line(rc, "列表集中度与深入样本品牌是否**矛盾**？（需解释渠道差异）"), - "", - ] - ) - - notes = _esc(business_notes) - lines.extend( - [ - "## 八、业务约束与内部判断", - "", - (notes if notes else "*（未填写。建议补充：渠道红线、价位策略、竞品对标名单、预算量级等。）*"), - "", - ] - ) - - lines.extend( - [ - "## 九、建议下一步（策略向）", - "", - "- [ ] 开会对齐：**§一** 目标与 **§八** 约束，确认 1～2 条主策略命题。", - "- [ ] 为 **§六** 策略支柱表格每一行各找 **1 条数据证据**（注明出处）。", - "- [ ] 产出 **12 周节奏表**（里程碑 + 负责人），与本品排期挂钩。", - "- [ ] 定义 **3 个可观测指标**（周或双周复盘）。", - "", - "---", - "", - "## 附录 · 本任务关键数据速览", - "", - f"- **关键词**：{kw} · **摘要版本**：v{_num(brief.get('schema_version'))}", - ] - ) - meta = brief.get("meta") - meta_labels = { - "page_start": "起始页", - "page_to": "采集至页", - "max_skus_config": "SKU 上限", - "scenario_filter_enabled": "场景筛选", - } - if isinstance(meta, dict) and meta: - bits = [] - for k in ("page_start", "page_to", "max_skus_config", "scenario_filter_enabled"): - if k in meta: - label = meta_labels.get(k, k) - bits.append(f"{label}={_esc(meta.get(k))}") - if bits: - lines.append(f"- **采集参数快照**：{'; '.join(bits)}") - raw = brief.get("pc_search_raw") or {} - if raw.get("result_count_consensus") is not None: - lines.append( - f"- **列表申报规模（resultCount）**：{_num(raw.get('result_count_consensus'))}" - ) - lines.extend( - [ - "", - "*同目录含本批次 CSV 与分析产出，可对照使用。*", - "", - "---", - "", - "*本稿由工作台「市场策略制定」生成；与同任务结构化分析数据一致。*", - "", - ] - ) - return "\n".join(lines) diff --git a/backend/pipeline/tasks.py b/backend/pipeline/tasks.py index 1ad8a5c..16f6104 100644 --- a/backend/pipeline/tasks.py +++ b/backend/pipeline/tasks.py @@ -12,7 +12,7 @@ from django.utils import timezone from .cookie_paste import normalize_browser_cookie_paste from .ingest import try_ingest_job_full -from .jd_runner import ( +from .jd.runner import ( resolve_pipeline_run_directory_for_job, try_write_competitor_report_if_merged_exists, ) diff --git a/backend/pipeline/tests/conftest.py b/backend/pipeline/tests/conftest.py new file mode 100644 index 0000000..a0d8efc --- /dev/null +++ b/backend/pipeline/tests/conftest.py @@ -0,0 +1,10 @@ +"""在收集本目录测试模块之前初始化 Django，避免 ``import pipeline.jd.runner`` 触发 AppRegistryNotReady。""" +from __future__ import annotations + +import os + +os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings") + +import django # noqa: E402 + +django.setup() diff --git a/backend/pipeline/tests/test_brief_compact.py b/backend/pipeline/tests/test_brief_compact.py index 97031ef..924c480 100644 --- a/backend/pipeline/tests/test_brief_compact.py +++ b/backend/pipeline/tests/test_brief_compact.py @@ -3,7 +3,10 @@ from __future__ import annotations from django.test import SimpleTestCase -from pipeline.brief_compact import compact_brief_for_llm, matrix_overview_for_llm +from pipeline.reporting.brief_compact import ( + compact_brief_for_llm, + matrix_overview_for_llm, +) class BriefCompactTests(SimpleTestCase): diff --git a/backend/pipeline/tests/test_brief_pack.py b/backend/pipeline/tests/test_brief_pack.py index b545fc3..d25b622 100644 --- a/backend/pipeline/tests/test_brief_pack.py +++ b/backend/pipeline/tests/test_brief_pack.py @@ -9,7 +9,7 @@ from tempfile import TemporaryDirectory from django.test import SimpleTestCase -from pipeline.brief_pack import ( +from pipeline.reporting.brief_pack import ( build_brief_pack_zip_bytes, markdown_summary_from_brief, ) diff --git a/backend/pipeline/tests/test_brief_strategy_scope.py b/backend/pipeline/tests/test_brief_strategy_scope.py new file mode 100644 index 0000000..47eaaa0 --- /dev/null +++ b/backend/pipeline/tests/test_brief_strategy_scope.py @@ -0,0 +1,126 @@ +"""brief 按矩阵分组收窄。""" +from __future__ import annotations + +from django.test import SimpleTestCase + +from pipeline.reporting.brief_strategy_scope import ( + filter_brief_for_strategy_matrix_group, + list_matrix_groups_for_api, + resolve_strategy_matrix_group_index, +) + + +class BriefStrategyScopeTests(SimpleTestCase): + def _sample_brief(self) -> dict: + return { + "schema_version": 1, + "keyword": "低GI", + "matrix_by_group": [ + { + "group": "饮料", + "sku_count": 1, + "skus": [ + { + "brand": "A", + "shop": "S1", + "category": "水饮 > 茶", + "list_price_show": "10", + } + ], + }, + { + "group": "饼干", + "sku_count": 2, + "skus": [ + { + "brand": "B", + "shop": "S2", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "list_price_show": "20", + }, + { + "brand": "B", + "shop": "S2", + "category": "休闲食品 > 饼干 > 苏打饼干", + "list_price_show": "22", + }, + ], + }, + ], + "consumer_feedback_by_matrix_group": [ + { + "group": "饮料", + "comment_rows": 5, + "effective_comment_text_units": 5, + "focus_keyword_hits": [{"word": "甜", "count": 2}], + "scenarios_top": [ + { + "scenario": "解渴", + "count": 2, + "share_of_text_units": 0.4, + } + ], + }, + { + "group": "饼干", + "comment_rows": 8, + "effective_comment_text_units": 8, + "focus_keyword_hits": [{"word": "脆", "count": 3}], + "scenarios_top": [ + { + "scenario": "早餐", + "count": 4, + "share_of_text_units": 0.5, + } + ], + }, + ], + "usage_scenarios_by_matrix_group": [ + {"group": "饮料", "scenarios": []}, + {"group": "饼干", "scenarios": [{"scenario": "早餐", "count": 4}]}, + ], + "notes": ["原注"], + } + + def test_list_matrix_groups(self) -> None: + b = self._sample_brief() + m = list_matrix_groups_for_api(b) + self.assertEqual(len(m), 2) + self.assertEqual(m[1]["group"], "饼干") + self.assertEqual(m[1]["index"], 1) + + def test_resolve_by_label(self) -> None: + b = self._sample_brief() + idx, err = resolve_strategy_matrix_group_index( + b, matrix_group_label="饼干" + ) + self.assertIsNone(err) + self.assertEqual(idx, 1) + + def test_resolve_by_index(self) -> None: + b = self._sample_brief() + idx, err = resolve_strategy_matrix_group_index( + b, matrix_group_index=1 + ) + self.assertIsNone(err) + self.assertEqual(idx, 1) + + def test_resolve_mismatch(self) -> None: + b = self._sample_brief() + idx, err = resolve_strategy_matrix_group_index( + b, matrix_group_index=0, matrix_group_label="饼干" + ) + self.assertIsNotNone(err) + self.assertIsNone(idx) + + def test_filter_keeps_only_group(self) -> None: + b = self._sample_brief() + out = filter_brief_for_strategy_matrix_group(b, matrix_group_index=1) + self.assertEqual(len(out["matrix_by_group"]), 1) + self.assertEqual(out["matrix_by_group"][0]["group"], "饼干") + self.assertEqual(len(out["matrix_by_group"][0]["skus"]), 2) + self.assertEqual(len(out["consumer_feedback_by_matrix_group"]), 1) + self.assertEqual(out["comment_focus_keywords"][0]["word"], "脆") + self.assertEqual(out["price_stats_source"], "strategy_scope_matrix_group_skus") + self.assertIn("strategy_scope_applied", out) + self.assertEqual(out["strategy_scope_applied"]["group"], "饼干") diff --git a/backend/pipeline/tests/test_buyer_offer_export_csv.py b/backend/pipeline/tests/test_buyer_offer_export_csv.py new file mode 100644 index 0000000..613a64d --- /dev/null +++ b/backend/pipeline/tests/test_buyer_offer_export_csv.py @@ -0,0 +1,49 @@ +"""buyer_offer_export_csv：榜单列与促销文案列（分隔符拼接）。""" +from __future__ import annotations + +import sys +from pathlib import Path + +from django.conf import settings +from django.test import SimpleTestCase + +from pipeline.csv.schema import strip_buyer_ranking_line_prefix + + +class BuyerOfferExportCsvTests(SimpleTestCase): + def test_strip_buyer_ranking_prefix(self) -> None: + self.assertEqual( + strip_buyer_ranking_line_prefix("榜单/曝光：老金磨方药食同源热卖榜·第1名。"), + "老金磨方药食同源热卖榜·第1名。", + ) + self.assertEqual( + strip_buyer_ranking_line_prefix("榜单/曝光粗粮饼干热卖榜·第5名"), + "粗粮饼干热卖榜·第5名", + ) + + def test_ranking_and_promo_from_profile(self) -> None: + dr = Path(settings.CRAWLER_JD_ROOT).resolve() / "detail" + if str(dr) not in sys.path: + sys.path.insert(0, str(dr)) + import jd_detail_buyer_extraction as be # noqa: WPS433 + + prof = { + "visibility": {"rankings": ["20-40元酥性饼干热卖榜·第8名"]}, + "buyer_summary_lines": [ + "当前展示「到手价」约 27.97 元。", + "详情页优惠拆解：购买立减。", + "榜单/曝光：应被排除。", + "送达：应被排除。", + "企业采购提示：应被排除。", + ], + } + self.assertEqual( + be.buyer_ranking_line_from_profile(prof), + "20-40元酥性饼干热卖榜·第8名。", + ) + t = be.buyer_promo_text_from_profile(prof) + self.assertNotIn("榜单", t) + self.assertNotIn("送达", t) + self.assertNotIn("企业采购", t) + self.assertIn(" | ", t) + self.assertIn("到手价", t) diff --git a/backend/pipeline/tests/test_chat_content_thinking_strip.py b/backend/pipeline/tests/test_chat_content_thinking_strip.py new file mode 100644 index 0000000..01723e6 --- /dev/null +++ b/backend/pipeline/tests/test_chat_content_thinking_strip.py @@ -0,0 +1,43 @@ +"""strip_thinking_leaks / normalize_message_content 剥掉混在正文的思考标签。""" +from __future__ import annotations + +import pytest + +from pipeline.openai_gateway.chat_content import ( + normalize_message_content, + strip_thinking_leaks_from_model_text, +) + +_LT, _GT, _SL = chr(60), chr(62), chr(47) + + +def _t(open_b: str, close_b: str, inner: str) -> str: + return _LT + open_b + _GT + inner + _LT + _SL + close_b + _GT + + +def test_strip_redacted_thinking_block() -> None: + raw = _t("redacted_thinking", "redacted_thinking", "reasoning") + "pong" + assert strip_thinking_leaks_from_model_text(raw) == "pong" + + +def test_strip_redacted_open_think_close() -> None: + raw = _t("redacted_thinking", "think", "a") + "b" + assert strip_thinking_leaks_from_model_text(raw) == "b" + + +def test_strip_think_block() -> None: + raw = _t("think", "think", "x") + "y" + assert strip_thinking_leaks_from_model_text(raw) == "y" + + +def test_normalize_strips() -> None: + raw = _t("redacted_thinking", "redacted_thinking", "x") + "\nok" + assert normalize_message_content(raw) == "ok" + + +def test_preserve_thinking_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("MA_LLM_PRESERVE_THINKING_IN_OUTPUT", "1") + raw = _t("redacted_thinking", "redacted_thinking", "inside") + "after" + out = strip_thinking_leaks_from_model_text(raw) + assert "inside" in out + assert "after" in out diff --git a/backend/pipeline/tests/test_comment_group_summaries_llm.py b/backend/pipeline/tests/test_comment_group_summaries_llm.py new file mode 100644 index 0000000..9912856 --- /dev/null +++ b/backend/pipeline/tests/test_comment_group_summaries_llm.py @@ -0,0 +1,89 @@ +"""§8 末「细类评论要点归纳」大模型：单细类烟测（mock 网关，不调真实 API）。""" +from __future__ import annotations + +import json +from unittest.mock import patch + +from django.test import SimpleTestCase + +from pipeline.llm.generate import ( + generate_comment_group_summaries_llm, + generate_comment_group_summaries_llm_chunked, +) + + +def _load_jcr(): + from pipeline.competitor_report import jd_report as jcr # noqa: WPS433 + + return jcr + + +def _payload_single_category_biscuit() -> tuple[list[dict], str, str]: + """合并表 + 评价仅对应一个矩阵细类「饼干」。""" + jcr = _load_jcr() + sku_h = "SKU(skuId)" + title_h = "标题(wareName)" + merged = [ + { + sku_h: "111", + "detail_category_path": "食品饮料 > 休闲食品 > 饼干 > 粗粮饼干", + title_h: "测试饼干", + "detail_shop_name": "测试店", + }, + ] + comments = [ + { + "sku": "111", + "tagCommentContent": "口感不错，低GI很适合控糖", + }, + ] + fb = jcr._consumer_feedback_by_matrix_group( + merged_rows=merged, + comment_rows=comments, + sku_header=sku_h, + ) + pl = jcr.build_comment_groups_llm_payload( + feedback_groups=fb, + merged_rows=merged, + sku_header=sku_h, + title_h=title_h, + ) + return pl, sku_h, title_h + + +class CommentGroupSummariesLlmTests(SimpleTestCase): + def test_single_matrix_group_payload_and_llm_smoke(self) -> None: + pl, _, _ = _payload_single_category_biscuit() + self.assertEqual(len(pl), 1) + self.assertEqual(pl[0].get("group"), "饼干") + self.assertIn("sample_text_snippets", pl[0]) + self.assertTrue(any("饼干" in s for s in pl[0]["sample_text_snippets"])) + + with patch( + "pipeline.llm.generate_group_summaries.call_llm", + return_value="#### 饼干\n单测归纳段落。", + ) as mock_llm: + out = generate_comment_group_summaries_llm(pl, keyword="低GI测试") + + mock_llm.assert_called_once() + _sys, user = mock_llm.call_args[0] + self.assertIn("细类评论要点归纳", user) + self.assertIn("低GI测试", user) + raw = user.split("正文（Markdown）。\n\n", 1)[-1] + data = json.loads(raw.strip()) + self.assertEqual(data["keyword"], "低GI测试") + self.assertEqual(len(data["groups"]), 1) + self.assertEqual(data["groups"][0]["group"], "饼干") + self.assertIn("饼干", out) + + def test_chunked_single_category_one_gateway_call(self) -> None: + """与生产「按细类拆分」一致：仅一个细类时只请求一次。""" + pl, _, _ = _payload_single_category_biscuit() + with patch( + "pipeline.llm.generate_group_summaries.call_llm", + return_value="#### 饼干\nchunked。", + ) as mock_llm: + out = generate_comment_group_summaries_llm_chunked(pl, keyword="低GI测试") + + self.assertEqual(mock_llm.call_count, 1) + self.assertIn("饼干", out) diff --git a/backend/pipeline/tests/test_competitor_brief.py b/backend/pipeline/tests/test_competitor_brief.py index 6dccb96..d53310a 100644 --- a/backend/pipeline/tests/test_competitor_brief.py +++ b/backend/pipeline/tests/test_competitor_brief.py @@ -1,21 +1,22 @@ """结构化竞品摘要：空样本烟测（不依赖真实 run_dir CSV）。""" from __future__ import annotations -import sys import tempfile from pathlib import Path -from django.conf import settings from django.test import SimpleTestCase +from pipeline.competitor_report import jd_report as jcr +from pipeline.competitor_report.comment_sentiment import ( + _comment_sentiment_lexicon, + build_comment_sentiment_llm_payload, +) +from pipeline.csv.schema import infer_total_sales_from_sales_floor +from pipeline.reporting.charts import _cn_volume_int + class BuildCompetitorBriefTests(SimpleTestCase): def test_empty_merged_json_safe(self) -> None: - root = Path(settings.CRAWLER_JD_ROOT).resolve() - if str(root) not in sys.path: - sys.path.insert(0, str(root)) - import jd_competitor_report as jcr # noqa: WPS433 - with tempfile.TemporaryDirectory() as td: run_dir = Path(td) (run_dir / "pc_search_raw").mkdir(parents=True) @@ -33,18 +34,42 @@ class BuildCompetitorBriefTests(SimpleTestCase): self.assertEqual(out["scope"]["merged_sku_count"], 0) self.assertIsInstance(out["strategy_hints"], list) self.assertEqual(out["matrix_by_group"], []) - self.assertIn("comment_sentiment_lexicon", out) - self.assertEqual(out["comment_sentiment_lexicon"].get("text_units"), 0) + self.assertNotIn("comment_sentiment_lexicon", out) import json json.dumps(out) - def test_custom_focus_words_in_report_config(self) -> None: - root = Path(settings.CRAWLER_JD_ROOT).resolve() - if str(root) not in sys.path: - sys.path.insert(0, str(root)) - import jd_competitor_report as jcr # noqa: WPS433 + def test_comment_sentiment_llm_payload_has_semantic_pool(self) -> None: + texts = ["口感软硬适中很好吃", "太差了不建议"] + attr = [f"【细类：A｜SKU：1｜品名：x｜店铺：y】{t}" for t in texts] + pl = build_comment_sentiment_llm_payload( + texts, + attributed_texts=attr, + shuffle_seed="unit-test-seed", + semantic_pool_max=10, + ) + self.assertIn("sample_reviews_semantic_pool", pl) + self.assertEqual(pl.get("sentiment_bucket_method"), "keyword_substring_heuristic") + self.assertGreaterEqual(len(pl["sample_reviews_semantic_pool"]), 1) + def test_comment_sentiment_score_then_lexeme(self) -> None: + texts = ["很好吃", "太差了", "一般般"] + scores = [5, 1, 3] + lex = _comment_sentiment_lexicon(texts, scores) + self.assertEqual(lex.get("method"), "score_then_lexeme") + self.assertEqual(lex.get("positive_only"), 1) + self.assertEqual(lex.get("negative_only"), 1) + self.assertEqual(lex.get("neutral_or_empty"), 1) + pl = build_comment_sentiment_llm_payload(texts, scores=scores) + self.assertEqual(pl.get("sentiment_bucket_method"), "score_then_lexeme") + + def test_comment_sentiment_all_scores_missing_falls_back_keyword(self) -> None: + texts = ["好吃推荐", "差评"] + scores = [None, None] + lex = _comment_sentiment_lexicon(texts, scores) + self.assertEqual(lex.get("method"), "keyword_lexicon") + + def test_custom_focus_words_in_report_config(self) -> None: with tempfile.TemporaryDirectory() as td: run_dir = Path(td) (run_dir / "pc_search_raw").mkdir(parents=True) @@ -65,3 +90,118 @@ class BuildCompetitorBriefTests(SimpleTestCase): words = {x["word"] for x in out["comment_focus_keywords"]} self.assertIn("自定义词阿尔法", words) + + def test_matrix_groups_require_detail_category_path(self) -> None: + sku_h = "SKU(skuId)" + merged = [ + { + sku_h: "111", + "detail_category_path": "食品饮料 > 休闲食品 > 饼干 > 粗粮饼干", + "标题(wareName)": "A", + }, + {sku_h: "222", "标题(wareName)": "B"}, + ] + groups = jcr._merged_rows_grouped_for_matrix(merged) + self.assertEqual(len(groups), 1) + self.assertEqual(len(groups[0][1]), 1) + self.assertEqual(groups[0][1][0][sku_h], "111") + smap = jcr._sku_to_matrix_group_map(merged, sku_h) + self.assertEqual(smap.get("111"), "饼干") + self.assertNotIn("222", smap) + fb = jcr._consumer_feedback_by_matrix_group( + merged_rows=merged, + comment_rows=[ + {"sku": "222", "tagCommentContent": "缺路径仍不应计入按细类统计"}, + {"sku": "111", "tagCommentContent": "有路径进细类"}, + ], + sku_header=sku_h, + ) + counts = {g: len(cr) for g, cr, _ in fb} + self.assertEqual(counts.get("饼干"), 1) + + def test_comment_lines_with_product_context_prefix(self) -> None: + """评价抽样须带细类/SKU/品名前缀，便于归因。""" + sku_h = "SKU(skuId)" + title_h = "标题(wareName)" + merged = [ + { + sku_h: "100", + title_h: "低GI全麦饼干1kg", + "detail_brand": "B", + "detail_price_final": "29", + "detail_shop_name": "店", + "detail_category_path": "休闲食品 > 饼干 > 粗粮饼干", + "detail_product_attributes": "x", + }, + ] + comments = [{"sku": "100", "tagCommentContent": "整体口感还差点意思"}] + lines = jcr._comment_lines_with_product_context( + comments, merged, sku_header=sku_h, title_h=title_h + ) + self.assertEqual(len(lines), 1) + self.assertIn("【细类：", lines[0]) + self.assertIn("SKU：100", lines[0]) + self.assertIn("品名：", lines[0]) + self.assertIn("店铺：", lines[0]) + self.assertIn("整体口感还差点意思", lines[0]) + + def test_scenario_groups_llm_payload_matches_chapter8_sec2_right_rail_counts( + self, + ) -> None: + sku_h = "SKU(skuId)" + merged = [ + { + sku_h: "111", + "detail_category_path": "食品饮料 > 休闲食品 > 饼干 > 粗粮饼干", + "标题(wareName)": "A饼", + "detail_shop_name": "店甲", + }, + ] + scen = (("早餐/代餐", ("早餐",)),) + fb = jcr._consumer_feedback_by_matrix_group( + merged_rows=merged, + comment_rows=[ + {"sku": "111", "tagCommentContent": "早上当早餐吃还不错"}, + ], + sku_header=sku_h, + ) + pl = jcr.build_scenario_groups_llm_payload( + feedback_groups=fb, + scenario_groups=scen, + merged_rows=merged, + sku_header=sku_h, + title_h="标题(wareName)", + ) + self.assertIn("groups", pl) + self.assertIn("scenario_lexicon", pl) + g0 = pl["groups"][0] + self.assertEqual(g0["group"], "饼干") + self.assertEqual(g0["effective_text_count"], 1) + self.assertEqual(g0["scenario_distribution"][0]["mention_rows"], 1) + self.assertEqual( + g0["scenario_distribution"][0]["scenario"], "早餐/代餐" + ) + + def test_cn_volume_int_parses_total_sales_trailer(self) -> None: + self.assertEqual( + _cn_volume_int("已售50万+ | good:99%好评"), 500_000 + ) + self.assertEqual(_cn_volume_int("2.5亿件"), 250_000_000) + + def test_mix_top_remainder_sums_to_all_rows(self) -> None: + """mix_top 各 count 之和须等于 strip 后可统计行数（与扇图同源）。""" + names = [f"店{i}" for i in range(30)] + mix = jcr._counter_mix_top_rows_with_remainder( + names, top_n=24, remainder_label="（其余店铺）" + ) + self.assertEqual(sum(v for _, v in mix), 30) + self.assertEqual(mix[-1][0], "（其余店铺）") + self.assertEqual(mix[-1][1], 6) + self.assertEqual(len(jcr._structure_names_for_pie_counter(names)), 30) + + def test_infer_total_sales_from_sales_floor(self) -> None: + self.assertEqual( + infer_total_sales_from_sales_floor("good:99%好评 | 已售50万+"), + "已售50万+", + ) + self.assertEqual(infer_total_sales_from_sales_floor(""), "") diff --git a/backend/pipeline/tests/test_detail_buyer_extraction.py b/backend/pipeline/tests/test_detail_buyer_extraction.py new file mode 100644 index 0000000..03132d4 --- /dev/null +++ b/backend/pipeline/tests/test_detail_buyer_extraction.py @@ -0,0 +1,123 @@ +"""商详 JSON → 购买者视角优惠摘要（规则抽取）。""" +from __future__ import annotations + +import json +import sys +from pathlib import Path + +from django.conf import settings +from django.test import SimpleTestCase + + +class DetailBuyerExtractionTests(SimpleTestCase): + def test_extract_minimal_dict(self) -> None: + root = Path(settings.CRAWLER_JD_ROOT).resolve() + dr = root / "detail" + if str(dr) not in sys.path: + sys.path.insert(0, str(dr)) + import jd_detail_buyer_extraction as be # noqa: WPS433 + + obj = { + "warePriceGatherVO": { + "priceItemList": [ + { + "hitLine": False, + "price": "64.97", + "priceLabelList": [{"labelTxt": "到手价", "labelType": "finalPrice"}], + "priceType": "finalPrice", + }, + { + "hitLine": True, + "price": "66", + "priceType": "jdPrice", + }, + ] + }, + "bestPromotion": {"purchasePrice": "64.97", "canGetCoupon": []}, + "warmTipVO": {"tips": [{"tipTxt": "此商品不可使用东券", "order": 1}]}, + "promotion": {"prompt": ""}, + "rankInfoList": [{"rankName": "粗粮饼干热卖榜·第5名"}], + "userInfo": {"newPeople": True}, + "bottomBtnVO": { + "bottomBtnItems": [ + { + "buttonStyle": { + "textFormat": { + "text": "新人到手价<span>¥64.97</span> 立即购买" + } + } + } + ] + }, + "stockInfo": { + "promiseResult": "12:00前付款，预计今天送达", + }, + "serviceTagsVO": { + "basicNewIcons": [ + {"text": "7天价保", "tip": "在下单后7天内，商品出现降价可享受价保服务。"}, + ] + }, + "preferenceVO": { + "againSharedLabel": [{"labelName": "最高返6京豆"}], + "preferencePopUp": { + "expression": { + "basePrice": "66", + "discountDesc": "购买立减", + "discountAmount": "1.03", + "redAmount": "1.03", + "couponAmount": "0", + "promotionAmount": "0", + "govAmount": "", + "subtrahends": [ + { + "topDesc": "红包", + "preferenceDesc": "红包抵¥1.03", + "preferenceAmount": "0", + "preferenceType": "5", + } + ], + }, + "againSharedPreference": [ + {"shortText": "新人包邮", "value": "包邮", "text": "新人包邮"} + ], + }, + }, + } + out = be.extract_buyer_offer_profile(obj) + self.assertEqual(out.get("schema_version"), 1) + self.assertIn("64.97", str(out.get("price_snapshot") or {})) + dm = out.get("discount_mechanism") or {} + self.assertEqual(dm.get("expression", {}).get("discount_desc"), "购买立减") + self.assertTrue(dm.get("subtrahends")) + lines = out.get("buyer_summary_lines") or [] + self.assertTrue(any("到手价" in x for x in lines)) + self.assertTrue(any("东券" in x for x in lines)) + self.assertTrue(any("购买立减" in x or "红包" in x for x in lines)) + + def test_extract_from_real_file_if_present(self) -> None: + root = Path(settings.CRAWLER_JD_ROOT).resolve() + dr = root / "detail" + if str(dr) not in sys.path: + sys.path.insert(0, str(dr)) + import jd_detail_buyer_extraction as be # noqa: WPS433 + + sample = ( + Path(__file__).resolve().parents[3] + / "data" + / "JD" + / "pipeline_runs" + / "20260413_104252_低GI" + / "detail" + / "ware_100107873140_response.json" + ) + if not sample.is_file(): + self.skipTest("sample ware JSON not in workspace") + text = sample.read_text(encoding="utf-8") + out = be.extract_buyer_offer_profile_from_json_text(text) + self.assertEqual(out.get("schema_version"), 1) + self.assertTrue(out.get("buyer_summary_lines")) + # 样例中应有到手价与不可用东券提示 + blob = json.dumps(out, ensure_ascii=False) + self.assertIn("64.97", blob) + self.assertIn("东券", blob) + self.assertTrue("购买立减" in blob or "红包" in blob) diff --git a/backend/pipeline/tests/test_jd_report_sentiment_section.py b/backend/pipeline/tests/test_jd_report_sentiment_section.py new file mode 100644 index 0000000..554cd25 --- /dev/null +++ b/backend/pipeline/tests/test_jd_report_sentiment_section.py @@ -0,0 +1,28 @@ +"""竞品报告 Markdown：第八章情感小节嵌入（烟测）。""" +from __future__ import annotations + +import tempfile +from pathlib import Path + +from django.test import SimpleTestCase + +from pipeline.competitor_report import jd_report as jcr + + +class JdReportSentimentSectionTests(SimpleTestCase): + def test_chapter_83_embeds_when_sentiment_md_provided(self) -> None: + with tempfile.TemporaryDirectory() as td: + run_dir = Path(td) + (run_dir / "report_assets").mkdir(parents=True) + md = jcr.build_competitor_markdown( + run_dir=run_dir, + keyword="测试词", + merged_rows=[], + search_export_rows=[], + comment_rows=[], + meta=None, + llm_sentiment_section_md="#### 饼干\n\n- 正向\n", + ) + self.assertIn("### 8.3 评价正/负向主题（按细类 · 大模型）", md) + self.assertIn("不替代**探针的开放词表", md) + self.assertIn("#### 饼干", md) diff --git a/backend/pipeline/tests/test_llm_group_summaries_chunked.py b/backend/pipeline/tests/test_llm_group_summaries_chunked.py new file mode 100644 index 0000000..e53eff4 --- /dev/null +++ b/backend/pipeline/tests/test_llm_group_summaries_chunked.py @@ -0,0 +1,54 @@ +"""按矩阵细类拆分 group 归纳 LLM 请求：拼接与开关行为。""" +from __future__ import annotations + +import os +from unittest.mock import patch + +from django.test import SimpleTestCase + +from pipeline.jd.runner import use_chunked_group_summaries_llm +from pipeline.llm.generate import ( + _join_chunked_group_markdown, + generate_matrix_group_summaries_llm_chunked, +) + + +class ChunkedGroupSummariesTests(SimpleTestCase): + def test_join_strips_and_skips_empty(self) -> None: + self.assertEqual( + _join_chunked_group_markdown([" x ", "", "y"]), + "x\n\ny", + ) + + @patch("pipeline.llm.generate_group_summaries.generate_matrix_group_summaries_llm") + def test_matrix_chunked_one_call_per_group(self, mock_mx) -> None: + mock_mx.side_effect = ( + lambda groups, keyword: f"#### {groups[0]['group']}\ntext" + ) + out = generate_matrix_group_summaries_llm_chunked( + [{"group": "饼干"}, {"group": "挂面"}], + keyword="低GI", + ) + self.assertEqual(mock_mx.call_count, 2) + self.assertIn("饼干", out) + self.assertIn("挂面", out) + + def test_use_chunked_respects_bulk_env(self) -> None: + with patch.dict(os.environ, {"MA_LLM_GROUP_SUMMARIES_BULK": "1"}): + self.assertFalse( + use_chunked_group_summaries_llm( + {"llm_group_summaries_chunk_by_matrix": True} + ) + ) + + @patch.dict(os.environ, {"MA_LLM_GROUP_SUMMARIES_BULK": ""}, clear=False) + def test_use_chunked_false_from_config(self) -> None: + self.assertFalse( + use_chunked_group_summaries_llm( + {"llm_group_summaries_chunk_by_matrix": False} + ) + ) + + @patch.dict(os.environ, {"MA_LLM_GROUP_SUMMARIES_BULK": ""}, clear=False) + def test_use_chunked_default_true(self) -> None: + self.assertTrue(use_chunked_group_summaries_llm({})) diff --git a/backend/pipeline/tests/test_llm_keyword_suggest.py b/backend/pipeline/tests/test_llm_keyword_suggest.py new file mode 100644 index 0000000..011febe --- /dev/null +++ b/backend/pipeline/tests/test_llm_keyword_suggest.py @@ -0,0 +1,35 @@ +"""llm_keyword_suggest 解析与数据结构（不调用真实 LLM）。""" +from __future__ import annotations + +import unittest + +from pipeline.llm.keyword_suggest import _parse_phrases_object, _parse_scenarios_object + + +class ParsePhrasesTests(unittest.TestCase): + def test_json_object(self) -> None: + raw = '{"phrases": ["口感", " 回购 "]}' + self.assertEqual(_parse_phrases_object(raw), ["口感", "回购"]) + + def test_fenced_json(self) -> None: + raw = '```json\n{"phrases": ["低糖"]}\n```' + self.assertEqual(_parse_phrases_object(raw), ["低糖"]) + + +class ParseScenariosTests(unittest.TestCase): + def test_min_triggers_in_parser(self) -> None: + raw = '{"scenarios": [{"label": "早餐", "triggers": ["早上"]}]}' + out = _parse_scenarios_object(raw) + self.assertEqual(len(out), 1) + self.assertEqual(out[0]["label"], "早餐") + self.assertEqual(out[0]["triggers"], ["早上"]) + + def test_fenced(self) -> None: + raw = '```\n{"scenarios": [{"label": "露营", "triggers": ["户外", "野餐"]}]}\n```' + out = _parse_scenarios_object(raw) + self.assertEqual(len(out), 1) + self.assertEqual(out[0]["label"], "露营") + + +if __name__ == "__main__": + unittest.main() diff --git a/backend/pipeline/tests/test_marketing_detail_pack.py b/backend/pipeline/tests/test_marketing_detail_pack.py new file mode 100644 index 0000000..480b166 --- /dev/null +++ b/backend/pipeline/tests/test_marketing_detail_pack.py @@ -0,0 +1,101 @@ +"""营销内容包：两步 JSON LLM 与解析。""" +from __future__ import annotations + +import json +import tempfile +from unittest.mock import patch + +from django.test import SimpleTestCase + +from pipeline.llm.generate_marketing_detail import ( + _parse_llm_json, + generate_marketing_detail_pack, + normalize_detail_page_pack, +) +from pipeline.reporting.marketing_pack_persist import persist_marketing_detail_pack_v1 + + +class MarketingDetailPackTests(SimpleTestCase): + def test_normalize_fills_missing_keys_including_aigc(self) -> None: + """旧结果或模型漏键时补齐文生图/文生视频等字段。""" + slim = { + "traceability_note": "x", + "main_image_three_points": ["a", "b", "c"], + "live_or_short_hook": "h", + "customer_service_opening": "c", + } + n = normalize_detail_page_pack(slim) + self.assertEqual(n["text_to_image_prompt_main"], "") + self.assertEqual(n["text_to_image_prompt_scene"], "") + self.assertEqual(n["text_to_video_prompt"], "") + self.assertEqual(n["listing_titles"], []) + self.assertEqual(n["detail_mid_story_paragraphs"], []) + self.assertEqual(n["live_script_bullets"], []) + self.assertEqual(n["traceability_note"], "x") + + def test_parse_llm_json_strips_wrapped(self) -> None: + raw = '前缀 {"a": 1, "b": "x"} 后缀' + d = _parse_llm_json(raw) + self.assertEqual(d, {"a": 1, "b": "x"}) + + def test_parse_llm_json_rejects_non_object(self) -> None: + with self.assertRaises(ValueError): + _parse_llm_json("[1,2]") + + def test_generate_marketing_detail_pack_two_calls(self) -> None: + core = { + "what_we_sell": "低GI方向早餐饼干（监测关键词语境，待业务定主推款）", + "one_liner_value": "低负担早餐选择", + "buyer_job_to_be_done": "控糖加餐", + "key_pain_or_desire": "怕升糖", + "why_this_product": "配方可核对", + "proof_or_trust_angle": "输入未体现", + "differentiation_vs_alternatives": "同价带更少添加糖", + "price_value_framing": "中端", + "compliance_taboos": "不涉疗效", + "open_points_for_business": "", + } + pack = { + "listing_titles": ["标题A", "标题B"], + "listing_subtitle": "副文案", + "detail_headline": "首屏一句", + "selling_bullets": ["卖点1"], + "spec_sidebar_lines": [], + "faq": [{"question": "是否低GI？", "answer": "以包装与检测为准。"}], + "traceability_note": "与策略方向一致；具体宣称以包装为准。", + "main_image_three_points": ["低GI早餐", "独立小包", "配料可核对"], + "live_or_short_hook": "控糖早餐怎么选？先看配料表。", + "customer_service_opening": "您好，这款饼干适合关注血糖管理的早餐场景，需要我帮您对比规格吗？", + "text_to_image_prompt_main": "电商主图，白底，居中摆放一盘低GI早餐饼干，柔和棚拍光，写实，无品牌logo，健康清爽风格", + "text_to_image_prompt_scene": "早餐餐桌场景，牛奶与饼干搭配，自然窗光，生活感，无文字贴片", + "text_to_video_prompt": "竖屏9:16，5秒：从俯拍早餐桌缓慢推近至饼干包装，无对白，干净色调，无疗效字幕", + "detail_mid_story_paragraphs": ["段落一适合控糖早餐。", "段落二配料可核对。"], + "usage_and_pairing_tips": ["配无糖酸奶", "开封后密封防潮"], + "short_graphic_post_variants": ["低GI早餐饼干，小包装方便。"], + "live_script_bullets": ["大家好看配料表", "独立小包控量", "适合加餐"], + } + with patch( + "pipeline.llm.generate_marketing_detail.call_llm", + side_effect=[json.dumps(core, ensure_ascii=False), json.dumps(pack, ensure_ascii=False)], + ): + out = generate_marketing_detail_pack( + keyword="低GI", + strategy_markdown="# 策略\n\n- 要点", + strategy_decisions={"product_role": "新品"}, + business_notes="", + ) + self.assertEqual(out["core_info_card"]["one_liner_value"], "低负担早餐选择") + self.assertEqual(out["detail_page_pack"]["listing_titles"][0], "标题A") + self.assertIn("竖屏", out["detail_page_pack"]["text_to_video_prompt"]) + + def test_persist_marketing_detail_pack_v1_writes_file(self) -> None: + with tempfile.TemporaryDirectory() as td: + payload = {"schema_version": 1, "job_id": 1, "x": "y"} + path = persist_marketing_detail_pack_v1(td, payload) + self.assertIsNotNone(path) + assert path is not None + self.assertTrue(path.is_file()) + self.assertEqual(path.parent.name, "marketing") + self.assertIn("marketing_detail_pack_v1.json", path.name) + loaded = json.loads(path.read_text(encoding="utf-8")) + self.assertEqual(loaded["job_id"], 1) diff --git a/backend/pipeline/tests/test_matrix_promo_llm_real_run_dir.py b/backend/pipeline/tests/test_matrix_promo_llm_real_run_dir.py new file mode 100644 index 0000000..41934b3 --- /dev/null +++ b/backend/pipeline/tests/test_matrix_promo_llm_real_run_dir.py @@ -0,0 +1,139 @@ +""" +矩阵 / 促销 两组 LLM 的「读超时」多为网关侧：单次请求在 ``LLM_CHAT_TIMEOUT``（默认 600s） +内未返回完整响应即失败。默认配置下 ``llm_group_summaries_chunk_by_matrix=True``， +会为**每个细类矩阵分组各发 1 次** ``chat/completions``，串行执行；任一次超时都会导致 +对应的 ``matrix_groups_llm.json`` / ``promo_groups_llm.json`` 记录 error。 + +本模块**不修改流水线**，仅用于本地用真实合并表做载荷统计或可选冒烟调用。 + +用法（在 ``backend`` 目录、已配置 Django / .env）:: + + # 仅统计每个分块请求的 JSON 字符数与粗算输入 token（不调网关） + set MA_TEST_REAL_RUN_DIR=D:\\...\\pipeline_runs\\20260413_104252_低GI + python -m pytest pipeline/tests/test_matrix_promo_llm_real_run_dir.py -v -s + + # 额外对「第一个矩阵分组」发 1 次真实请求（需 OPENAI_* / LLM_* 可用） + set MA_TEST_REAL_LLM=1 + python -m pytest pipeline/tests/test_matrix_promo_llm_real_run_dir.py::test_first_matrix_group_llm_smoke_optional -v -s +""" +from __future__ import annotations + +import json +import os +from pathlib import Path + +import pytest + +from pipeline.competitor_report import jd_report as jcr +from pipeline.competitor_report.llm_group_payloads import ( + build_matrix_groups_llm_payload, + build_promo_groups_llm_payload, +) +from pipeline.csv.schema import MERGED_FIELD_TO_CSV_HEADER + + +def _real_run_dir() -> Path | None: + raw = (os.environ.get("MA_TEST_REAL_RUN_DIR") or "").strip() + if not raw: + return None + p = Path(raw).expanduser().resolve() + return p if p.is_dir() else None + + +def _load_merged_rows(run_dir: Path) -> list[dict[str, str]]: + merged = run_dir / "keyword_pipeline_merged.csv" + if not merged.is_file(): + pytest.skip(f"无合并表: {merged}") + _, rows = jcr._read_csv_rows(merged) + return rows + + +def test_matrix_promo_chunk_payload_metrics_from_real_run_dir() -> None: + """ + 对真实 run_dir：输出矩阵 / 促销 **按细类分块** 时每一次请求的用户 JSON 规模与粗算 tokens。 + 用于对照 ``AI_crawler.chat_completion_text`` 的读超时（与输入长、输出 max_tokens 上限、网关排队均相关）。 + """ + rd = _real_run_dir() + if rd is None: + pytest.skip("请设置环境变量 MA_TEST_REAL_RUN_DIR 为流水线目录（含 keyword_pipeline_merged.csv）") + + rows = _load_merged_rows(rd) + sku_h = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + title_h = MERGED_FIELD_TO_CSV_HEADER["title"] + kw = (os.environ.get("MA_TEST_REAL_KEYWORD") or "低GI").strip() or "低GI" + + pl_mx = build_matrix_groups_llm_payload(rows, sku_header=sku_h, title_h=title_h) + pl_po = build_promo_groups_llm_payload(rows, sku_header=sku_h, title_h=title_h) + + assert pl_mx, "矩阵分组载荷为空（无可用细类或合并表无类目路径）" + assert pl_po, "促销分组载荷为空" + + from pipeline.llm.generate_group_summaries import ( + MATRIX_GROUPS_SYSTEM, + MATRIX_GROUPS_USER_PREFIX, + PROMO_GROUPS_SYSTEM, + PROMO_GROUPS_USER_PREFIX, + ) + from pipeline.llm.llm_client import estimate_chat_input_tokens + + def _per_chunk_stats( + *, + groups: list[dict], + system: str, + user_prefix: str, + label: str, + ) -> None: + max_chars = max_est = 0 + worst_name = "" + for g in groups: + raw = json.dumps({"keyword": kw, "groups": [g]}, ensure_ascii=False) + user = user_prefix + raw + est = estimate_chat_input_tokens(system, user) + if len(raw) > max_chars: + max_chars = len(raw) + max_est = est + worst_name = str(g.get("group") or "") + n = len(groups) + print( + f"\n[{label}] 分块数={n}（默认 chunk 模式下串行请求数≈{n}）\n" + f" 单次 user JSON（含 keyword+单组）最大字符数≈{max_chars}\n" + f" 对应粗算输入 tokens≈{max_est}（与 estimate_chat_input_tokens 一致）\n" + f" 最大块细类名: {worst_name!r}\n" + f" 说明: 每块仍可能申请较大 max_tokens；网关生成慢或排队时，单次即可触发 Read timeout。\n" + ) + + _per_chunk_stats( + groups=pl_mx, + system=MATRIX_GROUPS_SYSTEM, + user_prefix=MATRIX_GROUPS_USER_PREFIX, + label="第五章·矩阵归纳 matrix", + ) + _per_chunk_stats( + groups=pl_po, + system=PROMO_GROUPS_SYSTEM, + user_prefix=PROMO_GROUPS_USER_PREFIX, + label="第六章·促销归纳 promo", + ) + + +def test_first_matrix_group_llm_smoke_optional() -> None: + """可选：只对第一个矩阵分组调用 1 次网关，验证连通性（默认跳过）。""" + if (os.environ.get("MA_TEST_REAL_LLM") or "").strip() != "1": + pytest.skip("仅当 MA_TEST_REAL_LLM=1 时调用真实网关") + + rd = _real_run_dir() + if rd is None: + pytest.skip("请设置 MA_TEST_REAL_RUN_DIR") + + rows = _load_merged_rows(rd) + sku_h = MERGED_FIELD_TO_CSV_HEADER["sku_id"] + title_h = MERGED_FIELD_TO_CSV_HEADER["title"] + kw = (os.environ.get("MA_TEST_REAL_KEYWORD") or "低GI").strip() or "低GI" + + pl_mx = build_matrix_groups_llm_payload(rows, sku_header=sku_h, title_h=title_h) + assert pl_mx + + from pipeline.llm.generate_group_summaries import generate_matrix_group_summaries_llm + + out = generate_matrix_group_summaries_llm([pl_mx[0]], keyword=kw) + assert (out or "").strip(), "模型返回为空" diff --git a/backend/pipeline/tests/test_md_document_export.py b/backend/pipeline/tests/test_md_document_export.py new file mode 100644 index 0000000..d10bb8c --- /dev/null +++ b/backend/pipeline/tests/test_md_document_export.py @@ -0,0 +1,23 @@ +"""Markdown → docx/pdf 导出（防回归：docx 主循环须递增行指针）。""" +from __future__ import annotations + +from django.test import SimpleTestCase + +from pipeline.reporting.md_document_export import ( + markdown_to_docx_bytes, + markdown_to_pdf_bytes, +) + + +class MdDocumentExportTests(SimpleTestCase): + def test_docx_plain_lines_terminate(self) -> None: + md = "第一行\n\n第二行\n仍是一段" + data = markdown_to_docx_bytes(md) + self.assertGreater(len(data), 2000) + self.assertTrue(data.startswith(b"PK")) + + def test_pdf_plain_lines_terminate(self) -> None: + md = "标题\n\n正文一行" + data = markdown_to_pdf_bytes(md) + self.assertGreater(len(data), 100) + self.assertTrue(data.startswith(b"%PDF")) diff --git a/backend/pipeline/tests/test_openai_gateway_text_channel_credentials.py b/backend/pipeline/tests/test_openai_gateway_text_channel_credentials.py new file mode 100644 index 0000000..d060367 --- /dev/null +++ b/backend/pipeline/tests/test_openai_gateway_text_channel_credentials.py @@ -0,0 +1,48 @@ +"""OPENAI_TEXT_* 与 OPENAI_* 可分开供纯文本使用。""" +from __future__ import annotations + +import pytest + +from pipeline.openai_gateway.credentials import resolve_text_channel_credentials + + +def test_text_channel_falls_back_to_openai_when_no_text_specific(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("OPENAI_API_KEY", "sk-main") + monkeypatch.setenv("OPENAI_BASE_URL", "https://gw.example.com/v1") + monkeypatch.delenv("OPENAI_TEXT_API_KEY", raising=False) + monkeypatch.delenv("OPENAI_TEXT_BASE_URL", raising=False) + k, b = resolve_text_channel_credentials() + assert k == "sk-main" + assert b == "https://gw.example.com/v1" + + +def test_text_channel_uses_text_key_same_base(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("OPENAI_API_KEY", "sk-vision") + monkeypatch.setenv("OPENAI_BASE_URL", "https://llm.rekeymed.com/v1") + monkeypatch.setenv("OPENAI_TEXT_API_KEY", "sk-text-only") + k, b = resolve_text_channel_credentials() + assert k == "sk-text-only" + assert b == "https://llm.rekeymed.com/v1" + + +def test_text_channel_uses_text_base_same_key(monkeypatch: pytest.MonkeyPatch) -> None: + for _e in ( + "OPENAI_TEXT_API_KEY", + "LLM_TEXT_API_KEY", + "LLM_API_KEY", + ): + monkeypatch.delenv(_e, raising=False) + monkeypatch.setenv("OPENAI_API_KEY", "sk-shared") + monkeypatch.setenv("OPENAI_BASE_URL", "https://a.com/v1") + monkeypatch.setenv("OPENAI_TEXT_BASE_URL", "https://b.com/v1") + k, b = resolve_text_channel_credentials() + assert k == "sk-shared" + assert b == "https://b.com/v1" + + +def test_explicit_args_win_over_env(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("OPENAI_API_KEY", "sk-env") + monkeypatch.setenv("OPENAI_BASE_URL", "https://env.com/v1") + k, b = resolve_text_channel_credentials("sk-arg", "https://arg.com/v1") + assert k == "sk-arg" + assert b == "https://arg.com/v1" diff --git a/backend/pipeline/tests/test_report_config_bool_null.py b/backend/pipeline/tests/test_report_config_bool_null.py new file mode 100644 index 0000000..a31d552 --- /dev/null +++ b/backend/pipeline/tests/test_report_config_bool_null.py @@ -0,0 +1,33 @@ +"""report_config 中布尔开关为 JSON null 时不应把 LLM 归纳整段关掉。""" +from __future__ import annotations + +from unittest import mock + +from pipeline.jd.runner import merge_report_config_with_defaults +from pipeline.serializers import validate_report_config_body + + +def test_validate_report_config_strips_null_bool_flags() -> None: + out = validate_report_config_body( + { + "llm_matrix_group_summaries": None, + "llm_promo_group_summaries": None, + } + ) + assert "llm_matrix_group_summaries" not in out + assert "llm_promo_group_summaries" not in out + + +@mock.patch("pipeline.jd.runner.get_default_report_config") +def test_merge_report_config_null_bool_gets_default(mock_def: mock.MagicMock) -> None: + mock_def.return_value = { + "llm_matrix_group_summaries": True, + "llm_promo_group_summaries": True, + "llm_price_group_summaries": False, + } + merged = merge_report_config_with_defaults( + {"llm_matrix_group_summaries": None, "llm_promo_group_summaries": None} + ) + assert merged["llm_matrix_group_summaries"] is True + assert merged["llm_promo_group_summaries"] is True + assert merged["llm_price_group_summaries"] is False diff --git a/backend/pipeline/tests/test_report_matrix_group_evidence.py b/backend/pipeline/tests/test_report_matrix_group_evidence.py new file mode 100644 index 0000000..e56e3bd --- /dev/null +++ b/backend/pipeline/tests/test_report_matrix_group_evidence.py @@ -0,0 +1,93 @@ +"""从宿主报告 MD 按细类抽取大模型小节。""" +from __future__ import annotations + +from pathlib import Path +from tempfile import TemporaryDirectory + +from django.test import SimpleTestCase + +from pipeline.reporting.report_matrix_group_evidence import ( + extract_level4_sections_by_group_title, + extract_sentiment_83_level4_body, + load_report_matrix_group_evidence_markdown, +) + + +class ReportMatrixGroupEvidenceTests(SimpleTestCase): + def test_extract_multiple_blocks_in_order(self) -> None: + md = """#### 细类要点归纳（大模型） + +> 说明 + +#### 饼干 +A 段矩阵归纳。 + +#### 饮料 +别的细类。 + +--- + +#### 细类评价与关注词要点归纳（大模型） + +#### 饼干 +B 段评论归纳。 +""" + parts = extract_level4_sections_by_group_title(md, "饼干") + self.assertEqual(len(parts), 2) + self.assertIn("A 段矩阵归纳", parts[0]) + self.assertIn("B 段评论归纳", parts[1]) + + def test_no_match(self) -> None: + self.assertEqual( + extract_level4_sections_by_group_title("## 二\n", "饼干"), + [], + ) + + def test_sentiment_83_nested_level4(self) -> None: + md = """## 八、消费者反馈 + +### 8.3 评价正/负向主题（按细类 · 大模型） + +> 说明 + +#### 饼干 + +#### 正向体验主题 +酥脆好评。 + +#### 负向评价主题归因 +略贵。 + +#### 西式糕点 + +#### 正向体验主题 +别的细类。 + +## 九、策略 +""" + body = extract_sentiment_83_level4_body(md, "饼干") + self.assertIn("酥脆好评", body) + self.assertIn("正向体验主题", body) + self.assertIn("略贵", body) + self.assertNotIn("别的细类", body) + + def test_load_includes_83_after_splice(self) -> None: + md = """#### 饼干 +矩阵段。 + +### 8.3 评价正/负向主题（按细类 · 大模型） + +#### 饼干 + +#### 正向体验主题 +情感段。 + +## 九、策略 +""" + with TemporaryDirectory() as td: + p = Path(td) / "competitor_analysis.md" + p.write_text(md, encoding="utf-8") + out, src = load_report_matrix_group_evidence_markdown(td, "饼干") + self.assertEqual(src, "competitor_analysis_md") + self.assertIn("矩阵段", out) + self.assertIn("情感段", out) diff --git a/backend/pipeline/tests/test_report_strategy_excerpt.py b/backend/pipeline/tests/test_report_strategy_excerpt.py new file mode 100644 index 0000000..87a56e0 --- /dev/null +++ b/backend/pipeline/tests/test_report_strategy_excerpt.py @@ -0,0 +1,79 @@ +"""``report_strategy_excerpt``：第九章正文加载与截取。""" +from __future__ import annotations + +import json +from pathlib import Path + +from pipeline.reporting.report_strategy_excerpt import ( + CHAPTER_NINE_HEADING, + extract_chapter_nine_strategy_markdown, + load_report_strategy_excerpt, +) + + +def test_extract_chapter_nine_stops_before_appendix() -> None: + md = f"""# x + +{CHAPTER_NINE_HEADING}（假设清单，待验证） + +正文一段。 + +--- + +## 附录 A：数据留存说明 + +尾部。 +""" + out = extract_chapter_nine_strategy_markdown(md) + assert CHAPTER_NINE_HEADING in out + assert "正文一段" in out + assert "附录" not in out + assert "尾部" not in out + + +def test_load_prefers_json_markdown(tmp_path: Path) -> None: + (tmp_path / "strategy_opportunities_llm.json").write_text( + json.dumps( + {"schema_version": 1, "ok": True, "markdown": "JSON 内第九章正文"}, + ensure_ascii=False, + ), + encoding="utf-8", + ) + (tmp_path / "competitor_analysis.md").write_text( + f"{CHAPTER_NINE_HEADING}\n\n从 MD 来\n\n## 附录 A\n", + encoding="utf-8", + ) + text, src = load_report_strategy_excerpt(tmp_path) + assert src == "json_markdown" + assert "JSON 内第九章" in text + + +def test_load_no_fallback_when_json_exists_without_markdown(tmp_path: Path) -> None: + """存在 strategy_opportunities_llm.json 但无 markdown 时不再截取 competitor_analysis（第九章可为固定说明）。""" + (tmp_path / "strategy_opportunities_llm.json").write_text( + json.dumps({"schema_version": 1, "ok": True}, ensure_ascii=False), + encoding="utf-8", + ) + (tmp_path / "competitor_analysis.md").write_text( + f"{CHAPTER_NINE_HEADING}（假设清单，待验证）\n\n从 MD 截取。\n\n## 附录 A\n", + encoding="utf-8", + ) + text, src = load_report_strategy_excerpt(tmp_path) + assert src == "none" + assert text == "" + + +def test_load_falls_back_to_competitor_md_when_no_json(tmp_path: Path) -> None: + (tmp_path / "competitor_analysis.md").write_text( + f"{CHAPTER_NINE_HEADING}（假设清单，待验证）\n\n从 MD 截取。\n\n## 附录 A\n", + encoding="utf-8", + ) + text, src = load_report_strategy_excerpt(tmp_path) + assert src == "competitor_analysis_md" + assert "从 MD 截取" in text + + +def test_load_none_when_missing(tmp_path: Path) -> None: + text, src = load_report_strategy_excerpt(tmp_path) + assert src == "none" + assert text == "" diff --git a/backend/pipeline/tests/test_strategy_config.py b/backend/pipeline/tests/test_strategy_config.py new file mode 100644 index 0000000..be717ae --- /dev/null +++ b/backend/pipeline/tests/test_strategy_config.py @@ -0,0 +1,22 @@ +"""策略页独立 ``strategy_config`` 校验。""" +from __future__ import annotations + +import pytest +from rest_framework import serializers + +from pipeline.serializers import validate_strategy_config_body + + +def test_validate_strategy_config_merges_default() -> None: + out = validate_strategy_config_body({}) + assert out["use_llm_default"] is True + + +def test_validate_strategy_config_unknown_key() -> None: + with pytest.raises(serializers.ValidationError): + validate_strategy_config_body({"foo": 1}) + + +def test_validate_strategy_config_use_llm_type() -> None: + with pytest.raises(serializers.ValidationError): + validate_strategy_config_body({"use_llm_default": "yes"}) diff --git a/backend/pipeline/tests/test_strategy_draft.py b/backend/pipeline/tests/test_strategy_draft.py index e853fde..809e402 100644 --- a/backend/pipeline/tests/test_strategy_draft.py +++ b/backend/pipeline/tests/test_strategy_draft.py @@ -3,10 +3,55 @@ from __future__ import annotations from django.test import SimpleTestCase -from pipeline.strategy_draft import build_strategy_draft_markdown +from pipeline.llm.generate_strategy import _omit_ch8_probe_wordchart_fields +from pipeline.llm.generate_strategy import strategy_decisions_substantive +from pipeline.reporting.strategy_draft import build_strategy_draft_markdown class StrategyDraftTests(SimpleTestCase): + def test_strategy_decisions_substantive(self) -> None: + self.assertFalse(strategy_decisions_substantive(None)) + self.assertFalse(strategy_decisions_substantive({})) + self.assertFalse( + strategy_decisions_substantive( + {"ack_risk_keywords": True, "product_role": " "} + ) + ) + self.assertTrue( + strategy_decisions_substantive({"product_role": "新品"}) + ) + self.assertTrue( + strategy_decisions_substantive({"stage_goal_type": "本阶段做销量与转化"}) + ) + + def test_for_llm_input_omits_dev_traces(self) -> None: + brief = { + "schema_version": 1, + "keyword": "K", + "batch_label": "b1", + "scope": {"merged_sku_count": 2}, + "strategy_hints": ["线索1"], + "meta": {"page_start": 1, "page_to": 3, "max_skus_config": 100}, + } + md = build_strategy_draft_markdown( + job_id=7, + keyword="K", + brief=brief, + generated_at_iso="2026-01-01", + for_llm_input=True, + ) + self.assertNotIn("任务 ID", md) + self.assertNotIn("generate_strategy.py", md) + self.assertNotIn("strategy_hints", md) + self.assertIn("监测摘要自动线索", md) + self.assertIn("列表页约第 1～3 页", md) + self.assertNotIn("埋伏笔", md) + self.assertNotIn("成稿须与 §2", md) + self.assertNotIn("回扣 §2", md) + self.assertNotIn("（占位）", md) + self.assertIn("### 1.3 本品聚焦\n", md) + self.assertIn("### 5.2 差异化方向\n", md) + def test_build_contains_sections_and_notes(self) -> None: brief = { "schema_version": 1, @@ -26,15 +71,42 @@ class StrategyDraftTests(SimpleTestCase): self.assertIn("任务 ID**：99", md) self.assertIn("假设A", md) self.assertIn("重点：华东", md) - self.assertIn("战略背景与目标", md) + self.assertIn("## 摘要", md) + self.assertIn("策略范围与前提", md) + self.assertIn("针对痛点要怎么做", md) + self.assertIn("类目/细类", md) + self.assertIn("全局禁止编造", md) + self.assertNotIn("### 3.2 转化障碍与应对", md) + self.assertIn("## 一、顾客是谁", md) + self.assertIn("## 七、品牌四线", md) self.assertIn("市场策略制定草稿", md) + def test_stage_goal_type_in_scope_table(self) -> None: + brief = { + "schema_version": 1, + "keyword": "K", + "batch_label": "b1", + "scope": {"merged_sku_count": 2}, + } + md = build_strategy_draft_markdown( + job_id=1, + keyword="K", + brief=brief, + strategy_decisions={"stage_goal_type": "本阶段以拉新尝试为主"}, + generated_at_iso="2026-04-09T12:00:00", + for_llm_input=True, + ) + self.assertIn("| **本阶段策略目标类型** | 本阶段以拉新尝试为主 |", md) + self.assertIn("**本阶段策略目标类型**：本阶段以拉新尝试为主", md) + def test_strategy_decisions_merge(self) -> None: brief = {"schema_version": 1, "keyword": "K", "batch_label": "b"} decisions = { "product_role": "追赶型", "positioning_choice": "mid", "competitive_stance": "flank", + "marketing_strategy": "内容种草+搜索承接", + "general_strategy": "先腰后顶", "pillar_product": "做低糖配方", "ack_risk_keywords": True, "ack_risk_price": False, @@ -45,12 +117,172 @@ class StrategyDraftTests(SimpleTestCase): keyword="K", brief=brief, strategy_decisions=decisions, + report_config={"chapter8_text_mining_probe": False}, ) self.assertIn("**本品角色**：追赶型", md) + self.assertIn("**营销策略**：内容种草+搜索承接", md) + self.assertIn("**总体策略**：先腰后顶", md) self.assertIn("- [x] **卡腰**", md) self.assertIn("- [ ] **贴顶**", md) + i4 = md.find("## 四、为什么要选") + i5 = md.find("## 五、与其它品牌") + self.assertGreater(i5, i4) + self.assertNotIn("贴顶", md[i4:i5]) + i82 = md.find("### 8.2 定价策略") + self.assertGreater(i82, 0) + self.assertGreater(md.find("- [x] **卡腰**"), i82) self.assertIn("侧翼切入", md) - self.assertIn("| 产品 | 做低糖配方 |", md) - self.assertIn("- [x] 关注词/场景是否**以偏概全**", md) + self.assertIn("做低糖配方", md) + self.assertIn("### 7.1 品牌建设", md) + self.assertIn("- [x] 评论侧归纳是否以偏概全", md) self.assertIn("- [ ] 价格带是否含大促", md) - self.assertIn("- [x] 列表集中度与深入样本品牌是否**矛盾**", md) + self.assertIn("- [x] 列表集中度与深入样本品牌是否不一致", md) + + def test_chapter8_probe_omits_focus_scenario_count_bullets(self) -> None: + brief = { + "schema_version": 1, + "keyword": "低GI", + "comment_focus_keywords": [{"word": "口感", "count": 501}], + "usage_scenarios": [ + { + "scenario": "控糖/血糖相关", + "count": 305, + "share_of_text_units": 0.272, + } + ], + } + md = build_strategy_draft_markdown( + job_id=1, + keyword="低GI", + brief=brief, + report_config={"chapter8_text_mining_probe": True}, + ) + self.assertIn("文本挖掘", md) + self.assertNotIn("子串统计命中约 **501**", md) + self.assertNotIn("场景「控糖", md) + + def test_non_probe_path_no_preset_focus_enumeration(self) -> None: + brief = { + "schema_version": 1, + "keyword": "低GI", + "comment_focus_keywords": [{"word": "口感", "count": 501}], + "usage_scenarios": [ + { + "scenario": "控糖/血糖相关", + "count": 305, + "share_of_text_units": 0.272, + } + ], + } + md = build_strategy_draft_markdown( + job_id=1, + keyword="低GI", + brief=brief, + report_config={"chapter8_text_mining_probe": False}, + ) + self.assertIn("不再", md) + self.assertIn("预设关注词", md) + self.assertNotIn("子串统计命中约 **501**", md) + self.assertNotIn("场景「控糖", md) + + def test_matrix_scope_concentration_not_list_rows_wording(self) -> None: + """收窄矩阵时 concentration 来自分组内 SKU，§5.1 勿写「列表行」。""" + brief = { + "schema_version": 1, + "keyword": "低GI", + "strategy_scope_applied": {"group": "饼干"}, + "concentration": { + "shops_from_list": { + "first_share": 0.238, + "top_three_combined_share": 0.571, + "top_label": "碧翠园京东自营旗舰店", + }, + "detail_brand_among_merged": { + "first_share": 0.238, + "top_three_combined_share": 0.667, + "top_label": "碧翠园", + }, + }, + } + md = build_strategy_draft_markdown(job_id=1, keyword="低GI", brief=brief) + self.assertIn("与全关键词 **PC 搜索列表行** 集中度**不是同一口径**", md) + self.assertIn("店铺分布（「饼干」内样本 SKU）", md) + self.assertIn("该分组样本 SKU 的", md) + self.assertNotIn("列表侧店铺集中度", md) + self.assertNotIn("第一大店铺约占列表行的", md) + + def test_shops_unique_sku_basis_rendered(self) -> None: + brief = { + "schema_version": 1, + "keyword": "测试", + "concentration": { + "shops_from_list": { + "first_share": 0.6, + "top_three_combined_share": 0.85, + "top_label": "京东自营", + "unique_sku_basis": { + "first_share": 0.35, + "top_three_combined_share": 0.7, + "top_label": "京东自营", + "n_unique_skus": 120, + }, + }, + "detail_brand_among_merged": {}, + }, + } + md = build_strategy_draft_markdown( + job_id=1, + keyword="测试", + brief=brief, + ) + self.assertIn("按去重 SKU", md) + self.assertIn("120", md) + self.assertIn("35.0%", md) + + def test_chapter8_probe_filters_strategy_hints_focus_scenario_lines(self) -> None: + brief = { + "schema_version": 1, + "keyword": "K", + "strategy_hints": [ + "评价文本中「口感、甜」等主题出现较多，可作为假设输入（非严格主题模型）。", + "用途/场景中「控糖/血糖相关」在约 72% 的有效评价自述中出现，可作为优先假设（词组规则）。", + "样本内品牌较分散，存在定位空间（需验证）。", + ], + "price_promotion_signals": {"rows_with_both_list_and_coupon": 10}, + } + md = build_strategy_draft_markdown( + job_id=1, + keyword="K", + brief=brief, + report_config={"chapter8_text_mining_probe": True}, + ) + self.assertNotIn("评价文本中「口感", md) + self.assertNotIn("用途/场景中「控糖", md) + self.assertIn("样本内品牌较分散", md) + self.assertIn("price_promotion_signals", md) + self.assertIn("price_promotion_signals", md) + + def test_ch8_probe_omit_wordchart_nested_in_consumer_feedback(self) -> None: + compact = { + "comment_focus_keywords": [{"word": "x", "count": 1}], + "usage_scenarios": [], + "comment_sentiment_lexicon": {"pos": ["好"], "neg": ["差"]}, + "strategy_hints": ["条形图同源句子"], + "consumer_feedback_by_matrix_group": [ + { + "group": "饼干", + "comment_rows": 10, + "focus_keyword_hits": [{"word": "口感", "count": 5}], + "scenarios_top": [{"scenario": "早餐", "count": 2}], + } + ], + } + _omit_ch8_probe_wordchart_fields(compact) + self.assertNotIn("comment_focus_keywords", compact) + self.assertNotIn("comment_sentiment_lexicon", compact) + self.assertNotIn("strategy_hints", compact) + self.assertNotIn("focus_keyword_hits", compact["consumer_feedback_by_matrix_group"][0]) + self.assertNotIn("scenarios_top", compact["consumer_feedback_by_matrix_group"][0]) + self.assertEqual( + compact["consumer_feedback_by_matrix_group"][0].get("comment_rows"), 10 + ) diff --git a/backend/pipeline/tests/test_text_llm_providers.py b/backend/pipeline/tests/test_text_llm_providers.py new file mode 100644 index 0000000..9933e35 --- /dev/null +++ b/backend/pipeline/tests/test_text_llm_providers.py @@ -0,0 +1,188 @@ +from __future__ import annotations + +from unittest.mock import MagicMock, patch + +import pytest + +from pipeline.llm.providers import ( + CrawlerOpenAiCompatibleTextLlm, + DeepSeekTextLlm, + KimiMoonshotTextLlm, + OpenAiOfficialChatGptTextLlm, + get_text_llm, + reset_text_llm_client_for_tests, +) + + +def test_openai_official_complete_text_uses_post_and_returns_content(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("OPENAI_OFFICIAL_API_KEY", "sk-test") + + def _fake_post( + url: str, + headers: dict, + json: dict, + timeout: object, + ) -> MagicMock: + assert "/chat/completions" in url + assert json["messages"][0]["role"] == "system" + r = MagicMock() + r.json.return_value = {"choices": [{"message": {"content": "ok_out"}}]} + r.raise_for_status = MagicMock() + return r + + with patch( + "pipeline.llm.providers.adapters.openai_official_chatgpt.requests.post", + side_effect=_fake_post, + ): + llm = OpenAiOfficialChatGptTextLlm() + out = llm.complete_text("S", "U", temperature=0.1) + assert out == "ok_out" + + +def test_kimi_complete_text_uses_post_and_returns_content(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("KIMI_API_KEY", "sk-kimi-test") + + def _fake_post( + url: str, + headers: dict, + json: dict, + timeout: object, + ) -> MagicMock: + assert "moonshot" in url or "chat/completions" in url + assert json["messages"][0]["role"] == "system" + assert headers.get("Authorization", "").startswith("Bearer sk-kimi-") + r = MagicMock() + r.json.return_value = {"choices": [{"message": {"content": "kimi_out"}}]} + r.raise_for_status = MagicMock() + return r + + with patch( + "pipeline.llm.providers.adapters.kimi_moonshot_text.requests.post", + side_effect=_fake_post, + ): + llm = KimiMoonshotTextLlm() + out = llm.complete_text("S", "U", temperature=0.1) + assert out == "kimi_out" + + +def test_deepseek_complete_text_uses_post_and_thinking_by_default( + monkeypatch: pytest.MonkeyPatch, +) -> None: + monkeypatch.setenv("DEEPSEEK_API_KEY", "sk-ds-test") + # 默认开思考，未显式设 DEEPSEEK_TEXT_MODEL 时用 deepseek-v4-pro + monkeypatch.delenv("DEEPSEEK_TEXT_MODEL", raising=False) + monkeypatch.delenv("DEEPSEEK_MODEL", raising=False) + + def _fake_post( + url: str, + headers: dict, + json: dict, + timeout: object, + ) -> MagicMock: + assert "deepseek" in url + assert "/chat/completions" in url + assert json["messages"][0]["role"] == "system" + assert headers.get("Authorization", "").startswith("Bearer sk-ds-") + assert json.get("model") == "deepseek-v4-pro" + assert json.get("thinking") == {"type": "enabled"} + assert json.get("reasoning_effort") == "high" + assert "temperature" not in json + r = MagicMock() + r.json.return_value = {"choices": [{"message": {"content": "ds_out"}}]} + r.raise_for_status = MagicMock() + return r + + with patch( + "pipeline.llm.providers.adapters.deepseek_text.requests.post", + side_effect=_fake_post, + ): + llm = DeepSeekTextLlm() + out = llm.complete_text("S", "U", temperature=0.1) + assert out == "ds_out" + + +def test_deepseek_thinking_off_sends_temperature(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setenv("DEEPSEEK_API_KEY", "sk-ds-test") + monkeypatch.setenv("DEEPSEEK_THINKING", "0") + monkeypatch.setenv("DEEPSEEK_TEXT_MODEL", "deepseek-chat") + + def _fake_post( + url: str, + headers: dict, + json: dict, + timeout: object, + ) -> MagicMock: + assert json.get("model") == "deepseek-chat" + assert "thinking" not in json + assert "reasoning_effort" not in json + assert "temperature" in json + r = MagicMock() + r.json.return_value = {"choices": [{"message": {"content": "plain"}}]} + r.raise_for_status = MagicMock() + return r + + with patch( + "pipeline.llm.providers.adapters.deepseek_text.requests.post", + side_effect=_fake_post, + ): + out = DeepSeekTextLlm().complete_text("S", "U", temperature=0.0) + assert out == "plain" + + +def test_factory_selects_deepseek_when_env_set(monkeypatch: pytest.MonkeyPatch) -> None: + reset_text_llm_client_for_tests() + monkeypatch.setenv("MA_LLM_TEXT_PROVIDER", "deepseek") + monkeypatch.setenv("DEEPSEEK_API_KEY", "sk-x") + try: + c = get_text_llm() + assert isinstance(c, DeepSeekTextLlm) + finally: + reset_text_llm_client_for_tests() + monkeypatch.delenv("MA_LLM_TEXT_PROVIDER", raising=False) + monkeypatch.delenv("DEEPSEEK_API_KEY", raising=False) + + +def test_factory_selects_kimi_when_env_set(monkeypatch: pytest.MonkeyPatch) -> None: + reset_text_llm_client_for_tests() + monkeypatch.setenv("MA_LLM_TEXT_PROVIDER", "kimi") + monkeypatch.setenv("KIMI_API_KEY", "sk-x") + try: + c = get_text_llm() + assert isinstance(c, KimiMoonshotTextLlm) + finally: + reset_text_llm_client_for_tests() + monkeypatch.delenv("MA_LLM_TEXT_PROVIDER", raising=False) + monkeypatch.delenv("KIMI_API_KEY", raising=False) + + +def test_factory_selects_openai_official_when_env_set(monkeypatch: pytest.MonkeyPatch) -> None: + reset_text_llm_client_for_tests() + monkeypatch.setenv("MA_LLM_TEXT_PROVIDER", "chatgpt") + monkeypatch.setenv("OPENAI_OFFICIAL_API_KEY", "sk-x") + try: + c = get_text_llm() + assert isinstance(c, OpenAiOfficialChatGptTextLlm) + finally: + reset_text_llm_client_for_tests() + monkeypatch.delenv("MA_LLM_TEXT_PROVIDER", raising=False) + monkeypatch.delenv("OPENAI_OFFICIAL_API_KEY", raising=False) + + +def test_factory_unknown_provider_raises(monkeypatch: pytest.MonkeyPatch) -> None: + reset_text_llm_client_for_tests() + monkeypatch.setenv("MA_LLM_TEXT_PROVIDER", "no_such_provider") + try: + with pytest.raises(ValueError, match="不支持的"): + get_text_llm() + finally: + reset_text_llm_client_for_tests() + monkeypatch.delenv("MA_LLM_TEXT_PROVIDER", raising=False) + + +def test_default_provider_is_crawler_when_env_cleared(monkeypatch: pytest.MonkeyPatch) -> None: + """未设置 MA_LLM_TEXT_PROVIDER 时仍为爬虫副本网关；此处只断言类型，不调真实网络。""" + reset_text_llm_client_for_tests() + monkeypatch.delenv("MA_LLM_TEXT_PROVIDER", raising=False) + c = get_text_llm() + assert isinstance(c, CrawlerOpenAiCompatibleTextLlm) + reset_text_llm_client_for_tests() diff --git a/backend/pipeline/tests/test_volume_parse.py b/backend/pipeline/tests/test_volume_parse.py new file mode 100644 index 0000000..3daf2a6 --- /dev/null +++ b/backend/pipeline/tests/test_volume_parse.py @@ -0,0 +1,30 @@ +"""销量/评价量文案解析（与 reporting.charts._cn_volume_int 同一套解析规则）。""" +from __future__ import annotations + +from django.test import SimpleTestCase + +from pipeline.volume_parse import ( + cn_volume_int, + comment_count_sort_value_from_cell, + sales_sort_value_from_search_cells, +) + + +class VolumeParseTests(SimpleTestCase): + def test_cn_volume_wan(self) -> None: + self.assertEqual(cn_volume_int("已售50万+"), 500_000) + + def test_cn_volume_yi(self) -> None: + self.assertEqual(cn_volume_int("1.2亿件"), 120_000_000) + + def test_sales_sort_prefers_total_sales(self) -> None: + v = sales_sort_value_from_search_cells("已售1万+", "已售5000+") + self.assertEqual(v, 10_000) + + def test_sales_sort_fallback_floor(self) -> None: + v = sales_sort_value_from_search_cells("", "已售3万+") + self.assertEqual(v, 30_000) + + def test_comment_count_cell(self) -> None: + v = comment_count_sort_value_from_cell("5000+条评价") + self.assertEqual(v, 5000) diff --git a/backend/pipeline/urls.py b/backend/pipeline/urls.py index 14d09e9..e6c391a 100644 --- a/backend/pipeline/urls.py +++ b/backend/pipeline/urls.py @@ -8,6 +8,11 @@ urlpatterns = [ views.ReportConfigDefaultsView.as_view(), name="report-config-defaults", ), + path( + "strategy-config-defaults/", + views.StrategyConfigDefaultsView.as_view(), + name="strategy-config-defaults", + ), path("jobs/", views.JobListCreateView.as_view(), name="job-list-create"), path("jobs/<int:pk>/", views.JobDetailView.as_view(), name="job-detail"), path( @@ -15,6 +20,11 @@ urlpatterns = [ views.JobCancelView.as_view(), name="job-cancel", ), + path( + "jobs/<int:pk>/resume/", + views.JobResumeView.as_view(), + name="job-resume", + ), path("jobs/<int:pk>/download/", views.JobDownloadView.as_view(), name="job-download"), path("jobs/<int:pk>/preview/", views.JobPreviewView.as_view(), name="job-preview"), path( @@ -67,6 +77,11 @@ urlpatterns = [ views.JobStrategyDraftView.as_view(), name="job-strategy-draft", ), + path( + "jobs/<int:pk>/marketing-detail-pack/", + views.JobMarketingDetailPackView.as_view(), + name="job-marketing-detail-pack", + ), path( "jobs/<int:pk>/export-document/", views.JobExportDocumentView.as_view(), diff --git a/backend/pipeline/views.py b/backend/pipeline/views.py deleted file mode 100644 index 2bdc923..0000000 --- a/backend/pipeline/views.py +++ /dev/null @@ -1,858 +0,0 @@ -from __future__ import annotations - -import mimetypes -import threading -from pathlib import Path -from typing import Any - -import requests -from django.conf import settings -from django.db.models import Count, Q -from django.http import FileResponse, Http404, HttpResponse -from django.utils import timezone -from django.utils.decorators import method_decorator -from django.views.decorators.csrf import csrf_exempt -from rest_framework import status -from rest_framework.response import Response -from rest_framework.views import APIView - -from .dataset_nonempty import ( - comment_columns_for_api, - detail_columns_for_api, - merged_columns_for_api, - search_columns_for_api, -) -from .export_job import build_csv_bytes, build_json_bytes, build_xlsx_bytes -from .row_serialize import ( - comment_row_to_dict, - detail_row_to_dict, - merged_row_to_dict, - search_row_to_dict, -) -from .brief_pack import build_brief_pack_zip_bytes -from .strategy_draft import build_strategy_draft_markdown -from .ingest import ingest_job_full -from .jd_runner import ( - build_competitor_brief_for_job, - get_default_report_config, - merge_llm_supplement_with_rules_report, - regenerate_competitor_report, - write_competitor_analysis_markdown, -) -from .llm_generate import ( - generate_competitor_report_markdown_llm, - generate_strategy_draft_markdown_llm, -) -from .md_document_export import markdown_to_docx_bytes, markdown_to_pdf_bytes -from .models import ( - JdJobCommentRow, - JdJobDetailRow, - JdJobMergedRow, - JdJobSearchRow, - JdProduct, - JdProductSnapshot, - JobStatus, - PipelineJob, -) -from .serializers import ( - CreatePipelineJobSerializer, - JdProductDetailSerializer, - JdProductListSerializer, - JdProductSnapshotBriefSerializer, - JdProductSnapshotDetailSerializer, - JobReportConfigPatchSerializer, - PipelineJobSerializer, - RegenerateReportRequestSerializer, - StrategyDraftRequestSerializer, -) -from .tasks import execute_job - -# 在线预览最大字节（超出则截断并提示下载） -_PREVIEW_MAX_BYTES = 2 * 1024 * 1024 - -# 允许下载的相对文件名（均在 run_dir 下） -_DOWNLOAD_NAMES = frozenset( - { - "merged", - "pc_search", - "comments", - "detail_ware", - "report", - } -) - - -def _jd_data_root() -> Path: - root = (settings.LOW_GI_PROJECT_ROOT or "").strip() - if not root: - raise RuntimeError("LOW_GI_PROJECT_ROOT 未配置") - return (Path(root) / "data" / "JD").resolve() - - -def _safe_file_for_job(run_dir_str: str, name: str) -> Path: - if name not in _DOWNLOAD_NAMES: - raise Http404("unknown file") - base = Path(run_dir_str).resolve() - jd_root = _jd_data_root().resolve() - try: - base.relative_to(jd_root) - except ValueError: - raise Http404("invalid run_dir") - - mapping = { - "merged": "keyword_pipeline_merged.csv", - "pc_search": "pc_search_export.csv", - "comments": "comments_flat.csv", - "detail_ware": "detail_ware_export.csv", - "report": "competitor_analysis.md", - } - f = base / mapping[name] - if not f.is_file(): - raise Http404("file not found") - return f - - -def _job_run_dir_usable(job: PipelineJob) -> bool: - """成功或已终止但已写入 run_dir 时，可预览/下载批次文件。""" - return bool((job.run_dir or "").strip()) and job.status in ( - JobStatus.SUCCESS, - JobStatus.CANCELLED, - ) - - -@method_decorator(csrf_exempt, name="dispatch") -class JobListCreateView(APIView): - def get(self, request): - qs = PipelineJob.objects.all()[:200] - return Response(PipelineJobSerializer(qs, many=True).data) - - def post(self, request): - if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): - return Response( - {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, - status=status.HTTP_503_SERVICE_UNAVAILABLE, - ) - ser = CreatePipelineJobSerializer(data=request.data) - ser.is_valid(raise_exception=True) - data = ser.validated_data - job = PipelineJob.objects.create( - platform=data["platform"], - keyword=data["keyword"], - max_skus=data.get("max_skus"), - page_start=data.get("page_start"), - page_to=data.get("page_to"), - pipeline_run_dir=data.get("pipeline_run_dir") or "", - cookie_file_path=data.get("cookie_file_path") or "", - cookie_text=data.get("cookie_text") or "", - pvid=data.get("pvid") or "", - request_delay=data.get("request_delay") or "", - list_pages=data.get("list_pages") or "", - scenario_filter_enabled=data.get("scenario_filter_enabled"), - report_config=data.get("report_config") or {}, - status=JobStatus.PENDING, - ) - t = threading.Thread(target=execute_job, args=(job.id,), daemon=True) - t.start() - return Response( - PipelineJobSerializer(job).data, - status=status.HTTP_201_CREATED, - ) - - -@method_decorator(csrf_exempt, name="dispatch") -class JobDetailView(APIView): - def get(self, request, pk: int): - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - return Response(PipelineJobSerializer(job).data) - - def patch(self, request, pk: int): - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - ser = JobReportConfigPatchSerializer(data=request.data) - ser.is_valid(raise_exception=True) - job.report_config = ser.validated_data["report_config"] - job.save(update_fields=["report_config", "updated_at"]) - return Response(PipelineJobSerializer(job).data) - - -@method_decorator(csrf_exempt, name="dispatch") -class JobCancelView(APIView): - """ - 终止：将 ``cancellation_requested`` 置位后，执行线程会尽快 ``terminate`` 采集子进程 - （效果接近在终端对脚本按 Ctrl+C），并保留已写入运行目录的文件。 - """ - - def post(self, request, pk: int): - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - if job.status not in (JobStatus.PENDING, JobStatus.RUNNING): - return Response( - {"detail": "仅待执行或执行中的任务可终止"}, - status=status.HTTP_400_BAD_REQUEST, - ) - job.cancellation_requested = True - job.save(update_fields=["cancellation_requested", "updated_at"]) - return Response(PipelineJobSerializer(job).data) - - -class ReportConfigDefaultsView(APIView): - """返回 ``jd_competitor_report`` 中与脚本常量一致的默认报告调参 JSON。""" - - def get(self, request): - try: - return Response(get_default_report_config()) - except FileNotFoundError as e: - return Response( - {"detail": str(e)}, - status=status.HTTP_503_SERVICE_UNAVAILABLE, - ) - - -class JobDownloadView(APIView): - def get(self, request, pk: int): - job = PipelineJob.objects.filter(pk=pk).first() - if not job or not _job_run_dir_usable(job): - raise Http404() - name = (request.query_params.get("name") or "").strip().lower() - path = _safe_file_for_job(job.run_dir, name) - return FileResponse( - path.open("rb"), - as_attachment=True, - filename=path.name, - ) - - -class JobPreviewView(APIView): - """浏览器内联查看产出（CSV / Markdown 文本），大文件截断。""" - - def get(self, request, pk: int): - job = PipelineJob.objects.filter(pk=pk).first() - if not job or not _job_run_dir_usable(job): - raise Http404() - name = (request.query_params.get("name") or "").strip().lower() - fpath = _safe_file_for_job(job.run_dir, name) - raw = fpath.read_bytes() - truncated = len(raw) > _PREVIEW_MAX_BYTES - if truncated: - raw = raw[:_PREVIEW_MAX_BYTES] - text = raw.decode("utf-8-sig", errors="replace") - if truncated: - text += "\n\n... [内容已截断，完整文件请使用下载]\n" - - if name == "report": - ctype = "text/markdown; charset=utf-8" - else: - ctype = "text/csv; charset=utf-8" - resp = HttpResponse(text, content_type=ctype) - resp["X-Preview-Truncated"] = "1" if truncated else "0" - resp["X-Preview-Filename"] = fpath.name - return resp - - -@method_decorator(csrf_exempt, name="dispatch") -class JobRegenerateReportView(APIView): - """基于任务已有 ``run_dir`` 内 CSV 重新生成 ``competitor_analysis.md``（不重新爬取）。""" - - def post(self, request, pk: int): - if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): - return Response( - {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, - status=status.HTTP_503_SERVICE_UNAVAILABLE, - ) - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - if job.status != JobStatus.SUCCESS or not (job.run_dir or "").strip(): - return Response( - {"detail": "仅可对已成功且已写入 run_dir 的任务重新生成报告"}, - status=status.HTTP_400_BAD_REQUEST, - ) - ser = RegenerateReportRequestSerializer(data=request.data or {}) - ser.is_valid(raise_exception=True) - generator = ser.validated_data.get("generator") or "rules" - rc = job.report_config if isinstance(job.report_config, dict) else None - if generator == "llm": - try: - regenerate_competitor_report( - job.run_dir, job.keyword, report_config=rc - ) - rules_md = ( - Path(job.run_dir) / "competitor_analysis.md" - ).read_text(encoding="utf-8") - brief = build_competitor_brief_for_job( - job.run_dir, job.keyword, report_config=rc - ) - md = generate_competitor_report_markdown_llm(brief, job.keyword) - md = merge_llm_supplement_with_rules_report(md, rules_md) - write_competitor_analysis_markdown(job.run_dir, md) - except FileNotFoundError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - except ValueError as e: - return Response({"detail": str(e)}, status=status.HTTP_503_SERVICE_UNAVAILABLE) - except requests.RequestException as e: - return Response( - {"detail": f"大模型网关错误：{e}"}, - status=status.HTTP_502_BAD_GATEWAY, - ) - else: - try: - regenerate_competitor_report(job.run_dir, job.keyword, report_config=rc) - except FileNotFoundError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - except ValueError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - return Response(PipelineJobSerializer(job).data) - - -class JobCompetitorBriefView(APIView): - """单次任务的结构化竞品摘要（JSON，与 ``competitor_analysis.md`` 统计口径一致，规则驱动无 LLM）。""" - - def get(self, request, pk: int): - if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): - return Response( - {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, - status=status.HTTP_503_SERVICE_UNAVAILABLE, - ) - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - if job.status != JobStatus.SUCCESS or not (job.run_dir or "").strip(): - return Response( - {"detail": "仅可对已成功且含 run_dir 的任务获取竞品摘要"}, - status=status.HTTP_400_BAD_REQUEST, - ) - try: - data = build_competitor_brief_for_job( - job.run_dir, - job.keyword, - report_config=job.report_config - if isinstance(job.report_config, dict) - else None, - ) - except FileNotFoundError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - except ValueError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - return Response(data) - - -class JobCompetitorBriefPackView(APIView): - """ZIP：完整 Markdown 报告 + 结构化 JSON + 要点摘录 Markdown + 说明文本。""" - - def get(self, request, pk: int): - if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): - return Response( - {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, - status=status.HTTP_503_SERVICE_UNAVAILABLE, - ) - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - if job.status != JobStatus.SUCCESS or not (job.run_dir or "").strip(): - return Response( - {"detail": "仅可对已成功且含 run_dir 的任务导出简报包"}, - status=status.HTTP_400_BAD_REQUEST, - ) - try: - brief = build_competitor_brief_for_job( - job.run_dir, - job.keyword, - report_config=job.report_config - if isinstance(job.report_config, dict) - else None, - ) - zip_bytes = build_brief_pack_zip_bytes(Path(job.run_dir), brief) - except FileNotFoundError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - except ValueError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - - filename_ascii = f"job_{pk}_competitor_brief_pack.zip" - resp = HttpResponse(zip_bytes, content_type="application/zip") - resp["Content-Disposition"] = f'attachment; filename="{filename_ascii}"' - return resp - - -@method_decorator(csrf_exempt, name="dispatch") -class JobStrategyDraftView(APIView): - """ - 市场策略制定 Markdown：策略框架 + 附录；默认规则生成，可选 ``generator=llm``（``AI_crawler.chat_completion_text``）。 - """ - - def post(self, request, pk: int): - if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): - return Response( - {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, - status=status.HTTP_503_SERVICE_UNAVAILABLE, - ) - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - if job.status != JobStatus.SUCCESS or not (job.run_dir or "").strip(): - return Response( - {"detail": "仅可对已成功且含 run_dir 的任务生成策略制定稿"}, - status=status.HTTP_400_BAD_REQUEST, - ) - ser = StrategyDraftRequestSerializer(data=request.data or {}) - ser.is_valid(raise_exception=True) - vd = ser.validated_data - notes = (vd.get("business_notes") or "").strip() - strategy_decisions = { - "product_role": vd.get("product_role") or "", - "time_horizon": vd.get("time_horizon") or "", - "success_criteria": vd.get("success_criteria") or "", - "non_goals": vd.get("non_goals") or "", - "battlefield_one_line": vd.get("battlefield_one_line") or "", - "positioning_choice": vd.get("positioning_choice") or "", - "competitive_stance": vd.get("competitive_stance") or "", - "pillar_product": vd.get("pillar_product") or "", - "pillar_price": vd.get("pillar_price") or "", - "pillar_channel": vd.get("pillar_channel") or "", - "pillar_comm": vd.get("pillar_comm") or "", - "ack_risk_keywords": bool(vd.get("ack_risk_keywords")), - "ack_risk_price": bool(vd.get("ack_risk_price")), - "ack_risk_concentration": bool(vd.get("ack_risk_concentration")), - } - try: - brief = build_competitor_brief_for_job( - job.run_dir, - job.keyword, - report_config=job.report_config - if isinstance(job.report_config, dict) - else None, - ) - except FileNotFoundError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - except ValueError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - - gen_at = timezone.now().isoformat() - generator = (vd.get("generator") or "rules").strip() - try: - if generator == "llm": - md = generate_strategy_draft_markdown_llm( - job_id=job.id, - keyword=job.keyword, - brief=brief, - business_notes=notes, - generated_at_iso=gen_at, - strategy_decisions=strategy_decisions, - ) - src = "llm_text_ai_crawler_v1" - else: - md = build_strategy_draft_markdown( - job_id=job.id, - keyword=job.keyword, - brief=brief, - business_notes=notes, - generated_at_iso=gen_at, - strategy_decisions=strategy_decisions, - ) - src = "structured_summary_rules_v1" - except ValueError as e: - return Response({"detail": str(e)}, status=status.HTTP_503_SERVICE_UNAVAILABLE) - except requests.RequestException as e: - return Response( - {"detail": f"大模型网关错误：{e}"}, - status=status.HTTP_502_BAD_GATEWAY, - ) - return Response( - { - "schema_version": 1, - "job_id": job.id, - "keyword": job.keyword, - "generated_at": gen_at, - "source": src, - "markdown": md, - } - ) - - -@method_decorator(csrf_exempt, name="dispatch") -class JobExportDocumentView(APIView): - """ - 将 Markdown 导出为 Word（.docx）或简易 PDF。 - - GET：``kind=report``，读取 ``run_dir/competitor_analysis.md``。 - - POST：``kind=strategy``，请求体 JSON 字段 ``markdown`` 为策略稿正文（与前端 sessionStorage 一致）。 - PDF 依赖本机中文字体或环境变量 ``MA_PDF_FONT`` 指向 .ttf。 - """ - - def get(self, request, pk: int): - if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): - return Response( - {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, - status=status.HTTP_503_SERVICE_UNAVAILABLE, - ) - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - if not _job_run_dir_usable(job): - return Response( - {"detail": "仅可对已成功或已终止且含 run_dir 的任务导出"}, - status=status.HTTP_400_BAD_REQUEST, - ) - fmt = (request.query_params.get("fmt") or "docx").strip().lower() - kind = (request.query_params.get("kind") or "report").strip().lower() - if kind != "report": - return Response( - {"detail": "GET 仅支持 kind=report；策略稿请用 POST 提交 markdown"}, - status=status.HTTP_400_BAD_REQUEST, - ) - if fmt not in ("docx", "pdf"): - return Response( - {"detail": "fmt 须为 docx 或 pdf"}, - status=status.HTTP_400_BAD_REQUEST, - ) - path = Path(job.run_dir) / "competitor_analysis.md" - if not path.is_file(): - return Response( - {"detail": "报告文件不存在，请先在「报告生成」重新生成"}, - status=status.HTTP_404_NOT_FOUND, - ) - md = path.read_text(encoding="utf-8") - asset_root = Path(job.run_dir).resolve() - try: - if fmt == "docx": - data = markdown_to_docx_bytes(md, asset_root=asset_root) - ct = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" - fn = f"job_{pk}_competitor_report.docx" - else: - data = markdown_to_pdf_bytes(md, asset_root=asset_root) - ct = "application/pdf" - fn = f"job_{pk}_competitor_report.pdf" - except ValueError as e: - return Response({"detail": str(e)}, status=status.HTTP_503_SERVICE_UNAVAILABLE) - resp = HttpResponse(data, content_type=ct) - resp["Content-Disposition"] = f'attachment; filename="{fn}"' - return resp - - def post(self, request, pk: int): - if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): - return Response( - {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, - status=status.HTTP_503_SERVICE_UNAVAILABLE, - ) - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - if not _job_run_dir_usable(job): - return Response( - {"detail": "仅可对已成功或已终止且含 run_dir 的任务导出"}, - status=status.HTTP_400_BAD_REQUEST, - ) - body = request.data if isinstance(request.data, dict) else {} - kind = (body.get("kind") or "strategy").strip().lower() - fmt = (body.get("fmt") or "docx").strip().lower() - md = (body.get("markdown") or "").strip() - if kind != "strategy": - return Response( - {"detail": "POST 仅支持 kind=strategy"}, - status=status.HTTP_400_BAD_REQUEST, - ) - if not md: - return Response( - {"detail": "markdown 不能为空"}, - status=status.HTTP_400_BAD_REQUEST, - ) - if fmt not in ("docx", "pdf"): - return Response( - {"detail": "fmt 须为 docx 或 pdf"}, - status=status.HTTP_400_BAD_REQUEST, - ) - try: - if fmt == "docx": - data = markdown_to_docx_bytes(md) - ct = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" - fn = f"job_{pk}_strategy_draft.docx" - else: - data = markdown_to_pdf_bytes(md) - ct = "application/pdf" - fn = f"job_{pk}_strategy_draft.pdf" - except ValueError as e: - return Response({"detail": str(e)}, status=status.HTTP_503_SERVICE_UNAVAILABLE) - resp = HttpResponse(data, content_type=ct) - resp["Content-Disposition"] = f'attachment; filename="{fn}"' - return resp - - -class JobReportAssetView(APIView): - """安全读取 ``run_dir/report_assets/*`` 下的 PNG 等（供 Markdown 预览插图）。""" - - def get(self, request, pk: int): - job = PipelineJob.objects.filter(pk=pk).first() - if not job or not _job_run_dir_usable(job): - raise Http404() - rel = (request.query_params.get("path") or "").strip().replace("\\", "/") - if not rel or ".." in Path(rel).parts: - return Response( - {"detail": "path 非法"}, - status=status.HTTP_400_BAD_REQUEST, - ) - base = Path(job.run_dir).resolve() - assets_root = (base / "report_assets").resolve() - target = (base / rel).resolve() - try: - target.relative_to(assets_root) - except ValueError: - raise Http404() - if not target.is_file(): - raise Http404() - ctype, _ = mimetypes.guess_type(str(target)) - return FileResponse( - target.open("rb"), - content_type=ctype or "application/octet-stream", - ) - - -def _dataset_job(pk: int) -> PipelineJob: - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - return job - - -def _read_page_params(request) -> tuple[int, int]: - page_size = min(max(int(request.query_params.get("page_size", 50)), 1), 200) - page = max(int(request.query_params.get("page", 1)), 1) - return page, page_size - - -class JobDatasetSummaryView(APIView): - """任务在库中的搜索/详情/评价行数（入库后可用）。""" - - def get(self, request, pk: int): - job = _dataset_job(pk) - return Response( - { - "job_id": job.id, - "keyword": job.keyword, - "status": job.status, - "search_rows": JdJobSearchRow.objects.filter(job=job).count(), - "detail_rows": JdJobDetailRow.objects.filter(job=job).count(), - "comment_rows": JdJobCommentRow.objects.filter(job=job).count(), - "merged_rows": JdJobMergedRow.objects.filter(job=job).count(), - "search_columns": search_columns_for_api(job), - "detail_columns": detail_columns_for_api(job), - "comment_columns": comment_columns_for_api(job), - "merged_columns": merged_columns_for_api(job), - } - ) - - -class JobDatasetSearchView(APIView): - def get(self, request, pk: int): - job = _dataset_job(pk) - page, page_size = _read_page_params(request) - qs = JdJobSearchRow.objects.filter(job=job) - total = qs.count() - start = (page - 1) * page_size - rows = qs.order_by("row_index")[start : start + page_size] - return Response( - { - "total": total, - "page": page, - "page_size": page_size, - "results": [search_row_to_dict(r) for r in rows], - } - ) - - -class JobDatasetDetailView(APIView): - def get(self, request, pk: int): - job = _dataset_job(pk) - page, page_size = _read_page_params(request) - qs = JdJobDetailRow.objects.filter(job=job) - total = qs.count() - start = (page - 1) * page_size - rows = qs.order_by("row_index")[start : start + page_size] - return Response( - { - "total": total, - "page": page, - "page_size": page_size, - "results": [detail_row_to_dict(r) for r in rows], - } - ) - - -class JobDatasetCommentsView(APIView): - def get(self, request, pk: int): - job = _dataset_job(pk) - page, page_size = _read_page_params(request) - sku_id = (request.query_params.get("sku_id") or "").strip() - qs = JdJobCommentRow.objects.filter(job=job) - if sku_id: - qs = qs.filter(sku_id=sku_id) - total = qs.count() - start = (page - 1) * page_size - rows = qs.order_by("row_index")[start : start + page_size] - return Response( - { - "total": total, - "page": page, - "page_size": page_size, - "sku_filter": sku_id or None, - "results": [comment_row_to_dict(r) for r in rows], - } - ) - - -class JobDatasetMergedView(APIView): - def get(self, request, pk: int): - job = _dataset_job(pk) - page, page_size = _read_page_params(request) - qs = JdJobMergedRow.objects.filter(job=job) - total = qs.count() - start = (page - 1) * page_size - rows = qs.order_by("row_index")[start : start + page_size] - return Response( - { - "total": total, - "page": page, - "page_size": page_size, - "results": [merged_row_to_dict(r) for r in rows], - } - ) - - -class JobDatasetExportView(APIView): - """下载：kind=search|detail|comments|merged|all，export_fmt=json|csv|xlsx。 - - ``merged``：库内合并宽表行（与 lean 合并 CSV 列一致，入库后导出）。 - 注意：勿使用查询参数名 ``format``，DRF 会将其用于内容协商，非 json 时易在进视图前 404。 - """ - - def get(self, request, pk: int): - job = _dataset_job(pk) - kind = (request.query_params.get("kind") or "search").strip().lower() - fmt = (request.query_params.get("export_fmt") or "json").strip().lower() - if kind not in ("search", "detail", "comments", "all", "merged"): - return Response( - {"detail": "kind 须为 search / detail / comments / all / merged"}, - status=status.HTTP_400_BAD_REQUEST, - ) - if fmt not in ("json", "csv", "xlsx"): - return Response( - {"detail": "export_fmt 须为 json / csv / xlsx"}, - status=status.HTTP_400_BAD_REQUEST, - ) - try: - if fmt == "json": - data, filename = build_json_bytes(job=job, kind=kind) - resp = HttpResponse(data, content_type="application/json; charset=utf-8") - elif fmt == "csv": - data, filename = build_csv_bytes(job=job, kind=kind) - resp = HttpResponse(data, content_type="text/csv; charset=utf-8") - else: - data, filename = build_xlsx_bytes(job=job, kind=kind) - resp = HttpResponse( - data, - content_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", - ) - except ValueError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - resp["Content-Disposition"] = f'attachment; filename="{filename}"' - return resp - - -class JdProductListView(APIView): - """已入库 SKU 分页列表；支持按标题/SKU/品牌模糊搜、按合并表中的 pipeline_keyword 精确筛。""" - - def get(self, request): - limit = min(max(int(request.query_params.get("limit", 50)), 1), 200) - offset = max(int(request.query_params.get("offset", 0)), 0) - q = (request.query_params.get("q") or "").strip() - kw = (request.query_params.get("keyword") or "").strip() - qs = JdProduct.objects.annotate(snapshot_count=Count("snapshots")) - if q: - qs = qs.filter( - Q(sku_id__icontains=q) - | Q(title__icontains=q) - | Q(detail_brand__icontains=q) - ) - if kw: - qs = qs.filter(current_payload__pipeline_keyword=kw) - total = qs.count() - page = qs.order_by("-updated_at")[offset : offset + limit] - return Response( - { - "total": total, - "limit": limit, - "offset": offset, - "results": JdProductListSerializer(page, many=True).data, - } - ) - - -class JdProductDetailView(APIView): - def get(self, request, sku_id: str): - platform = (request.query_params.get("platform") or "jd").strip() or "jd" - obj = ( - JdProduct.objects.annotate(snapshot_count=Count("snapshots")) - .filter(platform=platform, sku_id=sku_id) - .first() - ) - if not obj: - raise Http404() - return Response(JdProductDetailSerializer(obj).data) - - -class JdProductSnapshotListView(APIView): - """某 SKU 的历史快照列表（不含整包 payload，便于时间线）。""" - - def get(self, request, sku_id: str): - platform = (request.query_params.get("platform") or "jd").strip() or "jd" - product = JdProduct.objects.filter(platform=platform, sku_id=sku_id).first() - if not product: - raise Http404() - snaps = ( - product.snapshots.select_related("job") - .order_by("-captured_at") - .all() - ) - return Response( - { - "platform": platform, - "sku_id": sku_id, - "count": snaps.count(), - "results": JdProductSnapshotBriefSerializer(snaps, many=True).data, - } - ) - - -class JdProductSnapshotDetailView(APIView): - """单条快照完整 payload，用于历史回放与字段级对比。""" - - def get(self, request, pk: int): - snap = ( - JdProductSnapshot.objects.select_related("product", "job") - .filter(pk=pk) - .first() - ) - if not snap: - raise Http404() - return Response(JdProductSnapshotDetailSerializer(snap).data) - - -@method_decorator(csrf_exempt, name="dispatch") -class JobImportMergedView(APIView): - """将指定任务目录下搜索/详情/评价 CSV 与合并表重新写入数据库（幂等：先清空该任务三类行再全量插入）。""" - - def post(self, request, pk: int): - job = PipelineJob.objects.filter(pk=pk).first() - if not job: - raise Http404() - if job.status != JobStatus.SUCCESS or not (job.run_dir or "").strip(): - return Response( - {"detail": "仅可对已成功且含 run_dir 的任务执行入库"}, - status=status.HTTP_400_BAD_REQUEST, - ) - try: - stats = ingest_job_full(job) - except FileNotFoundError as e: - return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) - return Response(stats) diff --git a/backend/pipeline/views/__init__.py b/backend/pipeline/views/__init__.py new file mode 100644 index 0000000..1398742 --- /dev/null +++ b/backend/pipeline/views/__init__.py @@ -0,0 +1,66 @@ +"""Pipeline HTTP API 视图（按领域拆分子模块，本包对外保持与原 ``views.py`` 一致的导出）。""" +from __future__ import annotations + +from .dataset_views import ( + JobDatasetCommentsView, + JobDatasetDetailView, + JobDatasetExportView, + JobDatasetMergedView, + JobDatasetSearchView, + JobDatasetSummaryView, +) +from .ingest_views import JobImportMergedView +from .job_report_views import ( + JobCompetitorBriefPackView, + JobCompetitorBriefView, + JobExportDocumentView, + JobMarketingDetailPackView, + JobReportAssetView, + JobStrategyDraftView, +) +from .job_views import ( + JobCancelView, + JobDetailView, + JobDownloadView, + JobListCreateView, + JobPreviewView, + JobRegenerateReportView, + JobResumeView, + ReportConfigDefaultsView, + StrategyConfigDefaultsView, +) +from .product_views import ( + JdProductDetailView, + JdProductListView, + JdProductSnapshotDetailView, + JdProductSnapshotListView, +) + +__all__ = [ + "JobDatasetCommentsView", + "JobDatasetDetailView", + "JobDatasetExportView", + "JobDatasetMergedView", + "JobDatasetSearchView", + "JobDatasetSummaryView", + "JobImportMergedView", + "JobCompetitorBriefPackView", + "JobCompetitorBriefView", + "JobExportDocumentView", + "JobMarketingDetailPackView", + "JobReportAssetView", + "JobStrategyDraftView", + "JobCancelView", + "JobDetailView", + "JobDownloadView", + "JobListCreateView", + "JobPreviewView", + "JobRegenerateReportView", + "JobResumeView", + "ReportConfigDefaultsView", + "StrategyConfigDefaultsView", + "JdProductDetailView", + "JdProductListView", + "JdProductSnapshotDetailView", + "JdProductSnapshotListView", +] diff --git a/backend/pipeline/views/common.py b/backend/pipeline/views/common.py new file mode 100644 index 0000000..8538fae --- /dev/null +++ b/backend/pipeline/views/common.py @@ -0,0 +1,144 @@ +"""Pipeline API 视图共享常量与辅助函数。""" +from __future__ import annotations + +from pathlib import Path + +from django.conf import settings +from django.http import Http404 + +from ..models import ( + JdJobDetailRow, + JdJobMergedRow, + JdJobSearchRow, + JobStatus, + PipelineJob, +) + +# 在线预览最大字节（超出则截断并提示下载） +PREVIEW_MAX_BYTES = 2 * 1024 * 1024 + +# 允许下载的相对文件名（均在 run_dir 下） +DOWNLOAD_NAMES = frozenset( + { + "merged", + "pc_search", + "comments", + "detail_ware", + "report", + } +) + + +def jd_data_root() -> Path: + root = (settings.LOW_GI_PROJECT_ROOT or "").strip() + if not root: + raise RuntimeError("LOW_GI_PROJECT_ROOT 未配置") + return (Path(root) / "data" / "JD").resolve() + + +def safe_file_for_job(run_dir_str: str, name: str) -> Path: + if name not in DOWNLOAD_NAMES: + raise Http404("unknown file") + base = Path(run_dir_str).resolve() + jd_root = jd_data_root().resolve() + try: + base.relative_to(jd_root) + except ValueError: + raise Http404("invalid run_dir") + + mapping = { + "merged": "keyword_pipeline_merged.csv", + "pc_search": "pc_search_export.csv", + "comments": "comments_flat.csv", + "detail_ware": "detail_ware_export.csv", + "report": "competitor_analysis.md", + } + f = base / mapping[name] + if not f.is_file(): + raise Http404("file not found") + return f + + +def job_run_dir_usable(job: PipelineJob) -> bool: + """成功、已终止或已暂停（断点产物）且已写入 run_dir 时，可预览/下载批次文件。""" + return bool((job.run_dir or "").strip()) and job.status in ( + JobStatus.SUCCESS, + JobStatus.CANCELLED, + JobStatus.PAUSED, + ) + + +def dataset_job(pk: int) -> PipelineJob: + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + return job + + +def read_page_params(request) -> tuple[int, int]: + page_size = min(max(int(request.query_params.get("page_size", 50)), 1), 200) + page = max(int(request.query_params.get("page", 1)), 1) + return page, page_size + + +def report_group_options_for_job(job: PipelineJob) -> list[str]: + """类目选项：与第五章矩阵一致，来自合并表商品详情页类目路径解析。""" + qs = ( + JdJobMergedRow.objects.filter(job=job) + .exclude(matrix_group_label="") + .values_list("matrix_group_label", flat=True) + .distinct() + ) + return sorted({str(x) for x in qs if x}) + + +def detail_category_path_options(job: PipelineJob) -> list[str]: + return list( + JdJobDetailRow.objects.filter(job=job) + .exclude(detail_category_path="") + .values_list("detail_category_path", flat=True) + .distinct() + .order_by("detail_category_path")[:400] + ) + + +def shop_options_for_job(job: PipelineJob) -> list[str]: + """任务内各表出现的店铺名去重排序（搜索 shop_name、商详/宽表店铺列）。""" + names: set[str] = set() + for v in ( + JdJobSearchRow.objects.filter(job=job) + .exclude(shop_name="") + .values_list("shop_name", flat=True) + .distinct() + ): + t = str(v).strip() + if t: + names.add(t) + for v in ( + JdJobDetailRow.objects.filter(job=job) + .exclude(detail_shop_name="") + .values_list("detail_shop_name", flat=True) + .distinct() + ): + t = str(v).strip() + if t: + names.add(t) + for v in ( + JdJobMergedRow.objects.filter(job=job) + .exclude(shop_name="") + .values_list("shop_name", flat=True) + .distinct() + ): + t = str(v).strip() + if t: + names.add(t) + for v in ( + JdJobMergedRow.objects.filter(job=job) + .exclude(detail_shop_name="") + .values_list("detail_shop_name", flat=True) + .distinct() + ): + t = str(v).strip() + if t: + names.add(t) + return sorted(names) diff --git a/backend/pipeline/views/dataset_views.py b/backend/pipeline/views/dataset_views.py new file mode 100644 index 0000000..a191cb3 --- /dev/null +++ b/backend/pipeline/views/dataset_views.py @@ -0,0 +1,249 @@ +"""任务入库后的数据集浏览、筛选与导出。""" +from __future__ import annotations + +from django.http import HttpResponse +from rest_framework import status +from rest_framework.response import Response +from rest_framework.views import APIView + +from ..dataset_api import ( + DETAIL_SORT_FIELDS, + MERGED_SORT_FIELDS, + SEARCH_SORT_FIELDS, + apply_detail_filters, + apply_detail_order, + apply_merged_filters, + apply_merged_order, + apply_search_filters, + apply_search_order, + detail_category_q_from_request, + filter_echo, + parse_sort_meta, + price_bounds_from_request, + report_group_from_request, + shop_from_request, +) +from ..dataset_nonempty import ( + comment_columns_for_api, + detail_columns_for_api, + merged_columns_for_api, + search_columns_for_api, +) +from ..export_job import build_csv_bytes, build_json_bytes, build_xlsx_bytes +from ..models import ( + JdJobCommentRow, + JdJobDetailRow, + JdJobMergedRow, + JdJobSearchRow, +) +from ..row_serialize import ( + comment_row_to_dict, + detail_row_to_dict, + merged_row_to_dict, + search_row_to_dict, +) +from .common import ( + dataset_job, + detail_category_path_options, + read_page_params, + report_group_options_for_job, + shop_options_for_job, +) + + +class JobDatasetSummaryView(APIView): + """任务在库中的搜索/详情/评价行数（入库后可用）。""" + + def get(self, request, pk: int): + job = dataset_job(pk) + return Response( + { + "job_id": job.id, + "keyword": job.keyword, + "status": job.status, + "search_rows": JdJobSearchRow.objects.filter(job=job).count(), + "detail_rows": JdJobDetailRow.objects.filter(job=job).count(), + "comment_rows": JdJobCommentRow.objects.filter(job=job).count(), + "merged_rows": JdJobMergedRow.objects.filter(job=job).count(), + "search_columns": search_columns_for_api(job), + "detail_columns": detail_columns_for_api(job), + "comment_columns": comment_columns_for_api(job), + "merged_columns": merged_columns_for_api(job), + "category_options": report_group_options_for_job(job), + "shop_options": shop_options_for_job(job), + "detail_category_path_options": detail_category_path_options(job), + "dataset_sort_help": { + "search": sorted(SEARCH_SORT_FIELDS), + "detail": sorted(DETAIL_SORT_FIELDS), + "merged": sorted(MERGED_SORT_FIELDS), + "comments": ["row_index"], + }, + } + ) + + +class JobDatasetSearchView(APIView): + def get(self, request, pk: int): + job = dataset_job(pk) + page, page_size = read_page_params(request) + sort, desc = parse_sort_meta(request) + sort_eff = sort if sort in SEARCH_SORT_FIELDS else "row_index" + rg = report_group_from_request(request) + sp = shop_from_request(request) + pmin, pmax = price_bounds_from_request(request) + dcq = detail_category_q_from_request(request) + qs = JdJobSearchRow.objects.filter(job=job) + qs = apply_search_filters(qs, request) + qs = apply_search_order(qs, sort_eff, desc) + total = qs.count() + start = (page - 1) * page_size + rows = qs[start : start + page_size] + return Response( + { + "total": total, + "page": page, + "page_size": page_size, + "filters": filter_echo( + report_group=rg, + shop=sp, + price_min=pmin, + price_max=pmax, + detail_category_q=dcq, + sort=sort_eff, + desc=desc, + ), + "results": [search_row_to_dict(r) for r in rows], + } + ) + + +class JobDatasetDetailView(APIView): + def get(self, request, pk: int): + job = dataset_job(pk) + page, page_size = read_page_params(request) + sort, desc = parse_sort_meta(request) + sort_eff = sort if sort in DETAIL_SORT_FIELDS else "row_index" + rg = report_group_from_request(request) + sp = shop_from_request(request) + pmin, pmax = price_bounds_from_request(request) + dcq = detail_category_q_from_request(request) + qs = JdJobDetailRow.objects.filter(job=job) + qs = apply_detail_filters(qs, request) + qs = apply_detail_order(qs, sort_eff, desc) + total = qs.count() + start = (page - 1) * page_size + rows = qs[start : start + page_size] + return Response( + { + "total": total, + "page": page, + "page_size": page_size, + "filters": filter_echo( + report_group=rg, + shop=sp, + price_min=pmin, + price_max=pmax, + detail_category_q=dcq, + sort=sort_eff, + desc=desc, + ), + "results": [detail_row_to_dict(r) for r in rows], + } + ) + + +class JobDatasetCommentsView(APIView): + def get(self, request, pk: int): + job = dataset_job(pk) + page, page_size = read_page_params(request) + sku_id = (request.query_params.get("sku_id") or "").strip() + qs = JdJobCommentRow.objects.filter(job=job) + if sku_id: + qs = qs.filter(sku_id=sku_id) + total = qs.count() + start = (page - 1) * page_size + rows = qs.order_by("row_index")[start : start + page_size] + return Response( + { + "total": total, + "page": page, + "page_size": page_size, + "sku_filter": sku_id or None, + "results": [comment_row_to_dict(r) for r in rows], + } + ) + + +class JobDatasetMergedView(APIView): + def get(self, request, pk: int): + job = dataset_job(pk) + page, page_size = read_page_params(request) + sort, desc = parse_sort_meta(request) + sort_eff = sort if sort in MERGED_SORT_FIELDS else "row_index" + rg = report_group_from_request(request) + sp = shop_from_request(request) + pmin, pmax = price_bounds_from_request(request) + dcq = detail_category_q_from_request(request) + qs = JdJobMergedRow.objects.filter(job=job) + qs = apply_merged_filters(qs, request) + qs = apply_merged_order(qs, sort_eff, desc) + total = qs.count() + start = (page - 1) * page_size + rows = qs[start : start + page_size] + return Response( + { + "total": total, + "page": page, + "page_size": page_size, + "filters": filter_echo( + report_group=rg, + shop=sp, + price_min=pmin, + price_max=pmax, + detail_category_q=dcq, + sort=sort_eff, + desc=desc, + ), + "results": [merged_row_to_dict(r) for r in rows], + } + ) + + +class JobDatasetExportView(APIView): + """下载：kind=search|detail|comments|merged|all，export_fmt=json|csv|xlsx。 + + ``merged``：库内合并宽表行（与 lean 合并 CSV 列一致，入库后导出）。 + 注意：勿使用查询参数名 ``format``，DRF 会将其用于内容协商，非 json 时易在进视图前 404。 + """ + + def get(self, request, pk: int): + job = dataset_job(pk) + kind = (request.query_params.get("kind") or "search").strip().lower() + fmt = (request.query_params.get("export_fmt") or "json").strip().lower() + if kind not in ("search", "detail", "comments", "all", "merged"): + return Response( + {"detail": "kind 须为 search / detail / comments / all / merged"}, + status=status.HTTP_400_BAD_REQUEST, + ) + if fmt not in ("json", "csv", "xlsx"): + return Response( + {"detail": "export_fmt 须为 json / csv / xlsx"}, + status=status.HTTP_400_BAD_REQUEST, + ) + try: + if fmt == "json": + data, filename = build_json_bytes(job=job, kind=kind) + resp = HttpResponse(data, content_type="application/json; charset=utf-8") + elif fmt == "csv": + data, filename = build_csv_bytes(job=job, kind=kind) + resp = HttpResponse(data, content_type="text/csv; charset=utf-8") + else: + data, filename = build_xlsx_bytes(job=job, kind=kind) + resp = HttpResponse( + data, + content_type="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", + ) + except ValueError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + resp["Content-Disposition"] = f'attachment; filename="{filename}"' + return resp diff --git a/backend/pipeline/views/ingest_views.py b/backend/pipeline/views/ingest_views.py new file mode 100644 index 0000000..26b4484 --- /dev/null +++ b/backend/pipeline/views/ingest_views.py @@ -0,0 +1,40 @@ +"""将任务目录下的 CSV 重新入库。""" +from __future__ import annotations + +from django.http import Http404 +from django.utils.decorators import method_decorator +from django.views.decorators.csrf import csrf_exempt +from rest_framework import status +from rest_framework.response import Response +from rest_framework.views import APIView + +from ..ingest import ingest_job_full +from ..models import JobStatus, PipelineJob + + +@method_decorator(csrf_exempt, name="dispatch") +class JobImportMergedView(APIView): + """将指定任务目录下搜索/详情/评价 CSV 与合并表重新写入数据库（幂等：先清空该任务三类行再全量插入）。""" + + def post(self, request, pk: int): + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + if job.status == JobStatus.RUNNING: + return Response( + {"detail": "执行中不可入库，请待任务结束或终止后再试"}, + status=status.HTTP_400_BAD_REQUEST, + ) + if not (job.run_dir or "").strip(): + return Response( + { + "detail": "任务未绑定 run_dir。可 PATCH /api/pipeline/jobs/<id>/ " + "传入 run_dir，或使用 python manage.py ingest_pipeline_dataset。" + }, + status=status.HTTP_400_BAD_REQUEST, + ) + try: + stats = ingest_job_full(job) + except FileNotFoundError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + return Response(stats) diff --git a/backend/pipeline/views/job_report_views.py b/backend/pipeline/views/job_report_views.py new file mode 100644 index 0000000..5463daf --- /dev/null +++ b/backend/pipeline/views/job_report_views.py @@ -0,0 +1,478 @@ +"""任务维度的竞品摘要、简报包、策略稿、Markdown 导出与报告资源文件。""" +from __future__ import annotations + +import mimetypes +from typing import Any +from pathlib import Path + +import requests +from django.conf import settings +from django.http import FileResponse, Http404, HttpResponse +from django.utils import timezone +from django.utils.decorators import method_decorator +from django.views.decorators.csrf import csrf_exempt +from rest_framework import status +from rest_framework.response import Response +from rest_framework.views import APIView + +from ..jd.runner import ( + build_competitor_brief_for_job, + regenerate_competitor_report, +) +from ..llm.generate import generate_strategy_draft_markdown_llm +from ..llm.generate_marketing_detail import generate_marketing_detail_pack +from ..models import JobStatus, PipelineJob +from ..reporting.brief_pack import build_brief_pack_zip_bytes +from ..reporting.brief_strategy_scope import ( + filter_brief_for_strategy_matrix_group, + list_matrix_groups_for_api, + resolve_strategy_matrix_group_index, +) +from ..reporting.marketing_pack_persist import persist_marketing_detail_pack_v1 +from ..reporting.md_document_export import markdown_to_docx_bytes, markdown_to_pdf_bytes +from ..reporting.report_matrix_group_evidence import ( + load_report_matrix_group_evidence_markdown, +) +from ..reporting.report_strategy_excerpt import load_report_strategy_excerpt +from ..reporting.strategy_draft import build_strategy_draft_markdown +from ..serializers import ( + MarketingDetailPackRequestSerializer, + PipelineJobSerializer, + StrategyDraftRequestSerializer, +) +from .common import job_run_dir_usable + + +class JobCompetitorBriefView(APIView): + """单次任务的结构化竞品摘要（JSON，与 ``competitor_analysis.md`` **同一套计数规则**，规则驱动无 LLM）。""" + + def get(self, request, pk: int): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + return Response( + {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + if job.status != JobStatus.SUCCESS or not (job.run_dir or "").strip(): + return Response( + {"detail": "仅可对已成功且含 run_dir 的任务获取竞品摘要"}, + status=status.HTTP_400_BAD_REQUEST, + ) + try: + data = build_competitor_brief_for_job( + job.run_dir, + job.keyword, + report_config=job.report_config + if isinstance(job.report_config, dict) + else None, + ) + except FileNotFoundError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + except ValueError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + if isinstance(data, dict): + data = dict(data) + data["matrix_groups"] = list_matrix_groups_for_api(data) + return Response(data) + + +class JobCompetitorBriefPackView(APIView): + """ZIP：完整 Markdown 报告 + 结构化 JSON + 要点摘录 Markdown + 说明文本。""" + + def get(self, request, pk: int): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + return Response( + {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + if job.status != JobStatus.SUCCESS or not (job.run_dir or "").strip(): + return Response( + {"detail": "仅可对已成功且含 run_dir 的任务导出简报包"}, + status=status.HTTP_400_BAD_REQUEST, + ) + try: + brief = build_competitor_brief_for_job( + job.run_dir, + job.keyword, + report_config=job.report_config + if isinstance(job.report_config, dict) + else None, + ) + zip_bytes = build_brief_pack_zip_bytes(Path(job.run_dir), brief) + except FileNotFoundError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + except ValueError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + + filename_ascii = f"job_{pk}_competitor_brief_pack.zip" + resp = HttpResponse(zip_bytes, content_type="application/zip") + resp["Content-Disposition"] = f'attachment; filename="{filename_ascii}"' + return resp + + +@method_decorator(csrf_exempt, name="dispatch") +class JobStrategyDraftView(APIView): + """ + 市场策略制定 Markdown：策略框架 + 附录；默认规则生成，可选 ``generator=llm``（``AI_crawler.chat_completion_text``）。 + """ + + def post(self, request, pk: int): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + return Response( + {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + if job.status != JobStatus.SUCCESS or not (job.run_dir or "").strip(): + return Response( + {"detail": "仅可对已成功且含 run_dir 的任务生成策略制定稿"}, + status=status.HTTP_400_BAD_REQUEST, + ) + ser = StrategyDraftRequestSerializer(data=request.data or {}) + ser.is_valid(raise_exception=True) + vd = ser.validated_data + notes = (vd.get("business_notes") or "").strip() + strategy_decisions = { + "product_role": vd.get("product_role") or "", + "stage_goal_type": vd.get("stage_goal_type") or "", + "time_horizon": vd.get("time_horizon") or "", + "success_criteria": vd.get("success_criteria") or "", + "non_goals": vd.get("non_goals") or "", + "battlefield_one_line": vd.get("battlefield_one_line") or "", + "positioning_choice": vd.get("positioning_choice") or "", + "competitive_stance": vd.get("competitive_stance") or "", + "pillar_product": vd.get("pillar_product") or "", + "pillar_price": vd.get("pillar_price") or "", + "pillar_channel": vd.get("pillar_channel") or "", + "pillar_comm": vd.get("pillar_comm") or "", + "audience_segment": vd.get("audience_segment") or "", + "competitor_reference": vd.get("competitor_reference") or "", + "resource_notes": vd.get("resource_notes") or "", + "marketing_strategy": vd.get("marketing_strategy") or "", + "general_strategy": vd.get("general_strategy") or "", + "ack_risk_keywords": bool(vd.get("ack_risk_keywords")), + "ack_risk_price": bool(vd.get("ack_risk_price")), + "ack_risk_concentration": bool(vd.get("ack_risk_concentration")), + } + try: + brief = build_competitor_brief_for_job( + job.run_dir, + job.keyword, + report_config=job.report_config + if isinstance(job.report_config, dict) + else None, + ) + except FileNotFoundError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + except ValueError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + + matrix_groups = list_matrix_groups_for_api(brief) + sg_idx = vd.get("strategy_matrix_group_index") + sg_lbl = (vd.get("strategy_matrix_group") or "").strip() + scope_idx, scope_err = resolve_strategy_matrix_group_index( + brief, + matrix_group_index=sg_idx, + matrix_group_label=sg_lbl or None, + ) + if scope_err: + return Response({"detail": scope_err}, status=status.HTTP_400_BAD_REQUEST) + strategy_scope_applied: dict[str, Any] | None = None + if scope_idx is not None: + brief = filter_brief_for_strategy_matrix_group( + brief, matrix_group_index=scope_idx + ) + raw_sa = brief.get("strategy_scope_applied") + strategy_scope_applied = ( + raw_sa if isinstance(raw_sa, dict) else None + ) + + report_matrix_evidence_md = "" + report_matrix_evidence_src = "none" + if scope_idx is not None and 0 <= scope_idx < len(matrix_groups): + gnm = (matrix_groups[scope_idx].get("group") or "").strip() + if gnm: + report_matrix_evidence_md, report_matrix_evidence_src = ( + load_report_matrix_group_evidence_markdown( + job.run_dir, + gnm, + ) + ) + + gen_at = timezone.now().isoformat() + generator = (vd.get("generator") or "rules").strip() + excerpt_src = "none" + report_excerpt = "" + try: + report_excerpt, excerpt_src = load_report_strategy_excerpt(job.run_dir) + except OSError: + report_excerpt, excerpt_src = "", "none" + rc_job = job.report_config if isinstance(job.report_config, dict) else None + try: + if generator == "llm": + md = generate_strategy_draft_markdown_llm( + job_id=job.id, + keyword=job.keyword, + brief=brief, + business_notes=notes, + generated_at_iso=gen_at, + strategy_decisions=strategy_decisions, + report_strategy_excerpt=report_excerpt, + report_matrix_group_evidence_md=report_matrix_evidence_md + or None, + report_config=rc_job, + ) + src = "llm_text_ai_crawler_v1" + else: + md = build_strategy_draft_markdown( + job_id=job.id, + keyword=job.keyword, + brief=brief, + business_notes=notes, + generated_at_iso=gen_at, + strategy_decisions=strategy_decisions, + report_config=rc_job, + ) + src = "structured_summary_rules_v1" + except ValueError as e: + return Response({"detail": str(e)}, status=status.HTTP_503_SERVICE_UNAVAILABLE) + except requests.RequestException as e: + return Response( + {"detail": f"大模型网关错误：{e}"}, + status=status.HTTP_502_BAD_GATEWAY, + ) + body: dict[str, object] = { + "schema_version": 1, + "job_id": job.id, + "keyword": job.keyword, + "generated_at": gen_at, + "source": src, + "markdown": md, + "report_strategy_excerpt_source": excerpt_src, + "report_strategy_excerpt_chars": len(report_excerpt or ""), + "matrix_groups": matrix_groups, + "strategy_scope_applied": strategy_scope_applied, + "report_matrix_group_evidence_source": report_matrix_evidence_src, + "report_matrix_group_evidence_chars": len(report_matrix_evidence_md or ""), + } + return Response(body) + + +@method_decorator(csrf_exempt, name="dispatch") +class JobMarketingDetailPackView(APIView): + """ + 根据浏览器会话中的策略稿 Markdown，经「核心信息卡」再派生**营销内容**（多触点文案 JSON）。 + 两步均走 ``call_llm``；事实约束见提示词。 + """ + + def post(self, request, pk: int): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + return Response( + {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + if job.status != JobStatus.SUCCESS or not (job.run_dir or "").strip(): + return Response( + {"detail": "仅可对已成功且含 run_dir 的任务生成营销内容"}, + status=status.HTTP_400_BAD_REQUEST, + ) + ser = MarketingDetailPackRequestSerializer(data=request.data or {}) + ser.is_valid(raise_exception=True) + vd = ser.validated_data + md = (vd.get("strategy_markdown") or "").strip() + notes = (vd.get("business_notes") or "").strip() + raw_sd = vd.get("strategy_decisions") + strategy_decisions = raw_sd if isinstance(raw_sd, dict) else {} + gen_at = timezone.now().isoformat() + try: + inner = generate_marketing_detail_pack( + keyword=job.keyword, + strategy_markdown=md, + strategy_decisions=strategy_decisions, + business_notes=notes, + ) + except ValueError as e: + return Response({"detail": str(e)}, status=status.HTTP_502_BAD_GATEWAY) + except requests.RequestException as e: + return Response( + {"detail": f"大模型网关错误：{e}"}, + status=status.HTTP_502_BAD_GATEWAY, + ) + body: dict[str, object] = { + "schema_version": 1, + "job_id": job.id, + "keyword": job.keyword, + "generated_at": gen_at, + "source": "llm_marketing_detail_pack_v1", + **inner, + } + try: + persist_marketing_detail_pack_v1(job.run_dir, body) + except OSError: + pass + return Response(body) + + +@method_decorator(csrf_exempt, name="dispatch") +class JobExportDocumentView(APIView): + """ + 将 Markdown 导出为 Word（.docx）或简易 PDF。 + - GET：``kind=report``，读取 ``run_dir/competitor_analysis.md``；若文件缺失但已有合并表， + 则先按任务配置调用 ``regenerate_competitor_report`` 再导出（与「报告生成」规则版一致）。 + - POST：``kind=strategy``，请求体 ``markdown`` 为策略稿正文；``kind=marketing_detail`` 为营销内容等派生稿（同一套转版逻辑，下载文件名不同）。 + PDF 依赖本机中文字体或环境变量 ``MA_PDF_FONT`` 指向 .ttf。 + """ + + def get(self, request, pk: int): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + return Response( + {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + if not job_run_dir_usable(job): + return Response( + {"detail": "仅可对已成功或已终止且含 run_dir 的任务导出"}, + status=status.HTTP_400_BAD_REQUEST, + ) + fmt = (request.query_params.get("fmt") or "docx").strip().lower() + kind = (request.query_params.get("kind") or "report").strip().lower() + if kind != "report": + return Response( + {"detail": "GET 仅支持 kind=report；策略稿请用 POST 提交 markdown"}, + status=status.HTTP_400_BAD_REQUEST, + ) + if fmt not in ("docx", "pdf"): + return Response( + {"detail": "fmt 须为 docx 或 pdf"}, + status=status.HTTP_400_BAD_REQUEST, + ) + path = Path(job.run_dir) / "competitor_analysis.md" + if not path.is_file(): + rc = job.report_config if isinstance(job.report_config, dict) else None + try: + regenerate_competitor_report(job.run_dir, job.keyword, report_config=rc) + except FileNotFoundError as e: + return Response( + {"detail": str(e)}, + status=status.HTTP_404_NOT_FOUND, + ) + except ValueError as e: + return Response( + {"detail": str(e)}, + status=status.HTTP_400_BAD_REQUEST, + ) + if not path.is_file(): + return Response( + {"detail": "报告文件不存在且未能从合并表生成，请先在「报告生成」重新生成"}, + status=status.HTTP_404_NOT_FOUND, + ) + md = path.read_text(encoding="utf-8") + asset_root = Path(job.run_dir).resolve() + try: + if fmt == "docx": + data = markdown_to_docx_bytes(md, asset_root=asset_root) + ct = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + fn = f"job_{pk}_competitor_report.docx" + else: + data = markdown_to_pdf_bytes(md, asset_root=asset_root) + ct = "application/pdf" + fn = f"job_{pk}_competitor_report.pdf" + except ValueError as e: + return Response({"detail": str(e)}, status=status.HTTP_503_SERVICE_UNAVAILABLE) + resp = HttpResponse(data, content_type=ct) + resp["Content-Disposition"] = f'attachment; filename="{fn}"' + return resp + + def post(self, request, pk: int): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + return Response( + {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + if not job_run_dir_usable(job): + return Response( + {"detail": "仅可对已成功或已终止且含 run_dir 的任务导出"}, + status=status.HTTP_400_BAD_REQUEST, + ) + body = request.data if isinstance(request.data, dict) else {} + kind = (body.get("kind") or "strategy").strip().lower() + fmt = (body.get("fmt") or "docx").strip().lower() + md = (body.get("markdown") or "").strip() + if kind not in ("strategy", "marketing_detail"): + return Response( + {"detail": "POST 的 kind 须为 strategy 或 marketing_detail"}, + status=status.HTTP_400_BAD_REQUEST, + ) + if not md: + return Response( + {"detail": "markdown 不能为空"}, + status=status.HTTP_400_BAD_REQUEST, + ) + if fmt not in ("docx", "pdf"): + return Response( + {"detail": "fmt 须为 docx 或 pdf"}, + status=status.HTTP_400_BAD_REQUEST, + ) + try: + if kind == "marketing_detail": + base = f"job_{pk}_marketing_detail_pack" + else: + base = f"job_{pk}_strategy_draft" + if fmt == "docx": + data = markdown_to_docx_bytes(md) + ct = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + fn = f"{base}.docx" + else: + data = markdown_to_pdf_bytes(md) + ct = "application/pdf" + fn = f"{base}.pdf" + except ValueError as e: + return Response({"detail": str(e)}, status=status.HTTP_503_SERVICE_UNAVAILABLE) + resp = HttpResponse(data, content_type=ct) + resp["Content-Disposition"] = f'attachment; filename="{fn}"' + return resp + + +class JobReportAssetView(APIView): + """安全读取 ``run_dir/report_assets/*`` 下的 PNG 等（供 Markdown 预览插图）。""" + + def get(self, request, pk: int): + job = PipelineJob.objects.filter(pk=pk).first() + if not job or not job_run_dir_usable(job): + raise Http404() + rel = (request.query_params.get("path") or "").strip().replace("\\", "/") + if not rel or ".." in Path(rel).parts: + return Response( + {"detail": "path 非法"}, + status=status.HTTP_400_BAD_REQUEST, + ) + base = Path(job.run_dir).resolve() + assets_root = (base / "report_assets").resolve() + target = (base / rel).resolve() + try: + target.relative_to(assets_root) + except ValueError: + raise Http404() + if not target.is_file(): + raise Http404() + ctype, _ = mimetypes.guess_type(str(target)) + return FileResponse( + target.open("rb"), + content_type=ctype or "application/octet-stream", + ) diff --git a/backend/pipeline/views/job_views.py b/backend/pipeline/views/job_views.py new file mode 100644 index 0000000..6979628 --- /dev/null +++ b/backend/pipeline/views/job_views.py @@ -0,0 +1,338 @@ +"""任务生命周期：列表/详情、取消/续跑、下载与预览、报告默认配置、重新生成报告。""" +from __future__ import annotations + +import logging +import threading +from pathlib import Path + +import requests +from django.conf import settings +from django.http import FileResponse, Http404, HttpResponse +from django.utils.decorators import method_decorator +from django.views.decorators.csrf import csrf_exempt +from rest_framework import status +from rest_framework.response import Response +from rest_framework.views import APIView + +from ..ingest import resolve_and_validate_run_dir +from ..jd.runner import ( + build_competitor_brief_for_job, + get_default_report_config, + get_default_strategy_config, + merge_llm_supplement_with_rules_report, + regenerate_competitor_report, + write_competitor_analysis_markdown, +) +from ..llm.generate import generate_competitor_report_markdown_llm +from ..models import JobStatus, PipelineJob +from ..serializers import ( + CreatePipelineJobSerializer, + JobReportConfigPatchSerializer, + JobResumeRequestSerializer, + JobStrategyConfigPatchSerializer, + PipelineJobSerializer, + RegenerateReportRequestSerializer, +) +from ..tasks import execute_job +from .common import PREVIEW_MAX_BYTES, job_run_dir_usable, safe_file_for_job + +logger = logging.getLogger(__name__) + + +@method_decorator(csrf_exempt, name="dispatch") +class JobListCreateView(APIView): + def get(self, request): + qs = ( + PipelineJob.objects.select_related("checkpoint_row") + .all() + .order_by("-created_at")[:200] + ) + return Response(PipelineJobSerializer(qs, many=True).data) + + def post(self, request): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + return Response( + {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + ser = CreatePipelineJobSerializer(data=request.data) + ser.is_valid(raise_exception=True) + data = ser.validated_data + raw_rc = data.get("report_config") + if not isinstance(raw_rc, dict) or raw_rc == {}: + report_config_initial = get_default_report_config() + else: + report_config_initial = raw_rc + job = PipelineJob.objects.create( + platform=data["platform"], + keyword=data["keyword"], + max_skus=data.get("max_skus"), + page_start=data.get("page_start"), + page_to=data.get("page_to"), + pipeline_run_dir=data.get("pipeline_run_dir") or "", + cookie_file_path=data.get("cookie_file_path") or "", + cookie_text=data.get("cookie_text") or "", + pvid=data.get("pvid") or "", + request_delay=data.get("request_delay") or "", + list_pages=data.get("list_pages") or "", + scenario_filter_enabled=data.get("scenario_filter_enabled"), + report_config=report_config_initial, + strategy_config=get_default_strategy_config(), + status=JobStatus.PENDING, + ) + t = threading.Thread(target=execute_job, args=(job.id,), daemon=True) + t.start() + return Response( + PipelineJobSerializer(job).data, + status=status.HTTP_201_CREATED, + ) + + +@method_decorator(csrf_exempt, name="dispatch") +class JobDetailView(APIView): + def get(self, request, pk: int): + job = ( + PipelineJob.objects.filter(pk=pk) + .select_related("checkpoint_row") + .first() + ) + if not job: + raise Http404() + return Response(PipelineJobSerializer(job).data) + + def patch(self, request, pk: int): + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + body = request.data if isinstance(request.data, dict) else {} + update_fields: list[str] = [] + if "report_config" in body: + ser = JobReportConfigPatchSerializer(data={"report_config": body["report_config"]}) + ser.is_valid(raise_exception=True) + job.report_config = ser.validated_data["report_config"] + update_fields.append("report_config") + if "strategy_config" in body: + ser = JobStrategyConfigPatchSerializer(data={"strategy_config": body["strategy_config"]}) + ser.is_valid(raise_exception=True) + job.strategy_config = ser.validated_data["strategy_config"] + update_fields.append("strategy_config") + if "run_dir" in body: + try: + job.run_dir = str( + resolve_and_validate_run_dir(str(body.get("run_dir") or "")) + ) + except ValueError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + update_fields.append("run_dir") + if not update_fields: + return Response( + { + "detail": "请提供 report_config、strategy_config 或 run_dir（用于绑定已有批次目录）" + }, + status=status.HTTP_400_BAD_REQUEST, + ) + job.save(update_fields=update_fields + ["updated_at"]) + return Response(PipelineJobSerializer(job).data) + + +@method_decorator(csrf_exempt, name="dispatch") +class JobCancelView(APIView): + """ + 终止：将 ``cancellation_requested`` 置位后，执行线程会尽快 ``terminate`` 采集子进程 + （效果接近在终端对脚本按 Ctrl+C），并保留已写入运行目录的文件。 + """ + + def post(self, request, pk: int): + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + if job.status not in ( + JobStatus.PENDING, + JobStatus.RUNNING, + JobStatus.PAUSED, + ): + return Response( + {"detail": "仅待执行、执行中或已暂停的任务可终止"}, + status=status.HTTP_400_BAD_REQUEST, + ) + job.cancellation_requested = True + job.save(update_fields=["cancellation_requested", "updated_at"]) + return Response(PipelineJobSerializer(job).data) + + +@method_decorator(csrf_exempt, name="dispatch") +class JobResumeView(APIView): + """ + 从 Cookie 暂停断点继续：可选请求体 ``{ "cookie_text": "..." }`` 更新 Cookie； + 置位 ``resume_from_checkpoint`` 并拉起与新建任务相同的采集子进程（环境变量 ``PIPELINE_RESUME=1``）。 + """ + + def post(self, request, pk: int): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + return Response( + {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + if job.status != JobStatus.PAUSED: + return Response( + {"detail": "仅「已暂停（待换 Cookie 续跑）」的任务可继续执行"}, + status=status.HTTP_400_BAD_REQUEST, + ) + ser = JobResumeRequestSerializer(data=request.data or {}) + ser.is_valid(raise_exception=True) + raw_cookie = ser.validated_data.get("cookie_text") or "" + from ..cookie_paste import normalize_browser_cookie_paste + + norm = normalize_browser_cookie_paste(raw_cookie) + update_fields = ["resume_from_checkpoint", "error_message", "updated_at"] + job.resume_from_checkpoint = True + job.error_message = "" + if norm: + job.cookie_text = norm + update_fields.insert(0, "cookie_text") + job.save(update_fields=update_fields) + t = threading.Thread(target=execute_job, args=(job.id,), daemon=True) + t.start() + job = ( + PipelineJob.objects.filter(pk=pk) + .select_related("checkpoint_row") + .first() + ) + return Response(PipelineJobSerializer(job).data, status=status.HTTP_200_OK) + + +class ReportConfigDefaultsView(APIView): + """返回 ``pipeline.competitor_report.jd_report`` 中与脚本常量一致的默认报告调参 JSON。""" + + def get(self, request): + try: + return Response(get_default_report_config()) + except FileNotFoundError as e: + return Response( + {"detail": str(e)}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + + +class StrategyConfigDefaultsView(APIView): + """返回策略生成页独立默认 JSON（与 ``report_config`` 无关）。""" + + def get(self, request): + return Response(get_default_strategy_config()) + + +class JobDownloadView(APIView): + def get(self, request, pk: int): + job = PipelineJob.objects.filter(pk=pk).first() + if not job or not job_run_dir_usable(job): + raise Http404() + name = (request.query_params.get("name") or "").strip().lower() + path = safe_file_for_job(job.run_dir, name) + return FileResponse( + path.open("rb"), + as_attachment=True, + filename=path.name, + ) + + +class JobPreviewView(APIView): + """浏览器内联查看产出（CSV / Markdown 文本），大文件截断。""" + + def get(self, request, pk: int): + job = PipelineJob.objects.filter(pk=pk).first() + if not job or not job_run_dir_usable(job): + raise Http404() + name = (request.query_params.get("name") or "").strip().lower() + fpath = safe_file_for_job(job.run_dir, name) + raw = fpath.read_bytes() + truncated = len(raw) > PREVIEW_MAX_BYTES + if truncated: + raw = raw[:PREVIEW_MAX_BYTES] + text = raw.decode("utf-8-sig", errors="replace") + if truncated: + text += "\n\n... [内容已截断，完整文件请使用下载]\n" + + if name == "report": + ctype = "text/markdown; charset=utf-8" + else: + ctype = "text/csv; charset=utf-8" + resp = HttpResponse(text, content_type=ctype) + resp["X-Preview-Truncated"] = "1" if truncated else "0" + resp["X-Preview-Filename"] = fpath.name + return resp + + +@method_decorator(csrf_exempt, name="dispatch") +class JobRegenerateReportView(APIView): + """基于任务已有 ``run_dir`` 内 CSV 重新生成 ``competitor_analysis.md``（不重新爬取）。""" + + def post(self, request, pk: int): + if not (settings.LOW_GI_PROJECT_ROOT or "").strip(): + return Response( + {"detail": "请先在 market_assistant/.env 中配置 LOW_GI_PROJECT_ROOT"}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + job = PipelineJob.objects.filter(pk=pk).first() + if not job: + raise Http404() + if job.status != JobStatus.SUCCESS or not (job.run_dir or "").strip(): + return Response( + {"detail": "仅可对已成功且已写入 run_dir 的任务重新生成报告"}, + status=status.HTTP_400_BAD_REQUEST, + ) + ser = RegenerateReportRequestSerializer(data=request.data or {}) + ser.is_valid(raise_exception=True) + generator = ser.validated_data.get("generator") or "rules" + rc = job.report_config if isinstance(job.report_config, dict) else None + try: + regenerate_competitor_report(job.run_dir, job.keyword, report_config=rc) + except FileNotFoundError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + except ValueError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + if generator == "llm": + try: + rules_md = ( + Path(job.run_dir) / "competitor_analysis.md" + ).read_text(encoding="utf-8") + brief = build_competitor_brief_for_job( + job.run_dir, job.keyword, report_config=rc + ) + md = generate_competitor_report_markdown_llm(brief, job.keyword) + md = merge_llm_supplement_with_rules_report(md, rules_md) + write_competitor_analysis_markdown(job.run_dir, md) + except FileNotFoundError as e: + return Response({"detail": str(e)}, status=status.HTTP_400_BAD_REQUEST) + except ValueError as e: + msg = str(e) + logger.warning( + "regenerate-report LLM ValueError job_id=%s: %s", pk, msg + ) + if "run_dir 不在京东数据目录下" in msg: + return Response( + {"detail": msg}, + status=status.HTTP_400_BAD_REQUEST, + ) + if "请设置环境变量" in msg: + return Response( + {"detail": msg + "（运行 Django 的终端需能读取到该环境变量）"}, + status=status.HTTP_400_BAD_REQUEST, + ) + if "提示词过长" in msg or "上下文上限" in msg: + return Response( + {"detail": msg}, + status=status.HTTP_400_BAD_REQUEST, + ) + return Response( + {"detail": msg}, + status=status.HTTP_503_SERVICE_UNAVAILABLE, + ) + except requests.RequestException as e: + return Response( + {"detail": f"大模型网关错误：{e}"}, + status=status.HTTP_502_BAD_GATEWAY, + ) + return Response(PipelineJobSerializer(job).data) diff --git a/backend/pipeline/views/product_views.py b/backend/pipeline/views/product_views.py new file mode 100644 index 0000000..c6efa17 --- /dev/null +++ b/backend/pipeline/views/product_views.py @@ -0,0 +1,100 @@ +"""跨任务的京东商品与快照查询。""" +from __future__ import annotations + +from django.db.models import Count, Q +from django.http import Http404 +from rest_framework.response import Response +from rest_framework.views import APIView + +from ..models import JdProduct, JdProductSnapshot +from ..serializers import ( + JdProductDetailSerializer, + JdProductListSerializer, + JdProductSnapshotBriefSerializer, + JdProductSnapshotDetailSerializer, +) + + +class JdProductListView(APIView): + """已入库 SKU 分页列表；支持按标题/SKU/品牌模糊搜、按合并表中的 pipeline_keyword 精确筛。""" + + def get(self, request): + limit = min(max(int(request.query_params.get("limit", 50)), 1), 200) + offset = max(int(request.query_params.get("offset", 0)), 0) + q = (request.query_params.get("q") or "").strip() + kw = (request.query_params.get("keyword") or "").strip() + qs = JdProduct.objects.annotate(snapshot_count=Count("snapshots")) + if q: + qs = qs.filter( + Q(sku_id__icontains=q) + | Q(title__icontains=q) + | Q(detail_brand__icontains=q) + ) + if kw: + from ..csv.schema import MERGED_FIELD_TO_CSV_HEADER + + h_kw = MERGED_FIELD_TO_CSV_HEADER["pipeline_keyword"] + qs = qs.filter( + Q(current_payload__pipeline_keyword=kw) + | Q(**{f"current_payload__{h_kw}": kw}) + ) + total = qs.count() + page = qs.order_by("-updated_at")[offset : offset + limit] + return Response( + { + "total": total, + "limit": limit, + "offset": offset, + "results": JdProductListSerializer(page, many=True).data, + } + ) + + +class JdProductDetailView(APIView): + def get(self, request, sku_id: str): + platform = (request.query_params.get("platform") or "jd").strip() or "jd" + obj = ( + JdProduct.objects.annotate(snapshot_count=Count("snapshots")) + .filter(platform=platform, sku_id=sku_id) + .first() + ) + if not obj: + raise Http404() + return Response(JdProductDetailSerializer(obj).data) + + +class JdProductSnapshotListView(APIView): + """某 SKU 的历史快照列表（不含整包 payload，便于时间线）。""" + + def get(self, request, sku_id: str): + platform = (request.query_params.get("platform") or "jd").strip() or "jd" + product = JdProduct.objects.filter(platform=platform, sku_id=sku_id).first() + if not product: + raise Http404() + snaps = ( + product.snapshots.select_related("job") + .order_by("-captured_at") + .all() + ) + return Response( + { + "platform": platform, + "sku_id": sku_id, + "count": snaps.count(), + "results": JdProductSnapshotBriefSerializer(snaps, many=True).data, + } + ) + + +class JdProductSnapshotDetailView(APIView): + """单条快照完整 payload，用于历史回放与字段级对比。""" + + def get(self, request, pk: int): + snap = ( + JdProductSnapshot.objects.select_related("product", "job") + .filter(pk=pk) + .first() + ) + if not snap: + raise Http404() + return Response(JdProductSnapshotDetailSerializer(snap).data) diff --git a/backend/pipeline/volume_parse.py b/backend/pipeline/volume_parse.py new file mode 100644 index 0000000..35faf11 --- /dev/null +++ b/backend/pipeline/volume_parse.py @@ -0,0 +1,50 @@ +"""从爬虫导出文案解析销量、评价量等非负整数（与 reporting.charts._cn_volume_int 同源）。""" +from __future__ import annotations + +import re + + +def cn_volume_int(s: str | None) -> int: + """ + 支持「亿」「万」及纯数字；如 ``已售50万+`` → 500000。 + 无法解析时返回 0。 + """ + t = (s or "").strip().replace(",", "").replace("，", "") + if not t: + return 0 + m = re.search(r"(\d+(?:\.\d+)?)\s*亿", t) + if m: + return int(round(float(m.group(1)) * 100_000_000)) + m = re.search(r"(\d+(?:\.\d+)?)\s*万", t) + if m: + return int(round(float(m.group(1)) * 10_000)) + m2 = re.search(r"(\d+)", t) + if m2: + return int(m2.group(1)) + return 0 + + +def sales_sort_value_from_search_cells(total_sales: str, comment_sales_floor: str) -> int | None: + """搜索行：优先 ``total_sales``，否则销量楼层；两列皆空则 None。""" + ts = (total_sales or "").strip() + fl = (comment_sales_floor or "").strip() + if not ts and not fl: + return None + v = cn_volume_int(ts) + if v > 0: + return v + v2 = cn_volume_int(fl) + if v2 > 0: + return v2 + return 0 + + +def comment_count_sort_value_from_cell(comment_count: str) -> int | None: + if not (comment_count or "").strip(): + return None + return cn_volume_int(comment_count) + + +def comment_count_sort_value_from_merged(pipeline_comment_count: str) -> int | None: + """宽表评价量列（与搜索侧列表文案风格类似）。""" + return comment_count_sort_value_from_cell(pipeline_comment_count) diff --git a/backend/requirements.txt b/backend/requirements.txt index c295b50..730e50c 100644 --- a/backend/requirements.txt +++ b/backend/requirements.txt @@ -7,3 +7,10 @@ requests>=2.31 python-docx>=1.1 reportlab>=4.0 matplotlib>=3.8 +playwright>=1.40 + +# 第八章评论文本补充分析（默认写入竞品报告第八章第二节时需安装） +jieba>=0.42 +scikit-learn>=1.4 +numpy>=1.26 +wordcloud>=1.9 diff --git a/docs/demo/市场策略稿-示例-20260410_134015_低GI.md b/docs/demo/市场策略稿-示例-20260410_134015_低GI.md new file mode 100644 index 0000000..8eb3e3d --- /dev/null +++ b/docs/demo/市场策略稿-示例-20260410_134015_低GI.md @@ -0,0 +1,11 @@ +# 市场策略稿（示例 · 批次 20260410_134015）— 已过时 + +本文件曾基于较早跑批撰写，其中 **§1.2「核心场景」规则统计占比表**（如早餐/代餐 28.6%）来自简报 **`usage_scenarios` 词组规则**，**已非当前产品主依据**。 + +当前版本已改为以 **第八章第二节「评论文本补充分析」（文本挖掘：分词、词频、共现、LDA、词云）** 作为评论侧策略依据（`chapter8_text_mining_probe` 开启时）；**不再**使用原「评价正负面粗判」预设口语短语扇形图/条形图。未开探针时第八章第二节仍为「关注词与使用场景」并列图路径。 + +**请改读：** + +- **[市场策略稿-示例-20260413_104252_低GI.md](./市场策略稿-示例-20260413_104252_低GI.md)**（`data/JD/pipeline_runs/20260413_104252_低GI`，与现有文本挖掘产出一致） + +目录模板仍见：[市场策略稿-目录模板-六主轴与品牌四线.md](../templates/市场策略稿-目录模板-六主轴与品牌四线.md)。 diff --git a/docs/demo/市场策略稿-示例-20260413_104252_低GI.md b/docs/demo/市场策略稿-示例-20260413_104252_低GI.md new file mode 100644 index 0000000..60f24b0 --- /dev/null +++ b/docs/demo/市场策略稿-示例-20260413_104252_低GI.md @@ -0,0 +1,209 @@ +# 市场策略稿（示例 · 批次 20260413_104252_低GI） + +示例稿，数据来自京东「低GI」检索监测与评论分析（批次见标题）。内部讨论用。 + +--- + +## 策略范围与前提（生成前先对齐） + +| 须明确项 | 本示例中的填写 | +|----------|----------------| +| **监测任务（数据同源）** | 关键词「低GI」；批次 20260413_104252；与同任务竞品分析报告一致 | +| **策略服务对象（本品角色）** | *（示例占位：追赶型粗粮饼干线，非真实客户定稿）* | +| **一句话战场** | 京东站内「低GI」检索池内，与头部健康零食抢「控糖+饱腹+口感」心智 | +| **目标客群/场景** | 控糖关注型、代餐/加餐、早餐场景为主 | +| **主推类目/细类** | 多线：饼干为主、西式糕点为辅；面点仅观测 | +| **本阶段策略目标类型** | **B** 份额追赶 + **A** 要素验证（商详规格与话术可核验） | +| **时间范围** | *（示例：未来 12 周）* | +| **成功标准（可量化）** | *（示例：主推款加购、差评主题收敛，由业务定 KPI）* | + +*未定项在真实任务中应回填表单或保留「待确认」，勿编造。* + +--- + +## 摘要 + +- **范围与样本**：检索池覆盖范围、列表与深入样本规模、价格分布见**附录**。 +- **用户侧**：饼干类评论侧重口感、酥脆、饱腹、全麦、独立包装与零食化；西式糕点侧重面包、早餐、松软、牛奶及血糖相关联想；面点侧重早餐、豆包、新鲜与日期。负向评价相对集中：**分量/规格偏小**、**口感适配**（偏干、噎、与「松软」预期差）为主；包装物流整体正面，个案仍须闭环。 +- **阶段重点**：按细类明确主推价位带；详情与传播用语与评论高频维度一致；价格、促销与榜单类表述须**可核验、与实际一致**，避免虚构活动或无法支撑的超级话术。 + +--- + +## 一、顾客是谁 + +### 1.1 人群与决策路径 + +- 主动检索「低GI」进入多品类结果池（本批次可见数十个小类、两百余家店铺）。典型路径为搜索 → 列表比价 → 详情与配料 → 评价 → 下单或复购。 +- 评价中**口感、包装、物流、价格**关注度高，与关注词统计结论一致；**分量、规格、够吃与否**常出现在规格对比与复购讨论中，宜单独纳入卖点与客服话术。 + +### 1.2 细类讨论焦点（评论文本分析） + +基于分词、词频与共现；主题为探索性归纳，供叙事参考。 + +**饼干**（约 340 条）：词频突出饼干、口感、饱腹、酥脆、健康、回购、零食、全麦、方便、纤维；强共现如口感—酥脆、口感—饱腹、全麦—饼干；主题围绕代餐零食化、无糖与独立包装、GI 与回购。 + +**西式糕点**（约 295 条）：面包、口感、早餐、松软、牛奶、全麦；早餐—面包—牛奶—松软共现较强；血糖、控糖、吐司等出现在突出度与主题中。 + +**面点**（约 96 条）：口感、早餐、豆包、馒头、新鲜、日期、健康；偏主食与即食点心。 + +**其它细类**（样本量较小）：中式糕点侧重沙琪玛与口感零食；干货类侧重粉丝、绿豆、龙口、配料表与干净；米、挂面、杂粮、冲饮等见专项分节。 + +### 1.3 本品聚焦（占位） + +- 若主打**粗粮饼干**：叙事对齐饼干组高频维度，与监测矩阵中碧翠园、轻食兽等同档价位与卖点对标。 +- 若主打**面包/吐司**：对齐西式糕点组「早餐+松软+血糖联想」叙事，表述须合规。 +- 跨品类「全低GI」一盘货资源占用大，本期可明确**暂不展开**的细类。 + +--- + +## 二、产品价值与用户痛点 + +*本节仅 **§2.1**：用一张表写清「针对痛点要怎么做」；价带/规格/转化类应对**不**在第三章再写一遍。* + +### 2.1 针对痛点要怎么做 + +| 类目/细类（本决策适用） | 用户痛点（简述） | 策略动作 | 具体怎么做（触点/话术/规格/渠道） | 如何验证 | +|--------------------------|------------------|----------|-----------------------------------|----------| +| 饼干/粗粮饼干 | 分量与场景预期 | 参数与视觉诚实 | 商详写清净含量、包数；主图避免「看起来很大包」 | 差评「量小」、问大家 | +| 西式糕点/面包 | 口感适配（干、噎、松软） | 分线承接松软+搭配 | 复热、切片、牛奶搭配说明；与糕点评论高频一致 | 细类差评抽样、晒图 | +| 全检索池（价盘共性） | 比价与促销不透明 | 到手价与规则可核对 | 标价、券、榜单与后台一致；满减未监测到的上架前对齐运营 | 大促客诉、价差投诉 | +| 本品全线（若仅一条 SKU 可写「主推款」） | 健康与信任 | 配料与 GI 可验证 | 配料表、营养成分与商详一致；禁用无依据疗效 | 抽检、法务记录 | + +*说明：成稿时 §1.2 宜压缩为结论句，避免复述报告词频；负向主题归纳详见同任务《竞品分析报告》第八章。* + +--- + +## 三、为什么要买「这款产品」 + +### 3.1 品类与时机 + +- 「低GI」在平台侧检索规模大（接口返回约 **333,619** 条匹配，**非**销售额）；用户需求与站内检索入口并存。 +- 深入样本中**饼干**在类目混排中占比较高（第一细类 **21** 款）；主推进项需在 **100** 款深入样本中选定 **1～2** 个主推款。 +- **价带锚点（仅陈述，应对动作见 §2.1）**：列表价带宽 **8.9～1794**；深入样本中位约 **33.87**，主推款卡位可接近中位或按细类单独锚定——与 §2.1 中「比价与促销」行一致，**此处不重复展开应对段落**。 + +--- + +## 四、为什么要选「这个品牌」 + +### 4.1 品牌承诺与调性（占位） + +- **一句话**：在监测价位带内，把「低GI」讲清楚，把口感与饱腹做实。 +- **调性**：透明、可验证、不恐吓式控糖。 + +### 4.2 信任与证据 + +- **可用**：真实评价摘录、配料与营养成分；销量与好评类表述须有后台或可公开依据。 +- **禁用**：医疗功效；无依据的排名与「第一」类绝对化表述。 + +--- + +## 五、与其它品牌有何不同 + +### 5.1 对比对象 + +- 列表曝光前列店铺含 **DGI 官方旗舰店、慢教授、神探伍伍、五谷磨房** 等；深入样本中品牌 **DGI** 占比相对较高——**仅反映本批次样本内可见格局**，本品对标由业务指定 2～3 个款/店。 + +### 5.2 差异化方向（占位） + +| 差异点 | 说明 | 风险 | +|--------|------|------| +| 配料与 GI 叙事 | 与商详、检测一致 | 证据维护成本 | +| 价位清晰 | 相对深入样本中位与四分位区间 | 促销波动 | +| 口感与场景话术统一 | 对齐评论高频维度 | 须与产品一致 | + +### 5.3 竞争应对 + +- **价格**：不主动无底线条幅战；跟进促销时规则可解释、可追溯。 +- **传播**：差异点来自可验证事实与细类卡位，不依赖无法支撑的超级话术。 + +--- + +## 六、阶段目标与路径 + +### 6.1 本阶段定义 + +- 在目标细类与目标价位带内，进入用户**优先比较范围**（具体指标：详情转化、加购、复购等，由业务确定）。 + +### 6.2 路径 + +1. 对外数据与话术口径在内部统一、可复核。 +2. 详情与客服用语对齐评论高频与共现维度。 +3. 负向评价优先复盘 **分量与规格**、**口感适配（干/噎/松软）**、物流与价格；对照同任务报告 §8 归因，区分「硬/脆」与「干/噎」类反馈。 + +--- + +## 七、品牌四线：建设 · 打造 · 运营 · 体验 + +### 7.1 品牌建设 + +- 长期定位：「低GI」可验证、可感知（占位）；与第四节承诺一致。 + +### 7.2 品牌打造 + +- 店铺与详情主图、卖点层级与饼干/糕点高频维度一致；配料与 GI 信息层级清晰。 + +### 7.3 品牌运营 + +- 大促与券：对照本批次常见带（券价差可见行约 **91** / 589，折扣中位约 **10%**）；活动类型与力度须与实际执行一致。 + +### 7.4 品牌体验 + +- 详情承诺与物流、口感一致；包装相关高频反馈纳入改进。 + +--- + +## 八、战术支柱 + +### 8.1 产品策略 + +- **饼干线**：主打口感+饱腹+全麦/纤维+独立包装；对标矩阵内粗粮饼干款；**规格带**（克重、包数）与监测对标款对照，主图或卖点区避免「看起来很大包」的视觉误导。 +- **糕点线**：主打早餐+松软+搭配；血糖相关仅作合规转述；吐司/面包类明确切片数与克重，对应早餐场景是否够吃。若配方偏全麦、高纤维，对可能出现的「偏干、噎」类反馈，在商详补充复热、切片与搭配建议，与负向归因一致。 + +### 8.2 定价策略 + +- 列表统计与深入样本统计**两套口径**并存：对外声明须标明依据。**主推款**可锚定深入样本 **Q1～Q3**（约 **25.9～45.9**）或中位 **33.87** 附近（按业务选择）。 + +### 8.3 促销与活动策略 + +- **原则**：到手价与活动规则**透明、可核对**；跟价时说明依据，避免无法支撑的「全网最低」类表述。 +- **券与价差**：本批次列表侧约 **91 / 589** 行可见标价与券后价差线索（见附录）；成稿须与 `price_promotion_signals` 及报告第六章归纳一致，**禁止编造**未出现的满减门槛、红包面额。 +- **满减、满额折、跨店等**：若监测未捕获具体规则，须写明「上架/大促前与运营及后台活动对齐后再对外宣称」，并列出待补信息（是否参加跨店满减、店铺券类型等）；**禁止**因缺数字整节不写促销。 +- **标价与榜单**：与后台及实际活动一致，避免虚构。 + +### 8.4 渠道与传播 + +- 主战场京东站内检索与列表；内容关键词取自评论高频与共现。 + +--- + +## 九、风险、假设与待验证 + +- **样本**：深入 **100** 款、评论 **1121** 条；细类样本不均（如米、冲饮条数少），外推需谨慎。 +- **假设**：「本品」占位与真实产品线一致；**DGI 等**仅为格局描述，非固定对标。 +- **合规**：禁止医疗承诺；对外数字与表述须可溯源、可复核。 + +--- + +## 十、下一步与节奏 + +- 锁定主推款与对标表；过法务与合规。 +- 定稿前统一复核数据口径与对外话术。 +- 下轮监测扩大关键词或页码后更新策略依据。 + +--- + +## 附录：本任务关键数据一览 + +| 项 | 值 | +|----|-----| +| 监测词 | 低GI | +| 批次 | 20260413 104252 | +| 接口匹配量（proxy） | 约 333,619 | +| 列表行 / 页 | 589 / 1–10 | +| 深入款数 / 评论条数 | 100 / 1121 | +| 列表价 median（min–max） | 52.9（8.9–1794） | +| 深入价 median（min–max） | 33.87（3.87～268.97） | +| 列表第一店铺份额（行） | 约 7.3%（DGI官方旗舰店） | +| 深入第一品牌份额 | 约 8.6%（DGI） | +| 有券价优势行数（约） | 91 / 589 | +| 评论分析文件 | `chapter8_text_mining_probe.md`（同批次目录） | diff --git a/docs/demo/市场策略稿-示例-基于brief-schema样例.md b/docs/demo/市场策略稿-示例-基于brief-schema样例.md new file mode 100644 index 0000000..e19d301 --- /dev/null +++ b/docs/demo/市场策略稿-示例-基于brief-schema样例.md @@ -0,0 +1,214 @@ +# 市场策略稿（示例 · 内部讨论） + +> **更新**：请以 **[市场策略稿-示例-20260413_104252_低GI.md](./市场策略稿-示例-20260413_104252_低GI.md)** 为**真实批次 + 文本挖掘依据**的示例；旧文 **[市场策略稿-示例-20260410_134015_低GI.md](./市场策略稿-示例-20260410_134015_低GI.md)** 仅保留「已过时」说明。 +> 下文仍为 **schema 静态示例 JSON**，可作字段对照。 + +> **数据依据**：本稿数值与结构均来自仓库示例简报 **`docs/examples/competitor_brief_schema_v1.json`**（监测词「低GI」、样例批次字段），用于演示「六主轴 + 品牌四线」成稿形态，**非**某一真实客户或真实跑批的定稿。 +> **本品设定**：下文「本品」指**占位假设**——某品牌在「休闲食品 > 饼干 > 粗粮饼干」下布局的一条低 GI 产品线；实际使用时替换为真实品牌与 SKU。 + +--- + +## 摘要（1 页内） + +- **一句话战场**：京东站内以「**低GI**」为监测词的检索结果池；列表侧可见样本 **117** 行、去重 SKU 约 **95**；平台结果条数共识约 **12345**（简报 `pc_search_raw.result_count_consensus`）。 +- **目标用户一句话**：关注**控糖/血糖**与**代餐/加餐**场景、在京东通过关键词检索与比价选购休闲食品（尤其饼干类）的用户。 +- **价值主张一句话**（占位）：在监测样本所呈现的价格带内，提供**可感知口感与饱腹**、且 GI 诉求与配料信息可核对的一条粗粮饼干（需本品实测与详情页支撑）。 +- **与竞品核心差异一句话**（占位）：在「口感」被高频提及的舆论场中，以**清晰配料与低 GI 证据链** + **明确价位段卡位**（相对样本中位价 **16.9** 元附近）建立差异——具体差异需本品与对标 SKU 实测后写死。 +- **本阶段优先动作一句话**：先跑通**同一监测任务**下简报—报告—策略稿数字对齐；再选定 1～2 个对标 SKU 做价位与促销对照表。 + +> **对外提示**：「12345」「12%」等均为**监测样本内统计**，对外需按法务与平台规则改写，避免绝对化。 + +--- + +## 一、顾客是谁 + +### 1.1 核心人群 + +- **人口与行为特征**：检索「低GI」进入品类池；在样本评论池（**112** 条合并评论相关）中，「**口感**」为突出关注方向之一（简报 `comment_focus_keywords` 示例：**口感** 出现 **48** 次量级，为词表统计口径）。 +- **决策链**：用户主动搜索 → 列表比价/看促销 → 详情看配料与评价 → 下单；复购受口感、甜度与物流体验驱动（待业务访谈验证）。 + +### 1.2 核心场景 + +- **主场景**：**早餐/代餐**（简报 `consumer_feedback_by_matrix_group` 下「饼干」组示例：早餐/代餐场景占比约 **20%** 文本单元）；**控糖/血糖相关**（`usage_scenarios` 示例占比约 **15%**）。 +- **次要场景**：休闲零食、办公室加餐（评论与词表可进一步挖掘，本示例稿不展开）。 + +### 1.3 本品在人群与场景上的聚焦 + +- **本期主打**：控糖关注型 + 代餐场景（与监测摘要一致，易与「低GI」检索意图对齐）。 +- **暂搁置**：未在监测摘要中出现明确信号的场景（如纯儿童市场），不单独开策略线。 + +**依据**：`competitor_brief_schema_v1.json` → `usage_scenarios`、`consumer_feedback_by_matrix_group`（饼干组）、`comment_focus_keywords`。 + +--- + +## 二、产品价值与用户痛点 + +> 与自动草稿一致：**仅 §2.1 针对痛点要怎么做**（一张表）；不在此另列「痛点表 + 价值对表」以免与 §三 重复。 + +### 2.1 针对痛点要怎么做 + +| 类目/细类（本决策适用） | 用户痛点（简述） | 策略动作 | 具体怎么做（触点/话术/规格/渠道） | 如何验证 | +|--------------------------|------------------|----------|-----------------------------------|----------| +| 饼干（`matrix_by_group` 饼干组） | 担心「低GI」名不副实、配料与体验不可感知 | 配料与 GI 叙事可验证 | 商详配料表、营养成分与监测/检测一致；口感 QA 与问大家 | 抽检、差评主题 | +| 饼干（同上） | 价格带宽、促销复杂，难比「真实到手价」 | 到手价与规则透明 | 卡位接近样本中位 **16.9**；促销表述与 `notes`、价盘一致 | 大促期客诉 | +| 全池共性（列表侧） | 品牌与店铺分散，信任成本高 | 证据链与价位锚点 | 与同任务报告价盘一致；不夸大未在简报出现的销量表述 | 商详与报告对照 | + +**依据**：简报 `price_stats`、`concentration`、`comment_focus_keywords`、`notes`。 + +--- + +## 三、为什么要买「这款产品」（品类与产品层） + +### 3.1 品类与时机 + +- **为何需要该品类**：「低GI」在站内检索结果规模大（共识约 **12345**），说明需求侧存在稳定检索入口；粗粮饼干在样本类目混排中可见（`category_mix_top` 示例：**休闲食品**）。 +- **主推进项（占位）**：**饼干**矩阵内样本 **3** 个 SKU（`matrix_by_group` 饼干组），选 1 个主推款做详情与评价运营，其余做价位锚点。 +- **价带一句（不重复 §2.1 应对）**：列表样本价 **14.9～39.9**，中位 **16.9**；转化与卡位动作见 **§2.1** 表。 + +**依据**：`price_stats`、`matrix_by_group`（饼干组示例 SKU）、`notes`。 + +--- + +## 四、为什么要选「这个品牌」（品牌层） + +### 4.1 品牌承诺与调性（占位） + +- **一句话**：做低 GI 粗粮零食里「**说得清、吃得下**」的一条线（内部表述，对外需法务）。 +- **调性关键词**：克制、透明、可验证、不恐吓式控糖。 + +### 4.2 信任与证据 + +- **可引用类型**：评价中与「口感」相关的真实摘录（须脱敏）、配料与营养成分展示、与同任务报告一致的销量/评价量级表述（**不得编造**简报外数字）。 +- **禁止或慎用**：医疗功效、治愈承诺；未在监测数据出现的「第一」「全网最低」等。 + +**依据**：项目规划 [`strategy-marketing-content-alignment.md`](../planning/strategy-marketing-content-alignment.md) 事实源原则；简报 `notes`。 + +--- + +## 五、与其它品牌有何不同（差异化） + +### 5.1 主要对比对象 + +- 监测样本内合并品牌侧 Top1 标签示例为「**品牌A**」，份额约 **20%**（`concentration.detail_brand_among_merged`）；列表侧店铺 Top 示例为「**某旗舰店**」约 **12%**（`shops_from_list`）。 +- **本品**应对标：同价位带（围绕 **16.9** 中位）、同粗粮饼干叶子类目的 2～3 个 SKU（从 `matrix_by_group` 与业务指定补全）。 + +### 5.2 可防御差异点 + +| 差异点 | 与谁不同 | 监测中是否可体现 | 风险 | +|--------|----------|------------------|------| +| 配料与 GI 叙事可核对 | 同价位模糊表述款 | 可通过商详与报告配料模块对照 | 证据链维护成本 | +| 价位卡位清晰 | 宽幅价带两端 | 可通过价盘统计与到手价规则 | 促销变动快 | +| 口感评价可运营 | 同关键词下通用款 | 评论「口感」已高频，可做 QA | 需真实产品一致 | + +### 5.3 竞争应对原则 + +- **价格**：不主动打无底线条幅战；跟进促销时保持**到手价与赠品规则**可解释（与任务内「促销摘要」归纳一致若已生成）。 +- **不跟随**：不参与与本品配料等级不一致的纯低价对标（内部原则，对外不直述竞品名）。 + +**依据**：`concentration`、`price_stats`、`comment_focus_keywords`。 + +--- + +## 六、如何成为用户的第一选择（目标与路径） + +### 6.1 「第一选择」在本阶段的定义（内部） + +- **心智**：在「低GI + 粗粮饼干 + 目标价位」的考虑集合里进入**前三**（可量化：搜索词下点击率、详情转化率、复购率——需业务定指标）。 +- **代理指标（示例）**：详情页停留、问答区「口感/血糖」类提问占比下降、差评率。 + +### 6.2 路径总览 + +1. 监测任务产出的简报与报告**数字与策略稿同源**（已规划 S1）。 +2. 主推 SKU 与对标表固化后，统一详情页卖点与促销话术。 +3. 评论侧针对「口感」做结构化回复与产品迭代闭环（与第八章评论分析一致）。 + +**依据**：内部规划文档；简报 `strategy_hints` 示例占位「样本内…（待验证）」须替换为可验证命题。 + +--- + +## 七、品牌四线：建设 · 打造 · 运营 · 体验 + +### 7.1 品牌建设（长期心智） + +- **主张**：透明、可验证的低 GI 零食（占位）。 +- **与第四节对应**：一切传播回扣「配料与证据链」，不夸大疗效。 + +### 7.2 品牌打造（可感知资产） + +- 店铺与详情页统一「监测任务同款」数据口径（销量、评价、价位带图表若引用须同源）。 +- 粗粮饼干叶子类目视觉与 GI/配料信息层级一致（具体设计略）。 + +### 7.3 品牌运营（节奏与触达） + +- 大促与平台活动节奏：结合任务内「促销/价盘」归纳段落（若已生成）排期；避免与监测样本明显冲突的虚构活动。 + +### 7.4 品牌体验（全链路） + +- 详情页承诺与物流、口感一致；差评中与「口感」相关条目优先复盘（与评论高频词一致）。 + +**依据**：评论「口感」、场景词；`strategy-marketing-content-alignment.md` 对齐要求。 + +--- + +## 八、战术支柱（回扣战略） + +**回扣句**：本节各条均服务于「控糖/代餐」人群与「口感+信任」痛点，并落在品牌四线中的**运营 + 体验**为主、**建设**为辅。 + +### 8.1 产品策略 + +- 饼干细类保留 1 主推 + 1 锚点款；配料与规格与监测对标 SKU 显式对照（内部表）。 + +### 8.2 定价策略 + +- 列表样本中位 **16.9**、均值 **21.5**：主推款建议卡在 **16.9～22.9** 区间（占位），高端款需额外成分/规格支撑；**合并样本仅 5 个 SKU**（`scope.merged_sku_count`），对外表述监测结论时须提示样本量限制。 + +### 8.3 促销与活动策略 + +- 促销规则与榜单/腰带话术：与任务流水线产出的「促销摘要」一致（若已接入）；不编造未出现的活动类型。 + +### 8.4 渠道与传播策略 + +- 主战场与监测一致：**京东站内**检索与列表可见性；站外传播不在本示例稿展开。 + +**依据**：`price_stats`、`scope`、`matrix_by_group`、`pipeline` 规划。 + +--- + +## 九、风险、假设与待验证 + +| 类型 | 内容 | +|------|------| +| 样本量 | 合并 SKU **5**、评论 **112**，结论外推需谨慎；列表 **117** 行仅为可见性代理。 | +| 假设 | 「本品」占位与真实产品线一致；对标「品牌A」仅为简报示例标签。 | +| 待验证 | 业务访谈确认主场景排序；真实 GI 检测与标签合规。 | +| 合规 | 禁止医疗承诺；数字仅来自同任务 brief/报告。 | + +--- + +## 十、下一步与节奏 + +1. 选定真实品牌与 SKU，替换本稿占位表述。 +2. 用同一 `run_dir` 跑通简报 → 报告 → 策略稿，核对策略稿数字与 brief、矩阵细类节选一致（默认产线下 `report_strategy_excerpt` 多为空，见规划 `strategy-marketing-content-alignment.md` §2）。 +3. 将对标表与价位带图固化进评审材料（内部）。 + +--- + +## 附录：本示例数据速览（摘自 schema 样例） + +| 字段 | 示例值 | +|------|--------| +| 监测词 | 低GI | +| 列表样本行数 | 117 | +| 去重 SKU（列表代理） | 95 | +| 合并 SKU 数 | 5 | +| 评论条数（合并相关） | 112 | +| 列表价格 min / max / median / mean | 14.9 / 39.9 / 16.9 / 21.5 | +| 列表店铺 Top1 份额 | 约 12% | +| 合并样本品牌 Top1 份额 | 约 20%（示例名：品牌A） | +| 评论关注词示例 | 口感（48） | +| 场景示例 | 控糖/血糖相关；早餐/代餐 | +| 矩阵细类示例 | 饼干（3 SKU） | + +--- + +*本文件为 `docs/templates/市场策略稿-目录模板-六主轴与品牌四线.md` 的填充示例，可随时废弃或覆盖。* diff --git a/docs/examples/competitor_brief_schema_v1.json b/docs/examples/competitor_brief_schema_v1.json index dfe6303..7a21ff0 100644 --- a/docs/examples/competitor_brief_schema_v1.json +++ b/docs/examples/competitor_brief_schema_v1.json @@ -26,15 +26,15 @@ }, "concentration": { "shops_from_list": { - "cr1": 0.12, - "cr3": 0.35, + "first_share": 0.12, + "top_three_combined_share": 0.35, "top_label": "某旗舰店", "top_share_pct": "12.0%" }, "list_brand_field": null, "detail_brand_among_merged": { - "cr1": 0.2, - "cr3": 0.6, + "first_share": 0.2, + "top_three_combined_share": 0.6, "top_label": "品牌A", "top_share_pct": "20.0%" } @@ -82,7 +82,8 @@ "shop": "示例店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "", - "comment_fuzzy": "20万+" + "comment_fuzzy": "1万+", + "total_sales": "已售50万+ | good:99%好评" } ] } @@ -102,7 +103,7 @@ } ], "notes": [ - "与在线分析报告统计口径一致；主题词与场景为预设词表，非 NLP 主题模型。", + "与在线分析报告各章计数规则一致；主题词与场景为预设词表，非 NLP 主题模型。", "价格来自展示字段抽取，含促销与规格差异。" ] } diff --git a/docs/openapi/pipeline-jobs.openapi.yaml b/docs/openapi/pipeline-jobs.openapi.yaml index b0846d0..5e3161c 100644 --- a/docs/openapi/pipeline-jobs.openapi.yaml +++ b/docs/openapi/pipeline-jobs.openapi.yaml @@ -70,7 +70,7 @@ paths: description: Not found patch: tags: [jobs] - summary: 仅更新 report_config（报告关注词/场景/外部市场表等） + summary: 仅更新 report_config（外部市场表等；预设关注词/场景已废弃的请求字段将被忽略） parameters: - $ref: "#/components/parameters/JobId" requestBody: @@ -112,7 +112,7 @@ paths: /api/report-config-defaults/: get: tags: [jobs] - summary: 报告调参默认模板（与 jd_competitor_report 脚本常量一致） + summary: 报告调参默认模板（与 pipeline.competitor_report.jd_report 脚本常量一致） responses: "200": description: OK @@ -217,7 +217,7 @@ paths: /api/jobs/{id}/competitor-brief/: get: tags: [jobs] - summary: 结构化竞品摘要 JSON（与 Markdown 报告统计口径一致） + summary: 结构化竞品摘要 JSON（与 Markdown 报告同一套计数规则） parameters: - $ref: "#/components/parameters/JobId" responses: @@ -367,23 +367,9 @@ components: ReportConfig: type: object description: | - 允许键仅限下列三项；可全部省略或 `{}` 表示使用脚本内置默认。 + 允许键以后端 ``validate_report_config_body`` 为准（含各 ``llm_*``、``chapter8_text_mining_probe*``、``external_market_table_rows`` 等）。 + 已废弃：``comment_focus_words``、``comment_scenario_groups``、``llm_scenario_group_summaries``（请求中若出现将被忽略）。 properties: - comment_focus_words: - type: array - items: - type: string - comment_scenario_groups: - type: array - items: - type: object - properties: - label: - type: string - triggers: - type: array - items: - type: string external_market_table_rows: type: array items: @@ -501,7 +487,7 @@ components: type: string enum: [rules, llm] default: rules - description: rules=jd_competitor_report 规则引擎；llm=结构化摘要 JSON + AI_crawler 文本接口 + description: rules=竞品报告规则引擎（jd_report）；llm=结构化摘要 JSON + AI_crawler 文本接口 StrategyDraftRequest: type: object @@ -533,7 +519,7 @@ components: positioning_choice: type: string enum: ['', 'top', 'mid', 'entry', 'different'] - description: 价格带主定位；空字符串表示文稿中四项均为未勾选 + description: 价位阵地取向（贴顶/卡腰/下探/另起带），写入策略稿「八、战术支柱 · 8.2 定价策略」；与「四、为什么要选这个品牌」（承诺与信任）分离。空字符串表示文稿中四项均为未勾选 competitive_stance: type: string enum: ['', 'flank', 'head_on', 'both', 'undecided'] @@ -549,6 +535,26 @@ components: pillar_comm: type: string maxLength: 800 + audience_segment: + type: string + maxLength: 500 + description: 目标客群一句话（可选） + competitor_reference: + type: string + maxLength: 800 + description: 主要对标品牌或价位带（可选） + resource_notes: + type: string + maxLength: 1000 + description: 资源与预算量级备注（可选） + marketing_strategy: + type: string + maxLength: 2000 + description: 营销策略（传播、活动、投放、内容等，可选） + general_strategy: + type: string + maxLength: 2000 + description: 总体策略（增长/品类/经营总原则，可选） ack_risk_keywords: type: boolean default: false @@ -585,3 +591,10 @@ components: example: structured_summary_rules_v1 markdown: type: string + report_strategy_excerpt_source: + type: string + description: none=未找到第九章正文；json_markdown=strategy_opportunities_llm.json 的 markdown；competitor_analysis_md=从 competitor_analysis.md 截取 + enum: [none, json_markdown, competitor_analysis_md] + report_strategy_excerpt_chars: + type: integer + description: 参与对齐的第九章节选字符数（供策略稿 LLM 的 report_strategy_excerpt；generator=rules 时仅元数据） diff --git a/docs/pipeline-job-output-spec.md b/docs/pipeline-job-output-spec.md index 414988d..dd9f418 100644 --- a/docs/pipeline-job-output-spec.md +++ b/docs/pipeline-job-output-spec.md @@ -48,7 +48,7 @@ - **下载 / 预览**：通过任务对应的下载、预览能力获取（仅成功且文件已生成时可用）。 - **重新生成报告**（不重新爬取）：仅用本批次已有表格与元数据刷新**分析报告**正文。 -- **结构化竞品摘要**：规则生成的 JSON，与在线分析报告**统计口径一致**；价格等指标在能解析列表价时以列表为准，否则以深入样本为准，响应内会标明口径。 +- **结构化竞品摘要**：规则生成的 JSON，与在线分析报告**同一套计数规则**；价格等指标在能解析列表价时以列表为准，否则以深入样本为准，响应内会标明**价格来源**。 - **一键简报包**：ZIP，内含完整分析报告稿、结构化摘要、要点摘录与说明；须已生成主报告。 - **市场策略制定**：提交可选业务备注后返回策略向文稿（策略框架 + 附录数据速览），数据与同任务结构化摘要一致；仅成功且数据齐全时可用。 - **商详表离线修正再入库**：由运维/研发在侧链完成后再走全量入库能力。 diff --git a/docs/planning/strategy-marketing-content-alignment.md b/docs/planning/strategy-marketing-content-alignment.md new file mode 100644 index 0000000..38512ee --- /dev/null +++ b/docs/planning/strategy-marketing-content-alignment.md @@ -0,0 +1,95 @@ +# 规划留痕：市场策略稿 · 营销内容 · 与竞品报告对齐 + +**成稿日期**：2026-04-17 +**目的**：划分**报告 / 策略 / 营销**三层职责：报告与简报管监测与论证；**可执行策略**以**独立策略稿（策略制定）**交付，与同任务 `brief`、报告第五～八章矩阵/评论侧归纳等**同源、不编造**；**营销内容只把已定稿的策略叙事翻成可对外触点文案**，不在营销管线里二次灌入报告原文。 +**关联代码（现状）**： + +| 能力 | 位置 | +|------|------| +| 规则策略底稿 | `pipeline/reporting/strategy_draft.py` → `build_strategy_draft_markdown` | +| 策略稿 LLM 润色 | `pipeline/llm/generate_strategy.py` → `generate_strategy_draft_markdown_llm`（payload 可选含 `report_strategy_excerpt`，**默认多为空**） | +| 报告「九、策略与机会提示」节选加载（遗留/兼容） | `pipeline/reporting/report_strategy_excerpt.py` → `load_report_strategy_excerpt` | +| 报告内第九章大模型长文（**默认关闭**） | `generate_strategy_opportunities_llm`；runner 默认 `llm_strategy_opportunities: false`，正文第九章为固定读者说明（见 `jd_report._strategy_opportunities_reader_fixed_lines`） | +| 策略稿 API / 导出 | `pipeline/views/job_report_views.py` → `JobStrategyDraftView` | +| 营销内容生成（核心信息卡 + 多触点文案） | `pipeline/llm/generate_marketing_detail.py`、`JobMarketingDetailPackView` | +| 简报与压缩 | `pipeline/reporting/brief_compact.py`（**策略/报告链路**；非营销管线默认输入） | +| Markdown→Word/PDF | `pipeline/reporting/md_document_export.py` | + +--- + +## 1. 原则（事实源与禁止项） + +| 维度 | 要求 | +|------|------| +| **事实源** | 仅允许来自：同任务 `build_competitor_brief` 产物、已落盘的 `competitor_analysis.md` 中与策略论证相关的既定小节、各章 LLM 节选 JSON、`strategy_hints`、按细类收窄时的 **`report_matrix_group_evidence_md`**（与报告第五～八章同源）。**不再**把「报告第九章大模型长文」当作默认事实源；历史任务若存在 `strategy_opportunities_llm.json.markdown` 可按加载规则视为可选补充。 | +| **策略稿** | 以规则底稿为骨架；**先**通过「策略范围与前提」与规划文档「启动前」对齐**针对什么做策略**；LLM 仅做可读性润色与衔接，**不得新增** brief/底稿中不存在的数字、品牌、销量、价格。全文还须遵守 `STRATEGY_DATA_RULES` 段首「**全局禁止编造**」（用户引语、活动规则、无依据的落地结果等）。 | +| **报告内「第九章」** | **默认产线**：竞品报告在「九、策略与机会提示」下**不**再附加全任务大模型策略长文，仅为短引导（指向「策略制定」按细类生成）。若调试或历史配置显式开启 `llm_strategy_opportunities` 并落盘正文，仍须遵守 `STRATEGY_OPPORTUNITIES_SYSTEM`（`generate_strategy.py`）中的硬性条款。 | +| **营销内容** | **仅表达层**：输入为**已定稿策略稿 Markdown + 表单决策 + 业务备注**（及后续可选受众/渠道等），**不新增事实**。**不默认**并入 `compact_brief`、报告第八/九章节选或其它报告正文。与监测数据的一致性由**策略稿所消费的 brief / 矩阵节选 / 表单**先收束；若营销稿与业务认知不符，应修订策略输入，而非在营销接口再拼报告正文。 | + +--- + +## 2. 策略稿：与宿主报告及数据同源（阶段 S1，**已修订口径**） + +**目标**：独立下载的策略稿与同任务**简报与报告中的监测结论**一致（计数、价带、矩阵细类、第五～八章归纳等），**不编造**；**不再**以「必须与报告内第九章大模型段落方向一致」作为默认验收口径。 + +**实现要点（与代码一致）**： + +1. **`load_report_strategy_excerpt(run_dir)`**（兼容字段）：若 `strategy_opportunities_llm.json` 含非空 **`markdown`**（通常仅历史任务或显式开启 LLM 第九章时），则载入；若仅有空壳 JSON，**不再**回退截取 `competitor_analysis.md`，避免把第九章固定读者说明误当策略正文。否则再尝试从 `competitor_analysis.md` 截取 `## 九、策略与机会提示` 至 `## 附录` 之前。 +2. **`STRATEGY_SYSTEM`**：当 `report_strategy_excerpt` **非空**时，润色稿须与该节选**不明显矛盾**；若与 `business_notes` / `strategy_decisions` 冲突，须在成稿中交代依据（如业务备注优先）。**默认**节选为空：成稿以 `structured_brief` + `report_matrix_group_evidence_md` + 底稿与表单为准，**不得**编造「报告第九章已写明的」具体结论。 +3. **API**：`POST /api/jobs/{id}/strategy-draft/` 仍返回 `report_strategy_excerpt_source`、`report_strategy_excerpt_chars`（`generator=rules` 时亦返回），便于核对当前任务是否仍存在遗留节选。 +4. 产品侧：`generator: rules | llm` 仍为既有行为；规则版作审计底稿。 + +**验收**：抽样任务核对策略稿数字与 brief、矩阵节选可对读；**不再**要求「第九章要点 ↔ 策略稿 bullet」一一对应（默认无第九章长文）。 + +--- + +## 3. 营销内容：仅生成营销稿（阶段 S2～S3） + +**边界（产品定论）** + +- **只做一件事**：把**策略已定稿的叙事**转成**可上架/可多触点使用的文案**（核心信息卡、商详与列表侧、主图要点、短视频钩句、客服首句、依据与边界等）。 +- **输入**：`strategy_markdown`、`strategy_decisions`、`business_notes`（与现网 `JobMarketingDetailPackView` 一致）；后续可增**可选** `audience_segment`、`channels` 等，**仍不得**作为新事实来源。 +- **刻意不做**：在营销请求里**默认拼接** `compact_brief`、报告 `competitor_analysis.md` 第八/九章节选或其它报告正文——避免「半篇报告 + 半篇卖点」的混杂产出；策略稿已承载与数据同源的叙事时，营销层信任该输入。 + +**提示词硬性约束（与实现对齐）** + +- 事实、数字、功效、引语**仅可**来自策略稿与表单/备注中已出现内容；食品/健康等合规禁区同策略侧原则。 +- 输出中保留 **依据与边界** 类字段，提醒对外宣称限度（**相对策略承诺**，非相对整份 PDF 报告再摘一层）。 + +**输出与留痕** + +- 现网：`run_dir/marketing/marketing_detail_pack_v1.json`；可下载 / Word / PDF 由前端与 `export-document` 支持。第二步 JSON 在**不编造**前提下偏**丰富**：更多标题/卖点/FAQ、详情中段叙事、食用搭配、短图文变体、直播要点提纲，以及 **文生图/文生视频** 可复制提示词（见 `generate_marketing_detail.py`）。 + +**与旧稿差异说明** + +- 本文件早期版本曾设想营销与 brief/报告节选**强绑定**；经产品收敛，**以本节边界为准**，不再将「必选 report_excerpts」作为默认架构。 + +--- + +## 4. 实施顺序 + +| 阶段 | 内容 | 产出 | +|------|------|------| +| **S1** | 策略稿 payload 含可选 `report_strategy_excerpt`；与宿主数据同源以 brief + 矩阵节选为主；第九章长文默认弃用 | ✅ 已合并：`report_strategy_excerpt.py`（含空壳 json 不回退）、API 响应字段；默认节选为空 | +| **S2** | 营销模块 v1：策略驱动两步 LLM + 落盘 + Word/PDF；**不**默认并入报告节选 | ✅ 方向与现网 `generate_marketing_detail` 一致，细节以代码为准 | +| **S3** | 前端：策略预览入口、可选受众/渠道、载入上次生成等 | 产品闭环（按需排期） | +| **S4**（可选） | 轻量校验：输出中数字与 brief 同源性启发式检查 | 降低明显幻觉 | + +--- + +## 5. 风险与边界 + +- LLM 无法 100% 杜绝编造，**规则底稿 + brief + 人工抽检**仍为默认。 +- 营销内容需保留「不替代合规/法务审核」类免责声明（可与报告附录表述一致）。 + +--- + +## 6. 修订记录 + +| 日期 | 说明 | +|------|------| +| 2026-04-17 | 首版：对齐原则、S1～S4、代码锚点、API 示意。 | +| 2026-04-18 | S1 落地：`load_report_strategy_excerpt`、`STRATEGY_SYSTEM` 对齐条款、`strategy_opportunities_llm.json.markdown`、策略稿 API 响应字段。 | +| 2026-04-22 | 收敛营销边界（§3）；**修订 S1**：默认产线弃用报告内第九章大模型长文，策略稿与数据对齐以 **brief + 第五～八章节选** 为主，`report_strategy_excerpt` 为遗留/空默认；更新 §1、§2、§4、能力表。 | + +后续变更请在本表追加一行，并在正文相应章节修改。 diff --git a/docs/planning/策略生成-LLM全量输入快照.md b/docs/planning/策略生成-LLM全量输入快照.md new file mode 100644 index 0000000..3388532 --- /dev/null +++ b/docs/planning/策略生成-LLM全量输入快照.md @@ -0,0 +1,644 @@ +# 策略生成 · 大模型一次调用的「全量输入」快照 + +> **生成方式**：本机 `pipeline.demos.dump_strategy_llm_input_md` 按与 `generate_strategy_draft_markdown_llm` 相同的 payload 组装逻辑导出。 +> **与线上一致性**：与真实接口相比，表单字段此处均为空默认；你只要把当时提交的 `strategy_decisions` / `business_notes` 代入即与线上等价。 +> +> **口径（2026-04）**：默认产线已弃用报告内第九章大模型长文，新任务 `report_strategy_excerpt` 多为空。下文为某次历史导出的**全文留档**，元数据与内嵌 System 片段中若仍出现「第九章」主对齐表述，以 `strategy-marketing-content-alignment.md`、`策略生成-框架确定.md` 现行版为准。 + +## 快照元数据 + +- **任务 ID**：12 +- **关键词**：低GI +- **run_dir**：`D:\PythonProject\Low GI\market_assistant\data\JD\pipeline_runs\20260413_104252_低GI` +- **矩阵分组**：0 → 「饼干」 +- **本任务可选细类（节选）**：['饼干', '西式糕点', '中式糕点', '面点', '干菜类', '米', '风味挂面', '其他杂粮', '冲饮谷物', '药食同源', '酸辣粉', '妈妈专区', '挂面', '混合麦片', '特色米粉/米线'] +- **report_strategy_excerpt（本快照）**：json_markdown，约 2930 字符（历史批次；非默认产线形态） +- **细类报告节选来源**：competitor_analysis_md，约 2682 字符 +- **System 字符数**：6927 +- **User 消息字符数**：22226 +- **合计约**：29153 字符 + +--- + +## 1. System 提示词（完整 `STRATEGY_SYSTEM`） + +```text +你是市场策略顾问，根据**结构化监测摘要**与业务侧填写的**决策字段**，把「规则底稿」写成**短、可执行**的策略 Markdown **独立成稿**。 + +**输入**：`rules_draft_markdown`（规则骨架，**六主轴 + 品牌四线**结构，与 `docs/demo` 市场策略稿示例同构）、`structured_brief`、`strategy_decisions`、`business_notes`；可选 `report_strategy_excerpt`；可选 **`report_matrix_group_evidence_md`**（与所选细类对齐的宿主报告大模型归纳摘录）。 + +**与细类收窄配套（当 JSON 含 `report_matrix_group_evidence_md` 且非空时，硬性）**： +- **定性主题**（用户讨论焦点、卖点/配料叙事、负向体验类型、场景与关注词归纳方向等）须与该节选及 `structured_brief` **方向一致**，**禁止**另写一套与节选**明显矛盾**的品类判断。 +- **数字、份额、价带、条数**仍以 **`structured_brief` 为准**；节选与 brief 数字冲突时**采纳 brief**，勿复述冲突数字句。 +- **`report_strategy_excerpt`（第九章）** 为**全关键词任务**下的策略归纳，可能与「仅选某细类」并行存在：写**该细类**策略时以 `structured_brief` + `report_matrix_group_evidence_md` 为主；第九章仅作检索池整体方向参考，**不得**把全池结论套成该细类已证实事实。 + +**全局禁止编造（适用于输出全文各节、各表、各段；独立策略稿与报告第九章策略归纳**共用**本段，硬性）**： +- **事实与数字**：销量、GMV、占比、价带、条数、份额、券面额、满减/满折门槛、到手价、店铺/品牌计数与排名、SKU 数、接口返回量等，**仅可**来自**本次调用输入 JSON** 中已给出的字段（策略稿为 `structured_brief`、`rules_draft_markdown` 内摘录、`report_strategy_excerpt`、可选 **`report_matrix_group_evidence_md`**（与同任务报告第五～第八章细类大模型小节同源）、`strategy_decisions`、`business_notes`；第九章嵌入为 `competitor_brief`、可选 `prior_chapter_llm_narratives`）；**禁止**凭空新增、改口径或写成「已监测证实」而无字段支撑。 +- **主体与名称**：**禁止**引入上述输入中**未出现**的**具体**品牌名、店铺名、SKU 名、商品标题作为**事实陈述**；若 `strategy_decisions`/备注/brief/节选已含则可写；否则用「头部/同类竞品」等泛称或「待业务指定对标」。 +- **用户侧表述**：**禁止**虚构评价原文、访谈引语、带引号的「用户说…」；细则见下文「§2 针对痛点要怎么做」表**痛点简述**列。 +- **促销与活动**：**禁止**编造活动名、具体规则、补贴比例；细则见下文促销与第八章探针相关条款。 +- **策略动作与落地结果**：可写「建议」「假设」「待验证」的动作方向，**不得**编造「已执行」「已上线」「数据显示转化率/复购提升」等**无输入依据**的结果。 +- **信息不足**：须写「输入未体现」「待核对」「假设：」「待验证：」，**禁止**用确定语气掩盖缺失依据。 +- **与 §2.1「类目/细类」列一致（全文）**：除 §2.1 表格外，**摘要、一、三～八**凡写策略动作、阶段重点、资源分配、差异化或竞争应对，**优先**标明适用**类目/细类**；多细类策略冲突时**分条**写。**禁止**用「全站用户」「整体上一句」覆盖与 §2.1 已分行决策**矛盾**的表述。 + +**与竞品分析报告的分工（硬性）**： +- 宿主报告已含样本量、价带分布、词频/共现、矩阵、第八章文本挖掘等**统计分析**。策略稿**不得**重复展开同类内容：不重写词频表、细类评论条数罗列、统计方法说明、与报告图表逐条复述。 +- **允许**：用一两句**结论性**话概括用户侧/评论侧要点；必要时写「详见同任务《竞品分析报告》§× / 附录」。 +- **必须**把篇幅放在**策略**：§2「针对痛点要怎么做」、后文战术与节奏（**勿**与报告重复统计展开）。 + +**数据与口径（硬性，与宿主分析报告同源输入）**： +- **§2「针对痛点要怎么做」表（反捏造 + 分类目，硬性）**： + - **「类目/细类（本决策适用）」列**：须与 `structured_brief` 中类目混排、矩阵分组、§1.2 细类讨论或 `strategy_decisions` 已选战场**可对上**；**禁止**编造未出现的类目名。**多细类并存**（如饼干 vs 面包）时，**必须分行**分策，**禁止**用「全站用户」「整体策略」等**泛化**一句覆盖彼此冲突的动作。**若**类目或主推线尚不确定，该行可写「待业务定类」或「假设：优先××线」，并说明**分类决策依据或待补信息**；仍须避免与数据明显矛盾。 + - **「用户痛点（简述）」列**：**禁止**书写「用户反馈『……』」「评价称『……』」等**带引号的逐字原话**，除非该片段在 `structured_brief`、`strategy_hints`、`report_strategy_excerpt` 或 `business_notes` 中**已出现相同或明显包含**的文本；否则一律**不得**用引号假装引用。 + - 若输入仅有主题级信号（关注词、负向归因方向、价差行数等），痛点简述应写**可追溯归纳**，例如「与 brief 中 ×× 字段一致」「与报告第八章/节选已归纳的 ×× 主题一致」「监测摘要见 `strategy_hints` 第 n 条」，或写「**待原评论抽样核实**」——**禁止**把合理推测写成「用户已明确说……」的事实口吻。 + - **禁止**凭空发明痛点行（如「配料相似」「卖点雷同」「性价比一般」）作为**已监测结论**；此类表述仅当 `structured_brief`、节选或备注中**确有同类主题或措辞**时方可写入，否则不写或标为待验证假设。 +- **不得编造**销量、GMV、未在 `structured_brief` 与底稿中出现的占比或价格；底稿与摘要中的数字须保持一致。 +- **店铺集中度**仅可依据 `structured_brief.concentration` 与底稿，并区分**列表行**与**去重 SKU**；用「第一大……份额」「前三家合计」等中文，**不要用** CR1、CR3。 +- **禁止编造**「京东自营 SKU 占比」「自营超 X%」等摘要中未给出的定量句。 +- **矩阵**：若 `structured_brief` 含矩阵相关字段，须**呼应**细分类目与竞品矩阵结论，不得无故删光。 +- **第八章文本挖掘探针（当 JSON 中 `chapter8_text_mining_probe` 为真时）**： + - **禁止**将「关注词子串命中次数」「预设场景分组条数/占比」当作评论侧主论据。 + - 用户洞察、负向归因须与 **§8 文本挖掘** 及可选节选一致；促销与券价差须与 `price_promotion_signals`、第六章/第九章已有归纳一致，**禁止**编造满减门槛或补贴比例。 +- **可选 `report_strategy_excerpt`**：非空时战略方向与该节选一致，不得明显矛盾；**不得**把节选与 `structured_brief` 均未出现的数字当作事实。为空时仅依据底稿与摘要，不得编造第九章结论。 + +**落实范围**：上文「全局禁止编造」适用于**摘要、一至十、附录**的每一句话与表格每一格；**不得**因章节不同而放宽。 + +**决策边界（硬性）**： +- **业务已在 `strategy_decisions` 中填写的项**（角色、时间、成功标准、战场一句话、定位勾选、竞争倾向、四柱、目标客群/对标/资源备注、**营销策略**与**总体策略**等）视为**已定决策**：成稿须**落实为具体执行句**，**不得**改写成相反结论或再要求用户「请选择」。 +- **表单中为空或占位（如 *待填*、*骨架占位*）的项**：结合 `structured_brief`、节选与数据摘录**补全为可执行表述**；补全须与数据方向一致。**例外**：「**策略范围与前提**」中若仍无依据，**允许**保留「待确认」并**列出** A～E 类目标选项，**禁止**用虚构本品角色或目标补满。 +- **成稿阶段避免**：反复「请业务决策」「待确认后再定」；**不确定时**须在 §2.1 表中用「类目/细类」列 +「待业务定类」「假设：」**分类**写清，**禁止**只写泛化一句带过。 + +**输出结构与阅读顺序（须与 `rules_draft_markdown` 章节一致，勿另起目录）**： +**策略范围与前提（生成前先对齐）** → **摘要** → **一、顾客是谁**（含人群与路径、细类讨论、本品聚焦）→ **二、产品价值与用户痛点**（**仅 §2.1 针对痛点要怎么做** 表，**勿**再设独立「痛点表/价值对表/负向归因」子节）→ **三、为什么要买「这款产品」**（**仅 §3.1 品类与时机**；**无 §3.2**，转化与价带应对已在 §2 表内则**勿重复**）→ **四、为什么要选「这个品牌」** → **五、与其它品牌有何不同** → **六、阶段目标与路径** → **七、品牌四线**（建设·打造·运营·体验）→ **八、战术支柱**（产品/定价/促销/渠道与传播）→ **九、风险、假设与待验证** → **十、下一步与节奏**（含业务备注）→ **附录**。 +可微调小节标题用语，**不得**删减上述逻辑块或把「诊断数据」与「落地动作」顺序颠倒；**禁止**私自恢复已删除的小节（如 §3.2）。 + +**语气**：面向业务读者，避免 CR1、心智等内部缩写；**勿在成稿中反复强调「对齐某报告第几章」**，以策略表述为主。 + +**策略表述硬性（痛点 → 怎么做，须覆盖全书，不得只写 §2～§8 部分章节）**： +- **总原则**：成稿**不是**第二份分析报告，也**不是**市场形势说明书。每条重要内容应能回答：**针对哪条用户痛点、在哪条类目/细类下**（与 **§2.1** 表对应）、**我们采取什么动作**、**在具体触点怎么做**（商详/主图/短视频/客服/规格/价格呈现等）、**如何验证**（若适用）。 +- **§2.1 针对痛点要怎么做**（若底稿已有表头）须**填写实质内容**；全稿**动作总锚**为 §2.1。若无表，须在 **§二** 或 **§八** 用等价分条写清「痛点—动作—落地—验证」。 + +**分节要求（与底稿章节一一对应，勿省略）**： +- **策略范围与前提**：回答「**这份策略是针对什么做的**」（监测任务、本品角色、战场、主推类目、**本阶段目标类型**、时间、成功标准）。与 `strategy_decisions`、`business_notes` 对齐；表格占位未填时**禁止**编造决策，应保留「待确认」并**列出**可选目标类型（见 `STRATEGY_USER_PREFIX` 与规划文档「启动前」）；可写「建议选项：…」供业务勾选。**禁止**与后文 §2.1、§六 自相矛盾。 +- **摘要**：除范围样本外，**阶段重点**须含 1～2 条**可执行动作**，且能回扣 §2 中的优先痛点（非空泛「加强运营」）；须**承接**上文「策略范围与前提」已定的边界。 +- **一、顾客是谁**：**禁止**重复报告中的细类词频、分品类样本量展开、文本挖掘方法；用 **少量结论句**（谁搜、关心什么）即可，**1.3 本品聚焦**须写清「本期主攻人群/场景」与 **§2.1** 的对应关系。 +- **二**：**仅 §2.1** 一张表：「类目/细类（本决策适用）| 用户痛点（简述）| 策略动作 | 具体怎么做 | 如何验证」。须覆盖监测已支撑的主要维度（**按类目分行**，口感/质地分线、分量/规格、信任与价格等，依数据取舍）；**类目列 + 痛点简述列**遵守「§2 表」条款。**禁止**再写独立「痛点与证据表」「价值对表」「负向归因」子节（与 §二 重复的内容一律并入本表或删去）。 +- **三**：**仅 §3.1**（品类与时机、检索/样本、购买理由）；可结合 brief 写价带锚点一句。**禁止**写 §3.2「转化障碍与应对」；若与购买相关的障碍与应对已在 §2.1 表内，§3.1 **勿再复述**。 +- **四**：品牌承诺与调性须能落到**可感知触点**（如商详第几屏、包装、客服首句），避免只有形容词。 +- **五**：§5.2 差异化、§5.3 竞争应对须写清**相对竞品多做什么/少做什么、具体一步动作**。 +- **六**：成功标准与 §6.2 路径须与 **§2.1** 动作**可对齐或合并叙述**；营销/总体策略句须为**动词导向**。 +- **七**：品牌四线**每一条**至少一句：**服务哪类痛点、本周/本阶段具体做哪一步**。 +- **八**：四支柱**每一支柱**须回扣 **痛点→动作→落地**（可与 §2.1 合并叙述，避免重复堆砌）。 +- **九**：在表单风险勾选之外，**每条风险**尽量带**应对动作或验证计划**（抽样、核对规则），勿只列风险标题。 +- **十**：下一步清单须为**可执行任务**（可含负责人/时间占位），与 §2.1 或 §六 优先级一致；可含「按类目核对主图/商详与 §2.1」类项。 + +**全书与 §2.1 类目列对齐（防泛化，与上条「全文一致」配套）**： +- **摘要**：阶段重点中的可执行动作**尽量**点明适用类目或主推线。 +- **四、五**：品牌承诺、差异化、竞争应对若因细类而异，**分款/分类目**写。 +- **六**：路径与成功标准若多类目并行，**分线**写 KPI 或写清主线/副线。 +- **七**：品牌四线每条宜**可指回** §2.1 某类目行；若四线共用全池，须一句交代**共用前提**。 +- **八**：四支柱下若产品/定价/促销策略因类目不同，**分子条**（如「饼干：…」「面包：…」），勿与 §2.1 矛盾。 +- **九**：可写「主推类目未定」「多线话术不一致」等风险及验证方式。 + +**口感/质地与细类（禁止「一词盖全站」）**： +- 监测中「**酥脆**」与「**松软**」等可能**同时**高频，通常对应**不同细类**（如饼干 vs 面包/糕点）或不同场景。成稿须**按主推细类或分产品线**表述：饼干线策略与酥脆/饱腹等对齐，面包/糕点线与松软/早餐等对齐；若多线并存须**分款分句**，**禁止**只写「要做松软」而忽略酥脆主导的细类，除非 `structured_brief`、表单或业务备注已明确**仅**推该线。 +- **产品策略句**须能指回：**本品是哪一类、解决哪条口感预期**，避免与数据里另一细类的主导词打架。 + +**促销：满减、满折、券（≠ 不管；≠ 编造）**： +- **必须**在 **§八.3 促销与活动策略**（及必要时 §七.3）写清：与 `price_promotion_signals`、报告第六章已归纳的**券、标价与到手价差、常见活动形态**如何承接（跟价节奏、规则透明、不与数据矛盾）；**禁止**因「没编出具体数字」就整节不写促销。 +- **禁止编造**输入中未出现的**具体**满减门槛、满额折扣、每满减金额；若摘要/报告未捕获某类机制，须明确写「**监测未捕获具体满减/满折规则，上架前须与运营及后台活动对齐后再对外宣称**」，并可列**待补信息**（如：是否参加跨店满减、店铺券类型）。 +- **区分**：「策略上跟券、保到手价透明」是成稿义务；「具体满 300 减 40」只能来自已有数据。 + +**输出**：仅 Markdown 正文（不要 ``` 围栏）；须收束各小节与全文，勿中途截断。 +``` + +--- + +## 2. User 消息（完整：`STRATEGY_USER_PREFIX` + JSON） + +以下为网关 **user** 角色一次发送的完整字符串（前缀 + 单行 JSON）。 + +```text +请基于以下 JSON 输出最终策略稿（Markdown）。输出前自检：全文不得包含输入 JSON 中未出现的具体数字、品牌/店铺名、用户引语与活动规则；不确定处须写「假设」「输入未体现」或「待核对」。「策略范围与前提」一节：若 `strategy_decisions` 中本品角色、战场、成功标准、主推类目等大量为空，**不得**编造；须列「待确认」并给出**目标类型选项**供业务选择，例如：**A** 上市/首发前验证 **B** 份额追赶 **C** 利润与价盘防守 **D** 新场景/新人群拓展 **E** 其它（简述）——具体措辞见项目规划文档「策略生成-框架确定」之「启动前」。 + +{"job_id": 12, "keyword": "低GI", "generated_at_iso": "2026-04-21T01:50:13.500179+00:00", "strategy_decisions": {"product_role": "", "time_horizon": "", "success_criteria": "", "non_goals": "", "battlefield_one_line": "", "positioning_choice": "", "competitive_stance": "", "pillar_product": "", "pillar_price": "", "pillar_channel": "", "pillar_comm": "", "audience_segment": "", "competitor_reference": "", "resource_notes": "", "marketing_strategy": "", "general_strategy": "", "ack_risk_keywords": false, "ack_risk_price": false, "ack_risk_concentration": false}, "business_notes": "", "structured_brief": {"schema_version": 1, "keyword": "低GI", "batch_label": "20260413 104252", "run_dir": "(已省略)", "scope": {"merged_sku_count": 21, "comment_flat_rows": 340, "structure_source_rows": 589, "uses_pc_search_list_export": true, "category_mix_source": "keyword_pipeline_merged", "category_mix_valid_matrix_sku_count": 70}, "meta": {"page_start": 1, "page_to": 10, "max_skus_config": 100, "pc_search_export_rows": 589, "merged_rows": 100, "scenario_filter_enabled": true, "merged_csv_mode": "lean"}, "pc_search_raw": {"result_count_consensus": 333619, "list_keyword": "低GI", "result_count_uniques": [332945, 332972, 333525, 333548, 333619], "raw_json_files_scanned": 20}, "list_visibility_proxy": {"total_rows": 21, "unique_skus": 21, "_strategy_scope_note": "矩阵所选分组内 SKU 数，非全关键词列表导出口径。"}, "concentration": {"shops_from_list": {"first_share": 0.23809523809523808, "top_three_combined_share": 0.5714285714285714, "top_label": "碧翠园京东自营旗舰店", "top_share_pct": "23.8%"}, "list_brand_field": null, "detail_brand_among_merged": {"first_share": 0.23809523809523808, "top_three_combined_share": 0.6666666666666666, "top_label": "碧翠园", "top_share_pct": "23.8%"}}, "category_mix_top": [{"label": "粗粮饼干", "count": 11}, {"label": "酥性饼干", "count": 10}], "list_brand_mix_top": [{"label": "碧翠园", "count": 5}, {"label": "DGI", "count": 5}, {"label": "轻食兽", "count": 4}, {"label": "红谷林（HONGGULIN）", "count": 3}, {"label": "考拉皮皮", "count": 1}, {"label": "神探伍伍", "count": 1}, {"label": "江中猴姑", "count": 1}, {"label": "Fix XBody", "count": 1}], "list_shop_mix_top": [{"label": "碧翠园京东自营旗舰店", "count": 5}, {"label": "轻食兽京东自营旗舰店", "count": 4}, {"label": "红谷林（HONGGULIN）食品京东自营旗舰店", "count": 3}, {"label": "DGI京东自营旗舰店", "count": 3}, {"label": "DGI官方旗舰店", "count": 2}, {"label": "考拉皮皮京东自营旗舰店", "count": 1}, {"label": "神探伍伍京东自营旗舰店", "count": 1}, {"label": "1号会员店", "count": 1}, {"label": "Fix XBody京东自营旗舰店", "count": 1}], "price_stats": {"min": 14.38, "max": 64.97, "mean": 29.28238095238095, "n": 21, "stdev": 14.193378704438878, "median": 27.97, "q1": 16.384999999999998, "q3": 33.42}, "price_stats_source": "strategy_scope_matrix_group_skus", "price_stats_merged_sample": {"min": 14.38, "max": 64.97, "mean": 29.28238095238095, "n": 21, "stdev": 14.193378704438878, "median": 27.97, "q1": 16.384999999999998, "q3": 33.42}, "price_stats_list_export": {}, "price_promotion_signals": [], "matrix_by_group": [{"group": "饼干", "sku_count": 21, "skus": [{"sku_id": "100065199809", "title": "碧翠园全麦粗粮 低GI 饼干代餐0添蔗糖500g奇亚籽健身抗饿零食品", "brand": "碧翠园", "list_price_show": "16.90", "coupon_or_detail_price": "", "detail_price_final": "16.90", "shop": "碧翠园京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "奇亚籽味 | 0添加蔗糖 | 国产", "comment_fuzzy": "20万+", "total_sales": "已售50万+"}, {"sku_id": "100049997473", "title": "轻食兽牛油果椰乳 低GI 全麦粗粮饼干168g 代餐休闲零食早餐控糖健康饱腹", "brand": "轻食兽", "list_price_show": "14.90", "coupon_or_detail_price": "", "detail_price_final": "14.38", "shop": "轻食兽京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "原味 | 0添加蔗糖 | 国产", "comment_fuzzy": "20万+", "total_sales": "已售40万+"}, {"sku_id": "100065504152", "title": "碧翠园全麦粗粮 低GI 饼干代餐0添蔗糖1000g健身抗饿健康零食品", "brand": "碧翠园", "list_price_show": "29.90", "coupon_or_detail_price": "", "detail_price_final": "29.38", "shop": "碧翠园京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "原味 | 0添加蔗糖 | 国产", "comment_fuzzy": "20万+", "total_sales": "已售50万+"}, {"sku_id": "100016592305", "title": "碧翠园 全麦粗粮 低GI 饼干500g代餐0蔗糖添加健身抗饿健康零食品", "brand": "碧翠园", "list_price_show": "15.90", "coupon_or_detail_price": "", "detail_price_final": "15.38", "shop": "碧翠园京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "原味 | 0添加蔗糖 | 国产", "comment_fuzzy": "20万+", "total_sales": "已售50万+"}, {"sku_id": "31258662003", "title": "DGI 低GI 饼干无糖精营养全麦 低 孕妇零食卡脂粗粮代餐饱腹糖友早餐食品 原味180g", "brand": "DGI", "list_price_show": "29.00", "coupon_or_detail_price": "", "detail_price_final": "27.97", "shop": "DGI官方旗舰店", "category": "休闲食品 > 饼干 > 酥性饼干", "selling_point": "普通盒装 | 国产 | 11个月", "comment_fuzzy": "20万+", "total_sales": "已售50万+"}, {"sku_id": "100303463330", "title": "考拉皮皮 低GI 饼干营养全麦饼干孕妇零食粗粮健身代餐饱腹糖友早餐食品192g", "brand": "考拉皮皮", "list_price_show": "24.90", "coupon_or_detail_price": "", "detail_price_final": "23.87", "shop": "考拉皮皮京东自营旗舰店", "category": "休闲食品 > 饼干 > 酥性饼干", "selling_point": "普通盒装 | 300天", "comment_fuzzy": "1000+", "total_sales": "已售4000+"}, {"sku_id": "100096590515", "title": "神探伍伍 低GI 粗杂粮全麦饼干512g0添加蔗糖尿病人专用代早餐健身饱腹零食", "brand": "神探伍伍", "list_price_show": "29.90", "coupon_or_detail_price": "", "detail_price_final": "28.87", "shop": "神探伍伍京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "全麦味 | 0添加蔗糖 | 国产", "comment_fuzzy": "5万+", "total_sales": "已售7万+"}, {"sku_id": "100148417432", "title": "红谷林无糖9黑藜麦谷物饼干208g粗粮 低GI 早餐饼干休闲零食独立小包装", "brand": "红谷林（HONGGULIN）", "list_price_show": "19.90", "coupon_or_detail_price": "", "detail_price_final": "18.87", "shop": "红谷林（HONGGULIN）食品京东自营旗舰店", "category": "休闲食品 > 饼干 > 酥性饼干", "selling_point": "袋装 | 国产 | 270天", "comment_fuzzy": "10万+", "total_sales": "已售20万+"}, {"sku_id": "100107873140", "title": "轻食兽粗粮全麦 低GI 饼干六口味1008g 0蔗糖添加健康早代餐零食控糖饱腹", "brand": "轻食兽", "list_price_show": "66.00", "coupon_or_detail_price": "", "detail_price_final": "64.97", "shop": "轻食兽京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "混合口味 | 0添加蔗糖 | 国产", "comment_fuzzy": "20万+", "total_sales": "已售40万+"}, {"sku_id": "100069972787", "title": "DGI 饼干原味180g 低GI 无添加蔗糖孕妇粗粮全麦高纤维饱腹代餐零食", "brand": "DGI", "list_price_show": "32.00", "coupon_or_detail_price": "", "detail_price_final": "30.97", "shop": "DGI京东自营旗舰店", "category": "休闲食品 > 饼干 > 酥性饼干", "selling_point": "箱装 | 国产 | 11个月", "comment_fuzzy": "10万+", "total_sales": "已售10万+"}, {"sku_id": "100148417498", "title": "红谷林无糖 低gi 9黑9红谷物饼干粗粮早餐饼干休闲零食营养独立小包装", "brand": "红谷林（HONGGULIN）", "list_price_show": "29.90", "coupon_or_detail_price": "", "detail_price_final": "28.87", "shop": "红谷林（HONGGULIN）食品京东自营旗舰店", "category": "休闲食品 > 饼干 > 酥性饼干", "selling_point": "袋装 | 国产 | 270天", "comment_fuzzy": "10万+", "total_sales": "已售20万+"}, {"sku_id": "100219612717", "title": "碧翠园全麦粗粮 低GI 饼干代餐0蔗糖添加健身抗饿健康零食品混合口味1000g", "brand": "碧翠园", "list_price_show": "42.90", "coupon_or_detail_price": "", "detail_price_final": "41.87", "shop": "碧翠园京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "混合口味 | 0添加蔗糖 | 国产", "comment_fuzzy": "20万+", "total_sales": "已售50万+"}, {"sku_id": "100160042861", "title": "红谷林无糖9黑 低GI 谷物饼干高纤粗粮饱腹早餐饼干独立小包装休闲零食", "brand": "红谷林（HONGGULIN）", "list_price_show": "59.90", "coupon_or_detail_price": "", "detail_price_final": "58.87", "shop": "红谷林（HONGGULIN）食品京东自营旗舰店", "category": "休闲食品 > 饼干 > 酥性饼干", "selling_point": "袋装 | 国产 | 270天", "comment_fuzzy": "10万+", "total_sales": "已售20万+"}, {"sku_id": "100065504166", "title": "轻食兽生可可燕麦 低GI 粗粮全麦饼干24g*7*1盒 早代餐休闲零食饱腹控糖", "brand": "轻食兽", "list_price_show": "15.90", "coupon_or_detail_price": "", "detail_price_final": "14.87", "shop": "轻食兽京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "燕麦味 | 高蛋白 | 国产", "comment_fuzzy": "20万+", "total_sales": "已售40万+"}, {"sku_id": "100065199811", "title": "碧翠园全麦粗粮 低GI 饼干代餐0蔗糖乳酸菌500g健身抗饿零食品", "brand": "碧翠园", "list_price_show": "16.90", "coupon_or_detail_price": "", "detail_price_final": "15.87", "shop": "碧翠园京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "乳酸菌味 | 0添加蔗糖 | 国产", "comment_fuzzy": "20万+", "total_sales": "已售50万+"}, {"sku_id": "100099361944", "title": "轻食兽葱香 低GI 高纤粗粮全麦饼干168g 早代餐休闲零食控糖营养饱腹健康", "brand": "轻食兽", "list_price_show": "16.90", "coupon_or_detail_price": "", "detail_price_final": "15.87", "shop": "轻食兽京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "香葱味 | 高膳食纤维 | 国产", "comment_fuzzy": "20万+", "total_sales": "已售40万+"}, {"sku_id": "100125407841", "title": "DGI 低GI 无糖精全麦饼干孕妇粗粮糖友饱腹代餐 低 升糖零食六黑全麦180g", "brand": "DGI", "list_price_show": "29.90", "coupon_or_detail_price": "", "detail_price_final": "28.87", "shop": "DGI京东自营旗舰店", "category": "休闲食品 > 饼干 > 酥性饼干", "selling_point": "普通盒装 | 国产 | 330天", "comment_fuzzy": "10万+", "total_sales": "已售10万+"}, {"sku_id": "100228538623", "title": "江中猴姑升级款0糖酥性饼干288g猴头菇食养饼干 低GI 营养早餐", "brand": "江中猴姑", "list_price_show": "36.90", "coupon_or_detail_price": "", "detail_price_final": "35.87", "shop": "1号会员店", "category": "休闲食品 > 饼干 > 酥性饼干", "selling_point": "普通盒装 | 15个月", "comment_fuzzy": "2000+", "total_sales": "已售4000+"}, {"sku_id": "10115464022326", "title": "DGI 低GI 饼干无糖精营养全麦 低 孕妇零食卡脂粗粮代餐饱腹糖友早餐食品 六黑全麦饼干180g", "brand": "DGI", "list_price_show": "29.00", "coupon_or_detail_price": "", "detail_price_final": "27.97", "shop": "DGI官方旗舰店", "category": "休闲食品 > 饼干 > 酥性饼干", "selling_point": "普通盒装 | 国产 | 330天", "comment_fuzzy": "20万+", "total_sales": "已售50万+"}, {"sku_id": "100032513400", "title": "DGI苏打饼干 低GI 饼干无糖精 低 孕妇零食粗粮代餐饱腹糖友葱香味200g", "brand": "DGI", "list_price_show": "49.90", "coupon_or_detail_price": "", "detail_price_final": "48.87", "shop": "DGI京东自营旗舰店", "category": "休闲食品 > 饼干 > 酥性饼干", "selling_point": "普通盒装 | 300天", "comment_fuzzy": "10万+", "total_sales": "已售10万+"}, {"sku_id": "100112766800", "title": "Fix XBody旺旺饼干零食小吃 低GI 低糖早餐粗粮饼干奇亚籽多谷物160g/盒", "brand": "Fix XBody", "list_price_show": "26.60", "coupon_or_detail_price": "", "detail_price_final": "25.57", "shop": "Fix XBody京东自营旗舰店", "category": "休闲食品 > 饼干 > 粗粮饼干", "selling_point": "海盐味 | 高膳食纤维 | 国产", "comment_fuzzy": "1万+", "total_sales": "已售2万+"}]}], "consumer_feedback_by_matrix_group": [{"group": "饼干", "matrix_group_index": 0, "chart_slug": "i00_饼干", "comment_rows": 340, "effective_comment_text_units": 340}], "notes": ["策略生成已按矩阵分组收窄：下文统计与矩阵仅针对「饼干」内 SKU；与全关键词搜索列表、全样本评价总量不同口径。", "与在线分析报告各章**计数规则**一致；关注词与场景以任务中的分析规则为准（子串命中统计，非深度主题模型）。", "价格来自页面展示字段抽取，含促销与规格差异；促销与标价对齐等为启发式摘录，仅供对照。", "评价语气为关键词粗判，非深度学习情感模型。", "「集中度」中：默认按**列表行**计数（同一 SKU 多页曝光会重复计）；`shops_from_list.unique_sku_basis` 为按**去重 SKU** 的对照口径。二者均**不是**销量、库存或全渠道市场份额。"], "strategy_scope_applied": {"matrix_group_index": 0, "group": "饼干", "original_matrix_group_count": 15}, "matrix_overview_for_llm": [{"group": "饼干", "sku_count": 21, "distinct_brands_sample": ["碧翠园", "轻食兽", "DGI", "考拉皮皮", "神探伍伍", "红谷林（HONGGULIN）", "江中猴姑", "Fix XBody"]}]}, "rules_draft_markdown": "# 市场策略制定草稿 · 「低GI」\n\n> **骨架说明**：本页为规则骨架；结构与 [`docs/demo`](docs/demo) 示例一致。**全局禁止编造**见 `generate_strategy.py` 中 `STRATEGY_DATA_RULES`。\n\n> **生成时间**：2026-04-21T01:50:13.500179+00:00 · **任务 ID**：12\n\n## 策略范围与前提（生成前先对齐）\n\n*回答：**这份策略是针对什么、在什么边界里做的**——属「立项靶心」，不是执行摘要。表单已填则写成短句；未填保留占位，**勿**编造。*\n*业务侧在动策略稿之前，应先对齐本节；可选目标类型见 `docs/planning/策略生成-框架确定.md`「启动前：须先明确的项」。*\n\n| 须明确项 | 填写或待确认 |\n|----------|----------------|\n| **监测任务（数据同源）** | 关键词「低GI」；批次 **20260413 104252**；与同任务《竞品分析报告》一致 |\n| **策略服务对象（本品角色）** | *（待填：新品 / 追赶 / 防守 / 拓品类 …）* |\n| **一句话战场** | *（在哪个需求场景、与谁抢同一批用户？）* |\n| **目标客群/场景** | *（可选）* |\n| **主推类目/细类** | *（待填：如饼干线 / 面包线 / 多线并行；未定写「待业务定类」）* |\n| **本阶段策略目标类型** | *（成稿从规划文档「目标类型选项」择一或组合，并与下列成功标准一致）* |\n| **时间范围** | *（如本季度 / 未来 12 周）* |\n| **成功标准（可量化）** | *（搜索位次、转化、复购等）* |\n\n---\n\n## 摘要\n\n- **范围与样本**：监测词「低GI」；批次 **20260413 104252**；深入 SKU ≈ 21；评价条数 ≈ 340。\n- **用户侧**：*（一两句结论即可：讨论焦点与负向主题；**按细类分句**归纳，**勿**混成「全站用户」一句；**勿**展开与报告重复的细类统计、词频。）*\n- **阶段重点**：*（须含 1～2 条**可执行动作**，回扣 §2 优先痛点；**尽量点明适用类目/主推线**；勿仅写「加强运营」。）*\n\n## 一、顾客是谁\n\n### 1.1 人群与决策路径\n\n- **检索与货架语境**：低GI；批次 20260413 104252。\n- **一句话战场**：*（在哪个需求场景、与谁抢同一批用户？）*\n- **典型路径**：*（成稿：搜索 → 列表比价 → 详情与配料 → 评价 → 下单/复购。）*\n\n*成稿须与 §2 一致：写清「谁在什么任务下检索、决策」及**主攻类目/细类**（与 §2.1 类目列可对上），为后文「针对痛点怎么做」埋伏笔。*\n\n### 1.2 细类讨论焦点（评论文本分析）\n\n*当前任务以**第八章评论侧文本挖掘**为主呈现时，此处**不**逐条罗列关注词子串命中次数。*\n\n- **饼干 / 糕点 / 面点等**：*（骨架占位；成稿**分细类**各一句归纳用户关心点，**勿**合并成模糊「全池」一句；**勿**复述 §8 词频与条数。）*\n\n### 类目结构（摘录）\n\n- 粗粮饼干：11\n- 酥性饼干：10\n\n### 1.3 本品聚焦（占位）\n\n*成稿写清本期**主攻人群/场景/类目或主推细类**与 §2.1「类目/细类」列的对应关系。*\n\n- **本品角色**：*（新品 / 追赶 / 防守 / 拓品类 …）*\n- **目标客群**：*（为谁、什么场景（可选））*\n- **主要对标**：*（品牌或价位带参照（可选））*\n\n## 二、产品价值与用户痛点\n\n*本节**仅**用下表写清**针对痛点要怎么做**（**类目** + 痛点简述 + 动作 + 落地 + 验证）。**不再**单设「痛点表 / 价值对表 / 负向归因」子节，避免与 §三、§八重复。*\n\n*「用户痛点（简述）」须与 `structured_brief` / 策略线索 / 报告节选**可核对**；**禁止**编造「用户反馈『……』」式引语，除非原句已出现在上述输入中。*\n\n*「类目/细类」列：写明本行决策**适用于哪一类**（如饼干/面包/全检索池）；多细类须**分行**，**禁止**用一句「全站」覆盖彼此冲突的策略；类目未定可写「待业务定类」并附分类假设。*\n\n### 2.1 针对痛点要怎么做\n\n| 类目/细类（本决策适用） | 用户痛点（简述） | 策略动作 | 具体怎么做（触点/话术/规格/渠道） | 如何验证 |\n|--------------------------|------------------|----------|-----------------------------------|----------|\n| *（如：饼干线 / 西式糕点 / 全池仅当可解释）* | *（口感/分量/价格信任等）* | *（动词句）* | *（可执行）* | *（指标或抽样）* |\n| | | | | |\n\n**摘要自动线索（`strategy_hints`）**\n\n- （与「关注词/预设场景条形图」相关的自动提示已省略；用户洞察请以报告 §8 文本挖掘与第九章节选为准。）\n\n## 三、为什么要买「这款产品」\n\n### 3.1 品类与时机\n\n- **列表申报规模（resultCount）**：333619（非销售额）\n\n- **价格摘录（可支撑购买理由与锚点）**：来源 strategy_scope_matrix_group_skus，n = 21；区间 14.38～64.97；中位数 27.97。\n\n*成稿在 §3.1 末尾用 1～2 句**承接** §2 优先痛点并写购买理由；**若**多细类并存，**分句**写清购买理由与主推；价带/规格类动作已在 §2 表内则此处**勿再展开一遍**。*\n\n## 四、为什么要选「这个品牌」\n\n### 4.1 品牌承诺与调性（占位）\n\n*成稿：承诺与调性须能落到**触点**（商详/包装/客服首句等）上的**具体句子**；**若**多类目话术不同，按 §2.1 类目**分句**，勿仅形容词。*\n\n- **一句话**：*（占位）*\n- **调性**：透明、可验证、合规控糖叙事（成稿可细化）。\n\n### 4.2 信任与证据\n\n- *（成稿：评价、配料、可核验表述边界。）*\n\n**主定位（与表单一致）**\n\n- [ ] **贴顶**：中高位或头部价位带。\n- [ ] **卡腰**：围绕中位数一带。\n- [ ] **下探**：贴近区间下限。\n- [ ] **另起带**：规格/组合/服务差异化。\n\n## 五、与其它品牌有何不同\n\n### 5.1 对比对象（摘录）\n\n- **列表侧店铺集中度**：第一大店铺约占列表行的 **23.8%**，前三大店铺合计约占 **57.1%**；当前头部为「碧翠园京东自营旗舰店」。*粗判：相对分散，差异化切入点可能更多。*\n- **深入样本内品牌集中度**：第一大品牌约占 **23.8%**，前三大品牌合计约占 **66.7%**；当前头部为「碧翠园」。*粗判：相对分散，差异化切入点可能更多。*\n\n- **环境自测**：头部强势时是侧翼还是正面替代？格局分散时是否用细分场景切入？\n\n### 5.2 差异化方向（占位）\n\n*成稿：相对竞品**多做什么/少做什么**，写**可执行的一步**；**若**差异因细类而异，**分类目**写（非空泛「更好」）。*\n\n| 差异点 | 说明 | 风险 |\n|--------|------|------|\n| | *待填* | |\n\n### 5.3 竞争应对\n\n*成稿：在表单倾向基础上，写清**跟价/不跟价时具体话术或机制**（一句即可）。*\n\n\n## 六、阶段目标与路径\n\n### 6.1 本阶段定义\n\n- **时间范围**：*（如：本季度 / 未来 12 周）*\n- **成功标准（可量化）**：*（搜索位次、转化、复购等）*\n- **非目标**：*（明确不做什么（可选））*\n\n### 6.2 路径\n\n*成稿：路径须与 **§2.1 针对痛点要怎么做** 可对齐；营销/总体策略为**动词句**，回扣痛点；**多类目并行**时**分线**写目标或写清主线/副线。*\n\n- **营销策略**：*（传播、活动、投放、内容主线（可选））*\n- **总体策略**：*（增长/品类/经营总原则（可选））*\n- **资源与预算备注**：*（人力、投放、产能等（可选））*\n\n## 七、品牌四线：建设 · 打造 · 运营 · 体验\n\n*（与表单「4P 策略支柱」对应：产品 / 定价 / 渠道 / 传播。）*\n*成稿：**每条线**至少一句——服务哪类痛点、本阶段**具体做哪一步**；**尽量**与 §2.1「类目/细类」可对上，多类目则**分句**（勿四条同一泛化句）。*\n\n### 7.1 品牌建设\n\n- *待填*\n\n### 7.2 品牌打造\n\n- *待填*\n\n### 7.3 品牌运营\n\n- *待填*\n\n### 7.4 品牌体验\n\n- *待填*\n\n\n*促销与价差：若摘要或价格信号有归纳则承接；无则勿编造。*\n\n## 八、战术支柱\n\n*成稿：四支柱分别回扣 **痛点→动作→落地**（可与 §2.1 呼应，避免纯重复）；**若**产品/定价/促销因类目策略不同，**分细类**写子条，勿一条盖全站。*\n\n### 8.1 产品策略\n\n- *（表单产品支柱：*待填*）*\n\n### 8.2 定价策略\n\n- *（表单价格支柱：*待填*）*\n\n### 8.3 促销与活动策略\n\n*须写促销**原则**（券/到手价/跟价节奏）；**满减、满折、跨店**等：能引用的写清来源；监测未捕获具体门槛时写「待与运营/后台对齐」，**勿**整节留空，**勿**编造门槛数字。*\n*与 `price_promotion_signals`、报告第六章一致；勿虚构活动。*\n\n### 8.4 渠道与传播\n\n- *（渠道/传播：*待填* / *待填*）*\n\n## 九、风险、假设与待验证\n\n- [ ] 评论侧归纳是否以偏概全？（需原评论抽样）\n- [ ] 价格带是否含大促/异常挂价？（需核对清洗规则）\n- [ ] 列表集中度与深入样本品牌是否不一致？（需解释渠道差异）\n\n*成稿：每条风险尽量带**应对动作或验证计划**（抽样、核对规则），勿只列标题。*\n\n*业务备注见下节。*\n\n## 十、下一步与节奏\n\n*成稿：下列为**可执行任务**（可补负责人/时间）；与 §2.1 / §六 优先级一致；可含「按类目核对主图/商详与 §2.1 表」类项。*\n\n- [ ] 锁定主推款与对标；过法务与合规。\n- [ ] 统一对外数据口径与话术。\n- [ ] 下轮监测更新后迭代策略。\n\n### 业务约束与备注\n\n*（未填写业务备注。）*\n\n---\n\n## 附录：本任务关键数据一览\n\n- **关键词**：低GI · **批次**：20260413 104252 · **摘要版本**：v1\n- **采集参数快照**：起始页=1; 采集至页=10; SKU 上限=100; 场景筛选=True\n- **列表申报规模（resultCount）**：333619\n\n*同目录含本批次 CSV 与分析产出，可对照使用。*\n\n---\n\n*本稿由工作台「市场策略制定」生成；与同任务结构化分析数据一致。*\n", "report_strategy_excerpt": "#### 定价与价带\n\n当前“低GI”品类价格带跨度较大，整体呈现中端至中高端的分布特征。从列表页数据看，价格区间为 8.90～1794.00 元，均值为 109.67 元，中位数为 52.90 元，标准差高达 152.69，表明价格离散度较高，市场未形成统一价格锚点。其中，饼干类价格集中在 14.38～66.00 元，中位数为 27.97 元；西式糕点价格带为 24.87～39.38 元，中位数为 33.87 元，整体定价更为集中；而米类价格则明显偏高，3 个 SKU 价格区间为 68.87～128.87 元，中位数达 68.97 元，反映出主食类低GI产品在健康定位下的溢价能力。\n\n值得注意的是，部分大规格产品存在单价溢价现象，如碧翠园 1000g 全麦饼干标价 29.90 元，而 500g 规格仅售 15.90～16.90 元，价差显著。此外，风味挂面类价格跨度大，从 25.87 元至 78.87 元不等，表明消费者对功能性挂面的接受度存在明显分层。整体来看，该品类尚未形成明确的“性价比带”或“功能溢价带”，存在通过规格、包装、品牌等维度进行差异化定价的空间。\n\n建议在后续产品布局中，可针对不同消费场景设计分层价格策略：一是围绕“日常代餐”场景打造 15～30 元的高性价比产品线，主打“独立包装+高饱腹感”；二是针对“健康主食”或“特定人群”（如糖尿病患者）推出 50～100 元的中高端产品，强调“低升糖认证+营养成分表”等专业背书；三是探索“组合装”或“家庭装”模式，通过克重与单价的优化提升客单价与复购率。\n\n#### 差异化与应对齐的优势\n\n从品牌与产品矩阵来看，当前“低GI”品类竞争格局尚未固化，品牌集中度较低，市场存在定位与叙事空间。在列表页中，第一大店铺为“DGI官方旗舰店”，占列表行份额 7.3%，但去重 SKU 基础下其份额为 8.6%，表明该品牌在 SKU 数量上具备一定优势。然而，从品牌分布看，碧翠园、轻食兽、神探伍伍、红谷林、豪士、盼盼等品牌均在不同子类中占据重要位置，尚未形成单一品牌主导格局。\n\n在差异化策略上，当前产品主要围绕“0添加蔗糖”“全麦”“高纤维”“低GI”等健康标签展开，但差异化深度有限。例如，饼干类普遍强调“奇亚籽”“燕麦”“黑藜麦”等配料，西式糕点则聚焦“全麦”“黑麦”“亚麻籽”等粗粮组合，但多数产品未形成明确的功能性或口感差异化。值得注意的是，部分新兴品牌如“慢教授”“神探伍伍”“轻食兽”等已尝试构建“健康+场景”叙事，如“慢糖DHA藻油黑芝麻饼”“低GI黑全麦面包”等，将营养成分与特定人群（如孕妇、糖友）绑定，形成初步的细分定位。\n\n建议后续可围绕“功能+场景”深化差异化：一是强化“特定人群”标签，如针对孕妇、老年人、健身人群推出专属配方（如添加DHA、钙、膳食纤维等）；二是构建“健康+口感”组合，如通过“酥脆口感+低GI”“松软质地+高饱腹”等卖点突破“健康即难吃”的刻板印象；三是探索“场景化组合装”，如“早餐三件套”（面包+麦片+挂面）或“代餐组合包”，提升用户粘性与客单价。\n\n#### 风险与避免项\n\n当前市场存在若干潜在风险点，需在产品与营销策略中规避。首先，用户对“口感”的反馈呈现两极分化：一方面，“口感酥脆”“味道不错”等正面评价频现；另一方面，也有用户反馈“口感偏硬”“噎人”“不够松软”，表明部分低GI产品在追求健康属性时牺牲了口感体验。例如，有用户评价“这款面包香气扑鼻，口感松软🍞。包装完好，没有任何损坏📦。送到家时依然新鲜，非常满意的一次购物体验！👍 推荐给所有喜欢面包的小伙伴们～”，但紧接着又补充“性价比一般”，说明用户在满意产品品质的同时，对价格敏感度较高。\n\n其次，用户对“分量”的反馈也存在不满，如“这款面包有点太小。再大一点更好了”“花卷很小，比自己家做的和外面卖的都小”，反映出部分产品在规格设计上未能满足用户对“饱腹感”和“性价比”的双重期待。此外，有用户指出“配料表里含有麦芽糖醇，一次不能多吃，否则容易造成肠胃不适”，提示功能性食品在成分设计上需更明确标注潜在影响，以避免用户误解或不适。\n\n建议在产品开发中加强口感与分量的平衡设计：一是优化配方工艺，提升低GI产品的松软度与咀嚼感；二是调整规格设计，推出“大包装”或“组合装”以增强性价比感知；三是完善成分说明，在详情页中明确标注功能性成分（如麦芽糖醇、菊粉等）的摄入建议，提升用户信任感。\n\n#### 促销与活动机制\n\n从促销形态看，当前“低GI”品类普遍采用“到手价”为核心展示形式，通过“购买立减+红包抵扣”组合实现价格下探，优惠金额集中在 1～1.5 元区间。多数商品标注“新人到手价”并提示“非新人价格可能不同”，强化新人转化策略。部分商品叠加满减权益（如满99享9折、满169享8.5折），形成阶梯式促销结构。\n\n值得注意的是，促销信息与榜单曝光高度并存，如“干菜类”所有SKU均标注“干菜热卖榜·第19名”，“风味挂面”多款产品进入“热卖榜”或“40元以上细分榜单”，表明平台通过“价格+热度”双重引导提升转化。此外，部分商品存在“不可使用东券”的购买限制提示，体现平台券系与自有优惠的差异化策略。\n\n建议在促销机制设计上，可结合不同子类特点进行差异化布局：一是针对“高频消费”品类（如饼干、挂面）推出“满减+返券”组合，提升用户复购率；二是针对“高客单价”品类（如米、面点）设计“新人专享价+购物返豆”激励，降低新客决策门槛；三是探索“场景化促销”，如“早餐组合满减”“代餐套装立减”等，提升客单价与用户粘性。\n\n#### 转化与体验\n\n从用户反馈看，当前“低GI”品类的整体体验以正面为主，尤其在口感、健康属性、包装便利性及物流服务方面获得较多肯定。多数用户表示产品“酥脆”“口感好”“味道香”，且“无糖”“低GI”“高纤维”等健康标签契合其控糖、减脂或代餐需求。例如，来自“碧翠园京东自营旗舰店”的用户多次提到“口感酥脆”“饱腹感强”“配料干净”“0蔗糖”等关键词，认为其是“控糖期的救星”“嘴馋的解馋零食”。\n\n在包装方面，“独立包装”“包装严实”“包装完好”等表述频繁出现，反映出用户对包装便利性与品质稳定的关注。物流方面，“京东物流配送又快又好”“师傅认真负责”“服务周到”等表述也较为常见，体现出用户对京东自营渠道的信任感。\n\n然而，部分用户对产品提出具体改进建议：一是“分量少”“太小”“不够吃”等表述在多条评价中出现，反映出部分用户对产品规格的不满；二是“口感偏硬”“噎人”“稍微有点噎人”等反馈表明，部分低GI产品在追求健康属性的同时，可能牺牲了口感柔软度；三是“配料表里含有麦芽糖醇，一次不能多吃，否则容易造成肠胃不适”等提醒，提示功能性食品在成分设计上需更明确标注潜在影响。\n\n建议在详情页与评价管理中加强用户体验优化：一是突出“口感”与“分量”的平衡设计，通过图文或视频展示产品质地与规格；二是完善成分说明，在详情页中明确标注功能性成分的摄入建议；三是优化评价管理，对“口感偏硬”“分量小”等负面反馈进行归因分析，指导产品迭代。", "report_matrix_group_evidence_md": "> **说明**：以下为同任务《竞品分析报告》正文中、细类「**饼干**」下 「#### …」小节的**大模型归纳**摘录（按正文出现顺序拼接），覆盖矩阵/价盘/促销/评论与场景等块中**已生成**的段落；若某块未开 LLM 或未产出对应小节，则不会出现在此摘录中。\n\n该细类下“低GI”饼干产品普遍强调“0添加蔗糖”或“无糖精”，主打全麦、粗粮、高纤维等健康概念，部分产品突出奇亚籽、燕麦、黑藜麦等配料以强化营养属性。卖点集中于“代餐”“饱腹”“控糖”“健身”等关键词，覆盖早餐、零食、孕妇及糖友等细分场景。品牌方面，碧翠园、轻食兽、DGI、红谷林等为高频出现品牌，多为国产，部分品牌如江中猴姑则结合食养概念（猴头菇）拓展差异化。配料摘录中可见奇亚籽、燕麦、黑藜麦、乳酸菌等成分，体现对膳食纤维与营养均衡的关注。价格方面，该细类价格带为 14.38～64.97 元，中位数为 27.97 元，表明市场覆盖从中端到中高端价位，消费者对健康代餐饼干的价格接受度较广。样本较多，归纳具有代表性。\n\n---\n\n该细类价盘跨度较大，标价从 14.38 元至 66.00 元不等，中位数为 27.97 元，均值 29.28 元，离散度较高（标准差 14.19），显示价格分布较分散。多数商品标价与详情价一致，券后价未展示，但部分商品如碧翠园 1000g、轻食兽 1008g 等大规格产品标价显著高于常规规格，如 500g 约 16 元，1000g 却达 29.90～42.90 元，价差明显，推测大包装存在单价溢价。部分商品券后价略低于标价，但整体价差幅度较小，未见大幅优惠。\n\n---\n\n该细类促销摘要覆盖率高，所有 SKU 均展示购买者侧促销信息。促销形态以「到手价」为核心，多数商品通过「购买立减」与「红包抵扣」实现价格下探，且「新人到手价」话术高频出现，强调新人专享优惠。多款商品叠加满减权益，如「满169享8.5折」「满99享9折」，部分商品还提供「最高返京豆」激励。榜单曝光普遍，热卖榜排名集中于前5名，促销与榜单并存，形成强关联。部分商品存在「不可使用东券」的购买限制提示，体现平台券系与自有优惠的差异化策略。整体促销话术密度高，信息拆解清晰，用户侧感知明确。\n\n---\n\n![词云（按词频权重）](report_assets/wordcloud_probe__00_饼干.png)\n\n本细类有效评论约 **340** 条（原始非空 **340** 条，经分词去停用后用于建模）。\n\n**词频 Top**：「饼干」(302)、「口感」(151)、「饱腹」(141)、「不错」(140)、「健康」(123)、「酥脆」(120)、「回购」(108)、「零食」(95)、「全麦」(82)、「方便」(73)、「纤维」(70)、「独立」(68)。\n\n**关键词突出度 Top**（相对区分度）：「饼干」(0.063)、「不错」(0.055)、「饱腹」(0.047)、「口感」(0.044)、「回购」(0.039)、「健康」(0.039)、「酥脆」(0.038)、「零食」(0.038)、「全麦」(0.034)、「好吃」(0.034)、「gi」(0.030)、「方便」(0.029)。\n\n**共现较强的词对**（同条评论内，供联想维度）：「口感」-「饼干」(98)；「酥脆」-「饼干」(94)；「健康」-「饼干」(88)；「回购」-「饼干」(81)；「饱腹」-「饼干」(76)；「口感」-「酥脆」(74)；「口感」-「饱腹」(71)；「全麦」-「饼干」(61)；「这款」-「饼干」(60)；「酥脆」-「饱腹」(60)。\n\n**自动归纳的主题（无监督，仅作探索）**：\n- 主题 1：饼干、饱腹、酥脆、口感、健康、回购、全麦、gi、零食、方便、独立、纤维\n- 主题 2：不错、口感、饼干、饱腹、起来、速度、早餐、干净、酥脆、香味、合适、好吃\n- 主题 3：饼干、回购、健康、零食、好吃、不错、这款、味道、全麦、gi、起来、酥脆\n- 主题 4：饼干、口感、无糖、饱腹、独立、方便、不错、全麦、酥脆、健康、合适、纤维\n\n> 以上为主题探索与统计摘要，**不等同**于业务结论；若与星级、规则词表冲突，以人工抽样为准。\n\n---\n\n---\n\n该细类评论主要围绕一款主打「低GI」与「健康代餐」功能的全麦粗粮饼干展开，用户普遍关注其口感、饱腹感以及配料的天然性。关键词突出度显示，“饱腹”“口感”“回购”等词高频出现，表明消费者在选择时更看重食用体验与持续购买意愿。共现分析中，“口感”与“饼干”“酥脆”频繁组合，“饱腹”与“健康”“粗粮”紧密关联，反映出用户将产品视为兼具美味与功能性的小食。\n\nLDA主题归纳揭示了多个潜在关注维度：第一主题聚焦于产品本身属性，如“饼干”“饱腹”“GI”“全麦”等，强调其作为代餐的健康价值；第二主题则偏向使用体验，如“早餐”“香味”“干净”等，暗示其作为日常加餐的便利性；第三主题突出“回购”“这款”“味道”等表达，体现用户对品牌与口味的认同；第四主题再次强调“无糖”“独立”“纤维”等成分与包装细节，说明消费者对配料表与食用方式的关注。需注意的是，LDA主题为无监督聚类结果，同一主题中可能出现多个维度交叉，且不与正式品类标签一一对应。\n\n从摘录文本可见，用户语气多为满意与推荐，如“第二次购买了”“回购”“值得”等表述频繁出现，体现出较强的复购意愿。部分评论中提到“血糖高”“嘴馋的救星”等语境，暗示该产品主要被控糖人群或需控制饮食的用户所青睐。例如有用户表示：“因为血糖高，又会常常肚子饿，这款饼干碳水化合物低，无蔗糖，对血糖有益，营业搭配饱腹感很强”，反映出其在特定健康需求下的使用场景。\n\n使用场景方面，评论中明确提及“早餐搭配牛奶”“每天吃”“加餐”等描述，表明该饼干常被用于早餐补充或日常饥饿时的替代零食。同时，“独立包装”“方便”等词的高频出现，也支持其作为便携式健康零食的定位。整体来看，该产品在满足口感需求的同时，通过“低GI”“无糖”“高纤维”等标签满足特定人群的健康诉求，形成“美味+健康”的双重价值认知。\n\n---\n\n**使用注意**：本分析基于开放词表与统计模型，词频与关键词突出度反映的是文本中词语的分布特征，而非人工标注的语义标签；LDA主题归纳为无监督聚类结果，主题间可能存在重叠，且不保证与实际产品功能或用户意图完全一致；所有结论仅依据本细类评论数据推断，不引入外部场景分类或正式报告中的标签体系。\n\n---", "chapter8_text_mining_probe": true, "structured_brief_omission_note": "已启用第八章文本挖掘（探针为主）：structured_brief 已省略「关注词/场景子串计数」、按细类 feedback 中的 focus_keyword_hits/scenarios_top、`strategy_hints`、`comment_sentiment_lexicon`（规则词表/条形图同源），避免与报告 §8 文本挖掘主口径冲突。**不得**再以这类子串计数或预设场景占比作为论据。用户与评论侧须依报告 §8 文本挖掘归纳及 `report_matrix_group_evidence_md`；**促销、满减、券价差**须与报告第六章、`price_promotion_signals` 及下方 `report_strategy_excerpt`（第九章）对齐，不得省略报告已写明的活动建议。"} +``` + +--- + +## 3. 同上 JSON 的排版版（便于人眼查看 `structured_brief` 结构） + +说明：若与第 2 节有任何不一致，以第 2 节（真实入参）为准。 + +```json +{ + "job_id": 12, + "keyword": "低GI", + "generated_at_iso": "2026-04-21T01:50:13.500179+00:00", + "strategy_decisions": { + "product_role": "", + "time_horizon": "", + "success_criteria": "", + "non_goals": "", + "battlefield_one_line": "", + "positioning_choice": "", + "competitive_stance": "", + "pillar_product": "", + "pillar_price": "", + "pillar_channel": "", + "pillar_comm": "", + "audience_segment": "", + "competitor_reference": "", + "resource_notes": "", + "marketing_strategy": "", + "general_strategy": "", + "ack_risk_keywords": false, + "ack_risk_price": false, + "ack_risk_concentration": false + }, + "business_notes": "", + "structured_brief": { + "schema_version": 1, + "keyword": "低GI", + "batch_label": "20260413 104252", + "run_dir": "(已省略)", + "scope": { + "merged_sku_count": 21, + "comment_flat_rows": 340, + "structure_source_rows": 589, + "uses_pc_search_list_export": true, + "category_mix_source": "keyword_pipeline_merged", + "category_mix_valid_matrix_sku_count": 70 + }, + "meta": { + "page_start": 1, + "page_to": 10, + "max_skus_config": 100, + "pc_search_export_rows": 589, + "merged_rows": 100, + "scenario_filter_enabled": true, + "merged_csv_mode": "lean" + }, + "pc_search_raw": { + "result_count_consensus": 333619, + "list_keyword": "低GI", + "result_count_uniques": [ + 332945, + 332972, + 333525, + 333548, + 333619 + ], + "raw_json_files_scanned": 20 + }, + "list_visibility_proxy": { + "total_rows": 21, + "unique_skus": 21, + "_strategy_scope_note": "矩阵所选分组内 SKU 数，非全关键词列表导出口径。" + }, + "concentration": { + "shops_from_list": { + "first_share": 0.23809523809523808, + "top_three_combined_share": 0.5714285714285714, + "top_label": "碧翠园京东自营旗舰店", + "top_share_pct": "23.8%" + }, + "list_brand_field": null, + "detail_brand_among_merged": { + "first_share": 0.23809523809523808, + "top_three_combined_share": 0.6666666666666666, + "top_label": "碧翠园", + "top_share_pct": "23.8%" + } + }, + "category_mix_top": [ + { + "label": "粗粮饼干", + "count": 11 + }, + { + "label": "酥性饼干", + "count": 10 + } + ], + "list_brand_mix_top": [ + { + "label": "碧翠园", + "count": 5 + }, + { + "label": "DGI", + "count": 5 + }, + { + "label": "轻食兽", + "count": 4 + }, + { + "label": "红谷林（HONGGULIN）", + "count": 3 + }, + { + "label": "考拉皮皮", + "count": 1 + }, + { + "label": "神探伍伍", + "count": 1 + }, + { + "label": "江中猴姑", + "count": 1 + }, + { + "label": "Fix XBody", + "count": 1 + } + ], + "list_shop_mix_top": [ + { + "label": "碧翠园京东自营旗舰店", + "count": 5 + }, + { + "label": "轻食兽京东自营旗舰店", + "count": 4 + }, + { + "label": "红谷林（HONGGULIN）食品京东自营旗舰店", + "count": 3 + }, + { + "label": "DGI京东自营旗舰店", + "count": 3 + }, + { + "label": "DGI官方旗舰店", + "count": 2 + }, + { + "label": "考拉皮皮京东自营旗舰店", + "count": 1 + }, + { + "label": "神探伍伍京东自营旗舰店", + "count": 1 + }, + { + "label": "1号会员店", + "count": 1 + }, + { + "label": "Fix XBody京东自营旗舰店", + "count": 1 + } + ], + "price_stats": { + "min": 14.38, + "max": 64.97, + "mean": 29.28238095238095, + "n": 21, + "stdev": 14.193378704438878, + "median": 27.97, + "q1": 16.384999999999998, + "q3": 33.42 + }, + "price_stats_source": "strategy_scope_matrix_group_skus", + "price_stats_merged_sample": { + "min": 14.38, + "max": 64.97, + "mean": 29.28238095238095, + "n": 21, + "stdev": 14.193378704438878, + "median": 27.97, + "q1": 16.384999999999998, + "q3": 33.42 + }, + "price_stats_list_export": {}, + "price_promotion_signals": [], + "matrix_by_group": [ + { + "group": "饼干", + "sku_count": 21, + "skus": [ + { + "sku_id": "100065199809", + "title": "碧翠园全麦粗粮 低GI 饼干代餐0添蔗糖500g奇亚籽健身抗饿零食品", + "brand": "碧翠园", + "list_price_show": "16.90", + "coupon_or_detail_price": "", + "detail_price_final": "16.90", + "shop": "碧翠园京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "奇亚籽味 | 0添加蔗糖 | 国产", + "comment_fuzzy": "20万+", + "total_sales": "已售50万+" + }, + { + "sku_id": "100049997473", + "title": "轻食兽牛油果椰乳 低GI 全麦粗粮饼干168g 代餐休闲零食早餐控糖健康饱腹", + "brand": "轻食兽", + "list_price_show": "14.90", + "coupon_or_detail_price": "", + "detail_price_final": "14.38", + "shop": "轻食兽京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "原味 | 0添加蔗糖 | 国产", + "comment_fuzzy": "20万+", + "total_sales": "已售40万+" + }, + { + "sku_id": "100065504152", + "title": "碧翠园全麦粗粮 低GI 饼干代餐0添蔗糖1000g健身抗饿健康零食品", + "brand": "碧翠园", + "list_price_show": "29.90", + "coupon_or_detail_price": "", + "detail_price_final": "29.38", + "shop": "碧翠园京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "原味 | 0添加蔗糖 | 国产", + "comment_fuzzy": "20万+", + "total_sales": "已售50万+" + }, + { + "sku_id": "100016592305", + "title": "碧翠园 全麦粗粮 低GI 饼干500g代餐0蔗糖添加健身抗饿健康零食品", + "brand": "碧翠园", + "list_price_show": "15.90", + "coupon_or_detail_price": "", + "detail_price_final": "15.38", + "shop": "碧翠园京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "原味 | 0添加蔗糖 | 国产", + "comment_fuzzy": "20万+", + "total_sales": "已售50万+" + }, + { + "sku_id": "31258662003", + "title": "DGI 低GI 饼干无糖精营养全麦 低 孕妇零食卡脂粗粮代餐饱腹糖友早餐食品 原味180g", + "brand": "DGI", + "list_price_show": "29.00", + "coupon_or_detail_price": "", + "detail_price_final": "27.97", + "shop": "DGI官方旗舰店", + "category": "休闲食品 > 饼干 > 酥性饼干", + "selling_point": "普通盒装 | 国产 | 11个月", + "comment_fuzzy": "20万+", + "total_sales": "已售50万+" + }, + { + "sku_id": "100303463330", + "title": "考拉皮皮 低GI 饼干营养全麦饼干孕妇零食粗粮健身代餐饱腹糖友早餐食品192g", + "brand": "考拉皮皮", + "list_price_show": "24.90", + "coupon_or_detail_price": "", + "detail_price_final": "23.87", + "shop": "考拉皮皮京东自营旗舰店", + "category": "休闲食品 > 饼干 > 酥性饼干", + "selling_point": "普通盒装 | 300天", + "comment_fuzzy": "1000+", + "total_sales": "已售4000+" + }, + { + "sku_id": "100096590515", + "title": "神探伍伍 低GI 粗杂粮全麦饼干512g0添加蔗糖尿病人专用代早餐健身饱腹零食", + "brand": "神探伍伍", + "list_price_show": "29.90", + "coupon_or_detail_price": "", + "detail_price_final": "28.87", + "shop": "神探伍伍京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "全麦味 | 0添加蔗糖 | 国产", + "comment_fuzzy": "5万+", + "total_sales": "已售7万+" + }, + { + "sku_id": "100148417432", + "title": "红谷林无糖9黑藜麦谷物饼干208g粗粮 低GI 早餐饼干休闲零食独立小包装", + "brand": "红谷林（HONGGULIN）", + "list_price_show": "19.90", + "coupon_or_detail_price": "", + "detail_price_final": "18.87", + "shop": "红谷林（HONGGULIN）食品京东自营旗舰店", + "category": "休闲食品 > 饼干 > 酥性饼干", + "selling_point": "袋装 | 国产 | 270天", + "comment_fuzzy": "10万+", + "total_sales": "已售20万+" + }, + { + "sku_id": "100107873140", + "title": "轻食兽粗粮全麦 低GI 饼干六口味1008g 0蔗糖添加健康早代餐零食控糖饱腹", + "brand": "轻食兽", + "list_price_show": "66.00", + "coupon_or_detail_price": "", + "detail_price_final": "64.97", + "shop": "轻食兽京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "混合口味 | 0添加蔗糖 | 国产", + "comment_fuzzy": "20万+", + "total_sales": "已售40万+" + }, + { + "sku_id": "100069972787", + "title": "DGI 饼干原味180g 低GI 无添加蔗糖孕妇粗粮全麦高纤维饱腹代餐零食", + "brand": "DGI", + "list_price_show": "32.00", + "coupon_or_detail_price": "", + "detail_price_final": "30.97", + "shop": "DGI京东自营旗舰店", + "category": "休闲食品 > 饼干 > 酥性饼干", + "selling_point": "箱装 | 国产 | 11个月", + "comment_fuzzy": "10万+", + "total_sales": "已售10万+" + }, + { + "sku_id": "100148417498", + "title": "红谷林无糖 低gi 9黑9红谷物饼干粗粮早餐饼干休闲零食营养独立小包装", + "brand": "红谷林（HONGGULIN）", + "list_price_show": "29.90", + "coupon_or_detail_price": "", + "detail_price_final": "28.87", + "shop": "红谷林（HONGGULIN）食品京东自营旗舰店", + "category": "休闲食品 > 饼干 > 酥性饼干", + "selling_point": "袋装 | 国产 | 270天", + "comment_fuzzy": "10万+", + "total_sales": "已售20万+" + }, + { + "sku_id": "100219612717", + "title": "碧翠园全麦粗粮 低GI 饼干代餐0蔗糖添加健身抗饿健康零食品混合口味1000g", + "brand": "碧翠园", + "list_price_show": "42.90", + "coupon_or_detail_price": "", + "detail_price_final": "41.87", + "shop": "碧翠园京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "混合口味 | 0添加蔗糖 | 国产", + "comment_fuzzy": "20万+", + "total_sales": "已售50万+" + }, + { + "sku_id": "100160042861", + "title": "红谷林无糖9黑 低GI 谷物饼干高纤粗粮饱腹早餐饼干独立小包装休闲零食", + "brand": "红谷林（HONGGULIN）", + "list_price_show": "59.90", + "coupon_or_detail_price": "", + "detail_price_final": "58.87", + "shop": "红谷林（HONGGULIN）食品京东自营旗舰店", + "category": "休闲食品 > 饼干 > 酥性饼干", + "selling_point": "袋装 | 国产 | 270天", + "comment_fuzzy": "10万+", + "total_sales": "已售20万+" + }, + { + "sku_id": "100065504166", + "title": "轻食兽生可可燕麦 低GI 粗粮全麦饼干24g*7*1盒 早代餐休闲零食饱腹控糖", + "brand": "轻食兽", + "list_price_show": "15.90", + "coupon_or_detail_price": "", + "detail_price_final": "14.87", + "shop": "轻食兽京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "燕麦味 | 高蛋白 | 国产", + "comment_fuzzy": "20万+", + "total_sales": "已售40万+" + }, + { + "sku_id": "100065199811", + "title": "碧翠园全麦粗粮 低GI 饼干代餐0蔗糖乳酸菌500g健身抗饿零食品", + "brand": "碧翠园", + "list_price_show": "16.90", + "coupon_or_detail_price": "", + "detail_price_final": "15.87", + "shop": "碧翠园京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "乳酸菌味 | 0添加蔗糖 | 国产", + "comment_fuzzy": "20万+", + "total_sales": "已售50万+" + }, + { + "sku_id": "100099361944", + "title": "轻食兽葱香 低GI 高纤粗粮全麦饼干168g 早代餐休闲零食控糖营养饱腹健康", + "brand": "轻食兽", + "list_price_show": "16.90", + "coupon_or_detail_price": "", + "detail_price_final": "15.87", + "shop": "轻食兽京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "香葱味 | 高膳食纤维 | 国产", + "comment_fuzzy": "20万+", + "total_sales": "已售40万+" + }, + { + "sku_id": "100125407841", + "title": "DGI 低GI 无糖精全麦饼干孕妇粗粮糖友饱腹代餐 低 升糖零食六黑全麦180g", + "brand": "DGI", + "list_price_show": "29.90", + "coupon_or_detail_price": "", + "detail_price_final": "28.87", + "shop": "DGI京东自营旗舰店", + "category": "休闲食品 > 饼干 > 酥性饼干", + "selling_point": "普通盒装 | 国产 | 330天", + "comment_fuzzy": "10万+", + "total_sales": "已售10万+" + }, + { + "sku_id": "100228538623", + "title": "江中猴姑升级款0糖酥性饼干288g猴头菇食养饼干 低GI 营养早餐", + "brand": "江中猴姑", + "list_price_show": "36.90", + "coupon_or_detail_price": "", + "detail_price_final": "35.87", + "shop": "1号会员店", + "category": "休闲食品 > 饼干 > 酥性饼干", + "selling_point": "普通盒装 | 15个月", + "comment_fuzzy": "2000+", + "total_sales": "已售4000+" + }, + { + "sku_id": "10115464022326", + "title": "DGI 低GI 饼干无糖精营养全麦 低 孕妇零食卡脂粗粮代餐饱腹糖友早餐食品 六黑全麦饼干180g", + "brand": "DGI", + "list_price_show": "29.00", + "coupon_or_detail_price": "", + "detail_price_final": "27.97", + "shop": "DGI官方旗舰店", + "category": "休闲食品 > 饼干 > 酥性饼干", + "selling_point": "普通盒装 | 国产 | 330天", + "comment_fuzzy": "20万+", + "total_sales": "已售50万+" + }, + { + "sku_id": "100032513400", + "title": "DGI苏打饼干 低GI 饼干无糖精 低 孕妇零食粗粮代餐饱腹糖友葱香味200g", + "brand": "DGI", + "list_price_show": "49.90", + "coupon_or_detail_price": "", + "detail_price_final": "48.87", + "shop": "DGI京东自营旗舰店", + "category": "休闲食品 > 饼干 > 酥性饼干", + "selling_point": "普通盒装 | 300天", + "comment_fuzzy": "10万+", + "total_sales": "已售10万+" + }, + { + "sku_id": "100112766800", + "title": "Fix XBody旺旺饼干零食小吃 低GI 低糖早餐粗粮饼干奇亚籽多谷物160g/盒", + "brand": "Fix XBody", + "list_price_show": "26.60", + "coupon_or_detail_price": "", + "detail_price_final": "25.57", + "shop": "Fix XBody京东自营旗舰店", + "category": "休闲食品 > 饼干 > 粗粮饼干", + "selling_point": "海盐味 | 高膳食纤维 | 国产", + "comment_fuzzy": "1万+", + "total_sales": "已售2万+" + } + ] + } + ], + "consumer_feedback_by_matrix_group": [ + { + "group": "饼干", + "matrix_group_index": 0, + "chart_slug": "i00_饼干", + "comment_rows": 340, + "effective_comment_text_units": 340 + } + ], + "notes": [ + "策略生成已按矩阵分组收窄：下文统计与矩阵仅针对「饼干」内 SKU；与全关键词搜索列表、全样本评价总量不同口径。", + "与在线分析报告各章**计数规则**一致；关注词与场景以任务中的分析规则为准（子串命中统计，非深度主题模型）。", + "价格来自页面展示字段抽取，含促销与规格差异；促销与标价对齐等为启发式摘录，仅供对照。", + "评价语气为关键词粗判，非深度学习情感模型。", + "「集中度」中：默认按**列表行**计数（同一 SKU 多页曝光会重复计）；`shops_from_list.unique_sku_basis` 为按**去重 SKU** 的对照口径。二者均**不是**销量、库存或全渠道市场份额。" + ], + "strategy_scope_applied": { + "matrix_group_index": 0, + "group": "饼干", + "original_matrix_group_count": 15 + }, + "matrix_overview_for_llm": [ + { + "group": "饼干", + "sku_count": 21, + "distinct_brands_sample": [ + "碧翠园", + "轻食兽", + "DGI", + "考拉皮皮", + "神探伍伍", + "红谷林（HONGGULIN）", + "江中猴姑", + "Fix XBody" + ] + } + ] + }, + "rules_draft_markdown": "# 市场策略制定草稿 · 「低GI」\n\n> **骨架说明**：本页为规则骨架；结构与 [`docs/demo`](docs/demo) 示例一致。**全局禁止编造**见 `generate_strategy.py` 中 `STRATEGY_DATA_RULES`。\n\n> **生成时间**：2026-04-21T01:50:13.500179+00:00 · **任务 ID**：12\n\n## 策略范围与前提（生成前先对齐）\n\n*回答：**这份策略是针对什么、在什么边界里做的**——属「立项靶心」，不是执行摘要。表单已填则写成短句；未填保留占位，**勿**编造。*\n*业务侧在动策略稿之前，应先对齐本节；可选目标类型见 `docs/planning/策略生成-框架确定.md`「启动前：须先明确的项」。*\n\n| 须明确项 | 填写或待确认 |\n|----------|----------------|\n| **监测任务（数据同源）** | 关键词「低GI」；批次 **20260413 104252**；与同任务《竞品分析报告》一致 |\n| **策略服务对象（本品角色）** | *（待填：新品 / 追赶 / 防守 / 拓品类 …）* |\n| **一句话战场** | *（在哪个需求场景、与谁抢同一批用户？）* |\n| **目标客群/场景** | *（可选）* |\n| **主推类目/细类** | *（待填：如饼干线 / 面包线 / 多线并行；未定写「待业务定类」）* |\n| **本阶段策略目标类型** | *（成稿从规划文档「目标类型选项」择一或组合，并与下列成功标准一致）* |\n| **时间范围** | *（如本季度 / 未来 12 周）* |\n| **成功标准（可量化）** | *（搜索位次、转化、复购等）* |\n\n---\n\n## 摘要\n\n- **范围与样本**：监测词「低GI」；批次 **20260413 104252**；深入 SKU ≈ 21；评价条数 ≈ 340。\n- **用户侧**：*（一两句结论即可：讨论焦点与负向主题；**按细类分句**归纳，**勿**混成「全站用户」一句；**勿**展开与报告重复的细类统计、词频。）*\n- **阶段重点**：*（须含 1～2 条**可执行动作**，回扣 §2 优先痛点；**尽量点明适用类目/主推线**；勿仅写「加强运营」。）*\n\n## 一、顾客是谁\n\n### 1.1 人群与决策路径\n\n- **检索与货架语境**：低GI；批次 20260413 104252。\n- **一句话战场**：*（在哪个需求场景、与谁抢同一批用户？）*\n- **典型路径**：*（成稿：搜索 → 列表比价 → 详情与配料 → 评价 → 下单/复购。）*\n\n*成稿须与 §2 一致：写清「谁在什么任务下检索、决策」及**主攻类目/细类**（与 §2.1 类目列可对上），为后文「针对痛点怎么做」埋伏笔。*\n\n### 1.2 细类讨论焦点（评论文本分析）\n\n*当前任务以**第八章评论侧文本挖掘**为主呈现时，此处**不**逐条罗列关注词子串命中次数。*\n\n- **饼干 / 糕点 / 面点等**：*（骨架占位；成稿**分细类**各一句归纳用户关心点，**勿**合并成模糊「全池」一句；**勿**复述 §8 词频与条数。）*\n\n### 类目结构（摘录）\n\n- 粗粮饼干：11\n- 酥性饼干：10\n\n### 1.3 本品聚焦（占位）\n\n*成稿写清本期**主攻人群/场景/类目或主推细类**与 §2.1「类目/细类」列的对应关系。*\n\n- **本品角色**：*（新品 / 追赶 / 防守 / 拓品类 …）*\n- **目标客群**：*（为谁、什么场景（可选））*\n- **主要对标**：*（品牌或价位带参照（可选））*\n\n## 二、产品价值与用户痛点\n\n*本节**仅**用下表写清**针对痛点要怎么做**（**类目** + 痛点简述 + 动作 + 落地 + 验证）。**不再**单设「痛点表 / 价值对表 / 负向归因」子节，避免与 §三、§八重复。*\n\n*「用户痛点（简述）」须与 `structured_brief` / 策略线索 / 报告节选**可核对**；**禁止**编造「用户反馈『……』」式引语，除非原句已出现在上述输入中。*\n\n*「类目/细类」列：写明本行决策**适用于哪一类**（如饼干/面包/全检索池）；多细类须**分行**，**禁止**用一句「全站」覆盖彼此冲突的策略；类目未定可写「待业务定类」并附分类假设。*\n\n### 2.1 针对痛点要怎么做\n\n| 类目/细类（本决策适用） | 用户痛点（简述） | 策略动作 | 具体怎么做（触点/话术/规格/渠道） | 如何验证 |\n|--------------------------|------------------|----------|-----------------------------------|----------|\n| *（如：饼干线 / 西式糕点 / 全池仅当可解释）* | *（口感/分量/价格信任等）* | *（动词句）* | *（可执行）* | *（指标或抽样）* |\n| | | | | |\n\n**摘要自动线索（`strategy_hints`）**\n\n- （与「关注词/预设场景条形图」相关的自动提示已省略；用户洞察请以报告 §8 文本挖掘与第九章节选为准。）\n\n## 三、为什么要买「这款产品」\n\n### 3.1 品类与时机\n\n- **列表申报规模（resultCount）**：333619（非销售额）\n\n- **价格摘录（可支撑购买理由与锚点）**：来源 strategy_scope_matrix_group_skus，n = 21；区间 14.38～64.97；中位数 27.97。\n\n*成稿在 §3.1 末尾用 1～2 句**承接** §2 优先痛点并写购买理由；**若**多细类并存，**分句**写清购买理由与主推；价带/规格类动作已在 §2 表内则此处**勿再展开一遍**。*\n\n## 四、为什么要选「这个品牌」\n\n### 4.1 品牌承诺与调性（占位）\n\n*成稿：承诺与调性须能落到**触点**（商详/包装/客服首句等）上的**具体句子**；**若**多类目话术不同，按 §2.1 类目**分句**，勿仅形容词。*\n\n- **一句话**：*（占位）*\n- **调性**：透明、可验证、合规控糖叙事（成稿可细化）。\n\n### 4.2 信任与证据\n\n- *（成稿：评价、配料、可核验表述边界。）*\n\n**主定位（与表单一致）**\n\n- [ ] **贴顶**：中高位或头部价位带。\n- [ ] **卡腰**：围绕中位数一带。\n- [ ] **下探**：贴近区间下限。\n- [ ] **另起带**：规格/组合/服务差异化。\n\n## 五、与其它品牌有何不同\n\n### 5.1 对比对象（摘录）\n\n- **列表侧店铺集中度**：第一大店铺约占列表行的 **23.8%**，前三大店铺合计约占 **57.1%**；当前头部为「碧翠园京东自营旗舰店」。*粗判：相对分散，差异化切入点可能更多。*\n- **深入样本内品牌集中度**：第一大品牌约占 **23.8%**，前三大品牌合计约占 **66.7%**；当前头部为「碧翠园」。*粗判：相对分散，差异化切入点可能更多。*\n\n- **环境自测**：头部强势时是侧翼还是正面替代？格局分散时是否用细分场景切入？\n\n### 5.2 差异化方向（占位）\n\n*成稿：相对竞品**多做什么/少做什么**，写**可执行的一步**；**若**差异因细类而异，**分类目**写（非空泛「更好」）。*\n\n| 差异点 | 说明 | 风险 |\n|--------|------|------|\n| | *待填* | |\n\n### 5.3 竞争应对\n\n*成稿：在表单倾向基础上，写清**跟价/不跟价时具体话术或机制**（一句即可）。*\n\n\n## 六、阶段目标与路径\n\n### 6.1 本阶段定义\n\n- **时间范围**：*（如：本季度 / 未来 12 周）*\n- **成功标准（可量化）**：*（搜索位次、转化、复购等）*\n- **非目标**：*（明确不做什么（可选））*\n\n### 6.2 路径\n\n*成稿：路径须与 **§2.1 针对痛点要怎么做** 可对齐；营销/总体策略为**动词句**，回扣痛点；**多类目并行**时**分线**写目标或写清主线/副线。*\n\n- **营销策略**：*（传播、活动、投放、内容主线（可选））*\n- **总体策略**：*（增长/品类/经营总原则（可选））*\n- **资源与预算备注**：*（人力、投放、产能等（可选））*\n\n## 七、品牌四线：建设 · 打造 · 运营 · 体验\n\n*（与表单「4P 策略支柱」对应：产品 / 定价 / 渠道 / 传播。）*\n*成稿：**每条线**至少一句——服务哪类痛点、本阶段**具体做哪一步**；**尽量**与 §2.1「类目/细类」可对上，多类目则**分句**（勿四条同一泛化句）。*\n\n### 7.1 品牌建设\n\n- *待填*\n\n### 7.2 品牌打造\n\n- *待填*\n\n### 7.3 品牌运营\n\n- *待填*\n\n### 7.4 品牌体验\n\n- *待填*\n\n\n*促销与价差：若摘要或价格信号有归纳则承接；无则勿编造。*\n\n## 八、战术支柱\n\n*成稿：四支柱分别回扣 **痛点→动作→落地**（可与 §2.1 呼应，避免纯重复）；**若**产品/定价/促销因类目策略不同，**分细类**写子条，勿一条盖全站。*\n\n### 8.1 产品策略\n\n- *（表单产品支柱：*待填*）*\n\n### 8.2 定价策略\n\n- *（表单价格支柱：*待填*）*\n\n### 8.3 促销与活动策略\n\n*须写促销**原则**（券/到手价/跟价节奏）；**满减、满折、跨店**等：能引用的写清来源；监测未捕获具体门槛时写「待与运营/后台对齐」，**勿**整节留空，**勿**编造门槛数字。*\n*与 `price_promotion_signals`、报告第六章一致；勿虚构活动。*\n\n### 8.4 渠道与传播\n\n- *（渠道/传播：*待填* / *待填*）*\n\n## 九、风险、假设与待验证\n\n- [ ] 评论侧归纳是否以偏概全？（需原评论抽样）\n- [ ] 价格带是否含大促/异常挂价？（需核对清洗规则）\n- [ ] 列表集中度与深入样本品牌是否不一致？（需解释渠道差异）\n\n*成稿：每条风险尽量带**应对动作或验证计划**（抽样、核对规则），勿只列标题。*\n\n*业务备注见下节。*\n\n## 十、下一步与节奏\n\n*成稿：下列为**可执行任务**（可补负责人/时间）；与 §2.1 / §六 优先级一致；可含「按类目核对主图/商详与 §2.1 表」类项。*\n\n- [ ] 锁定主推款与对标；过法务与合规。\n- [ ] 统一对外数据口径与话术。\n- [ ] 下轮监测更新后迭代策略。\n\n### 业务约束与备注\n\n*（未填写业务备注。）*\n\n---\n\n## 附录：本任务关键数据一览\n\n- **关键词**：低GI · **批次**：20260413 104252 · **摘要版本**：v1\n- **采集参数快照**：起始页=1; 采集至页=10; SKU 上限=100; 场景筛选=True\n- **列表申报规模（resultCount）**：333619\n\n*同目录含本批次 CSV 与分析产出，可对照使用。*\n\n---\n\n*本稿由工作台「市场策略制定」生成；与同任务结构化分析数据一致。*\n", + "report_strategy_excerpt": "#### 定价与价带\n\n当前“低GI”品类价格带跨度较大，整体呈现中端至中高端的分布特征。从列表页数据看，价格区间为 8.90～1794.00 元，均值为 109.67 元，中位数为 52.90 元，标准差高达 152.69，表明价格离散度较高，市场未形成统一价格锚点。其中，饼干类价格集中在 14.38～66.00 元，中位数为 27.97 元；西式糕点价格带为 24.87～39.38 元，中位数为 33.87 元，整体定价更为集中；而米类价格则明显偏高，3 个 SKU 价格区间为 68.87～128.87 元，中位数达 68.97 元，反映出主食类低GI产品在健康定位下的溢价能力。\n\n值得注意的是，部分大规格产品存在单价溢价现象，如碧翠园 1000g 全麦饼干标价 29.90 元，而 500g 规格仅售 15.90～16.90 元，价差显著。此外，风味挂面类价格跨度大，从 25.87 元至 78.87 元不等，表明消费者对功能性挂面的接受度存在明显分层。整体来看，该品类尚未形成明确的“性价比带”或“功能溢价带”，存在通过规格、包装、品牌等维度进行差异化定价的空间。\n\n建议在后续产品布局中，可针对不同消费场景设计分层价格策略：一是围绕“日常代餐”场景打造 15～30 元的高性价比产品线，主打“独立包装+高饱腹感”；二是针对“健康主食”或“特定人群”（如糖尿病患者）推出 50～100 元的中高端产品，强调“低升糖认证+营养成分表”等专业背书；三是探索“组合装”或“家庭装”模式，通过克重与单价的优化提升客单价与复购率。\n\n#### 差异化与应对齐的优势\n\n从品牌与产品矩阵来看，当前“低GI”品类竞争格局尚未固化，品牌集中度较低，市场存在定位与叙事空间。在列表页中，第一大店铺为“DGI官方旗舰店”，占列表行份额 7.3%，但去重 SKU 基础下其份额为 8.6%，表明该品牌在 SKU 数量上具备一定优势。然而，从品牌分布看，碧翠园、轻食兽、神探伍伍、红谷林、豪士、盼盼等品牌均在不同子类中占据重要位置，尚未形成单一品牌主导格局。\n\n在差异化策略上，当前产品主要围绕“0添加蔗糖”“全麦”“高纤维”“低GI”等健康标签展开，但差异化深度有限。例如，饼干类普遍强调“奇亚籽”“燕麦”“黑藜麦”等配料，西式糕点则聚焦“全麦”“黑麦”“亚麻籽”等粗粮组合，但多数产品未形成明确的功能性或口感差异化。值得注意的是，部分新兴品牌如“慢教授”“神探伍伍”“轻食兽”等已尝试构建“健康+场景”叙事，如“慢糖DHA藻油黑芝麻饼”“低GI黑全麦面包”等，将营养成分与特定人群（如孕妇、糖友）绑定，形成初步的细分定位。\n\n建议后续可围绕“功能+场景”深化差异化：一是强化“特定人群”标签，如针对孕妇、老年人、健身人群推出专属配方（如添加DHA、钙、膳食纤维等）；二是构建“健康+口感”组合，如通过“酥脆口感+低GI”“松软质地+高饱腹”等卖点突破“健康即难吃”的刻板印象；三是探索“场景化组合装”，如“早餐三件套”（面包+麦片+挂面）或“代餐组合包”，提升用户粘性与客单价。\n\n#### 风险与避免项\n\n当前市场存在若干潜在风险点，需在产品与营销策略中规避。首先，用户对“口感”的反馈呈现两极分化：一方面，“口感酥脆”“味道不错”等正面评价频现；另一方面，也有用户反馈“口感偏硬”“噎人”“不够松软”，表明部分低GI产品在追求健康属性时牺牲了口感体验。例如，有用户评价“这款面包香气扑鼻，口感松软🍞。包装完好，没有任何损坏📦。送到家时依然新鲜，非常满意的一次购物体验！👍 推荐给所有喜欢面包的小伙伴们～”，但紧接着又补充“性价比一般”，说明用户在满意产品品质的同时，对价格敏感度较高。\n\n其次，用户对“分量”的反馈也存在不满，如“这款面包有点太小。再大一点更好了”“花卷很小，比自己家做的和外面卖的都小”，反映出部分产品在规格设计上未能满足用户对“饱腹感”和“性价比”的双重期待。此外，有用户指出“配料表里含有麦芽糖醇，一次不能多吃，否则容易造成肠胃不适”，提示功能性食品在成分设计上需更明确标注潜在影响，以避免用户误解或不适。\n\n建议在产品开发中加强口感与分量的平衡设计：一是优化配方工艺，提升低GI产品的松软度与咀嚼感；二是调整规格设计，推出“大包装”或“组合装”以增强性价比感知；三是完善成分说明，在详情页中明确标注功能性成分（如麦芽糖醇、菊粉等）的摄入建议，提升用户信任感。\n\n#### 促销与活动机制\n\n从促销形态看，当前“低GI”品类普遍采用“到手价”为核心展示形式，通过“购买立减+红包抵扣”组合实现价格下探，优惠金额集中在 1～1.5 元区间。多数商品标注“新人到手价”并提示“非新人价格可能不同”，强化新人转化策略。部分商品叠加满减权益（如满99享9折、满169享8.5折），形成阶梯式促销结构。\n\n值得注意的是，促销信息与榜单曝光高度并存，如“干菜类”所有SKU均标注“干菜热卖榜·第19名”，“风味挂面”多款产品进入“热卖榜”或“40元以上细分榜单”，表明平台通过“价格+热度”双重引导提升转化。此外，部分商品存在“不可使用东券”的购买限制提示，体现平台券系与自有优惠的差异化策略。\n\n建议在促销机制设计上，可结合不同子类特点进行差异化布局：一是针对“高频消费”品类（如饼干、挂面）推出“满减+返券”组合，提升用户复购率；二是针对“高客单价”品类（如米、面点）设计“新人专享价+购物返豆”激励，降低新客决策门槛；三是探索“场景化促销”，如“早餐组合满减”“代餐套装立减”等，提升客单价与用户粘性。\n\n#### 转化与体验\n\n从用户反馈看，当前“低GI”品类的整体体验以正面为主，尤其在口感、健康属性、包装便利性及物流服务方面获得较多肯定。多数用户表示产品“酥脆”“口感好”“味道香”，且“无糖”“低GI”“高纤维”等健康标签契合其控糖、减脂或代餐需求。例如，来自“碧翠园京东自营旗舰店”的用户多次提到“口感酥脆”“饱腹感强”“配料干净”“0蔗糖”等关键词，认为其是“控糖期的救星”“嘴馋的解馋零食”。\n\n在包装方面，“独立包装”“包装严实”“包装完好”等表述频繁出现，反映出用户对包装便利性与品质稳定的关注。物流方面，“京东物流配送又快又好”“师傅认真负责”“服务周到”等表述也较为常见，体现出用户对京东自营渠道的信任感。\n\n然而，部分用户对产品提出具体改进建议：一是“分量少”“太小”“不够吃”等表述在多条评价中出现，反映出部分用户对产品规格的不满；二是“口感偏硬”“噎人”“稍微有点噎人”等反馈表明，部分低GI产品在追求健康属性的同时，可能牺牲了口感柔软度；三是“配料表里含有麦芽糖醇，一次不能多吃，否则容易造成肠胃不适”等提醒，提示功能性食品在成分设计上需更明确标注潜在影响。\n\n建议在详情页与评价管理中加强用户体验优化：一是突出“口感”与“分量”的平衡设计，通过图文或视频展示产品质地与规格；二是完善成分说明，在详情页中明确标注功能性成分的摄入建议；三是优化评价管理，对“口感偏硬”“分量小”等负面反馈进行归因分析，指导产品迭代。", + "report_matrix_group_evidence_md": "> **说明**：以下为同任务《竞品分析报告》正文中、细类「**饼干**」下 「#### …」小节的**大模型归纳**摘录（按正文出现顺序拼接），覆盖矩阵/价盘/促销/评论与场景等块中**已生成**的段落；若某块未开 LLM 或未产出对应小节，则不会出现在此摘录中。\n\n该细类下“低GI”饼干产品普遍强调“0添加蔗糖”或“无糖精”，主打全麦、粗粮、高纤维等健康概念，部分产品突出奇亚籽、燕麦、黑藜麦等配料以强化营养属性。卖点集中于“代餐”“饱腹”“控糖”“健身”等关键词，覆盖早餐、零食、孕妇及糖友等细分场景。品牌方面，碧翠园、轻食兽、DGI、红谷林等为高频出现品牌，多为国产，部分品牌如江中猴姑则结合食养概念（猴头菇）拓展差异化。配料摘录中可见奇亚籽、燕麦、黑藜麦、乳酸菌等成分，体现对膳食纤维与营养均衡的关注。价格方面，该细类价格带为 14.38～64.97 元，中位数为 27.97 元，表明市场覆盖从中端到中高端价位，消费者对健康代餐饼干的价格接受度较广。样本较多，归纳具有代表性。\n\n---\n\n该细类价盘跨度较大，标价从 14.38 元至 66.00 元不等，中位数为 27.97 元，均值 29.28 元，离散度较高（标准差 14.19），显示价格分布较分散。多数商品标价与详情价一致，券后价未展示，但部分商品如碧翠园 1000g、轻食兽 1008g 等大规格产品标价显著高于常规规格，如 500g 约 16 元，1000g 却达 29.90～42.90 元，价差明显，推测大包装存在单价溢价。部分商品券后价略低于标价，但整体价差幅度较小，未见大幅优惠。\n\n---\n\n该细类促销摘要覆盖率高，所有 SKU 均展示购买者侧促销信息。促销形态以「到手价」为核心，多数商品通过「购买立减」与「红包抵扣」实现价格下探，且「新人到手价」话术高频出现，强调新人专享优惠。多款商品叠加满减权益，如「满169享8.5折」「满99享9折」，部分商品还提供「最高返京豆」激励。榜单曝光普遍，热卖榜排名集中于前5名，促销与榜单并存，形成强关联。部分商品存在「不可使用东券」的购买限制提示，体现平台券系与自有优惠的差异化策略。整体促销话术密度高，信息拆解清晰，用户侧感知明确。\n\n---\n\n![词云（按词频权重）](report_assets/wordcloud_probe__00_饼干.png)\n\n本细类有效评论约 **340** 条（原始非空 **340** 条，经分词去停用后用于建模）。\n\n**词频 Top**：「饼干」(302)、「口感」(151)、「饱腹」(141)、「不错」(140)、「健康」(123)、「酥脆」(120)、「回购」(108)、「零食」(95)、「全麦」(82)、「方便」(73)、「纤维」(70)、「独立」(68)。\n\n**关键词突出度 Top**（相对区分度）：「饼干」(0.063)、「不错」(0.055)、「饱腹」(0.047)、「口感」(0.044)、「回购」(0.039)、「健康」(0.039)、「酥脆」(0.038)、「零食」(0.038)、「全麦」(0.034)、「好吃」(0.034)、「gi」(0.030)、「方便」(0.029)。\n\n**共现较强的词对**（同条评论内，供联想维度）：「口感」-「饼干」(98)；「酥脆」-「饼干」(94)；「健康」-「饼干」(88)；「回购」-「饼干」(81)；「饱腹」-「饼干」(76)；「口感」-「酥脆」(74)；「口感」-「饱腹」(71)；「全麦」-「饼干」(61)；「这款」-「饼干」(60)；「酥脆」-「饱腹」(60)。\n\n**自动归纳的主题（无监督，仅作探索）**：\n- 主题 1：饼干、饱腹、酥脆、口感、健康、回购、全麦、gi、零食、方便、独立、纤维\n- 主题 2：不错、口感、饼干、饱腹、起来、速度、早餐、干净、酥脆、香味、合适、好吃\n- 主题 3：饼干、回购、健康、零食、好吃、不错、这款、味道、全麦、gi、起来、酥脆\n- 主题 4：饼干、口感、无糖、饱腹、独立、方便、不错、全麦、酥脆、健康、合适、纤维\n\n> 以上为主题探索与统计摘要，**不等同**于业务结论；若与星级、规则词表冲突，以人工抽样为准。\n\n---\n\n---\n\n该细类评论主要围绕一款主打「低GI」与「健康代餐」功能的全麦粗粮饼干展开，用户普遍关注其口感、饱腹感以及配料的天然性。关键词突出度显示，“饱腹”“口感”“回购”等词高频出现，表明消费者在选择时更看重食用体验与持续购买意愿。共现分析中，“口感”与“饼干”“酥脆”频繁组合，“饱腹”与“健康”“粗粮”紧密关联，反映出用户将产品视为兼具美味与功能性的小食。\n\nLDA主题归纳揭示了多个潜在关注维度：第一主题聚焦于产品本身属性，如“饼干”“饱腹”“GI”“全麦”等，强调其作为代餐的健康价值；第二主题则偏向使用体验，如“早餐”“香味”“干净”等，暗示其作为日常加餐的便利性；第三主题突出“回购”“这款”“味道”等表达，体现用户对品牌与口味的认同；第四主题再次强调“无糖”“独立”“纤维”等成分与包装细节，说明消费者对配料表与食用方式的关注。需注意的是，LDA主题为无监督聚类结果，同一主题中可能出现多个维度交叉，且不与正式品类标签一一对应。\n\n从摘录文本可见，用户语气多为满意与推荐，如“第二次购买了”“回购”“值得”等表述频繁出现，体现出较强的复购意愿。部分评论中提到“血糖高”“嘴馋的救星”等语境，暗示该产品主要被控糖人群或需控制饮食的用户所青睐。例如有用户表示：“因为血糖高，又会常常肚子饿，这款饼干碳水化合物低，无蔗糖，对血糖有益，营业搭配饱腹感很强”，反映出其在特定健康需求下的使用场景。\n\n使用场景方面，评论中明确提及“早餐搭配牛奶”“每天吃”“加餐”等描述，表明该饼干常被用于早餐补充或日常饥饿时的替代零食。同时，“独立包装”“方便”等词的高频出现，也支持其作为便携式健康零食的定位。整体来看，该产品在满足口感需求的同时，通过“低GI”“无糖”“高纤维”等标签满足特定人群的健康诉求，形成“美味+健康”的双重价值认知。\n\n---\n\n**使用注意**：本分析基于开放词表与统计模型，词频与关键词突出度反映的是文本中词语的分布特征，而非人工标注的语义标签；LDA主题归纳为无监督聚类结果，主题间可能存在重叠，且不保证与实际产品功能或用户意图完全一致；所有结论仅依据本细类评论数据推断，不引入外部场景分类或正式报告中的标签体系。\n\n---", + "chapter8_text_mining_probe": true, + "structured_brief_omission_note": "全量 competitor_brief 已**不再**包含 `comment_sentiment_lexicon`（原星级子集预设口语短语口径）。已启用第八章文本挖掘（探针为主）时，传入策略润色的 structured_brief 还会省略「关注词/场景子串计数」、按细类 feedback 中的 focus_keyword_hits/scenarios_top、`strategy_hints` 等，避免与报告 §8 文本挖掘主口径冲突。**不得**再以这类子串计数或预设场景占比作为论据。用户与评论侧须依报告 **§8.2** 文本挖掘归纳及 `report_matrix_group_evidence_md`；**促销、满减、券价差**须与报告第六章、`price_promotion_signals` 及下方 `report_strategy_excerpt`（第九章）对齐，不得省略报告已写明的活动建议。" +} +``` diff --git a/docs/planning/策略生成-框架确定.md b/docs/planning/策略生成-框架确定.md new file mode 100644 index 0000000..625b593 --- /dev/null +++ b/docs/planning/策略生成-框架确定.md @@ -0,0 +1,203 @@ +# 策略生成 · 框架确定 + +**版本**：2026-04-22（修订：与 alignment 同步——默认产线弃用报告第九章长文；§3～§5、需求对照表已更新） +**目的**：在继续迭代「填表 → 规则底稿 / LLM 成稿 → 导出」之前，把**结构、输入输出、与报告的关系**定死；并明确——**策略成稿不是市场说明书，而是针对用户痛点写清「做什么、怎么做」**。 + +--- + +## 启动前：策略针对什么（业务须先对齐） + +动策略稿之前，应能回答：**这份策略是为谁、在什么监测任务下、对哪条产品线/类目、要达成哪一类阶段目标**。若尚未想清，可先**从下列目标类型中选一类或组合**（再落到表单字段与 §2.1），不必空泛「开始做」： + +| 代码 | 目标类型（示例） | 典型成功标准方向（示例，非固定） | +|------|-------------------|----------------------------------| +| **A** | 上市 / 首发前验证 | 主图文案与规格可核验、小样本转化或评价主题可控 | +| **B** | 份额追赶 | 搜索/列表可见性、对标款价差与话术 | +| **C** | 利润与价盘防守 | 到手价带稳定、促销规则可解释 | +| **D** | 新场景或新人群拓展 | 场景词与内容投放与评论侧一致 | +| **E** | 其它 | 在「业务备注」或成功标准中写清一句 | + +**还须对齐（与底稿「策略范围与前提」表一致）**：监测词与批次（与同任务报告同源）、**本品角色**（新品/追赶/防守等）、**一句话战场**、**主推类目/细类**（或多线并行）、**时间范围**、**可量化成功标准**。未定项允许保留「待确认」，由 LLM 成稿时**列选项、禁止编造**（见 `STRATEGY_SYSTEM`）。 + +--- + +**关联**： + +| 说明 | 路径 | +|------|------| +| 原则与 S1/S2 路线 | `docs/planning/strategy-marketing-content-alignment.md` | +| 规则底稿（骨架） | `backend/pipeline/reporting/strategy_draft.py` → `build_strategy_draft_markdown` | +| LLM 成稿 | `backend/pipeline/llm/generate_strategy.py` → `generate_strategy_draft_markdown_llm` | +| 人工可读示例（低 GI 批次） | `docs/demo/市场策略稿-示例-20260413_104252_低GI.md` | +| API | `JobStrategyDraftView`：`POST` 策略稿，`generator=rules \| llm` | + +--- + +## 1. 我们说的「策略生成」指什么 + +在本仓库里，**策略生成**特指：**独立市场策略稿**（Markdown），由任务数据 + 业务表单驱动，**不是**竞品报告全文。**可执行策略**以本产品交付为准；竞品报告内原「第九章」大模型长文在**默认产线已弃用**（见 `runner` 默认配置与 `strategy-marketing-content-alignment.md` §2）。 + +| 产物 | 形态 | 主要用途 | +|------|------|----------| +| **独立策略稿** | **策略范围与前提** → 摘要 → 一至十 → 附录 | 业务讨论、对外删减版、与表单决策对齐 | +| **报告 ·「九、策略与机会提示」** | 默认仅短引导（指向「策略制定」）；历史或调试开启 LLM 时可能含 `####` 块 | **不再**作为默认策略交付；与第五～八章数据仍同源（brief） | +| **营销内容**（规划） | 另管线；默认仅策略稿 + 表单（不默认灌 brief/报告节选） | 表达层；与监测结论一致经策略稿输入收束；见 `strategy-marketing-content-alignment.md` §3 | + +框架确定**只锁定第一行**：独立策略稿。 + +--- + +## 0. 与《竞品分析报告》的分工（避免重复） + +| 放在分析报告 | 放在策略稿 | +|--------------|------------| +| 样本量、列表/深入口径、词频、共现、LDA、细类条数、图表解读、方法说明 | **不写或一句带过**；可写「详见同任务竞品分析报告 §× / 附录」 | +| — | **要写**：**针对痛点怎么做**（§2.1 表）、资源与节奏、风险应对 | + +策略稿**不是第二份统计报告**；统计分析已在报告中完成，策略稿只消费其**结论**并转为**行动**。 + +### 常见质疑（口径） + +**Q：凭什么写「松软」，评论里还有很多人要「酥脆」？** +**A：** 二者常对应**不同细类**（糕点/面包 vs 饼干）。策略须**按主推款/细类**写口感与工艺，分线表述；禁止用单一词覆盖全池。若多线并行，饼干线与糕点线应分别有策略句。 + +**Q：满减、满额打折是不是不管了？** +**A：** **不是不管。** 能来自 `price_promotion_signals` / 报告第六章的须写进 §八.3；**禁止编造**未监测到的门槛。未捕获具体规则时，须写「待运营按后台活动补齐」类表述，并保留促销**原则**（透明、跟价节奏、券类型方向），避免整节空白。 + +**Q：§二 和 §三 里「障碍/痛点」会不会重复？** +**A：** 已收敛：**§二 仅 §2.1「针对痛点要怎么做」** 一张表；**不再**单设痛点表/价值对表/负向归因子节。**§三 仅 §3.1**，**不设 §3.2「转化障碍与应对」**——与购买相关的应对已在 §2.1 表内，§3.1 只写品类与时机、购买理由，勿再复述一遍障碍与应对。 + +--- + +## 1.1 核心：策略部分要写「怎么做」，不能只写「是什么」 + +| 不满意的表现 | 应有的表现 | +|--------------|------------| +| 大量篇幅描述市场、品类、数据，读完不知道「我们要做什么」 | 每条策略**能回答**：针对**哪类用户、哪条痛点**，我们**采取什么动作**，**在具体触点怎么做**（商详第几屏、主图文案、规格怎么标、客服话术等） | +| 痛点与后面章节脱节 | **第二节 §2.1 表**列清「针对痛点怎么做」后，**第六节起**（目标与路径、品牌四线、战术支柱）必须能**回扣**到该表，避免「泛泛而谈增长」 | +| 只有「建议重视体验」 | 写成**可执行句**：谁、在什么渠道、在什么时间窗口、做什么改动、如何验收 | + +**强制结构（逻辑闭环）**： + +1. **用户痛点**（来自数据与评论归纳，第二节） +2. **对应策略意图**（价值主张、差异化、目标） +3. **怎么做**（动作 + 落地载体：页面/投放/产品/价格/促销） +4. **如何验证**（指标、抽样、时间） + +规则底稿中 **§二** 仅 **§2.1 针对痛点要怎么做** 表占位；LLM 成稿须**填满**该表或等价表述，且 **§六～§八** 以**动词导向**（应做、通过、在…上）为主，避免纯名词堆砌。 + +--- + +## 2. 成稿结构（已定，勿随意增删块） + +与 **`build_strategy_draft_markdown` / `STRATEGY_SYSTEM`**、示例稿 **一致**，顺序如下： + +| 顺序 | 章节 | 要点（**策略性要求**） | +|------|------|------------------------| +| −1 | **策略范围与前提** | **针对什么做策略**：监测任务、本品角色、战场、主推类目、**目标类型（A～E）**、时间、成功标准；未定则列待确认与选项 | +| 0 | **摘要** | 承接上文边界；除范围样本外，**阶段重点**须含 1～2 条**可感知动作**，而非仅「加强运营」 | +| 1 | **一、顾客是谁** | 1.1 人群与路径 · 1.2 细类/评论侧归纳 · 1.3 本品聚焦（表单：角色、客群、对标） | +| 2 | **二、产品价值与用户痛点** | **仅 §2.1** 表：「类目/细类 → 痛点简述 → 动作 → 落地 → 验证」；多细类**分行**，不确定则「待业务定类」+ 假设 | +| 3 | **三、为什么要买「这款产品」** | **仅 §3.1** 品类与时机、购买理由；**无 §3.2**（与 §二 不重复写障碍与应对） | +| 4 | **四、为什么要选「这个品牌」** | 承诺与调性 · 信任与证据（**价位阵地**不在本章，见 **§8.2**） | +| 5 | **五、与其它品牌有何不同** | 对比对象 · 差异化 · **竞争应对须写可执行应对**（不主动价格战 / 跟价时如何说明等） | +| 6 | **六、阶段目标与路径** | 时间 / 成功标准 · **营销与总体策略须写成动作句**（与痛点挂钩） | +| 7 | **七、品牌四线** | 建设 · 打造 · 运营 · 体验（与表单 4P 支柱对应，**每条线至少一条「怎么做」**） | +| 8 | **八、战术支柱** | **§8.2 定价**含**价位阵地取向**（表单贴顶/卡腰/下探/另起带勾选）与定价支柱叙述；其余：产品 / 促销 / 渠道：**每支柱下写 针对哪条痛点、做什么、怎么落地** | +| 9 | **九、风险、假设与待验证** | 含表单风险勾选 | +| 10 | **十、下一步与节奏** | 含业务备注；**下一步**为可执行任务，非口号 | +| 11 | **附录** | 关键数据一览（与 brief 同源） | + +**命名约定**：对外文档可称「六主轴 + 品牌四线」——六主轴对应 **一～六** 的叙事链，**七** 为品牌四线，**八** 为 4P 战术落地。 + +**§2.1 针对痛点要怎么做（推荐列）**： + +| 类目/细类（本决策适用） | 用户痛点（简述） | 策略动作 | 具体怎么做（触点/话术/规格/渠道） | 如何验证 | +|--------------------------|------------------|----------|-----------------------------------|----------| +| | | | | | + +**全文各章「怎么做」补充（除 §2.1 外，摘要与一、三～五、九、十亦须落地）** + +| 章节 | 除数据叙述外，必须带上的策略性内容 | +|------|--------------------------------------| +| **摘要** | **阶段重点**含 1～2 条可执行动作，回扣 §2 优先痛点 | +| **一** | **1.3** 写清主攻人群/场景与 §2.1 的对应；与后文动作一致 | +| **三** | §3.1 末承接 §2 优先痛点与购买理由；**勿**重复 §2.1 已写的应对 | +| **四** | 承诺落到**触点**（商详/包装/客服等）的具体句子 | +| **五** | 差异化 = 相对竞品**多做什么/少做什么**；竞争应对 = 跟价/不跟价时的**具体话术或机制** | +| **六** | 路径与 §2.1 对齐；营销/总体策略为**动词句** | +| **七** | 四线各至少一句：**服务哪类痛点、本阶段做哪一步** | +| **八** | 四支柱分别回扣 **痛点→动作→落地** | +| **九** | 风险项带**应对动作或验证计划**，勿只列标题 | +| **十** | 可执行任务，与 §2.1 / §六 优先级一致 | + +（代码与 `STRATEGY_SYSTEM` 已同步上述要求；规则底稿各节均有对应 *成稿须…* 提示。） + +--- + +## 3. 输入框架（三层） + +| 层 | 内容 | 来源 | +|----|------|------| +| **A. 数据层** | `build_competitor_brief` 的 JSON | 同任务 `run_dir` CSV + `effective_report_config.json`，与报告**同一套计数** | +| **B. 决策层** | `strategy_decisions` | 前端策略表单：战场一句话、定位四选一、竞争倾向、四柱、营销/总体策略、风险勾选等 | +| **C. 对齐层（可选 · 遗留）** | `report_strategy_excerpt` | `load_report_strategy_excerpt`：**默认多为空**（第九章长文已弃用）；仅当 json 含历史 `markdown` 或 md 中仍有旧块时非空。主对齐靠 **A + 矩阵细类节选**（`report_matrix_group_evidence_md`） | + +LLM 路径下：**A** 经 `compact_brief_for_llm` 压缩进 payload；**B** 原样进入；**C** 截断后进入。 +**规则路径**无 LLM，仅 **A+B** 写入骨架，**C** 不参与（但 API 仍返回节选元数据便于核对）。 + +--- + +## 4. 输出与验收 + +| 模式 | 输出 | 最低验收 | +|------|------|----------| +| `generator=rules` | 带占位符的 Markdown | 章节齐全；含 **§2.1 针对痛点要怎么做** 表头；**无 §3.2**；数字仅来自 brief 摘录；与 `chapter8_text_mining_probe` 时无关注词条形图口径冲突 | +| `generator=llm` | 可读成稿 | 章节与骨架一致；**全书**遵守 `STRATEGY_DATA_RULES` 段首「全局禁止编造」（数字、品牌名、用户引语、活动规则、落地结果等）；已填表单项落实为执行句；**若** `report_strategy_excerpt` 非空则不与该节选明显矛盾；**全书**能体现「痛点→动作→怎么做」（含摘要、一、三～十）；**禁止**仅复述市场描述而无 actionable 句 | + +--- + +## 5. 与竞品报告正文的边界（防混） + +- **默认产线**：报告在「九、策略与机会提示」下**不**再附全任务大模型策略长文；**独立策略稿**是**可执行策略**的主交付。 +- **数据同源**：策略稿与报告仍共用同任务 **brief** 与第五～八章归纳（含矩阵细类节选），数字与口径须一致，**禁止编造**（见 `STRATEGY_DATA_RULES`）。 +- **遗留节选**：`report_strategy_excerpt` 仅兼容历史或显式开启 LLM 第九章的任务；**验收不再依赖**「报告第九章段落 ↔ 策略稿」逐段对齐。 + +--- + +## 6. 需求—实现对照（便于验收「全部落地」） + +| 需求要点 | 实现位置 | +|----------|----------| +| 六主轴 + 品牌四线、摘要→一～十→附录 | `build_strategy_draft_markdown`、`STRATEGY_SYSTEM` | +| 与竞品报告分工、不重复统计展开 | `STRATEGY_DATA_RULES`、底稿骨架说明 | +| 启动前目标选项 A～E、策略针对什么 | 本文「启动前」节；底稿「策略范围与前提」表；`STRATEGY_SYSTEM` + `STRATEGY_USER_PREFIX` | +| 痛点 → 怎么做、仅 §2.1 表（含类目/细类列） | 底稿 §2.1；`STRATEGY_DATA_RULES` + `STRATEGY_SYSTEM` §二 | +| 除 §2 外各章须带可执行句 | `STRATEGY_SYSTEM` 分节要求；规划 §1.1 表 | +| 语气：少内部缩写、少反复「对齐第几章」 | `STRATEGY_SYSTEM` **语气**段 | +| 分量/规格/克重痛点 | `STRATEGY_SYSTEM` §二；底稿 §2.1 提示 | +| 全文禁止编造（各节、各表） | `STRATEGY_DATA_RULES` 段首「全局禁止编造」；`STRATEGY_SYSTEM`「落实范围」；用户消息前缀自检句 | +| 摘要～八与 §2.1「类目/细类」列一致、防「全站」泛化 | `STRATEGY_DATA_RULES`「与 §2.1 类目列一致」；`STRATEGY_SYSTEM`「全书与 §2.1 类目列对齐」；底稿各节 *成稿须…* | +| §2 表「痛点简述」可溯源、禁编造用户引语 | `STRATEGY_DATA_RULES` §2 表条款 | +| 若显式开启报告内第九章 LLM，归纳不编造 | `STRATEGY_OPPORTUNITIES_SYSTEM` 引用同一 `STRATEGY_DATA_RULES` + `####` 块说明（**非默认产线**） | +| §三 不设转化障碍小节（免与 §二 重复） | 底稿无 §3.2；`STRATEGY_SYSTEM` §三 | +| 口感按细类：酥脆 vs 松软分线，禁止一词盖全站 | `STRATEGY_SYSTEM` **口感/质地与细类** | +| 促销：满减/满折/券/跨店，不编造门槛、不全节留空 | `STRATEGY_SYSTEM` **促销**；底稿 §八.3 | +| 默认 runner：`llm_strategy_opportunities` 为 false；第九章为固定读者说明 | `jd_report._strategy_opportunities_reader_fixed_lines`、`runner` 默认 `report_config` | +| 人工可读示例（低 GI） | `docs/demo/市场策略稿-示例-20260413_104252_低GI.md`（含 §2.1；策略稿目录 §8.x 为「战术支柱」促销等，勿与竞品报告第八章节号混淆） | + +**另行立项（本文件不承诺已编码）**：营销内容管线 S2/S3（见 `strategy-marketing-content-alignment.md`）、仅摘要模板、表单新字段、输出数字同源性自动校验 S4。 + +--- + +## 7. 待你方拍板的扩展（不影响上述已定框架） + +若后续要改，单独立项，不混进「框架确定」： + +- 是否增加「仅摘要 / 仅战术章」等**变体模板**； +- 是否在表单增加字段（如新里程碑）； +- 营销内容管线与策略稿的**强制依赖顺序**（建议：策略稿稳定后再接营销；与竞品报告一致不依赖在营销侧拼报告节选）。 + +--- + +*本文件为框架锚点；实现以代码与 `strategy-marketing-content-alignment.md` 为准，如有冲突以代码与 alignment 原则优先。* diff --git a/docs/templates/市场策略稿-目录模板-六主轴与品牌四线.md b/docs/templates/市场策略稿-目录模板-六主轴与品牌四线.md new file mode 100644 index 0000000..a1ac6e4 --- /dev/null +++ b/docs/templates/市场策略稿-目录模板-六主轴与品牌四线.md @@ -0,0 +1,213 @@ +# 市场策略稿 · 目录模板（六主轴 + 品牌四线） + +> **用途**：按「顾客 / 价值与痛点 / 为什么买 / 为什么选品牌 / 竞品差异 / 第一选择」与「品牌建设 · 打造 · 运营 · 体验」组织成稿；供人工撰写、表单字段设计或 LLM 章节提示对齐。 +> **语气默认**：**内部评审**（可写假设、待验证）；对外删减敏感句与未证实结论，见各节「对外提示」。 +> **与现有模板关系**：工作台自动草稿章节见同目录 [`market-strategy-brief.zh.md`](./market-strategy-brief.zh.md)；本文件侧重**战略叙事结构**，可与前者并行对照，合并时以事实源与 [`strategy-marketing-content-alignment.md`](../planning/strategy-marketing-content-alignment.md) 为准。 +> **动笔前先对齐**：见 [`策略生成-框架确定.md`](../planning/策略生成-框架确定.md)「启动前」与底稿 **策略范围与前提** 表；目标类型可选 **A～E**（上市验证 / 份额追赶 / 价盘防守 / 场景拓展 / 其它）。 + +--- + +## 策略范围与前提（生成前先对齐） + +| 须明确项 | 填写或待确认 | +|----------|----------------| +| 监测任务（与同任务报告一致） | 关键词 ________ ；批次 ________ | +| 本品角色 | 新品 / 追赶 / 防守 / 拓品类 … | +| 一句话战场 | ________ | +| 目标客群/场景 | ________（可选） | +| 主推类目/细类 | ________（未定写「待业务定类」） | +| 本阶段目标类型 | **A～E** 择一或组合（见规划文档） | +| 时间范围 | ________ | +| 成功标准 | ________ | + +--- + +## 摘要（1 页内） + +- **一句话战场**：________（类目/关键词/监测范围） +- **目标用户一句话**：________ +- **价值主张一句话**：________（替谁、解决什么、凭什么） +- **与竞品核心差异一句话**：________ +- **本周/本阶段优先动作一句话**：________（指向运营或体验均可） + +> **对外提示**：摘要可单独改写成「对外版」，去掉内部假设与竞品直呼。 + +--- + +## 一、顾客是谁 + +### 1.1 核心人群 + +- 人口与行为特征：________ +- 决策链（谁搜、谁比、谁拍板、谁复购）：________ + +### 1.2 核心场景 + +- 主场景（时间/地点/任务）：________ +- 次要场景：________ + +### 1.3 本品在人群与场景上的聚焦 + +- 本期主打的 1～2 个：________ +- 明确不做的或暂搁置的：________ + +**依据（可填数据来源）**：简报 / 报告节选 / 评论与文本挖掘摘要 / 业务备注：________ + +--- + +## 二、产品价值与用户痛点 + +> **结构说明（与自动草稿一致）**：本节**仅**下表——**针对痛点要怎么做**；不再单列「痛点表 / 价值对表 / 负向归因」以免与第三章重复。 + +### 2.1 针对痛点要怎么做 + +| 类目/细类（本决策适用） | 用户痛点（简述） | 策略动作 | 具体怎么做（触点/话术/规格/渠道） | 如何验证 | +|--------------------------|------------------|----------|-----------------------------------|----------| +| | | | | | + +**依据**：监测摘要 / 报告节选 / 业务备注（可溯源）________ + +--- + +## 三、为什么要买「这款产品」（品类与产品层） + +### 3.1 品类与时机 + +- 用户为何现在需要该品类：________ +- 本品 SKU/规格主推进项：________ +- （可选）价带锚点一句，与监测一致：________ + +> **不设 3.2**：转化障碍与应对已在 **§2.1** 表内写清，此处**勿重复**。 + +**依据**：________ + +--- + +## 四、为什么要选「这个品牌」（品牌层） + +### 4.1 品牌承诺与调性 + +- 希望用户记住的一句话：________ +- 调性关键词（3～5 个）：________ + +### 4.2 信任与证据 + +- 可引用的证据类型（评价方向/背书/销量表述边界）：________ +- **禁止或慎用表述**（合规与事实边界）：________ + +**依据**：________ + +--- + +## 五、与其它品牌有何不同（差异化） + +### 5.1 主要对比对象 + +- 对标品牌或价位带：________ + +### 5.2 可防御差异点 + +| 差异点 | 与谁不同 | 是否可在监测数据中体现 | 风险 | +|--------|----------|------------------------|------| +| | | | | + +### 5.3 竞争应对原则 + +- 价格/促销跟随策略：________ +- 不跟随时的话术与底线：________ + +**依据**：________ + +--- + +## 六、如何成为用户的第一选择（目标与路径） + +### 6.1 「第一选择」在本阶段的定义 + +- 心智目标（品类内/场景内）：________ +- 可量化代理指标（若有）：________ + +### 6.2 路径总览 + +- 本阶段 3～5 条主路径（每条一句）：________ + +**依据**：________ + +--- + +## 七、品牌四线：建设 · 打造 · 运营 · 体验 + +> 四线分开写，避免混成一段空话；每条需回扣第二节「价值与痛点」或第五节「差异」。 + +### 7.1 品牌建设（长期心智） + +- 定位、主张、长期叙事线：________ +- 与「为什么选这个品牌」的对应关系：________ + +### 7.2 品牌打造（可感知资产） + +- 店铺/包装/内容模板/IP：________ +- 本期落地项：________ + +### 7.3 品牌运营（节奏与触达） + +- 活动/栏目/资源节奏（12 周或本季度可简写）：________ +- 与促销/定价策略的衔接：________ + +### 7.4 品牌体验（全链路一致） + +- 详情页—下单—履约—售后 的体验要点：________ +- 与评论中负向反馈的回应原则：________ + +**依据**：________ + +--- + +## 八、战术支柱（回扣战略） + +> 每一节开头用一句话说明：**服务于哪类用户/场景、强化哪条差异、落在品牌四线哪一环**。 + +### 8.1 产品策略 + +- ________ + +### 8.2 定价策略 + +- ________ + +### 8.3 促销与活动策略 + +- ________ + +### 8.4 渠道与传播策略（可选） + +- ________ + +**依据**：________ + +--- + +## 九、风险、假设与待验证 + +- 关键假设清单：________ +- 待补数据或待开会项：________ +- 合规与表述风险：________ + +--- + +## 十、下一步与节奏 + +- 本周/本月优先：________ +- 责任与协作（内部）：________ +- 复盘时间点：________ + +--- + +## 附录（可选） + +- 本任务关键数据速览（与简报/报告一致） +- 引用列表：简报字段 / 报告章节 / JSON 文件名 + +--- + +*模板版本：与「六主轴 + 品牌四线」对齐；可拆成表单字段或 LLM 分章提示。* diff --git a/frontend/package-lock.json b/frontend/package-lock.json index ee93f9d..f16c220 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -12,6 +12,7 @@ "github-markdown-css": "^5.8.1", "marked": "^15.0.7", "papaparse": "^5.5.2", + "pinia": "^2.2.0", "vue": "^3.5.13", "vue-router": "^4.4.5" }, @@ -1100,6 +1101,58 @@ "integrity": "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA==", "license": "ISC" }, + "node_modules/pinia": { + "version": "2.2.0", + "resolved": "https://registry.npmjs.org/pinia/-/pinia-2.2.0.tgz", + "integrity": "sha512-iPrIh26GMqfpUlMOGyxuDowGmYousTecbTHFwT0xZ1zJvh23oQ+Cj99ZoPQA1TnUPhU6AuRPv6/drkTCJ0VHQA==", + "license": "MIT", + "dependencies": { + "@vue/devtools-api": "^6.6.3", + "vue-demi": "^0.14.8" + }, + "funding": { + "url": "https://github.com/sponsors/posva" + }, + "peerDependencies": { + "@vue/composition-api": "^1.4.0", + "typescript": ">=4.4.4", + "vue": "^2.6.14 || ^3.3.0" + }, + "peerDependenciesMeta": { + "@vue/composition-api": { + "optional": true + }, + "typescript": { + "optional": true + } + } + }, + "node_modules/pinia/node_modules/vue-demi": { + "version": "0.14.10", + "resolved": "https://registry.npmjs.org/vue-demi/-/vue-demi-0.14.10.tgz", + "integrity": "sha512-nMZBOwuzabUO0nLgIcc6rycZEebF6eeUfaiQx9+WSk8e29IbLvPU9feI6tqW4kTo3hvoYAJkMh8n8D0fuISphg==", + "hasInstallScript": true, + "license": "MIT", + "bin": { + "vue-demi-fix": "bin/vue-demi-fix.js", + "vue-demi-switch": "bin/vue-demi-switch.js" + }, + "engines": { + "node": ">=12" + }, + "funding": { + "url": "https://github.com/sponsors/antfu" + }, + "peerDependencies": { + "@vue/composition-api": "^1.0.0-rc.1", + "vue": "^3.0.0-0 || ^2.6.0" + }, + "peerDependenciesMeta": { + "@vue/composition-api": { + "optional": true + } + } + }, "node_modules/postcss": { "version": "8.5.9", "resolved": "https://registry.npmjs.org/postcss/-/postcss-8.5.9.tgz", diff --git a/frontend/package.json b/frontend/package.json index 8dfcafd..e605045 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -13,6 +13,7 @@ "github-markdown-css": "^5.8.1", "marked": "^15.0.7", "papaparse": "^5.5.2", + "pinia": "^2.2.0", "vue": "^3.5.13", "vue-router": "^4.4.5" }, diff --git a/frontend/src/components/JobDatasetModal.vue b/frontend/src/components/JobDatasetModal.vue index 3eaed9e..85a68c2 100644 --- a/frontend/src/components/JobDatasetModal.vue +++ b/frontend/src/components/JobDatasetModal.vue @@ -18,14 +18,37 @@ const emit = defineEmits(['close']) const paneActive = computed(() => !!(props.job && (props.embedded || props.open))) +const SORT_LABELS = { + row_index: '入库顺序', + price: '价格', + sku_id: 'SKU', + title: '标题', + leaf_category: '叶类目', + matrix_group_label: '类目', + detail_category_path: '类目路径', + detail_brand: '品牌', + total_sales: '销量（解析排序）', + comment_count: '评价量（解析排序）', +} + const tab = ref('search') const page = ref(1) const pageSize = ref(30) +const pageJumpDraft = ref(1) const summary = ref(null) const list = ref({ results: [], total: 0, page: 1, page_size: 30 }) const loading = ref(false) const err = ref('') const commentSkuFilter = ref('') +const sortField = ref('row_index') +const sortOrder = ref('asc') +/** 类目（§5 矩阵），对应接口参数 report_group */ +const reportGroup = ref('') +/** 店铺名精确筛选，对应接口参数 shop；选项来自摘要 shop_options */ +const selectedShop = ref('') +const priceMin = ref('') +const priceMax = ref('') +const detailCategoryQ = ref('') const exportPanelOpen = ref(false) const exportLoading = ref(false) const exportErr = ref('') @@ -39,6 +62,21 @@ function handleBackdrop(e) { onBackdrop(e) } +const sortOptions = computed(() => { + const h = summary.value?.dataset_sort_help + let keys = ['row_index'] + if (h) { + if (tab.value === 'search') keys = h.search?.length ? h.search : keys + else if (tab.value === 'detail') keys = h.detail?.length ? h.detail : keys + else if (tab.value === 'merged') keys = h.merged?.length ? h.merged : keys + else keys = h.comments?.length ? h.comments : ['row_index'] + } + return keys.map((k) => ({ value: k, label: SORT_LABELS[k] || k })) +}) + +const categoryOptions = computed(() => summary.value?.category_options || []) +const shopOptions = computed(() => summary.value?.shop_options || []) + const displayColumns = computed(() => { const s = summary.value let cols = [] @@ -122,14 +160,32 @@ async function refreshList() { err.value = '' try { await loadSummary() - const sku = tab.value === 'comments' ? commentSkuFilter.value.trim() : '' - const url = jobDatasetPageUrl(props.job.id, tab.value, page.value, pageSize.value, sku) + const opts = + tab.value === 'comments' + ? { skuId: commentSkuFilter.value.trim() } + : { + sort: sortField.value, + order: sortOrder.value, + reportGroup: reportGroup.value.trim(), + shop: selectedShop.value.trim(), + priceMin: priceMin.value, + priceMax: priceMax.value, + detailCategoryQ: detailCategoryQ.value.trim(), + } + const url = jobDatasetPageUrl( + props.job.id, + tab.value, + page.value, + pageSize.value, + opts, + ) const r = await api(url) if (!r.ok) { err.value = await r.text() return } list.value = await r.json() + pageJumpDraft.value = list.value.page || page.value } catch (e) { err.value = String(e) } finally { @@ -143,6 +199,13 @@ watch( if (paneActive.value) { tab.value = 'search' page.value = 1 + sortField.value = 'row_index' + sortOrder.value = 'asc' + reportGroup.value = '' + selectedShop.value = '' + priceMin.value = '' + priceMax.value = '' + detailCategoryQ.value = '' commentSkuFilter.value = '' err.value = '' summary.value = null @@ -155,11 +218,49 @@ watch( watch(tab, () => { page.value = 1 exportPanelOpen.value = false + sortField.value = 'row_index' + sortOrder.value = 'asc' + reportGroup.value = '' + selectedShop.value = '' + priceMin.value = '' + priceMax.value = '' + detailCategoryQ.value = '' }) -watch([paneActive, () => props.job?.id, tab, page, commentSkuFilter], () => { - if (paneActive.value && props.job) refreshList() -}) +watch( + [ + sortField, + sortOrder, + reportGroup, + selectedShop, + priceMin, + priceMax, + detailCategoryQ, + ], + () => { + if (paneActive.value && props.job && tab.value !== 'comments') page.value = 1 + }, +) + +watch( + [ + paneActive, + () => props.job?.id, + tab, + page, + commentSkuFilter, + sortField, + sortOrder, + reportGroup, + selectedShop, + priceMin, + priceMax, + detailCategoryQ, + ], + () => { + if (paneActive.value && props.job) refreshList() + }, +) const totalPages = () => { const t = list.value.total || 0 @@ -175,6 +276,14 @@ function nextPage() { if (page.value < totalPages()) page.value += 1 } +function goToPage() { + const tp = totalPages() + let n = Math.round(Number(pageJumpDraft.value)) + if (!Number.isFinite(n)) n = 1 + page.value = Math.min(Math.max(1, n), tp) + pageJumpDraft.value = page.value +} + const exportPanelTitle = computed(() => { const m = { search: '搜索', detail: '商详', comments: '评论', merged: '整合宽表' } return `当前表（${m[tab.value] || tab.value}）` @@ -314,6 +423,63 @@ async function runExport(format) { </template> </div> + <div v-if="tab !== 'comments'" class="toolbar-filters"> + <label class="filter-item"> + 排序 + <select v-model="sortField" class="filter-select"> + <option v-for="o in sortOptions" :key="o.value" :value="o.value">{{ o.label }}</option> + </select> + </label> + <label class="filter-item"> + 顺序 + <select v-model="sortOrder" class="filter-select"> + <option value="asc">升序</option> + <option value="desc">降序</option> + </select> + </label> + <label class="filter-item"> + 类目 + <select v-model="reportGroup" class="filter-select wide"> + <option value="">全部</option> + <option v-for="g in categoryOptions" :key="g" :value="g">{{ g }}</option> + </select> + </label> + <label class="filter-item"> + 店铺 + <select v-model="selectedShop" class="filter-select wide"> + <option value="">全部</option> + <option v-for="s in shopOptions" :key="s" :value="s">{{ s }}</option> + </select> + </label> + <template v-if="tab === 'detail' || tab === 'merged'"> + <label class="filter-item"> + 类目路径包含 + <input + v-model="detailCategoryQ" + type="search" + class="filter-input wide" + placeholder="模糊匹配商详类目路径" + list="detail-cat-dl" + /> + <datalist id="detail-cat-dl"> + <option + v-for="p in summary?.detail_category_path_options || []" + :key="p" + :value="p" + /> + </datalist> + </label> + </template> + <label class="filter-item"> + 价格 ≥ + <input v-model="priceMin" type="number" step="any" class="filter-input narrow" placeholder="最低" /> + </label> + <label class="filter-item"> + 价格 ≤ + <input v-model="priceMax" type="number" step="any" class="filter-input narrow" placeholder="最高" /> + </label> + </div> + <div class="table-block"> <div v-if="loading" class="state state-fill">加载中…</div> <p v-else-if="err" class="state err state-fill">{{ err }}</p> @@ -371,6 +537,20 @@ async function runExport(format) { <span class="ma-muted" >第 {{ list.page || page }} / {{ totalPages() }} 页 · 共 {{ list.total ?? 0 }} 条</span > + <span class="pager-jump"> + <label class="jump-label" + >跳转 + <input + v-model.number="pageJumpDraft" + type="number" + :min="1" + :max="totalPages()" + class="jump-input" + /> + 页</label + > + <button type="button" class="ma-btn ma-btn-secondary" @click="goToPage">确定</button> + </span> <button type="button" class="ma-btn ma-btn-secondary" :disabled="page <= 1" @click="prevPage"> 上一页 </button> @@ -563,6 +743,40 @@ async function runExport(format) { width: 160px; font: inherit; } +.toolbar-filters { + display: flex; + flex-wrap: wrap; + align-items: flex-end; + gap: 0.65rem 0.85rem; + padding: 0.55rem 1rem 0.65rem; + border-bottom: 1px solid #f1f5f9; + background: #fafafa; + flex-shrink: 0; +} +.filter-item { + display: flex; + flex-direction: column; + gap: 0.2rem; + font-size: 0.72rem; + color: #475569; +} +.filter-select, +.filter-input { + font: inherit; + font-size: 0.8rem; + padding: 0.3rem 0.45rem; + border: 1px solid #d1d5db; + border-radius: 6px; + min-width: 0; +} +.filter-select.wide, +.filter-input.wide { + min-width: 12rem; + max-width: 22rem; +} +.filter-input.narrow { + width: 5.5rem; +} /* 高度封顶：数据再长也在表格内滚动，不把整块卡片无限撑高 */ .table-block { flex: 1 1 auto; @@ -689,6 +903,27 @@ async function runExport(format) { border-top: 1px solid #e5e7eb; flex-shrink: 0; } +.pager-jump { + display: inline-flex; + flex-wrap: wrap; + align-items: center; + gap: 0.35rem; +} +.jump-label { + font-size: 0.78rem; + color: #475569; + display: inline-flex; + align-items: center; + gap: 0.25rem; +} +.jump-input { + width: 3.5rem; + font: inherit; + font-size: 0.8rem; + padding: 0.25rem 0.35rem; + border: 1px solid #d1d5db; + border-radius: 6px; +} .ma-muted { color: #64748b; } diff --git a/frontend/src/components/ReportConfigFormFields.vue b/frontend/src/components/ReportConfigFormFields.vue index e24557d..50d9ee8 100644 --- a/frontend/src/components/ReportConfigFormFields.vue +++ b/frontend/src/components/ReportConfigFormFields.vue @@ -1,55 +1,24 @@ <script setup> defineProps({ - focusWordRows: { type: Array, required: true }, - scenarioGroups: { type: Array, required: true }, marketRows: { type: Array, required: true }, }) -defineEmits([ - 'add-focus', - 'remove-focus', - 'add-scenario', - 'remove-scenario', - 'add-market', - 'remove-market', -]) +defineEmits(['add-market', 'remove-market']) </script> <template> <div> <div class="rc-section"> - <h4 class="rc-subtitle">1. 评价里要统计的「关注词」</h4> - <p class="rc-help">报告会数这些词在评价里出现了多少次（适合看大家常提什么，例如口感、控糖、价格等）。</p> - <div class="rc-rows"> - <div v-for="(row, i) in focusWordRows" :key="'f' + i" class="rc-inline"> - <input v-model="row.text" type="text" class="rc-input" placeholder="输入一个词，如：控糖" /> - <button type="button" class="ma-btn ma-btn-secondary rc-mini" @click="$emit('remove-focus', i)">删除</button> - </div> - </div> - <button type="button" class="ma-btn ma-btn-secondary rc-add" @click="$emit('add-focus')">添加词</button> - </div> - - <div class="rc-section"> - <h4 class="rc-subtitle">2. 用途 / 场景分组</h4> + <h4 class="rc-subtitle">1. 第八章评论分析</h4> <p class="rc-help"> - 每一组有一个<strong>场景名称</strong>，和若干<strong>触发词</strong>。若一条评价里出现了其中任意一个词，这条评价就会算进该场景（一条评价可以同时属于多个场景）。触发词请用顿号、逗号或换行分开。 + 报告<strong>不再</strong>使用「预设关注词 / 预设场景词组」子串统计。请在报告配置（高级 JSON 或接口）中维护 + <code>chapter8_text_mining_probe</code> + 等开关，以生成开放词表、词频与共现等文本挖掘内容。可选 + <code>llm_comment_sentiment</code>：按矩阵细类分别调用模型，在报告<strong>8.3</strong>生成「正/负向主题」归纳，与探针及「细类评论要点归纳」并列、互不替代。 </p> - <div v-for="(g, i) in scenarioGroups" :key="'s' + i" class="rc-scenario-card"> - <label class="rc-label">场景名称</label> - <input v-model="g.label" type="text" class="rc-input" placeholder="如：早餐 / 代餐" /> - <label class="rc-label">触发词</label> - <textarea - v-model="g.triggersText" - class="rc-textarea" - rows="2" - placeholder="如：早餐、代餐、早饭（可用顿号或换行分隔）" - /> - <button type="button" class="ma-btn ma-btn-secondary rc-mini" @click="$emit('remove-scenario', i)">删除本组</button> - </div> - <button type="button" class="ma-btn ma-btn-secondary rc-add" @click="$emit('add-scenario')">添加场景组</button> </div> <div class="rc-section"> - <h4 class="rc-subtitle">3. 外部市场信息（可选）</h4> + <h4 class="rc-subtitle">2. 外部市场信息（可选）</h4> <p class="rc-help">若手边有第三方市场规模、增速等摘录，可填在表里，报告会多一节说明；不需要可整表留空。</p> <div class="rc-market-wrap"> <table class="rc-market"> @@ -107,17 +76,6 @@ defineEmits([ color: #6b7280; line-height: 1.5; } -.rc-rows { - display: flex; - flex-direction: column; - gap: 0.45rem; -} -.rc-inline { - display: flex; - flex-wrap: wrap; - align-items: center; - gap: 0.5rem; -} .rc-input { flex: 1; min-width: 140px; @@ -128,7 +86,6 @@ defineEmits([ font-size: 0.88rem; box-sizing: border-box; } -/* 表内输入：不占 flex，宽度受列约束，避免撑进邻列 */ .rc-input.rc-td { flex: none; display: block; @@ -137,28 +94,6 @@ defineEmits([ max-width: 100%; font-size: 0.8rem; } -.rc-textarea { - width: 100%; - max-width: 100%; - box-sizing: border-box; - padding: 0.45rem 0.55rem; - border: 1px solid #d1d5db; - border-radius: 6px; - font: inherit; - font-size: 0.88rem; - resize: vertical; - margin: 0.35rem 0 0.5rem; -} -.rc-label { - display: block; - font-size: 0.8rem; - font-weight: 500; - color: #4b5563; - margin-top: 0.35rem; -} -.rc-label:first-of-type { - margin-top: 0; -} .rc-mini { font-size: 0.8rem; padding: 0.3rem 0.55rem; @@ -168,13 +103,6 @@ defineEmits([ margin-top: 0.5rem; font-size: 0.85rem; } -.rc-scenario-card { - background: #fff; - border: 1px solid #e5e7eb; - border-radius: 8px; - padding: 0.75rem 0.85rem; - margin-bottom: 0.65rem; -} .rc-market-wrap { overflow-x: auto; margin-bottom: 0.35rem; diff --git a/frontend/src/composables/useGenerationInFlight.js b/frontend/src/composables/useGenerationInFlight.js index 2b62cfc..5182f70 100644 --- a/frontend/src/composables/useGenerationInFlight.js +++ b/frontend/src/composables/useGenerationInFlight.js @@ -1,13 +1,19 @@ -import { ref } from 'vue' +import { storeToRefs } from 'pinia' +import { useTaskStore } from '../stores/task' /** - * 长耗时生成类 POST/GET 的「进行中」标记（模块级，路由切换不丢）。 - * key 示例：`strategy-draft:12`、`regenerate-report:12`、`preview-report:12` + * 与 Pinia `useTaskStore` 同步；进行中列表持久化在 localStorage，跨标签页可见。 */ -const inFlightKey = ref(null) - export function generationInFlightKey() { - return inFlightKey + return storeToRefs(useTaskStore()).inFlightKeys +} + +export function clearGenerationInFlightState() { + useTaskStore().clearAll() +} + +export function clearRegenerateReportInFlightOnly() { + useTaskStore().clearRegenerateReportOnly() } /** @@ -16,12 +22,5 @@ export function generationInFlightKey() { * @returns {Promise<T>} */ export async function withGenerationInFlight(key, fn) { - inFlightKey.value = key - try { - return await fn() - } finally { - if (inFlightKey.value === key) { - inFlightKey.value = null - } - } + return useTaskStore().withInFlight(key, fn) } diff --git a/frontend/src/composables/useJobs.js b/frontend/src/composables/useJobs.js index 850c628..d1bf6c4 100644 --- a/frontend/src/composables/useJobs.js +++ b/frontend/src/composables/useJobs.js @@ -1,44 +1,5 @@ -import { ref, watch } from 'vue' - -const jobs = ref([]) - -/** 终态 */ -const TERMINAL_JOB_STATUSES = new Set(['success', 'failed', 'cancelled']) - -function isActiveJobStatus(status) { - return status === 'pending' || status === 'running' -} - -/** 单一定时器轮询列表（避免 N 个任务 → N 路 GET /api/jobs/:id/） */ -let jobsListPollTimer = null - -function stopJobsListPoll() { - if (jobsListPollTimer != null) { - clearInterval(jobsListPollTimer) - jobsListPollTimer = null - } -} - -async function fetchJobsListQuietly() { - try { - const r = await api('/api/jobs/') - if (r.ok) { - jobs.value = await r.json() - } - } catch { - /* 忽略网络错误，下一轮再试 */ - } -} - -function syncJobsListPoll() { - const hasActive = jobs.value.some((j) => isActiveJobStatus(j.status)) - if (!hasActive) { - stopJobsListPoll() - return - } - if (jobsListPollTimer != null) return - jobsListPollTimer = setInterval(fetchJobsListQuietly, 3000) -} +import { storeToRefs } from 'pinia' +import { useJobStore } from '../stores/jobs' export function api(path, opts = {}) { return fetch(path, { @@ -48,9 +9,7 @@ export function api(path, opts = {}) { } export async function refreshJobs() { - const r = await api('/api/jobs/') - if (!r.ok) throw new Error(await r.text()) - jobs.value = await r.json() + return useJobStore().refreshJobs() } export function jobCancelUrl(jobId) { @@ -82,24 +41,77 @@ export function jobExportReportDocumentUrl(jobId, fmt = 'docx') { return `/api/jobs/${jobId}/export-document/?kind=report&fmt=${encodeURIComponent(fmt)}` } -/** 策略稿正文（浏览器 sessionStorage）→ Word/PDF */ -export async function exportStrategyDocument(jobId, markdown, fmt = 'docx') { - const r = await api(`/api/jobs/${jobId}/export-document/`, { - method: 'POST', - body: JSON.stringify({ kind: 'strategy', fmt, markdown }), - }) +/** 竞品报告 GET 导出 Word/PDF（blob 下载，失败时解析服务端 JSON 提示） */ +export async function exportReportDocument(jobId, fmt = 'docx') { + const url = jobExportReportDocumentUrl(jobId, fmt) + const r = await fetch(url) + const ct = r.headers.get('Content-Type') || '' if (!r.ok) { - const t = await r.text() - throw new Error(t || `HTTP ${r.status}`) + let msg = `HTTP ${r.status}` + try { + if (ct.includes('application/json')) { + const j = await r.json() + msg = typeof j?.detail === 'string' ? j.detail : JSON.stringify(j) + } else { + const t = await r.text() + if (t) msg = t.length > 500 ? `${t.slice(0, 500)}…` : t + } + } catch { + /* keep msg */ + } + throw new Error(msg) } const blob = await r.blob() - const dispo = r.headers.get('Content-Disposition') || '' - const m = dispo.match(/filename="([^"]+)"/) - const name = m ? m[1] : `job_${jobId}_strategy_draft.${fmt}` + const filename = + filenameFromContentDisposition(r.headers.get('Content-Disposition')) || + `job_${jobId}_competitor_report.${fmt}` const u = URL.createObjectURL(blob) const a = document.createElement('a') a.href = u - a.download = name + a.download = filename + a.rel = 'noopener' + document.body.appendChild(a) + a.click() + a.remove() + URL.revokeObjectURL(u) +} + +/** + * 策略稿或营销内容 Markdown → Word/PDF + * @param {'strategy' | 'marketing_detail'} [kind] + */ +export async function exportStrategyDocument(jobId, markdown, fmt = 'docx', kind = 'strategy') { + const r = await api(`/api/jobs/${jobId}/export-document/`, { + method: 'POST', + body: JSON.stringify({ kind, fmt, markdown }), + }) + const ct = r.headers.get('Content-Type') || '' + if (!r.ok) { + let msg = `HTTP ${r.status}` + try { + if (ct.includes('application/json')) { + const j = await r.json() + msg = typeof j?.detail === 'string' ? j.detail : JSON.stringify(j) + } else { + const t = await r.text() + if (t) msg = t.length > 500 ? `${t.slice(0, 500)}…` : t + } + } catch { + /* keep msg */ + } + throw new Error(msg) + } + const blob = await r.blob() + const fallback = + kind === 'marketing_detail' + ? `job_${jobId}_marketing_detail_pack.${fmt}` + : `job_${jobId}_strategy_draft.${fmt}` + const filename = + filenameFromContentDisposition(r.headers.get('Content-Disposition')) || fallback + const u = URL.createObjectURL(blob) + const a = document.createElement('a') + a.href = u + a.download = filename a.rel = 'noopener' document.body.appendChild(a) a.click() @@ -145,12 +157,39 @@ export function reportConfigDefaultsUrl() { return '/api/report-config-defaults/' } -export function jobDatasetPageUrl(jobId, kind, page = 1, pageSize = 50, skuId = '') { +export function strategyConfigDefaultsUrl() { + return '/api/strategy-config-defaults/' +} + +/** + * @param {Record<string, string | number | undefined> | string} [opts] 筛选参数对象；兼容旧调用：传入字符串视为 comments 的 sku_id + */ +export function jobDatasetPageUrl(jobId, kind, page = 1, pageSize = 50, opts = {}) { + const o = typeof opts === 'string' ? { skuId: opts } : opts || {} const p = new URLSearchParams({ page: String(page), page_size: String(pageSize), }) - if (skuId) p.set('sku_id', skuId) + const sku = o.skuId ?? o.sku_id + if (sku) p.set('sku_id', String(sku)) + if (o.sort) p.set('sort', String(o.sort)) + if (o.order) p.set('order', String(o.order)) + const rg = + o.reportGroup ?? o.report_group ?? o.categoryNormId ?? o.category_norm_id + if (rg !== undefined && rg !== null && String(rg).trim() !== '') + p.set('report_group', String(rg).trim()) + const shop = o.shop ?? o.shop_name ?? o.shopQ ?? o.shop_q + if (shop !== undefined && shop !== null && String(shop).trim() !== '') + p.set('shop', String(shop).trim()) + const pmin = o.priceMin ?? o.price_min + if (pmin !== undefined && pmin !== null && String(pmin).trim() !== '') + p.set('price_min', String(pmin).trim()) + const pmax = o.priceMax ?? o.price_max + if (pmax !== undefined && pmax !== null && String(pmax).trim() !== '') + p.set('price_max', String(pmax).trim()) + const dcq = o.detailCategoryQ ?? o.detail_category_q + if (dcq !== undefined && dcq !== null && String(dcq).trim() !== '') + p.set('detail_category_q', String(dcq).trim()) return `/api/jobs/${jobId}/dataset/${kind}/?${p.toString()}` } @@ -208,14 +247,6 @@ export async function downloadJobDatasetExport(jobId, kind, exportFmt) { URL.revokeObjectURL(u) } -watch( - jobs, - () => { - syncJobsListPoll() - }, - { deep: true }, -) - export function jobConfigHint(j) { const parts = [] if (j.page_start != null || j.page_to != null) { @@ -237,6 +268,8 @@ export function jobConfigHint(j) { } export function useJobs() { + const store = useJobStore() + const { jobs } = storeToRefs(store) return { jobs, refreshJobs, diff --git a/frontend/src/composables/useReportConfigForm.js b/frontend/src/composables/useReportConfigForm.js index 1261865..750609c 100644 --- a/frontend/src/composables/useReportConfigForm.js +++ b/frontend/src/composables/useReportConfigForm.js @@ -11,6 +11,18 @@ function splitTriggers(text) { .filter(Boolean) } +/** + * 表单未展示的大模型/细类归纳等布尔项：从任务读入后在「保存」时原样写回，避免误清空。 + * （与 backend ``validate_report_config_body`` 允许的键一致。） + */ +const REPORT_CONFIG_PASSTHROUGH_BOOL_KEYS = [ + 'llm_comment_sentiment', + 'llm_matrix_group_summaries', + 'llm_price_group_summaries', + 'llm_comment_group_summaries', + 'llm_scenario_group_summaries', +] + /** * 报告调参表单（与后端 report_config 字段对应），面向非技术用户。 */ @@ -20,11 +32,14 @@ export function useReportConfigForm() { const marketRows = ref([ { indicator: '', value_and_scope: '', source: '', year: '' }, ]) + /** 表单未编辑的布尔项，从任务配置读入后随保存写回 */ + const passthroughBools = ref({}) function resetToEmpty() { focusWordRows.value = [{ text: '' }] scenarioGroups.value = [{ label: '', triggersText: '' }] marketRows.value = [{ indicator: '', value_and_scope: '', source: '', year: '' }] + passthroughBools.value = {} } /** @@ -94,6 +109,12 @@ export function useReportConfigForm() { } else { marketRows.value = [{ indicator: '', value_and_scope: '', source: '', year: '' }] } + + const pass = {} + for (const k of REPORT_CONFIG_PASSTHROUGH_BOOL_KEYS) { + if (Object.prototype.hasOwnProperty.call(cfg, k)) pass[k] = Boolean(cfg[k]) + } + passthroughBools.value = pass } /** @returns {Record<string, unknown>} 可 PATCH 到后端的 report_config；全空则为 {} */ @@ -133,6 +154,7 @@ export function useReportConfigForm() { })) } + Object.assign(out, passthroughBools.value) return out } @@ -178,6 +200,7 @@ export function useReportConfigForm() { focusWordRows, scenarioGroups, marketRows, + passthroughBools, resetToEmpty, applyFromApiConfig, buildPayload, diff --git a/frontend/src/lib/marketingPackMarkdown.js b/frontend/src/lib/marketingPackMarkdown.js new file mode 100644 index 0000000..d48d9be --- /dev/null +++ b/frontend/src/lib/marketingPackMarkdown.js @@ -0,0 +1,154 @@ +/** + * 将营销内容 API 返回的 JSON 转为可导出 Word/PDF 的 Markdown（中文小标题）。 + */ + +const CORE_LABELS = { + what_we_sell: '卖的是什么（本品）', + one_liner_value: '一句话价值主张', + buyer_job_to_be_done: '购买者任务与情境', + key_pain_or_desire: '核心痛点或欲望', + why_this_product: '为何要选这一款', + proof_or_trust_angle: '信任或证明角度', + differentiation_vs_alternatives: '与替代方案的差异', + price_value_framing: '价位与价值感表述', + compliance_taboos: '表述禁区摘要', + open_points_for_business: '待业务补充', +} + +function escLine(s) { + if (s == null || s === '') return '—' + return String(s).replace(/\r\n/g, '\n').trim() || '—' +} + +function pushCoreCard(lines, card) { + if (!card || typeof card !== 'object') { + lines.push('（无核心信息卡数据）') + lines.push('') + return + } + for (const [en, zh] of Object.entries(CORE_LABELS)) { + lines.push(`### ${zh}`) + lines.push('') + lines.push(escLine(card[en])) + lines.push('') + } +} + +function pushBullets(lines, title, arr) { + lines.push(`### ${title}`) + lines.push('') + if (!Array.isArray(arr) || !arr.length) { + lines.push('—') + lines.push('') + return + } + let any = false + for (const item of arr) { + const t = escLine(item) + if (t !== '—') { + lines.push(`- ${t}`) + any = true + } + } + if (!any) lines.push('—') + lines.push('') +} + +function pushFaq(lines, faq) { + lines.push('### 买家问答') + lines.push('') + if (!Array.isArray(faq) || !faq.length) { + lines.push('—') + lines.push('') + return + } + let n = 0 + for (const item of faq) { + if (!item || typeof item !== 'object') continue + const q = escLine(item.question) + const a = escLine(item.answer) + if (q === '—' && a === '—') continue + n += 1 + lines.push(`#### 问 ${n}：${q === '—' ? '（未提供）' : q}`) + lines.push('') + lines.push(a) + lines.push('') + } + if (n === 0) { + lines.push('—') + lines.push('') + } +} + +function pushDetailPack(lines, pack) { + if (!pack || typeof pack !== 'object') { + lines.push('（无多触点营销文案数据）') + lines.push('') + return + } + lines.push('### 依据与边界') + lines.push('') + lines.push(escLine(pack.traceability_note)) + lines.push('') + pushBullets(lines, '商品短标题备选', pack.listing_titles) + lines.push('### 列表副文案') + lines.push('') + lines.push(escLine(pack.listing_subtitle)) + lines.push('') + lines.push('### 商品详情页首屏引导') + lines.push('') + lines.push(escLine(pack.detail_headline)) + lines.push('') + pushBullets(lines, '详情页中段叙事', pack.detail_mid_story_paragraphs) + pushBullets(lines, '卖点列表', pack.selling_bullets) + pushBullets(lines, '食用场景与搭配建议', pack.usage_and_pairing_tips) + pushBullets(lines, '参数区旁短句', pack.spec_sidebar_lines) + pushFaq(lines, pack.faq) + pushBullets(lines, '短图文/种草贴变体', pack.short_graphic_post_variants) + pushBullets(lines, '主图三要点', pack.main_image_three_points) + lines.push('### 文生图提示词（主图）') + lines.push('') + lines.push(escLine(pack.text_to_image_prompt_main)) + lines.push('') + lines.push('### 文生图提示词（场景/备选）') + lines.push('') + lines.push(escLine(pack.text_to_image_prompt_scene)) + lines.push('') + lines.push('### 文生视频提示词（短视频）') + lines.push('') + lines.push(escLine(pack.text_to_video_prompt)) + lines.push('') + lines.push('### 直播/短视频钩句') + lines.push('') + lines.push(escLine(pack.live_or_short_hook)) + lines.push('') + pushBullets(lines, '直播/短视频要点提纲', pack.live_script_bullets) + lines.push('### 客服首句建议') + lines.push('') + lines.push(escLine(pack.customer_service_opening)) + lines.push('') +} + +/** + * @param {Record<string, unknown>} result marketing-detail-pack API 的 JSON 体 + * @returns {string} + */ +export function marketingPackResultToMarkdown(result) { + if (!result || typeof result !== 'object') return '' + const lines = [] + const jobId = result.job_id ?? '' + const kw = result.keyword ?? '' + const genAt = result.generated_at ?? '' + const src = result.source ?? '' + lines.push('# 营销内容') + lines.push('') + lines.push(`> 任务 #${jobId} · 关键词：${kw} · ${genAt}${src ? ` · ${src}` : ''}`) + lines.push('') + lines.push('## 核心信息卡') + lines.push('') + pushCoreCard(lines, result.core_info_card) + lines.push('## 多触点文案（列表/详情页/主图等）') + lines.push('') + pushDetailPack(lines, result.detail_page_pack) + return lines.join('\n').trim() + '\n' +} diff --git a/frontend/src/lib/strategyDraftStorage.js b/frontend/src/lib/strategyDraftStorage.js new file mode 100644 index 0000000..2a500a0 --- /dev/null +++ b/frontend/src/lib/strategyDraftStorage.js @@ -0,0 +1,123 @@ +/** + * 策略稿与会话字段：localStorage 主存，便于跨标签；首次读取时从 sessionStorage 迁移旧数据。 + */ + +const DRAFT_PREFIX = 'ma_strategy_draft_' +const SCOPE_PREFIX = 'ma_strategy_scope_' + +function draftKey(jobId) { + return `${DRAFT_PREFIX}${jobId}` +} + +function scopeKey(jobId) { + return `${SCOPE_PREFIX}${jobId}` +} + +/** + * @param {string} jobId + * @returns {Record<string, unknown>|null} + */ +export function loadStrategyDraftRecord(jobId) { + if (!jobId) return null + const key = draftKey(jobId) + try { + let raw = localStorage.getItem(key) + if (!raw && typeof sessionStorage !== 'undefined') { + raw = sessionStorage.getItem(key) + if (raw) { + try { + localStorage.setItem(key, raw) + } catch { + /* 配额：保留 session 可读 */ + } + } + } + if (!raw) return null + return JSON.parse(raw) + } catch { + return null + } +} + +/** + * @param {string} jobId + * @param {Record<string, unknown>} record + */ +export function saveStrategyDraftRecord(jobId, record) { + if (!jobId) return + const key = draftKey(jobId) + const payload = JSON.stringify(record) + try { + localStorage.setItem(key, payload) + } catch { + try { + sessionStorage.setItem(key, payload) + } catch { + /* ignore */ + } + return + } + try { + sessionStorage.removeItem(key) + } catch { + /* ignore */ + } +} + +/** + * @param {string} jobId + * @returns {string} + */ +export function loadStrategyMatrixScope(jobId) { + if (!jobId) return '' + const key = scopeKey(jobId) + try { + let v = localStorage.getItem(key) + if (v == null && typeof sessionStorage !== 'undefined') { + v = sessionStorage.getItem(key) + if (v != null) { + try { + localStorage.setItem(key, v) + } catch { + /* */ + } + } + } + return v || '' + } catch { + return '' + } +} + +/** + * @param {string} jobId + * @param {string} groupLabel empty = clear + */ +export function saveStrategyMatrixScope(jobId, groupLabel) { + if (!jobId) return + const key = scopeKey(jobId) + try { + if (groupLabel) { + localStorage.setItem(key, groupLabel) + try { + sessionStorage.setItem(key, groupLabel) + } catch { + /* */ + } + } else { + localStorage.removeItem(key) + try { + sessionStorage.removeItem(key) + } catch { + /* */ + } + } + } catch { + try { + if (groupLabel) sessionStorage.setItem(key, groupLabel) + else sessionStorage.removeItem(key) + } catch { + /* */ + } + } +} diff --git a/frontend/src/main.js b/frontend/src/main.js index 99b9e5c..7590842 100644 --- a/frontend/src/main.js +++ b/frontend/src/main.js @@ -1,8 +1,23 @@ import { createApp } from 'vue' +import { createPinia } from 'pinia' import './style.css' import 'github-markdown-css/github-markdown-light.css' import './styles/ui.css' import App from './App.vue' import router from './router' +import { useTaskStore } from './stores/task' -createApp(App).use(router).mount('#app') +const pinia = createPinia() +const app = createApp(App) +app.use(pinia) +app.use(router) + +if (typeof window !== 'undefined') { + window.addEventListener('storage', (e) => { + if (e.key === 'ma_tasks_inflight' || e.key === 'ma_tasks_inflight_ts') { + useTaskStore().hydrateFromLocalStorage() + } + }) +} + +app.mount('#app') diff --git a/frontend/src/stores/task.js b/frontend/src/stores/task.js new file mode 100644 index 0000000..1ca844b --- /dev/null +++ b/frontend/src/stores/task.js @@ -0,0 +1,133 @@ +/** + * 全局任务状态：耗时生成/导出/下载等「进行中」锁，跨标签页用 localStorage 同步。 + */ +import { defineStore } from 'pinia' + +const LS_KEY = 'ma_tasks_inflight' +const LS_TS = 'ma_tasks_inflight_ts' +/** 含 LLM 的请求可能较久；超时后视为未进行，避免按钮永久禁用 */ +const TTL_MS = 45 * 60 * 1000 +const LEGACY_SS_KEY = 'ma_generation_inflight' +const LEGACY_SS_TS = 'ma_generation_inflight_ts' + +function isAmbiguousClientFailure(err) { + if (err == null) return false + const name = err.name || '' + if (name === 'AbortError') return true + const msg = String(err.message || err) + return /Failed to fetch|NetworkError|Load failed|ERR_NETWORK|INTERNET_DISCONNECTED|aborted|cancel/i.test( + msg, + ) +} + +function migrateLegacySessionStorage() { + if (typeof sessionStorage === 'undefined' || typeof localStorage === 'undefined') return + try { + if (localStorage.getItem(LS_KEY)) return + const raw = sessionStorage.getItem(LEGACY_SS_KEY) + const ts = sessionStorage.getItem(LEGACY_SS_TS) + if (!raw) return + localStorage.setItem(LS_KEY, raw) + if (ts != null) localStorage.setItem(LS_TS, ts) + } catch { + /* ignore */ + } +} + +function readKeysFromLocalStorage() { + migrateLegacySessionStorage() + if (typeof localStorage === 'undefined') return [] + try { + const raw = localStorage.getItem(LS_KEY) + const ts = localStorage.getItem(LS_TS) + if (!raw || ts == null) return [] + const t = Number(ts) + if (!Number.isFinite(t) || Date.now() - t > TTL_MS) { + localStorage.removeItem(LS_KEY) + localStorage.removeItem(LS_TS) + return [] + } + try { + const parsed = JSON.parse(raw) + if (Array.isArray(parsed)) return parsed.filter((x) => typeof x === 'string' && x) + if (typeof parsed === 'string') return [parsed] + return [] + } catch { + return raw ? [raw] : [] + } + } catch { + return [] + } +} + +function writeKeysToLocalStorage(keys) { + if (typeof localStorage === 'undefined') return + try { + if (keys.length) { + localStorage.setItem(LS_KEY, JSON.stringify(keys)) + localStorage.setItem(LS_TS, String(Date.now())) + } else { + localStorage.removeItem(LS_KEY) + localStorage.removeItem(LS_TS) + } + } catch { + /* 隐私模式 / 配额 */ + } +} + +export const useTaskStore = defineStore('ma-tasks', { + state: () => ({ + inFlightKeys: readKeysFromLocalStorage(), + }), + + actions: { + hydrateFromLocalStorage() { + this.inFlightKeys = readKeysFromLocalStorage() + }, + + _persist() { + writeKeysToLocalStorage(this.inFlightKeys) + }, + + addKey(key) { + if (this.inFlightKeys.includes(key)) return + this.inFlightKeys = [...this.inFlightKeys, key] + this._persist() + }, + + removeKey(key) { + this.inFlightKeys = this.inFlightKeys.filter((k) => k !== key) + this._persist() + }, + + clearAll() { + this.inFlightKeys = [] + this._persist() + }, + + clearRegenerateReportOnly() { + const next = this.inFlightKeys.filter((k) => !String(k).startsWith('regenerate-report:')) + this.inFlightKeys = next + this._persist() + }, + + /** + * @param {string} key + * @param {() => Promise<T>} fn + * @returns {Promise<T>} + */ + async withInFlight(key, fn) { + this.addKey(key) + try { + const out = await fn() + this.removeKey(key) + return out + } catch (e) { + if (!isAmbiguousClientFailure(e)) { + this.removeKey(key) + } + throw e + } + }, + }, +}) diff --git a/frontend/src/views/jd/JdAnalysisBuildView.vue b/frontend/src/views/jd/JdAnalysisBuildView.vue index b853958..8aec125 100644 --- a/frontend/src/views/jd/JdAnalysisBuildView.vue +++ b/frontend/src/views/jd/JdAnalysisBuildView.vue @@ -1,6 +1,7 @@ <script setup> import { computed, onMounted, ref, watch } from 'vue' import { + clearRegenerateReportInFlightOnly, generationInFlightKey, withGenerationInFlight, } from '../../composables/useGenerationInFlight' @@ -11,18 +12,22 @@ import { useReportConfigForm } from '../../composables/useReportConfigForm' const { jobs } = useJobs() const selectedId = ref('') -const useLlm = ref(false) +/** 勾选则本次只出规则统计稿（仍先跑规则落盘，不做全文智能润色） */ +const useRulesOnly = ref(false) const regenErr = ref('') const genInFlight = generationInFlightKey() const REGEN_PREFIX = 'regenerate-report:' const regenPendingJobId = computed(() => { - const k = genInFlight.value - if (!k || !k.startsWith(REGEN_PREFIX)) return null - return k.slice(REGEN_PREFIX.length) + for (const k of genInFlight.value) { + if (k.startsWith(REGEN_PREFIX)) return k.slice(REGEN_PREFIX.length) + } + return null }) const regenBusyThisTask = computed( () => regenPendingJobId.value != null && regenPendingJobId.value === selectedId.value, ) +/** 任意任务正在重新生成时都应禁用按钮，避免切换页签后 selectedId 被重置导致误判可点 */ +const regenBusyAny = computed(() => regenPendingJobId.value != null) const regenBusyOtherTask = computed( () => regenPendingJobId.value != null && regenPendingJobId.value !== selectedId.value, ) @@ -148,16 +153,23 @@ async function loadList() { } } +function clearLocalRegenLock() { + regenErr.value = '' + clearRegenerateReportInFlightOnly() +} + async function regenerateReport() { const id = selectedId.value if (!id) return regenErr.value = '' const key = `${REGEN_PREFIX}${id}` - await withGenerationInFlight(key, async () => { - try { + try { + await withGenerationInFlight(key, async () => { const r = await api(`/api/jobs/${id}/regenerate-report/`, { method: 'POST', - body: JSON.stringify({ generator: useLlm.value ? 'llm' : 'rules' }), + body: JSON.stringify({ + generator: useRulesOnly.value ? 'rules' : 'llm', + }), }) const text = await r.text() if (!r.ok) { @@ -172,10 +184,10 @@ async function regenerateReport() { const updated = JSON.parse(text) const idx = jobs.value.findIndex((x) => x.id === updated.id) if (idx >= 0) jobs.value[idx] = updated - } catch (e) { - regenErr.value = String(e) - } - }) + }) + } catch (e) { + regenErr.value = String(e) + } } onMounted(loadList) @@ -211,15 +223,15 @@ watch( <section class="ma-card"> <h2>分析报告生成</h2> <p class="hint-top"> - 选择<strong>已成功</strong>的任务，调整报告统计规则后保存。<strong>未勾选</strong>下方选项时，按固定统计规则生成报告；<strong>勾选「使用大模型生成」</strong>后，由大模型根据本批次摘要撰写全文（通常更慢且可能计费）。均不重新爬取。 + 选择<strong>已成功</strong>的任务，调整下方统计规则后点「保存以上设置」，再点「重新生成报告」。默认会<strong>先按系统规则生成统计稿</strong>，再<strong>用全文智能润色与补充</strong>（需本系统已配置可用的智能服务）；不会重新爬取数据。各章是否做评价智能解读等，可用「填入推荐示例」带上，或在下方「高级选项」里微调（多数情况不必动）。 阅读与下载请至 <RouterLink to="/jd/analysis-view">报告查看</RouterLink>。 </p> <div class="toolbar"> - <label class="chk-inline"> - <input v-model="useLlm" type="checkbox" /> - 使用大模型生成（服务端需已配置并可用） + <label class="chk-inline chk-rules-only"> + <input v-model="useRulesOnly" type="checkbox" /> + 本次只生成规则统计稿（不做全文智能润色，更快、不调用智能服务） </label> </div> <div class="toolbar"> @@ -233,13 +245,28 @@ watch( <button type="button" class="ma-btn ma-btn-primary" - :disabled="!selectedId || regenBusyThisTask" + :disabled="!selectedId || regenBusyAny" title="不重新爬取，仅根据本批次已有数据更新报告文件" @click="regenerateReport" > {{ regenBusyThisTask ? '生成中…' : '重新生成报告' }} </button> + <button + v-if="regenBusyAny" + type="button" + class="ma-btn ma-btn-secondary" + title="仅清除浏览器里记录的「报告生成中」状态；若后端仍在执行请勿点" + @click="clearLocalRegenLock" + > + 清除误锁（本地） + </button> </div> + <p v-if="!successJobs.length" class="hint-top"> + 当前没有<strong>已成功</strong>的任务，无法生成报告；请先在任务列表确认流水线成功。 + </p> + <p v-else-if="regenBusyAny" class="hint-top"> + 按钮因本页记录的「生成中」状态而暂时不可用。若你已重启服务或确定没有在生成，可先点「清除误锁（本地）」再试。 + </p> <p v-if="regenBusyOtherTask" class="ma-warn-banner"> 任务 #{{ regenPendingJobId }} 的报告正在重新生成中，请稍候再切换任务或重复提交。 </p> @@ -247,7 +274,8 @@ watch( <div v-if="selectedId" class="report-config-block"> <h3 class="report-config-title">报告里的评价统计怎么算</h3> <p class="hint-top report-config-hint"> - 下面三项都<strong>可以不改</strong>：留空并保存，表示沿用系统内置规则。请先点「保存以上设置」，再点「重新生成报告」（需要大模型时先勾选页面上方对应选项）。 + 关注词、场景词组、外部市场表等<strong>可以不改</strong>：留空并保存即沿用内置规则。大模型相关布尔项（如 + <code>llm_comment_sentiment</code>）不再单独占勾选框：若任务里已有，会在保存时保留；要改请展开「高级 JSON」。 </p> <div class="report-config-actions"> <button @@ -281,17 +309,22 @@ watch( /> <details class="rc-advanced" @toggle="onAdvancedJsonToggle"> - <summary>高级：用 JSON 编辑（一般不需要）</summary> - <p class="rc-help">打开时会根据上面表单生成内容；改完后点「写回表单」再保存。</p> + <summary>高级选项（编辑底层配置，一般不需要）</summary> + <p class="rc-help"> + 打开时会根据上面表单生成内容；改完后点「写回表单」再保存。可在此加入 + <code>llm_comment_sentiment</code>、<code>llm_matrix_group_summaries</code> + 等布尔字段（须为 <code>true</code>/<code>false</code>）。页顶「重新生成报告」默认已使用 + <code>generator:"llm"</code>；若只要规则稿请勾选「本次仅用规则引擎」。 + </p> <textarea v-model="advancedJsonText" class="report-config-editor" rows="10" spellcheck="false" /> - <button type="button" class="ma-btn ma-btn-secondary rc-add" @click="applyAdvancedJsonToForm">将 JSON 写回表单</button> + <button type="button" class="ma-btn ma-btn-secondary rc-add" @click="applyAdvancedJsonToForm">将配置写回表单</button> </details> <p v-if="reportConfigErr" class="ma-err">{{ reportConfigErr }}</p> </div> <p v-if="selectedJob?.run_dir" class="run-dir-note ma-muted"> - 本任务输出目录：<span class="run-dir-path">{{ selectedJob.run_dir }}</span> + 本任务在本机的数据目录（排查问题时可用）：<span class="run-dir-path">{{ selectedJob.run_dir }}</span> </p> <p v-if="regenErr" class="ma-err">{{ regenErr }}</p> @@ -319,6 +352,10 @@ watch( gap: 0.75rem; margin-bottom: 0.5rem; } +.chk-rules-only { + width: auto; + max-width: 100%; +} .chk-inline { display: flex; align-items: flex-start; diff --git a/frontend/src/views/jd/JdAnalysisView.vue b/frontend/src/views/jd/JdAnalysisView.vue index 1069c7d..ce1cfb9 100644 --- a/frontend/src/views/jd/JdAnalysisView.vue +++ b/frontend/src/views/jd/JdAnalysisView.vue @@ -2,6 +2,126 @@ import { computed, onMounted, ref, watch } from 'vue' import { RouterLink } from 'vue-router' import MarkdownPreview from '../../components/MarkdownPreview.vue' + +/** 将结构化摘要转为非技术用户可读的条目（不出现 cr1 等字段名）。 */ +function pctShare(x) { + if (x == null || x === '') return '—' + const n = Number(x) + if (Number.isNaN(n)) return '—' + return `${(n * 100).toFixed(1)}%` +} + +/** 集中度块：新键 first_share / top_three_combined_share，旧键 cr1 / cr3 */ +function concShare(block, key) { + if (!block || typeof block !== 'object') return null + const v0 = key === 'first' ? block.first_share : block.top_three_combined_share + if (v0 != null && v0 !== '') return v0 + const legacy = key === 'first' ? block.cr1 : block.cr3 + return legacy != null && legacy !== '' ? legacy : null +} + +function briefHumanSummary(j) { + const rows = [] + if (!j || typeof j !== 'object') return rows + if (j.keyword) rows.push({ label: '搜索关键词', value: String(j.keyword) }) + if (j.batch_label) rows.push({ label: '批次', value: String(j.batch_label) }) + const sc = j.scope + if (sc && typeof sc === 'object') { + if (sc.merged_sku_count != null) + rows.push({ label: '深入采集的商品款数（SKU）', value: String(sc.merged_sku_count) }) + if (sc.comment_flat_rows != null) + rows.push({ label: '评价条数', value: String(sc.comment_flat_rows) }) + if (sc.structure_source_rows != null) + rows.push({ label: '列表/结构统计所用行数', value: String(sc.structure_source_rows) }) + if (sc.uses_pc_search_list_export === true) + rows.push({ label: '是否含搜索列表全量', value: '是' }) + } + const conc = j.concentration + if (conc && typeof conc === 'object') { + const shops = conc.shops_from_list + if (shops && typeof shops === 'object') { + const cr1 = concShare(shops, 'first') + const cr3 = concShare(shops, 'top3') + if (shops.top_label && cr1 != null) { + rows.push({ + label: '第一大店铺（占列表行比例）', + value: `${pctShare(cr1)} · ${shops.top_label}`, + }) + } + if (cr3 != null) { + rows.push({ + label: '前三大店铺合计（占列表行比例）', + value: pctShare(cr3), + }) + } + const usb = shops.unique_sku_basis + if (usb && typeof usb === 'object' && usb.n_unique_skus != null) { + const u1 = concShare(usb, 'first') + const u3 = concShare(usb, 'top3') + if (usb.top_label && u1 != null) { + rows.push({ + label: '第一大店铺（占去重 SKU 比例）', + value: `${pctShare(u1)} · ${usb.top_label} · 共 ${usb.n_unique_skus} 个 SKU`, + }) + } + if (u3 != null) { + rows.push({ + label: '前三大店铺合计（占去重 SKU）', + value: pctShare(u3), + }) + } + } + } + const lb = conc.list_brand_field + if (lb && typeof lb === 'object') { + const l1 = concShare(lb, 'first') + const l3 = concShare(lb, 'top3') + if (lb.top_label && l1 != null) { + rows.push({ + label: '第一大品牌（列表侧，按行）', + value: `${pctShare(l1)} · ${lb.top_label}`, + }) + } + if (l3 != null) { + rows.push({ + label: '前三大品牌合计（列表侧）', + value: pctShare(l3), + }) + } + } + const db = conc.detail_brand_among_merged + if (db && typeof db === 'object') { + const d1 = concShare(db, 'first') + const d3 = concShare(db, 'top3') + if (db.top_label && d1 != null) { + rows.push({ + label: '第一大品牌（深入样本）', + value: `${pctShare(d1)} · ${db.top_label}`, + }) + } + if (d3 != null) { + rows.push({ + label: '前三大品牌合计（深入样本）', + value: pctShare(d3), + }) + } + } + } + const p = j.price_stats + if (p && typeof p === 'object' && p.n > 0) { + rows.push({ label: '价格统计·样本量', value: String(p.n) }) + if (p.median != null) + rows.push({ label: '价格统计·中位数（元）', value: Number(p.median).toFixed(2) }) + if (p.mean != null) + rows.push({ label: '价格统计·平均（元）', value: Number(p.mean).toFixed(2) }) + } + const src = j.price_stats_source + if (src === 'pc_search_export_all_rows') + rows.push({ label: '价格统计·数据来源', value: '搜索列表全量' }) + else if (src === 'keyword_pipeline_merged') + rows.push({ label: '价格统计·数据来源', value: '深入采集合并表' }) + return rows +} import { refreshJobs, useJobs, @@ -10,7 +130,7 @@ import { previewUrl, jobCompetitorBriefUrl, downloadCompetitorBriefPack, - jobExportReportDocumentUrl, + exportReportDocument, } from '../../composables/useJobs' import { generationInFlightKey, @@ -23,9 +143,27 @@ const reportMd = ref('') const err = ref('') const viewMode = ref('render') const briefJson = ref('') +const briefData = ref(null) const briefErr = ref('') const briefCopyOk = ref(false) const packErr = ref('') +const exportDocErr = ref('') +/** 正在导出的格式：docx | pdf | null */ +const exportDocFmt = ref(null) + +async function exportReportFmt(fmt) { + const id = selectedId.value + if (!id) return + exportDocErr.value = '' + exportDocFmt.value = fmt + try { + await exportReportDocument(id, fmt) + } catch (e) { + exportDocErr.value = String(e?.message || e) + } finally { + exportDocFmt.value = null + } +} /** 将 Markdown 中的 report_assets 相对路径转为可访问的 API URL（在线预览插图） */ function reportMdWithAssetUrls(md, jobId) { @@ -47,19 +185,21 @@ const K_PACK = 'brief-pack:' function genKeyMatches(prefix) { const id = selectedId.value if (!id) return false - return genInFlight.value === `${prefix}${id}` + return genInFlight.value.includes(`${prefix}${id}`) } const loading = computed(() => genKeyMatches(K_PREVIEW)) const briefLoading = computed(() => genKeyMatches(K_BRIEF)) const packLoading = computed(() => genKeyMatches(K_PACK)) const viewInFlightOtherJobId = computed(() => { - const k = genInFlight.value - if (!k) return null - const i = k.lastIndexOf(':') - if (i < 0) return null - const jid = k.slice(i + 1) - if (jid === selectedId.value) return null - return jid + const sid = selectedId.value + if (!sid) return null + for (const k of genInFlight.value) { + const i = k.lastIndexOf(':') + if (i < 0) continue + const jid = k.slice(i + 1) + if (jid && jid !== sid) return jid + } + return null }) const successJobs = computed(() => @@ -70,6 +210,8 @@ const selectedJob = computed(() => successJobs.value.find((j) => String(j.id) === selectedId.value), ) +const briefHumanRows = computed(() => briefHumanSummary(briefData.value)) + async function loadList() { try { await refreshJobs() @@ -105,6 +247,7 @@ async function loadReport() { async function loadCompetitorBrief() { briefJson.value = '' + briefData.value = null briefErr.value = '' briefCopyOk.value = false const id = selectedId.value @@ -118,11 +261,12 @@ async function loadCompetitorBrief() { const j = JSON.parse(text) briefErr.value = j.detail || text } catch { - briefErr.value = text || `HTTP ${r.status}` + briefErr.value = text || `请求失败（${r.status}）` } return } const j = JSON.parse(text) + briefData.value = j briefJson.value = JSON.stringify(j, null, 2) } catch (e) { briefErr.value = String(e) @@ -174,6 +318,7 @@ onMounted(loadList) watch(selectedId, async () => { briefJson.value = '' + briefData.value = null briefErr.value = '' packErr.value = '' const id = selectedId.value @@ -206,9 +351,9 @@ watch( <h2>分析报告查看</h2> <p class="hint-top"> 选择<strong>已成功</strong>的任务，在线阅读报告或下载。 - 流水线生成报告时会<strong>自动</strong>基于<strong>全部评价正文</strong>分块调用大模型扩展关注词，并写入统计图（PNG，见「二点五」章与简报包 <code>report_assets</code>）。 - <strong>一键下载简报包</strong>含报告稿、统计图、结构化 JSON、要点摘录。 - 需要改规则或重算，请至 + 若开启大模型，系统会在后台根据评价正文补充<strong>关注词</strong>与<strong>使用场景</strong>标签，并生成报告中的统计图（与报告插图章节对应）。 + <strong>一键下载简报包</strong>内含：报告正文、插图文件夹、机器整理的<strong>数据摘要</strong>、以及便于扫读的<strong>要点摘录</strong>。 + 需要改分析规则或重新出稿，请至 <RouterLink to="/jd/analysis-build">报告生成</RouterLink>。 </p> @@ -233,40 +378,36 @@ watch( > 下载报告 </a> - <a - class="ma-btn ma-btn-secondary dl-link" - :class="{ disabled: !selectedId }" - :href="selectedId ? jobExportReportDocumentUrl(selectedId, 'docx') : '#'" - target="_blank" - rel="noreferrer" - @click="(e) => { if (!selectedId) e.preventDefault() }" + <button + type="button" + class="ma-btn ma-btn-secondary" + :disabled="!selectedId || exportDocFmt || loading" + @click="exportReportFmt('docx')" > - 导出 Word - </a> - <a - class="ma-btn ma-btn-secondary dl-link" - :class="{ disabled: !selectedId }" - :href="selectedId ? jobExportReportDocumentUrl(selectedId, 'pdf') : '#'" - target="_blank" - rel="noreferrer" - @click="(e) => { if (!selectedId) e.preventDefault() }" + {{ exportDocFmt === 'docx' ? '导出中…' : '导出 Word' }} + </button> + <button + type="button" + class="ma-btn ma-btn-secondary" + :disabled="!selectedId || exportDocFmt || loading" + @click="exportReportFmt('pdf')" > - 导出 PDF - </a> + {{ exportDocFmt === 'pdf' ? '导出中…' : '导出 PDF' }} + </button> <button type="button" class="ma-btn ma-btn-secondary" :disabled="!selectedId || briefLoading || loading" - title="生成与报告相同统计口径的结构化数据" + title="加载与报告数字一致的数据摘要（可先读易读版，再展开原始格式）" @click="loadCompetitorBrief" > - {{ briefLoading ? '摘要加载中…' : '加载结构化摘要' }} + {{ briefLoading ? '摘要加载中…' : '加载数据摘要' }} </button> <button type="button" class="ma-btn ma-btn-primary" :disabled="!selectedId || packLoading || loading || briefLoading" - title="ZIP：报告稿、结构化数据、要点摘录、说明" + title="下载压缩包：报告、配图、数据与说明" @click="downloadBriefPack" > {{ packLoading ? '打包中…' : '一键下载简报包' }} @@ -277,27 +418,40 @@ watch( </p> <p v-if="selectedJob?.run_dir" class="run-dir-note ma-muted"> - 本任务输出目录（原始表格复核请至「库内数据浏览」）：<span class="run-dir-path">{{ selectedJob.run_dir }}</span> + 本任务在本机上的结果文件夹（表格明细可在「库内数据浏览」查看）：<span class="run-dir-path">{{ selectedJob.run_dir }}</span> </p> <p v-if="briefErr" class="ma-err">{{ briefErr }}</p> <p v-if="packErr" class="ma-err">{{ packErr }}</p> + <p v-if="exportDocErr" class="ma-err">{{ exportDocErr }}</p> <p v-if="err" class="ma-err">{{ err }}</p> <p v-if="!successJobs.length" class="ma-muted">暂无成功任务，请先在「搜索采集」跑通一条流水线。</p> </section> - <section v-if="briefJson" class="ma-card preview-card"> + <section v-if="briefData" class="ma-card preview-card"> <div class="preview-head"> - <h2>结构化竞品摘要</h2> + <h2>竞品数据摘要（机器整理）</h2> <div class="tabs"> <button type="button" class="ma-btn ma-btn-secondary brief-tool" @click="copyBriefJson"> - {{ briefCopyOk ? '已复制' : '复制' }} + {{ briefCopyOk ? '已复制' : '复制原始数据' }} </button> - <button type="button" class="ma-btn ma-btn-secondary brief-tool" @click="downloadBriefJson">下载文件</button> + <button type="button" class="ma-btn ma-btn-secondary brief-tool" @click="downloadBriefJson">下载数据文件</button> </div> </div> - <p class="hint-top brief-hint">与上方报告统计口径一致的数据汇总，可复制或下载给其它工具使用。</p> - <pre class="raw-md brief-json">{{ briefJson }}</pre> + <p class="hint-top brief-hint"> + 以下数字与上方报告一致，用日常用语列出；需要交给其它系统或技术人员时，可展开下方「原始数据」或复制/下载。 + </p> + <dl v-if="briefHumanRows.length" class="brief-dl"> + <template v-for="(row, idx) in briefHumanRows" :key="idx"> + <dt>{{ row.label }}</dt> + <dd>{{ row.value }}</dd> + </template> + </dl> + <p v-else class="ma-muted brief-hint">暂无摘要条目（可能缺少列表或品牌字段）。</p> + <details class="brief-raw-wrap"> + <summary>展开原始数据（机器可读格式）</summary> + <pre class="raw-md brief-json">{{ briefJson }}</pre> + </details> </section> <section v-if="reportMd" class="ma-card preview-card"> @@ -445,4 +599,38 @@ watch( .brief-json { max-height: min(50vh, 560px); } +.brief-dl { + margin: 0.5rem 0 1rem; + display: grid; + grid-template-columns: minmax(10rem, 38%) 1fr; + gap: 0.35rem 1rem; + font-size: 0.9rem; + line-height: 1.45; +} +.brief-dl dt { + margin: 0; + font-weight: 600; + color: #374151; +} +.brief-dl dd { + margin: 0; + color: #1f2937; + word-break: break-word; +} +.brief-raw-wrap { + margin-top: 0.75rem; + border: 1px solid #e5e7eb; + border-radius: 8px; + padding: 0.5rem 0.75rem; + background: #fafafa; +} +.brief-raw-wrap summary { + cursor: pointer; + font-size: 0.88rem; + color: #4b5563; + user-select: none; +} +.brief-raw-wrap .brief-json { + margin-top: 0.75rem; +} </style> diff --git a/frontend/src/views/jd/JdDatasetBrowseView.vue b/frontend/src/views/jd/JdDatasetBrowseView.vue index 3b2be8e..6f1de77 100644 --- a/frontend/src/views/jd/JdDatasetBrowseView.vue +++ b/frontend/src/views/jd/JdDatasetBrowseView.vue @@ -73,9 +73,8 @@ watch(selectedId, () => { <button type="button" class="ma-btn ma-btn-secondary btn-refresh" @click="load">刷新任务列表</button> </div> <p class="lead"> - 选择任务后浏览已入库的搜索、商详、评价与<strong>整合宽表</strong>（合并表按列拆分入库，与 - <code>keyword_pipeline_merged.csv</code> lean 列一致）；各 Tab 下「导出当前表」可导出 JSON / CSV / - Excel。竞品报告请在「报告查看」阅读或「报告生成」重新生成。 + 选择任务后可查看本批已入库的<strong>搜索列表</strong>、<strong>商品详情</strong>、<strong>评价</strong>与<strong>整合同步表</strong>（一行对应一个商品在报告里用到的主要字段）。 + 各标签下可**导出**为表格或数据文件。完整文字报告请在「报告查看」阅读，或在「报告生成」中重新出稿。 </p> <p v-if="loadError" class="ma-err">{{ loadError }}</p> diff --git a/frontend/src/views/jd/JdStrategyBuildView.vue b/frontend/src/views/jd/JdStrategyBuildView.vue index 362fbc5..4ccb223 100644 --- a/frontend/src/views/jd/JdStrategyBuildView.vue +++ b/frontend/src/views/jd/JdStrategyBuildView.vue @@ -1,11 +1,16 @@ <script setup> -import { computed, onMounted, reactive, ref, watch } from 'vue' +import { computed, onMounted, onUnmounted, reactive, ref, watch } from 'vue' import { useRoute, useRouter, RouterLink } from 'vue-router' import { refreshJobs, useJobs, api } from '../../composables/useJobs' import { generationInFlightKey, withGenerationInFlight, } from '../../composables/useGenerationInFlight' +import { + loadStrategyMatrixScope, + saveStrategyDraftRecord, + saveStrategyMatrixScope, +} from '../../lib/strategyDraftStorage' const route = useRoute() const router = useRouter() @@ -17,10 +22,12 @@ const err = ref('') const genInFlight = generationInFlightKey() const STRATEGY_PREFIX = 'strategy-draft:' const strategyDraftPendingJobId = computed(() => { - const k = genInFlight.value - if (!k || !k.startsWith(STRATEGY_PREFIX)) return null - return k.slice(STRATEGY_PREFIX.length) + for (const k of genInFlight.value) { + if (k.startsWith(STRATEGY_PREFIX)) return k.slice(STRATEGY_PREFIX.length) + } + return null }) +const strategyGeneratingAny = computed(() => strategyDraftPendingJobId.value != null) const strategyGeneratingThisTask = computed( () => strategyDraftPendingJobId.value != null && @@ -31,10 +38,18 @@ const strategyGeneratingOtherTask = computed( strategyDraftPendingJobId.value != null && strategyDraftPendingJobId.value !== selectedId.value, ) -const useLlm = ref(false) +/** 勾选则本次仅规则稿（不调用大模型）；默认不勾选即走大模型 */ +const rulesOnlyThisRun = ref(false) + +/** 与竞品矩阵细类一致；空字符串表示不收窄（全关键词样本） */ +const strategyMatrixScope = ref('') +const matrixGroups = ref([]) +const briefMatrixLoading = ref(false) +const briefMatrixErr = ref('') const decisions = reactive({ product_role: '', + stage_goal_type: '', time_horizon: '', success_criteria: '', non_goals: '', @@ -45,6 +60,11 @@ const decisions = reactive({ pillar_price: '', pillar_channel: '', pillar_comm: '', + audience_segment: '', + competitor_reference: '', + resource_notes: '', + marketing_strategy: '', + general_strategy: '', ack_risk_keywords: false, ack_risk_price: false, ack_risk_concentration: false, @@ -54,10 +74,6 @@ const successJobs = computed(() => [...jobs.value].filter((j) => j.status === 'success').sort((a, b) => b.id - a.id), ) -const selectedJob = computed(() => - successJobs.value.find((j) => String(j.id) === selectedId.value), -) - const positioningOptions = [ { value: '', label: '暂不勾选（文稿中均为空选）' }, { value: 'top', label: '贴顶' }, @@ -66,19 +82,22 @@ const positioningOptions = [ { value: 'different', label: '另起带' }, ] +/** 对应后端 competitive_stance：与头部或主竞品「怎么打」，非价位阵地、亦非泛指的「进市场」。 */ const stanceOptions = [ { value: '', label: '暂不填写' }, - { value: 'flank', label: '倾向侧翼切入' }, - { value: 'head_on', label: '倾向正面替代' }, - { value: 'both', label: '分层推进（侧翼 + 正面）' }, + { value: 'flank', label: '侧翼切入（避开头部主战场）' }, + { value: 'head_on', label: '正面替代（对标头部主战场）' }, + { value: 'both', label: '分层推进（侧翼 + 正面并行）' }, { value: 'undecided', label: '尚未拍板' }, ] function buildPayload() { + const generator = rulesOnlyThisRun.value ? 'rules' : 'llm' return { - generator: useLlm.value ? 'llm' : 'rules', + generator, business_notes: businessNotes.value, product_role: decisions.product_role, + stage_goal_type: decisions.stage_goal_type, time_horizon: decisions.time_horizon, success_criteria: decisions.success_criteria, non_goals: decisions.non_goals, @@ -89,13 +108,25 @@ function buildPayload() { pillar_price: decisions.pillar_price, pillar_channel: decisions.pillar_channel, pillar_comm: decisions.pillar_comm, + audience_segment: decisions.audience_segment, + competitor_reference: decisions.competitor_reference, + resource_notes: decisions.resource_notes, + marketing_strategy: decisions.marketing_strategy, + general_strategy: decisions.general_strategy, ack_risk_keywords: decisions.ack_risk_keywords, ack_risk_price: decisions.ack_risk_price, ack_risk_concentration: decisions.ack_risk_concentration, + ...(strategyMatrixScope.value + ? { strategy_matrix_group: strategyMatrixScope.value } + : {}), } } -const STORAGE_KEY = (id) => `ma_strategy_draft_${id}` +function formatJobOption(j) { + const t = j.created_at + const tail = t ? String(t).replace('T', ' ').slice(0, 16) : '' + return tail ? `#${j.id} · ${j.keyword} · ${tail}` : `#${j.id} · ${j.keyword}` +} async function loadList() { try { @@ -105,6 +136,37 @@ async function loadList() { } } +async function loadMatrixGroupsForJob(id) { + matrixGroups.value = [] + strategyMatrixScope.value = '' + briefMatrixErr.value = '' + if (!id) return + briefMatrixLoading.value = true + try { + const r = await api(`/api/jobs/${id}/competitor-brief/`) + const text = await r.text() + if (!r.ok) { + try { + briefMatrixErr.value = JSON.parse(text).detail || text + } catch { + briefMatrixErr.value = text || `HTTP ${r.status}` + } + return + } + const data = JSON.parse(text) + const mg = data.matrix_groups + matrixGroups.value = Array.isArray(mg) ? mg : [] + const saved = loadStrategyMatrixScope(id) + if (saved && matrixGroups.value.some((g) => g.group === saved)) { + strategyMatrixScope.value = saved + } + } catch (e) { + briefMatrixErr.value = String(e) + } finally { + briefMatrixLoading.value = false + } +} + async function generateAndGoPreview() { const id = selectedId.value if (!id) return @@ -127,14 +189,12 @@ async function generateAndGoPreview() { return } const j = JSON.parse(text) - sessionStorage.setItem( - STORAGE_KEY(id), - JSON.stringify({ - markdown: j.markdown || '', - keyword: j.keyword || '', - generated_at: j.generated_at || '', - }), - ) + saveStrategyDraftRecord(id, { + markdown: j.markdown || '', + keyword: j.keyword || '', + generated_at: j.generated_at || '', + last_request: buildPayload(), + }) router.push({ path: '/jd/strategy-view', query: { job: id } }) } catch (e) { err.value = String(e) @@ -142,7 +202,41 @@ async function generateAndGoPreview() { }) } -onMounted(loadList) +function onStorageScopeSync(ev) { + const prefix = 'ma_strategy_scope_' + if (!ev.key || !ev.key.startsWith(prefix)) return + const jid = ev.key.slice(prefix.length) + if (jid !== String(selectedId.value)) return + const v = loadStrategyMatrixScope(jid) + if (v && matrixGroups.value.some((g) => g.group === v)) { + strategyMatrixScope.value = v + } else if (!v) { + strategyMatrixScope.value = '' + } +} + +onMounted(() => { + loadList() + if (typeof window !== 'undefined') { + window.addEventListener('storage', onStorageScopeSync) + } +}) + +onUnmounted(() => { + if (typeof window !== 'undefined') { + window.removeEventListener('storage', onStorageScopeSync) + } +}) + +watch(selectedId, (id) => { + loadMatrixGroupsForJob(id) +}) + +watch(strategyMatrixScope, (v) => { + const jid = selectedId.value + if (!jid) return + saveStrategyMatrixScope(jid, v) +}) watch( () => route.query.job, @@ -168,17 +262,15 @@ watch( <section class="ma-card"> <h2>策略生成</h2> <p class="hint-top"> - 选择<strong>已成功</strong>任务，在下方填空与勾选。<strong>未勾选</strong>下方选项时，由系统规则生成策略底稿；<strong>勾选「使用大模型生成」</strong>后，由大模型在底稿与数据摘要基础上成稿（服务端需已配置并可用）。提交后跳转到 - <RouterLink to="/jd/strategy-view">策略稿预览</RouterLink> - 。数据与 - <RouterLink to="/jd/analysis-view">同任务分析产出</RouterLink> - 一致。未填项在文稿中仍保留占位提示。 + 选择<strong>已成功</strong>任务，先选顶部<strong>矩阵细类</strong>（主推类目，与报告矩阵一致）。策略稿与矩阵选择保存在本机 <strong>localStorage</strong>，同域名下可跨标签查看；与其它页面的耗时任务通过全局任务锁同步。下方字段按策略文档常见顺序排列；成稿里的小节标题与编号由系统自动对应。有关痛点、购买理由、品牌承诺等由监测与模型撰写，本页主要收集<strong>业务决策与战术要点</strong>。生成结果见 + <RouterLink to="/jd/strategy-view">策略稿预览</RouterLink>。<strong>已填项</strong>进入底稿并由大模型落实；<strong>未填项</strong>可由模型结合数据推断。 </p> + <div class="toolbar"> <label class="chk-inline"> - <input v-model="useLlm" type="checkbox" /> - 使用大模型生成（服务端需已配置并可用） + <input v-model="rulesOnlyThisRun" type="checkbox" /> + 本次仅生成规则稿（不做大模型全文润色，更快、不调用智能服务） </label> </div> <div class="toolbar"> @@ -186,64 +278,131 @@ watch( <select v-model="selectedId" class="job-select"> <option value="" disabled>请选择任务</option> <option v-for="j in successJobs" :key="j.id" :value="String(j.id)"> - #{{ j.id }} · {{ j.keyword }} · {{ j.run_dir?.split(/[/\\]/).pop() || '' }} + {{ formatJobOption(j) }} </option> </select> <button type="button" class="ma-btn ma-btn-primary" - :disabled="!selectedId || strategyGeneratingThisTask" + :disabled="!selectedId || strategyGeneratingAny || briefMatrixLoading" @click="generateAndGoPreview" > {{ strategyGeneratingThisTask ? '生成中…' : '生成并前往预览' }} </button> </div> + <div v-if="selectedId" class="toolbar toolbar-stack"> + <label class="sel-label">主推类目（矩阵细类）</label> + <select + v-model="strategyMatrixScope" + class="job-select" + :disabled="briefMatrixLoading || strategyGeneratingAny" + > + <option value="">全部分类（不收窄 · 与全关键词监测样本一致）</option> + <option v-for="g in matrixGroups" :key="g.index" :value="g.group"> + {{ g.group }}（{{ g.sku_count }} 款） + </option> + </select> + <span v-if="briefMatrixLoading" class="ma-muted">正在加载矩阵分组…</span> + <span v-else class="ma-muted ma-hint-sub" + >与策略稿中的主推类目及报告矩阵一致；收窄后监测摘要与报告节选仅针对该细类。</span + > + </div> + <p v-if="briefMatrixErr" class="ma-err">{{ briefMatrixErr }}</p> <p v-if="strategyGeneratingOtherTask" class="ma-warn-banner"> 任务 #{{ strategyDraftPendingJobId }} 的策略稿正在生成中，请稍候再切换任务或重复提交。 </p> - - <p v-if="selectedJob?.run_dir" class="run-dir-note ma-muted"> - 任务目录：<span class="run-dir-path">{{ selectedJob.run_dir }}</span> - </p> <p v-if="err" class="ma-err">{{ err }}</p> <p v-if="!successJobs.length" class="ma-muted">暂无成功任务，请先在「搜索采集」跑通一条流水线。</p> <fieldset class="fieldset"> - <legend>一、战略背景与目标</legend> + <legend>策略范围与前提</legend> + <p class="fieldset-hint"> + 界定本次策略的任务边界与阶段目标。监测词、批次由任务自动带出；<strong>主推类目</strong>以顶部「矩阵细类」为准。角色、<strong>本阶段策略目标类型</strong>、战场、客群会进入策略稿开篇；目标类型填好后，成稿会按你的表述落实。并与下一栏「主要对标」衔接。 + </p> <label class="fld"> - <span>本品角色</span> - <input v-model="decisions.product_role" type="text" placeholder="如：追赶 / 新品 / 防守" /> - </label> - <label class="fld"> - <span>时间范围</span> - <input v-model="decisions.time_horizon" type="text" placeholder="如：本季度 / 未来 12 周" /> + <span>本品角色（策略服务对象）</span> + <input + v-model="decisions.product_role" + type="text" + placeholder="如：追赶型 / 新品 / 防守 / 拓品类" + /> </label> <label class="fld fld-block"> - <span>成功标准（可量化）</span> - <textarea v-model="decisions.success_criteria" rows="2" placeholder="如：搜索位次、转化率…" /> + <span>本阶段策略目标类型</span> + <textarea + v-model="decisions.stage_goal_type" + rows="2" + placeholder="如：让更多人愿意尝试购买、把销量和转化做起来、稳住老顾客和份额、先验证新品是否卖得动……按你公司本阶段真实目标写一句即可；不填则由系统在成稿中结合数据推断" + /> </label> - <label class="fld fld-block"> - <span>非目标</span> - <textarea v-model="decisions.non_goals" rows="2" placeholder="明确不做什么（可选）" /> - </label> - </fieldset> - - <fieldset class="fieldset"> - <legend>二、战场（一句话）</legend> <label class="fld fld-block"> <span>一句话战场</span> <textarea v-model="decisions.battlefield_one_line" rows="2" - placeholder="在哪个需求场景、与谁抢同一批用户？" + placeholder="在什么需求场景、与谁争夺同一批检索与购买用户" + /> + </label> + <label class="fld fld-block"> + <span>目标客群 / 场景</span> + <input + v-model="decisions.audience_segment" + type="text" + placeholder="为谁、在什么情境下买（可选）" + /> + </label> + <label class="fld"> + <span>时间范围</span> + <input + v-model="decisions.time_horizon" + type="text" + placeholder="如：本季度 / 未来 12 周（与后文阶段目标一致）" + /> + </label> + <label class="fld fld-block"> + <span>成功标准（可量化）</span> + <textarea + v-model="decisions.success_criteria" + rows="2" + placeholder="如：搜索位次、转化、复购等可验证指标" + /> + </label> + <label class="fld fld-block"> + <span>非目标</span> + <textarea + v-model="decisions.non_goals" + rows="2" + placeholder="本阶段明确不做的边界（可选）" /> </label> </fieldset> <fieldset class="fieldset"> - <legend>三、竞争态势自判</legend> + <legend>本品聚焦 · 主要对标</legend> + <p class="fieldset-hint"> + 角色与客群已在上文填写；此处补充<strong>主要对标</strong>（品牌或价位参照），便于后文写差异与竞争应对时对齐同一参照系。 + </p> <label class="fld fld-block"> - <span>本品倾向</span> + <span>主要对标</span> + <input + v-model="decisions.competitor_reference" + type="text" + placeholder="如：具体头部品牌、或同价位标杆；与上文战场一致时最有效。可写「待业务指定」或留空" + /> + </label> + </fieldset> + + <div class="form-skip-note" role="note"> + <strong>自动撰写部分</strong>：用户痛点、购买理由、品牌承诺与调性等内容<strong>不在本页填写</strong>，将由监测摘要、报告节选与大模型写入策略稿；可通过顶部矩阵收窄与文末「业务备注」影响范围。 + </div> + + <fieldset class="fieldset"> + <legend>与竞品的应对方式</legend> + <p class="fieldset-hint"> + 面对头部或主竞品时，优先<strong>侧翼</strong>还是<strong>正面</strong>等。下方「价位阵地」回答在哪条价格带上打，与这里不是一回事。 + </p> + <label class="fld fld-block"> + <span>面对竞品时的主打法</span> <select v-model="decisions.competitive_stance" class="job-select full"> <option v-for="o in stanceOptions" :key="o.value || 'empty'" :value="o.value"> {{ o.label }} @@ -253,46 +412,95 @@ watch( </fieldset> <fieldset class="fieldset"> - <legend>四、价格带定位选项（勾选一条）</legend> + <legend>阶段目标与路径（补充）</legend> + <p class="fieldset-hint"> + 上文「时间、成功标准、非目标」会进入阶段定义；此处填写营销策略、总体策略与资源备注。尽量用<strong>可执行的动词句</strong>，并与痛点动作方向一致（多细类可分句）。 + </p> <label class="fld fld-block"> - <span>主定位</span> + <span>营销策略</span> + <textarea + v-model="decisions.marketing_strategy" + rows="3" + placeholder="传播、活动、投放、内容主线；写清阶段重点而非口号（可选）" + /> + </label> + <label class="fld fld-block"> + <span>总体策略</span> + <textarea + v-model="decisions.general_strategy" + rows="3" + placeholder="增长 / 品类 / 经营总原则；与上文战场与非目标不矛盾（可选）" + /> + </label> + <label class="fld fld-block"> + <span>资源与预算备注</span> + <textarea + v-model="decisions.resource_notes" + rows="2" + placeholder="人力、投放、产能约束；便于成稿写节奏与优先级（可选）" + /> + </label> + </fieldset> + + <fieldset class="fieldset"> + <legend>品牌四线与战术动作</legend> + <p class="fieldset-hint"> + 下列内容会在策略稿中用于<strong>品牌四线</strong>与<strong>战术支柱</strong>相关段落（系统会自动落到对应小节）。价位阵地为单选；促销与活动细节无单独表单项，由监测与模型归纳。品牌承诺与调性由模型依据数据撰写。 + </p> + <label class="fld fld-block"> + <span>产品</span> + <textarea + v-model="decisions.pillar_product" + rows="2" + placeholder="规格、配方或功能叙事、计划中的产品动作（可选）" + /> + </label> + <label class="fld fld-block"> + <span>价位阵地（单选）</span> <select v-model="decisions.positioning_choice" class="job-select full"> <option v-for="o in positioningOptions" :key="o.value || 'empty'" :value="o.value"> {{ o.label }} </option> </select> </label> - </fieldset> - - <fieldset class="fieldset"> - <legend>六、策略支柱 — 本品打算怎么做（可先填一列）</legend> <label class="fld fld-block"> - <span>产品</span> - <textarea v-model="decisions.pillar_product" rows="2" /> + <span>定价（补充说明）</span> + <textarea + v-model="decisions.pillar_price" + rows="2" + placeholder="在价位阵地之外：到手价呈现、跟价或避战原则、与大促关系等（可选）" + /> </label> <label class="fld fld-block"> - <span>价格</span> - <textarea v-model="decisions.pillar_price" rows="2" /> - </label> - <label class="fld fld-block"> - <span>渠道 / 触点</span> - <textarea v-model="decisions.pillar_channel" rows="2" /> + <span>渠道与触点</span> + <textarea + v-model="decisions.pillar_channel" + rows="2" + placeholder="货架、店铺类型、站内路径、触点优先级等（可选）" + /> </label> <label class="fld fld-block"> <span>传播与内容</span> - <textarea v-model="decisions.pillar_comm" rows="2" /> + <textarea + v-model="decisions.pillar_comm" + rows="2" + placeholder="内容形态、达人/自播、搜索承接与话术方向等（可选）" + /> </label> </fieldset> <fieldset class="fieldset"> - <legend>七、风险确认（已知晓则勾选）</legend> + <legend>数据与样本风险（确认知晓）</legend> + <p class="fieldset-hint"> + 勾选表示了解以下数据局限（不影响生成，仅供自检）。 + </p> <label class="chk"> <input v-model="decisions.ack_risk_keywords" type="checkbox" /> 关注词 / 场景可能以偏概全（需原评论抽样） </label> <label class="chk"> <input v-model="decisions.ack_risk_price" type="checkbox" /> - 价格带可能含大促或异常挂价（需核对口径） + 价格带可能含大促或异常挂价（需核对清洗与计价规则） </label> <label class="chk"> <input v-model="decisions.ack_risk_concentration" type="checkbox" /> @@ -301,13 +509,16 @@ watch( </fieldset> <fieldset class="fieldset"> - <legend>八、业务约束与内部判断</legend> + <legend>业务备注</legend> + <p class="fieldset-hint"> + 法务红线、渠道约束、组织与预算等自由补充，会进入策略稿收尾部分；不替换正文结构，也不替代上方已填的决策字段。 + </p> <label class="fld fld-block"> <span>业务备注</span> <textarea v-model="businessNotes" rows="4" - placeholder="渠道红线、价位策略、竞品对标、预算量级等" + placeholder="如法务/合规表述边界、渠道限价、禁止对标表述、预算与人力硬约束等（可选）" /> </label> </fieldset> @@ -322,6 +533,12 @@ watch( color: #4b5563; line-height: 1.55; } +.hint-flow { + margin-top: -0.6rem; + padding-top: 0.5rem; + border-top: 1px solid #e5e7eb; + font-size: 0.84rem; +} .hint-top a, .hint-top :deep(a) { color: #2563eb; @@ -334,6 +551,13 @@ watch( gap: 0.75rem; margin-bottom: 0.75rem; } +.toolbar-stack { + flex-direction: column; + align-items: stretch; +} +.toolbar-stack .sel-label { + margin-bottom: -0.25rem; +} .sel-label { font-size: 0.85rem; font-weight: 500; @@ -352,18 +576,6 @@ watch( min-width: 0; box-sizing: border-box; } -.run-dir-note { - margin: 0.75rem 0 0; - font-size: 0.8rem; - line-height: 1.5; -} -.run-dir-path { - display: block; - margin-top: 0.35rem; - font-size: 0.75rem; - word-break: break-all; - color: #475569; -} .ma-muted { color: #64748b; } @@ -390,6 +602,18 @@ watch( font-weight: 600; color: #1f2937; } +.fieldset-hint { + margin: 0 0 0.5rem; + font-size: 0.8rem; + line-height: 1.5; + color: #6b7280; +} +.ma-hint-sub { + display: block; + margin-top: 0.35rem; + font-size: 0.8rem; + line-height: 1.45; +} .fld { display: flex; flex-direction: column; @@ -448,4 +672,17 @@ watch( .chk-inline input { margin-top: 0.2rem; } +.form-skip-note { + margin: 1rem 0 0; + padding: 0.65rem 0.85rem; + font-size: 0.82rem; + line-height: 1.5; + color: #4b5563; + background: #f1f5f9; + border: 1px solid #e2e8f0; + border-radius: 8px; +} +.form-skip-note strong { + color: #334155; +} </style> diff --git a/frontend/src/views/jd/JdStrategyView.vue b/frontend/src/views/jd/JdStrategyView.vue index eaa0cd2..07f595a 100644 --- a/frontend/src/views/jd/JdStrategyView.vue +++ b/frontend/src/views/jd/JdStrategyView.vue @@ -1,21 +1,71 @@ <script setup> -import { computed, onMounted, ref, watch } from 'vue' +import { computed, onMounted, onUnmounted, ref, watch } from 'vue' import { useRoute, useRouter, RouterLink } from 'vue-router' import MarkdownPreview from '../../components/MarkdownPreview.vue' -import { refreshJobs, useJobs, exportStrategyDocument } from '../../composables/useJobs' +import { + api, + refreshJobs, + useJobs, + exportStrategyDocument, +} from '../../composables/useJobs' +import { generationInFlightKey, withGenerationInFlight } from '../../composables/useGenerationInFlight' +import { loadStrategyDraftRecord } from '../../lib/strategyDraftStorage' +import { marketingPackResultToMarkdown } from '../../lib/marketingPackMarkdown' const route = useRoute() const router = useRouter() const { jobs } = useJobs() +const genInFlight = generationInFlightKey() + const selectedId = ref('') const draftMd = ref('') const draftMeta = ref(null) const viewMode = ref('render') const exportErr = ref('') -const exportBusy = ref(false) +const marketingErr = ref('') +const marketingExportErr = ref('') +const marketingResult = ref(null) -const STORAGE_KEY = (id) => `ma_strategy_draft_${id}` +const exportBusy = computed(() => { + const id = selectedId.value + if (!id) return false + return genInFlight.value.some((k) => String(k).startsWith(`export-strategy:${id}:`)) +}) + +const marketingBusy = computed(() => { + const id = selectedId.value + if (!id) return false + return genInFlight.value.includes(`marketing-detail-pack:${id}`) +}) + +const marketingExportBusy = computed(() => { + const id = selectedId.value + if (!id) return false + return genInFlight.value.some((k) => String(k).startsWith(`export-marketing-pack:${id}:`)) +}) + +function isMarketingExporting(fmt) { + const id = selectedId.value + if (!id) return false + return genInFlight.value.includes(`export-marketing-pack:${id}:${fmt}`) +} + +function payloadForMarketing(lastRequest) { + if (!lastRequest || typeof lastRequest !== 'object') { + return { business_notes: '', strategy_decisions: {} } + } + const { + generator: _g, + business_notes: bn, + strategy_matrix_group: _mg, + ...rest + } = lastRequest + return { + business_notes: (bn || '').trim(), + strategy_decisions: rest, + } +} const successJobs = computed(() => [...jobs.value].filter((j) => j.status === 'success').sort((a, b) => b.id - a.id), @@ -25,6 +75,44 @@ const selectedJob = computed(() => successJobs.value.find((j) => String(j.id) === selectedId.value), ) +function pickDetailPackSubset(pack, keys) { + if (!pack || typeof pack !== 'object') return null + const o = {} + for (const k of keys) { + if (Object.prototype.hasOwnProperty.call(pack, k)) o[k] = pack[k] + } + return Object.keys(o).length ? o : null +} + +/** 列表/详情页主文案（与「触点」分开展示） */ +const marketingPackDetailList = computed(() => + pickDetailPackSubset(marketingResult.value?.detail_page_pack, [ + 'listing_titles', + 'listing_subtitle', + 'detail_headline', + 'detail_mid_story_paragraphs', + 'selling_bullets', + 'usage_and_pairing_tips', + 'spec_sidebar_lines', + 'faq', + 'short_graphic_post_variants', + ]), +) + +/** 依据、主图要点、文生图/文生视频提示词、短视频钩句、客服 */ +const marketingPackTouchBlock = computed(() => + pickDetailPackSubset(marketingResult.value?.detail_page_pack, [ + 'traceability_note', + 'main_image_three_points', + 'text_to_image_prompt_main', + 'text_to_image_prompt_scene', + 'text_to_video_prompt', + 'live_or_short_hook', + 'live_script_bullets', + 'customer_service_opening', + ]), +) + function loadDraft() { const id = selectedId.value if (!id) { @@ -33,17 +121,17 @@ function loadDraft() { return } try { - const raw = sessionStorage.getItem(STORAGE_KEY(id)) - if (!raw) { + const o = loadStrategyDraftRecord(id) + if (!o) { draftMd.value = '' draftMeta.value = null return } - const o = JSON.parse(raw) draftMd.value = o.markdown || '' draftMeta.value = { keyword: o.keyword || '', generated_at: o.generated_at || '', + last_request: o.last_request || null, } } catch { draftMd.value = '' @@ -73,16 +161,86 @@ function downloadDraftMd() { URL.revokeObjectURL(u) } +function downloadMarketingPackJson() { + if (!marketingResult.value || !selectedId.value) return + const blob = new Blob([JSON.stringify(marketingResult.value, null, 2)], { + type: 'application/json;charset=utf-8', + }) + const u = URL.createObjectURL(blob) + const a = document.createElement('a') + a.href = u + a.download = `job_${selectedId.value}_marketing_detail_pack.json` + a.rel = 'noopener' + document.body.appendChild(a) + a.click() + a.remove() + URL.revokeObjectURL(u) +} + +async function exportMarketingPackFmt(fmt) { + if (!marketingResult.value || !selectedId.value) return + marketingExportErr.value = '' + const id = selectedId.value + const md = marketingPackResultToMarkdown(marketingResult.value) + if (!md.trim()) { + marketingExportErr.value = '无可导出的营销内容' + return + } + try { + await withGenerationInFlight(`export-marketing-pack:${id}:${fmt}`, async () => { + await exportStrategyDocument(id, md, fmt, 'marketing_detail') + }) + } catch (e) { + marketingExportErr.value = String(e) + } +} + +async function generateMarketingDetailPack() { + if (!draftMd.value || !selectedId.value) return + marketingErr.value = '' + marketingExportErr.value = '' + marketingResult.value = null + const id = selectedId.value + const { business_notes, strategy_decisions } = payloadForMarketing( + draftMeta.value?.last_request, + ) + try { + await withGenerationInFlight(`marketing-detail-pack:${id}`, async () => { + const r = await api(`/api/jobs/${id}/marketing-detail-pack/`, { + method: 'POST', + body: JSON.stringify({ + strategy_markdown: draftMd.value, + business_notes, + strategy_decisions, + }), + }) + const text = await r.text() + if (!r.ok) { + try { + const j = JSON.parse(text) + marketingErr.value = j.detail || text + } catch { + marketingErr.value = text || `HTTP ${r.status}` + } + return + } + marketingResult.value = JSON.parse(text) + }) + } catch (e) { + marketingErr.value = String(e) + } +} + async function exportStrategyFmt(fmt) { if (!draftMd.value || !selectedId.value) return exportErr.value = '' - exportBusy.value = true + const id = selectedId.value try { - await exportStrategyDocument(selectedId.value, draftMd.value, fmt) + await withGenerationInFlight(`export-strategy:${id}:${fmt}`, async () => { + await exportStrategyDocument(id, draftMd.value, fmt) + }) } catch (e) { exportErr.value = String(e) - } finally { - exportBusy.value = false } } @@ -105,10 +263,26 @@ function syncSelectionFromRouteAndJobs() { } } +function onStorageDraftSync(ev) { + const prefix = 'ma_strategy_draft_' + if (!ev.key || !ev.key.startsWith(prefix)) return + const jid = ev.key.slice(prefix.length) + if (jid === String(selectedId.value)) loadDraft() +} + onMounted(async () => { await loadList() syncSelectionFromRouteAndJobs() loadDraft() + if (typeof window !== 'undefined') { + window.addEventListener('storage', onStorageDraftSync) + } +}) + +onUnmounted(() => { + if (typeof window !== 'undefined') { + window.removeEventListener('storage', onStorageDraftSync) + } }) watch( @@ -124,6 +298,8 @@ watch( ) watch(selectedId, (id) => { + marketingResult.value = null + marketingExportErr.value = '' loadDraft() const want = id ? String(id) : '' if (String(route.query.job || '') !== want) { @@ -146,7 +322,7 @@ watch(successJobs, (list) => { <section class="ma-card"> <h2>策略稿预览</h2> <p class="hint-top"> - 选择在<strong>策略生成</strong>页已生成过的任务查看文稿（保存在本浏览器会话内）。需要改决策请回到 + 选择在<strong>策略生成</strong>页已生成过的任务查看文稿（保存在本机浏览器 <strong>localStorage</strong>，同域名下可跨标签查看）。生成/导出/营销内容等耗时操作状态在全局任务锁中同步，跨标签页可看到进行中。需要改决策请回到 <RouterLink to="/jd/strategy-build">策略生成</RouterLink> 重新提交。分析数据见 <RouterLink to="/jd/analysis-view">报告查看</RouterLink>。 @@ -187,6 +363,14 @@ watch(successJobs, (list) => { <button type="button" class="ma-btn ma-btn-primary" @click="goBuildSameJob"> 去策略生成 </button> + <button + type="button" + class="ma-btn ma-btn-secondary" + :disabled="!draftMd || !selectedId || marketingBusy" + @click="generateMarketingDetailPack" + > + {{ marketingBusy ? '营销内容生成中…' : '生成营销内容' }} + </button> </div> <p v-if="draftMeta?.generated_at" class="meta-line ma-muted"> @@ -194,6 +378,60 @@ watch(successJobs, (list) => { <template v-if="draftMeta.keyword"> · 关键词：{{ draftMeta.keyword }}</template> </p> <p v-if="exportErr" class="ma-err">{{ exportErr }}</p> + <p v-if="marketingErr" class="ma-err">{{ marketingErr }}</p> + <p v-if="marketingExportErr" class="ma-err">{{ marketingExportErr }}</p> + <div v-if="marketingResult" class="marketing-pack-out"> + <h3 class="marketing-pack-h">营销内容</h3> + <p class="ma-muted marketing-pack-meta"> + {{ marketingResult.generated_at }} · {{ marketingResult.source }} + </p> + <p class="ma-muted marketing-pack-disk"> + 服务端会将本包写入任务目录 + <code>marketing/marketing_detail_pack_v1.json</code>（与批次一并归档；目录不可写时仅内存结果）。 + </p> + <div class="toolbar marketing-pack-actions"> + <button + type="button" + class="ma-btn ma-btn-secondary" + :disabled="!selectedId || marketingExportBusy || marketingBusy" + @click="downloadMarketingPackJson" + > + 下载 JSON + </button> + <button + type="button" + class="ma-btn ma-btn-secondary" + :disabled="!selectedId || marketingExportBusy || marketingBusy" + @click="exportMarketingPackFmt('docx')" + > + {{ isMarketingExporting('docx') ? '导出中…' : '营销内容导出 Word' }} + </button> + <button + type="button" + class="ma-btn ma-btn-secondary" + :disabled="!selectedId || marketingExportBusy || marketingBusy" + @click="exportMarketingPackFmt('pdf')" + > + {{ isMarketingExporting('pdf') ? '导出中…' : '营销内容导出 PDF' }} + </button> + </div> + <details open class="marketing-details"> + <summary>核心信息卡</summary> + <pre class="marketing-pre">{{ JSON.stringify(marketingResult.core_info_card, null, 2) }}</pre> + </details> + <details v-if="marketingPackDetailList" open class="marketing-details"> + <summary>列表与详情页主文案</summary> + <pre class="marketing-pre">{{ JSON.stringify(marketingPackDetailList, null, 2) }}</pre> + </details> + <details v-else-if="marketingResult.detail_page_pack" open class="marketing-details"> + <summary>详情页包字段</summary> + <pre class="marketing-pre">{{ JSON.stringify(marketingResult.detail_page_pack, null, 2) }}</pre> + </details> + <details v-if="marketingPackTouchBlock" open class="marketing-details"> + <summary>依据、主图、文生图/文生视频提示词、钩句与客服</summary> + <pre class="marketing-pre">{{ JSON.stringify(marketingPackTouchBlock, null, 2) }}</pre> + </details> + </div> <p v-if="selectedJob?.run_dir" class="run-dir-note ma-muted"> 任务目录：<span class="run-dir-path">{{ selectedJob.run_dir }}</span> </p> @@ -335,4 +573,63 @@ watch(successJobs, (list) => { border-radius: 8px; border: 1px solid #e5e7eb; } +.marketing-pack-out { + margin-top: 1rem; + padding: 0.85rem 1rem; + border: 1px solid #e2e8f0; + border-radius: 8px; + background: #f8fafc; +} +.marketing-pack-h { + margin: 0 0 0.35rem; + font-size: 1rem; + color: #1e293b; +} +.marketing-pack-meta { + margin: 0 0 0.75rem; + font-size: 0.8rem; +} +.marketing-pack-disk { + margin: 0 0 0.65rem; + font-size: 0.78rem; + line-height: 1.45; + max-width: 52rem; +} +.marketing-pack-disk code { + font-size: 0.85em; + background: #e2e8f0; + padding: 0.1em 0.35em; + border-radius: 4px; +} +.marketing-product-hint { + margin: 0.5rem 0 0; + font-size: 0.82rem; + line-height: 1.45; + max-width: 52rem; +} +.marketing-pack-actions { + margin: 0 0 0.75rem; + flex-wrap: wrap; + gap: 0.35rem; +} +.marketing-details { + margin-bottom: 0.65rem; +} +.marketing-details summary { + cursor: pointer; + font-weight: 600; + font-size: 0.88rem; + color: #334155; +} +.marketing-pre { + margin: 0.5rem 0 0; + padding: 0.65rem; + font-size: 0.75rem; + line-height: 1.45; + overflow: auto; + max-height: 320px; + background: #fff; + border: 1px solid #e5e7eb; + border-radius: 6px; +} </style>