devops-pdt613/market-assistant
1
0
Fork 0
mirror of https://github.com/primedigitaltech/market-assistant.git synced 2026-07-23 00:21:41 +08:00
Code Issues Packages Projects Releases Wiki Activity
182 Commits 2 Branches 0 Tags
Commit Graph

53 Commits

Author SHA1 Message Date
hub-gif
5e70bb1be3 refactor: 删除 AI_crawler 薄壳，配料试跑用 python -m pipeline.openai_gateway 
移除 crawler_copy/jd_pc_search/AI_crawler.py；试跑与校验逻辑在 openai_gateway/__main__.py。更新 README 与 .env.example 说明。

Made-with: Cursor
 2026-04-27 10:26:49 +08:00
hub-gif
9fd32a3f84 refactor(pipeline): OpenAI 网关与配料视觉抽离至 openai_gateway 
新增 pipeline/openai_gateway：credentials、text_chat、ingredients_op、ingredients_defaults 等；AI_crawler 仅加载 .env 与重导出。CrawlerOpenAiCompatible 与京东详情/流水线改走 import pipeline.openai_gateway，不再依赖 sys.path 注入爬虫目录。llm/providers/shared 转重导 gateway。详情脚本补充 backend 入 path。

Made-with: Cursor
 2026-04-27 10:24:50 +08:00
hub-gif
e4a05f3ce6 refactor(pipeline): 移除 csv_schema 等根目录中转，统一从 pipeline.csv 子包导入 
- 删除 csv_schema.py、csv_header_rewrite.py、matrix_group_label.py（无引用）

- 全库改为 from pipeline.csv.schema / pipeline.csv.header_rewrite；文档字符串同步

- 与此前 jd_competitor_report 去中转同一策略

Made-with: Cursor
 2026-04-17 14:48:32 +08:00
hub-gif
0e224c6a6c refactor(pipeline): 移除 jd_competitor_report 中转，统一导入 competitor_report.jd_report 
- 删除 backend/pipeline/jd_competitor_report.py；runner/测试与演示改为 from pipeline.competitor_report import jd_report

- 爬虫目录 jd_competitor_report.py 仅调用 jd_report.main；文档与 OpenAPI 同步更新

- 命令行推荐：python -m pipeline.competitor_report.jd_report

Made-with: Cursor
 2026-04-17 14:46:21 +08:00
hub-gif
179c9b2659 refactor(jd_pc_search): 拆分搜索与评论的解析/落盘与请求脚本 
评论：新增 jd_item_comment_parse、jd_item_comment_export；jd_h5_item_comment_requests 仅保留 Node 与 Playwright 与 main。搜索：新增 jd_h5_search_parse；jd_h5_search_requests 精简并 re-export CSV_FIELDS 等以兼容流水线与 collect_pc_search_items。

Made-with: Cursor
 2026-04-17 14:06:14 +08:00
hub-gif
1431ed3204 refactor(jd detail): 拆分 wareBusiness 解析与 Playwright 采集 
新增 jd_detail_ware_parse（JSON 扁平、重试判定、落盘格式化）与 jd_detail_ware_fetch（拦截接口与 #detail-main 抽图）；jd_detail_ware_business_requests 保留配置、main、写盘与薄封装 fetch_ware_business，并继续 re-export 解析符号以兼容旧导入。

Made-with: Cursor
 2026-04-17 14:01:23 +08:00
hub-gif
b6a648af9c refactor(jd_pc_search): 将 CSV 落盘与采集流水线解耦 
新增 jd_pipeline_export 模块承载合并表、PC 搜索导出、详情表与 run_meta 的列名规范化与写入；jd_keyword_pipeline 仅保留采集编排与调用。

Made-with: Cursor
 2026-04-17 13:53:28 +08:00
hub-gif
0d8e1962d7 refactor: 竞品报告迁入 pipeline，爬虫目录仅保留采集与兼容入口 
competitor_report 子包与 jd_competitor_report 主模块移至 backend/pipeline；runner 与测试改为 from pipeline import jd_competitor_report；crawler_copy 下 jd_competitor_report 为转发 shim；jd_keyword_pipeline 文档更新。

Made-with: Cursor
 2026-04-17 13:47:34 +08:00
hub-gif
b6a52c04d2 refactor(jd): 反馈分组、列表统计、矩阵行与报告片段拆至 competitor_report 
新增 consumer_feedback、run_context、list_mix、matrix_md、report_md_helpers；jd_competitor_report 从配置块后直接进入 build_competitor_markdown；移除未再使用的 ingredients 直引。

Made-with: Cursor
 2026-04-17 13:39:57 +08:00
hub-gif
8bbb921552 refactor(jd): 矩阵分组、价统与 LLM 分组载荷拆至 competitor_report 
新增 matrix_group（复用 pipeline.matrix_group_label）、price_stats、ingredients、llm_group_payloads；jd_competitor_report 仅保留报告编排与 Markdown；更新 matrix_group_label 模块说明。

Made-with: Cursor
 2026-04-17 11:46:35 +08:00
hub-gif
f2a2f50ded refactor(jd): 将评论情感与 build_comment_sentiment_llm_payload 拆至 competitor_report 
从 jd_competitor_report 迁入关键词命中、lexicon 与 LLM payload 构建；入口显式导入；保留 _comment_lines 与 build_scenario_groups 于主文件。

Made-with: Cursor
 2026-04-17 11:42:33 +08:00
hub-gif
64dd1b149d refactor(crawler): 价盘促销分析迁入 competitor_report.price_promo 
- 抽出 _analyze_price_promotions 与 _markdown_price_promotion_section，依赖 constants 与 csv_io。

- jd_competitor_report 显式 import，对外行为不变。

Made-with: Cursor
 2026-04-17 11:38:27 +08:00
hub-gif
4687bcb907 refactor(crawler): 抽取 competitor_report 常量与 CSV 辅助 
- 新增 jd_pc_search/competitor_report：constants、config（resolve_report_tuning）、csv_io。

- jd_competitor_report 通过 import * 保持原有符号；KEYWORD/EXISTING_RUN_DIR/OVERRIDE 仍在本脚本修改。

- 修正 constants.__all__ 中评价列名为 _COMMENT_CSV_*，避免 star import 漏绑。

Made-with: Cursor
 2026-04-17 11:37:27 +08:00
hub-gif
7ed160235b feat(pipeline): 报告文案通俗化与 Markdown 导出可读性优化 
- 竞品报告与流水线模板：弱化技术表述，促销与策略相关说明同步调整。

- md_document_export：按标题层级解析 #～######，列表与引用去掉行首符号，PDF 使用表格组件与分级标题样式。

Made-with: Cursor
 2026-04-17 10:55:51 +08:00
hub-gif
b8516662a8 fix(charts): 探针模式下跳过 §8.3 关注词场景条图并清理旧 PNG 
Made-with: Cursor
 2026-04-16 16:32:05 +08:00
hub-gif
31ac2f1ac7 feat(pipeline): 报告可启用第八章文本挖掘探针替代原 §8.3 关注词与场景 
Made-with: Cursor
 2026-04-16 16:22:12 +08:00
hub-gif
0e353318e1 feat(report): §8.2 先按评分分桶再统计口语短语 
Made-with: Cursor
 2026-04-16 14:57:58 +08:00
hub-gif
9d4881a01c fix(report): 扩充正向包装短语并澄清口语条形图口径 
Made-with: Cursor
 2026-04-16 14:21:32 +08:00
hub-gif
0cb54f679b feat(llm): 策略稿促销节须对齐 price_promotion_signals 
策略大模型提示要求依据 brief 中活动话术命中与券后价差字段写满减/券/红包等假设；禁止编造面额；活动子串补充红包/京豆。

Made-with: Cursor
 2026-04-16 14:13:33 +08:00
hub-gif
1495161850 fix(report): 负向归纳对齐分量类抱怨并约束质地表述 
默认关注词与负向口语短语表补充分量/量少等；§8.2 情感大模型提示要求对照 negative_lexeme_hits 条数、勿将未命中的质地问题写成主诉；lexeme 说明补充预设表局限。

Made-with: Cursor
 2026-04-16 14:03:24 +08:00
hub-gif
0be4049860 docs(copy): 将「口径」改为计数规则、说明、价格来源等通俗表述 
报告正文、简报、策略草稿、API 说明与 LLM 提示中避免「统计口径」等行话；列表 CSV 表头「销量口径」改为「销量展示」；日报目录未改。

Made-with: Cursor
 2026-04-16 10:43:44 +08:00
hub-gif
a2523415d5 docs(copy): 用「归类」替换易误解的「桶」表述 
报告说明、docstring 与情感分析 LLM 提示中的「分桶/乱桶」等改为归类、关键词规则等通俗说法；JSON 字段名 sentiment_bucket_method 未改。

Made-with: Cursor
 2026-04-16 10:40:39 +08:00
hub-gif
30a6bc4405 fix(report): 细类扇图口径与版式统一 
_category_mix 增加（其余细类）使简报与扇形图合计等于有效矩阵 SKU 数；§4.3 去掉易与图不一致的 Markdown 表格与规则摘要，仅保留扇形图；全部扇形图默认画布统一为 4.9×3.65 英寸。

Made-with: Cursor
 2026-04-16 10:38:21 +08:00
hub-gif
9cea35a665 fix(report): drop §4 brand/shop tables, align CR with pie charts 
Remove duplicate Markdown tables under 4.1/4.2; compute CR from strip-normalized names matching list_*_mix_top pies. Smaller figsize for list brand/shop pies; default pie slightly compact. Deduplicate test_mix_top_remainder test; assert _structure_names_for_pie_counter length.

Made-with: Cursor
 2026-04-15 17:54:53 +08:00
hub-gif
6280e436d8 feat(pipeline): JD 流水线 CSV 纯中文表头与模块目录重构 
- 统一 csv_schema 与搜索/合并/评价/商详导出为纯中文列名，入库与视图按中文表头解析

- 竞品分析报告兼容新旧表头；新增 csv_header_rewrite 与 rewrite_pipeline_csv_headers 管理命令

- 调整 pipeline 至 jd、llm、reporting、demos 子包并更新任务与测试引用

- 新增购买者优惠摘要抽取、合并表 regen/ingest 命令、0016 迁移及相关测试

Made-with: Cursor
 2026-04-15 16:02:29 +08:00
hub-gif
43b72cb32e feat: 关注词与场景延伸、报告体验与导出修复 
- 流水线：评价关注词与使用场景的大模型延伸（jd_runner、竞品报告、测试）

- 界面：报告生成与查看的通俗化文案；策略稿集中度按店铺/品牌区分

- 导出：PDF 插图限制最大尺寸避免 LayoutError；报告 Word/PDF 改为 blob 下载；策略稿导出错误解析；PDF 补充 Linux 中文字体路径

Made-with: Cursor
 2026-04-14 18:23:25 +08:00
hub-gif
3d6193af06 feat(brief): plain-language concentration keys and copy 
- Add first_share/top_three_combined_share plus cr1/cr3 readers
- Brief/schema/footnotes and LLM prompts avoid dev jargon where possible
- Frontend JD analysis/dataset hints in business-friendly wording

Made-with: Cursor
 2026-04-14 17:28:52 +08:00
hub-gif
c3fdb2b970 fix(report): align shop/brand pie denominators with §4 CR table via mix tail bucket 
Made-with: Cursor
 2026-04-14 17:28:52 +08:00
hub-gif
36aa36c60b refactor(pipeline): inline total_sales fallback; remove temp backfill script; add refresh command 
Made-with: Cursor
 2026-04-14 17:28:52 +08:00
hub-gif
8611e78a26 feat(jd-pipeline): matrix chart and brief use totalSales for sales volume 
Made-with: Cursor
 2026-04-14 17:28:17 +08:00
hub-gif
d99d9d2ab0 feat(report): combine focus-keyword and scenario charts per matrix group 
Made-with: Cursor
 2026-04-14 17:27:38 +08:00
hub-gif
5fe4ac4ffe feat(pipeline): add LLM summaries for usage scenarios (§8.4) 
Made-with: Cursor
 2026-04-14 17:27:38 +08:00
hub-gif
d2135e9a6e feat(report): drop chapter 7 product claims (detail attributes summary) 
Made-with: Cursor
 2026-04-14 17:27:19 +08:00
hub-gif
f994555f9f feat(pipeline): remove llm section bridges from report generation 
Made-with: Cursor
 2026-04-14 17:27:19 +08:00
hub-gif
c244a2902e refactor(report): drop §5 markdown tables, charts + CSV only 
Made-with: Cursor
 2026-04-14 17:27:03 +08:00
hub-gif
9d1ff31b23 fix(report): default LLM flags, embed §5 matrix charts, slim table 
Merge missing keys from get_default_report_config into effective report_config so llm_* switches default on when absent. §5 embeds chart_matrix_prices_reviews per fine category (price + review-volume bars, Y=title) and replaces the wide matrix with a compact table. Matrix chart Y-labels allow longer titles (48 chars).

Made-with: Cursor
 2026-04-14 17:27:03 +08:00
hub-gif
c6560bd12b fix(report): use detail_category_path only for matrix and fine-category stats 
Made-with: Cursor
 2026-04-14 17:25:54 +08:00
hub-gif
f7a4754e56 feat(report): wire LLM group summaries, improve matrix labels and chart Y-axis 
Made-with: Cursor
 2026-04-14 17:13:01 +08:00
hub-gif
11b9c24334 docs(report): clarify LLM keyword/scenario extension in brief notes 
Made-with: Cursor
 2026-04-14 17:12:32 +08:00
hub-gif
4cae885200 Revert "docs(report): note LLM keyword/scenario extension in competitor brief" 
This reverts commit 70b0b247f4d8ffd66120237ee6154f993fb2d733.
 2026-04-14 17:12:32 +08:00
hub-gif
30a985f749 docs(report): note LLM keyword/scenario extension in competitor brief 
Made-with: Cursor
 2026-04-14 17:12:32 +08:00
hub-gif
c45ec66d4c feat(report): remove §5 competitor matrix table, keep per-group charts 
Made-with: Cursor
 2026-04-14 13:18:40 +08:00
hub-gif
a1d0fa6686 fix(report): shop/brand pie totals, matrix price charts, drop LLM §8.5 merge 
Made-with: Cursor
 2026-04-14 13:16:29 +08:00
hub-gif
1f6abe49ee feat(pipeline): wire matrix/price/comment group LLM into report body 
- build_matrix/price_groups_llm_payload from merged rows
- jd_runner: call generate_*_group_summaries_llm when report_config or
  MA_ENABLE_LLM_*; write matrix/price/comment_groups_llm.json sidecars
- build_competitor_markdown: insert §5.1, §6.2, §8.6 with disclaimers
- §1.3 note for optional llm_*_group_summaries flags

Made-with: Cursor
 2026-04-14 11:15:17 +08:00
hub-gif
dd5fa8a407 feat(comments): attribute sentiment LLM samples to shop and SKU 
- Add _comment_lines_with_product_context and build_comment_groups_llm_payload
- build_comment_sentiment_llm_payload: lexicon on plain text, samples with
  【细类｜SKU｜品名｜店铺】 prefix; jd_runner passes parallel attributed lines
- Tighten §8.2/§8.5 prompts so negatives tie to shop + product; tests assert 店铺

Made-with: Cursor
 2026-04-14 11:04:35 +08:00
hub-gif
708f28f40e fix(report): restore jd_competitor_report after mistaken chart revert 
9cc8de9 accidentally rewrote jd_competitor_report.py and merged an oversized
report_charts.py. Reset both to f19d112 baseline; keep only obsolete cleanup
for removed global bar chart filenames.

Made-with: Cursor
 2026-04-14 10:51:38 +08:00
hub-gif
9cc8de9006 revert(report): drop global focus/scenario bar charts 
Remove chart_comment_focus_global_bar / chart_usage_scenarios_global_bar
generation and markdown embed; delete leftovers on chart regen.

Made-with: Cursor
 2026-04-14 10:48:54 +08:00
hub-gif
be0215bfe2 fix(ai_crawler): cap max_tokens to remaining context window 
LiteLLM rejects when max_tokens exceeds context minus measured input
tokens. Estimate input from prompt length and clamp max_tokens; allow
LLM_CONTEXT_WINDOW override. Prevents ContextWindowExceededError on long
Chinese/JSON prompts (e.g. regenerate-report §8.5).

Made-with: Cursor
 2026-04-14 10:16:38 +08:00
hub-gif
9797796f2e fix(pipeline): cap LLM brief size on report regen; enrich chat HTTP errors 
Regenerate-report (llm) used compact_brief_for_llm default 350k chars,
unlike other LLM steps (80k–100k), which can cause gateway timeouts or 5xx
surfacing as 502. chat_completion_text now attaches response body snippet
to HTTPError for easier diagnosis.

Made-with: Cursor
 2026-04-14 09:55:42 +08:00
hub-gif
a8210ec933 feat(报告): 各章大模型衔接分析与 report_config 开关 
Made-with: Cursor
 2026-04-13 11:06:35 +08:00