feat: Obsidian 知识库批量导入 + 语义搜索体验升级
- 新增 import_obsidian_kb.py 批量导入脚本(164篇入库,知识库186条) - parse_md_file 补强:来源fallback、相关装备/应用领域实体提取、 Obsidian双链去括号、原始类别/源文件路径存metadata、TYPE_MAP扩充 - search_similar 改进:智能摘要(中文2-gram拆词+加权段落匹配)、 min_similarity=0.3过滤、top_k 5→10、返回完整content_md - 前端搜索卡片升级:展开全文、关键词加粗渲染、相关度分档样式 - CLAUDE.md 状态更新 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -13,8 +13,8 @@
|
|||||||
|
|
||||||
## 🔖 状态栏(每次结束 session 前必须更新这三行)
|
## 🔖 状态栏(每次结束 session 前必须更新这三行)
|
||||||
|
|
||||||
- **最后更新**:Claude Opus(顾问)| 2026-07-07
|
- **最后更新**:Claude Opus(顾问)| 2026-07-08
|
||||||
- **当前状态一句话**:**知识库冷启动完成**——doco 22 期融合A稿已全部导入知识库(含 embedding),下一步做语义搜索界面(Phase 4a)。
|
- **当前状态一句话**:**知识库语义搜索基本可用**——Obsidian 知识库 164 篇批量导入完成(总计 186 条含 doco 22 篇),搜索体验三项改进已落地(智能摘要/相关度阈值/展开全文),中文分词 snippet bug 已修。下一步:搜索体验微调 + 进入 Phase 4a 对话式 TPS。
|
||||||
- **下一个动手的人从这里开始**:见下方「⏩ 交接备注」
|
- **下一个动手的人从这里开始**:见下方「⏩ 交接备注」
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -76,8 +76,8 @@
|
|||||||
|
|
||||||
## 3. 当前进度(动态,核心交接区 — 以最新快照为准)
|
## 3. 当前进度(动态,核心交接区 — 以最新快照为准)
|
||||||
|
|
||||||
- **已完成至**:收视分析看板 L1-L4 + 知识库冷启动(doco 22 期融合A稿批量导入,含 embedding)。
|
- **已完成至**:收视分析看板 L1-L4 + 知识库语义搜索(Obsidian 164 篇 + doco 22 篇批量导入,搜索体验三项改进)。
|
||||||
- **正在做**:语义搜索界面(Phase 4a)。
|
- **正在做**:语义搜索体验微调(中文 snippet 已修,待制片人复测),准备进入 Phase 4a 对话式 TPS。
|
||||||
- **卡点/待解**:无硬卡点。
|
- **卡点/待解**:无硬卡点。
|
||||||
- **Schema 状态**:episodes 表已通过 003(+7 AI 标签列)和 004(+content_digest JSONB)迁移。知识库两表不再改。
|
- **Schema 状态**:episodes 表已通过 003(+7 AI 标签列)和 004(+content_digest JSONB)迁移。知识库两表不再改。
|
||||||
|
|
||||||
@@ -85,6 +85,7 @@
|
|||||||
|
|
||||||
## 4. 已完成(只追加,最新在上)
|
## 4. 已完成(只追加,最新在上)
|
||||||
|
|
||||||
|
- [2026-07-08] **Obsidian 知识库批量导入 + 语义搜索体验升级**:① `parse_md_file` 补强 6 项("来源"→source_detail fallback、"相关装备/应用领域"实体提取、Obsidian 双链 `[[]]` 去括号、原始类别存 metadata、源文件路径预埋、SOURCE_TYPE_MAP 扩充含"军报文章/装备/技术/动态/术语/厂商/索引");② 脚本 `backend/scripts/import_obsidian_kb.py`(递归遍历、跳过 .obsidian+原始素材+空文件+无 frontmatter、按相对路径查重、支持 --limit/--dry-run/--dir),成功导入 164 篇(7+157),知识库从 22 → 186 条;③ 搜索体验三项改进:`search_similar` 返回完整 content_md + 智能摘要 `_extract_smart_snippet`(中文 2-gram 拆词+加权段落匹配+关键词加粗)+ min_similarity=0.3 过滤 + top_k 5→10;④ 前端搜索卡片升级(展开全文 Set 状态管理、`renderSnippet` 加粗渲染、相关度分档样式:≥70%蓝左边框/<50%半透明灰底)。
|
||||||
- [2026-07-07] **知识库冷启动:doco 22 期融合A稿批量导入**:脚本 `backend/scripts/import_doco_transcripts.py`,docx → 提取纯文本 → 包 YAML frontmatter → 调 `store_md_file()` 入库(自动算 MiniMax embo-01 embedding)。22 篇全部成功(第02期~第23期,缺第01期无A稿),source_type=manuscript,知识库从 0 → 22 条节目文稿。
|
- [2026-07-07] **知识库冷启动:doco 22 期融合A稿批量导入**:脚本 `backend/scripts/import_doco_transcripts.py`,docx → 提取纯文本 → 包 YAML frontmatter → 调 `store_md_file()` 入库(自动算 MiniMax embo-01 embedding)。22 篇全部成功(第02期~第23期,缺第01期无A稿),source_type=manuscript,知识库从 0 → 22 条节目文稿。
|
||||||
- [2026-07-07] **期次一条龙录入子项目立项(只立项不开发)**:① PRD v1.0 写入 `episode-intake/PRD_期次一条龙录入_v1.md`(每期任务清单模式、4 状态点、抽屉四区块、6 个新 API、005 迁移只加 transcript_item_id 一列、看板只用 reviewed 期次、四刀分期+七条验收);② 制片人三项拍板:文稿口径=doco 融合A稿(CCA 是播出前工具不算终稿,doco 转常态运行)、22 期批量导入时回联期次、责编可触发 AI 处理;③ 子项目 CLAUDE.md + 寄存条建立,CCA 寄存条清单表同步更新。
|
- [2026-07-07] **期次一条龙录入子项目立项(只立项不开发)**:① PRD v1.0 写入 `episode-intake/PRD_期次一条龙录入_v1.md`(每期任务清单模式、4 状态点、抽屉四区块、6 个新 API、005 迁移只加 transcript_item_id 一列、看板只用 reviewed 期次、四刀分期+七条验收);② 制片人三项拍板:文稿口径=doco 融合A稿(CCA 是播出前工具不算终稿,doco 转常态运行)、22 期批量导入时回联期次、责编可触发 AI 处理;③ 子项目 CLAUDE.md + 寄存条建立,CCA 寄存条清单表同步更新。
|
||||||
- [2026-07-06] **三处 Bug 修复 + 全局宽度调优**:① AI 诊断摘要块 `extractSection` 从硬编码 `indexOf` 改为正则模糊匹配,解决 DeepSeek 返回标题格式不一致导致干条不显示的问题;② 仪表盘近 12 期柱状图加 `.reverse()` 改为从左到右按播出时间升序;③ 全局内容区宽度:定位到 `.app-content`(`AppLayout.css`)的 `max-width` 是唯一有效参数(必须搭配 `width: 100%`,否则 Ant Design flex 布局下 `margin: 0 auto` 会触发 shrink-to-fit),最终定为 **1190px**。
|
- [2026-07-06] **三处 Bug 修复 + 全局宽度调优**:① AI 诊断摘要块 `extractSection` 从硬编码 `indexOf` 改为正则模糊匹配,解决 DeepSeek 返回标题格式不一致导致干条不显示的问题;② 仪表盘近 12 期柱状图加 `.reverse()` 改为从左到右按播出时间升序;③ 全局内容区宽度:定位到 `.app-content`(`AppLayout.css`)的 `max-width` 是唯一有效参数(必须搭配 `width: 100%`,否则 Ant Design flex 布局下 `margin: 0 auto` 会触发 shrink-to-fit),最终定为 **1190px**。
|
||||||
@@ -103,10 +104,11 @@
|
|||||||
## 5. 待办(下一刀候选,开局前定先做哪件)
|
## 5. 待办(下一刀候选,开局前定先做哪件)
|
||||||
|
|
||||||
- [x] ~~L4 AI 诊断报告~~ → 已完成(2026-07-03)
|
- [x] ~~L4 AI 诊断报告~~ → 已完成(2026-07-03)
|
||||||
- [ ] **下一刀三选一**:① 语义搜索界面(不依赖任何材料,随时能开,是 Phase 4a 硬门槛);② PDF 原文关联 + 大文件存储架构(需 Opus 审方案,优先级较高);③ 界面像素级打磨+视觉规范统一。
|
- [x] ~~语义搜索界面~~ → 已完成(2026-07-08),Phase 4a 硬门槛已过
|
||||||
- [ ] **期次一条龙录入开发**(子项目已立项 2026-07-07,PRD 就绪在 `episode-intake/`,等制片人排期;四刀分期,第二刀含 doco 22 期导入回联,与既有待办「200+ md 批量录入」相互独立)。
|
- [x] ~~200+ Obsidian md 批量录入~~ → 已完成(2026-07-08),164 篇导入成功,知识库 186 条
|
||||||
- [ ] PDF 大文件存储:**大文件不入库,单独文件仓库,DB 只存地址指针**(md 正文+向量留库参与检索;pdf 仅按需调阅)。地基一次定对,避免上云返工。
|
- [ ] **下一刀**:Phase 4a 对话式 TPS 选题策划助手(左对话右报告、脚注式引用、知识库检索底座已就绪)。
|
||||||
- [ ] 200+ Obsidian md 批量录入(**建议在 PDF 存储方案定后做**;先试 10 篇验证解析/落位再全量,每篇都真调 MiniMax 耗额度)。
|
- [ ] **期次一条龙录入开发**(子项目已立项 2026-07-07,PRD 就绪在 `episode-intake/`,等制片人排期;四刀分期,第二刀含 doco 22 期导入回联)。
|
||||||
|
- [ ] PDF 大文件存储:**大文件不入库,单独文件仓库,DB 只存地址指针**(md 正文+向量留库参与检索;pdf 仅按需调阅)。地基一次定对,避免上云返工。Obsidian 导入时已把源文件路径存入 `metadata["source_files"]` 预埋。
|
||||||
- [ ] 「按编导看稿」独立筛选视图(路线 A 重构时一并处理,从来源树里迁出来)。
|
- [ ] 「按编导看稿」独立筛选视图(路线 A 重构时一并处理,从来源树里迁出来)。
|
||||||
- [ ] 操作留痕 schema(episodes 等加 created_by/updated_by/updated_at;涉 schema 须 Opus 审)。
|
- [ ] 操作留痕 schema(episodes 等加 created_by/updated_by/updated_at;涉 schema 须 Opus 审)。
|
||||||
- [ ] 界面像素级打磨(连同全栏目视觉规范统一弄,参考「色调字体倒角」图)。
|
- [ ] 界面像素级打磨(连同全栏目视觉规范统一弄,参考「色调字体倒角」图)。
|
||||||
@@ -142,6 +144,7 @@
|
|||||||
|
|
||||||
## 7. ⏩ 交接备注(换人/换账号 0 摩擦续上)
|
## 7. ⏩ 交接备注(换人/换账号 0 摩擦续上)
|
||||||
|
|
||||||
|
- **知识库语义搜索已可用**(2026-07-08):186 条知识库条目(doco 22 篇 + Obsidian 164 篇),搜索链路完整(MiniMax embo-01 query embedding → pgvector 余弦检索 → 智能摘要 + 关键词加粗 + 展开全文)。新增文件:`backend/scripts/import_obsidian_kb.py`(批量导入脚本)。`knowledge_service.py` 的 `parse_md_file` 已补强支持全部 Obsidian frontmatter 字段。搜索结果返回 `content_md` 完整正文供展开查看。Obsidian 源文件路径已存入 `metadata["source_files"]` 预埋 PDF 关联。
|
||||||
- **收视分析看板 L1-L4 全部完成**。页面布局:指标卡 → 走势图 → AI 诊断报告(摘要块) → 双引擎象限图 → 双列对比(左:季度+编导, 右:题材)。
|
- **收视分析看板 L1-L4 全部完成**。页面布局:指标卡 → 走势图 → AI 诊断报告(摘要块) → 双引擎象限图 → 双列对比(左:季度+编导, 右:题材)。
|
||||||
- **L4 AI 诊断报告新增文件**:后端 `backend/app/api/analytics.py`(POST 端点);前端 `DiagnosisSummary.jsx`(摘要块)+ `DiagnosisReport.jsx/.css`(详情页);Prompt 文件 `ai-labeling/prompts/prompt4_content_digest.md` + `prompt5_diagnosis_report.md`;批量脚本 `ai-labeling/scripts/gen_content_digest.py` + `scripts/import_content_digests.py`。
|
- **L4 AI 诊断报告新增文件**:后端 `backend/app/api/analytics.py`(POST 端点);前端 `DiagnosisSummary.jsx`(摘要块)+ `DiagnosisReport.jsx/.css`(详情页);Prompt 文件 `ai-labeling/prompts/prompt4_content_digest.md` + `prompt5_diagnosis_report.md`;批量脚本 `ai-labeling/scripts/gen_content_digest.py` + `scripts/import_content_digests.py`。
|
||||||
- **L4 依赖**:`backend/.env` 需配 `DEEPSEEK_API_KEY`;前端需 `react-markdown` 包(已装);后端需 `openai` 包(已装)。
|
- **L4 依赖**:`backend/.env` 需配 `DEEPSEEK_API_KEY`;前端需 `react-markdown` 包(已装);后端需 `openai` 包(已装)。
|
||||||
|
|||||||
@@ -109,7 +109,8 @@ def get_grouped_knowledge_items(
|
|||||||
|
|
||||||
class SearchRequest(BaseModel):
|
class SearchRequest(BaseModel):
|
||||||
query: str
|
query: str
|
||||||
top_k: int = 5
|
top_k: int = 10
|
||||||
|
min_similarity: float = 0.3
|
||||||
|
|
||||||
|
|
||||||
@router.post("/search")
|
@router.post("/search")
|
||||||
@@ -121,10 +122,15 @@ def search_knowledge(
|
|||||||
"""
|
"""
|
||||||
语义检索:输入一段文字,返回最相关的知识库条目及相似度。
|
语义检索:输入一段文字,返回最相关的知识库条目及相似度。
|
||||||
查询向量用 type="query"(区分于存入时的 type="db")。
|
查询向量用 type="query"(区分于存入时的 type="db")。
|
||||||
|
top_k 默认 10,min_similarity 默认 0.3 过滤低相关条目。
|
||||||
三角色均可读。
|
三角色均可读。
|
||||||
"""
|
"""
|
||||||
svc = KnowledgeService()
|
svc = KnowledgeService()
|
||||||
results = svc.search_similar(query_text=body.query, top_k=body.top_k)
|
results = svc.search_similar(
|
||||||
|
query_text=body.query,
|
||||||
|
top_k=body.top_k,
|
||||||
|
min_similarity=body.min_similarity,
|
||||||
|
)
|
||||||
return {
|
return {
|
||||||
"results": results,
|
"results": results,
|
||||||
"query": body.query,
|
"query": body.query,
|
||||||
|
|||||||
@@ -23,8 +23,15 @@ class KnowledgeService:
|
|||||||
SOURCE_TYPE_MAP = {
|
SOURCE_TYPE_MAP = {
|
||||||
"杂志文章": "military_report",
|
"杂志文章": "military_report",
|
||||||
"军报": "military_report",
|
"军报": "military_report",
|
||||||
|
"军报文章": "military_report",
|
||||||
"节目文稿": "manuscript",
|
"节目文稿": "manuscript",
|
||||||
"报题单": "baoti",
|
"报题单": "baoti",
|
||||||
|
"装备": "manual",
|
||||||
|
"技术": "manual",
|
||||||
|
"动态": "manual",
|
||||||
|
"术语": "manual",
|
||||||
|
"厂商": "manual",
|
||||||
|
"索引": "manual",
|
||||||
}
|
}
|
||||||
|
|
||||||
# 来源大类固定显示顺序(制片人 Obsidian 习惯)
|
# 来源大类固定显示顺序(制片人 Obsidian 习惯)
|
||||||
@@ -89,6 +96,12 @@ class KnowledgeService:
|
|||||||
else:
|
else:
|
||||||
source_detail = None
|
source_detail = None
|
||||||
|
|
||||||
|
# fallback:如果 source_detail 仍为空,取"来源"字段
|
||||||
|
if not source_detail:
|
||||||
|
raw_source = str(fm.get("来源", "") or "").strip()
|
||||||
|
if raw_source:
|
||||||
|
source_detail = raw_source
|
||||||
|
|
||||||
# —— 播出日期:容错 "待补充" 等非日期文本——
|
# —— 播出日期:容错 "待补充" 等非日期文本——
|
||||||
raw_date = str(fm.get("播出日期", "") or "").strip()
|
raw_date = str(fm.get("播出日期", "") or "").strip()
|
||||||
publish_date = None
|
publish_date = None
|
||||||
@@ -107,17 +120,22 @@ class KnowledgeService:
|
|||||||
# —— 权重(不展示,存 JSONB 备 Phase 4)——
|
# —— 权重(不展示,存 JSONB 备 Phase 4)——
|
||||||
weight = str(fm.get("权重", "") or "").strip() or None
|
weight = str(fm.get("权重", "") or "").strip() or None
|
||||||
|
|
||||||
# —— 相关实体(涉及装备/涉及技术/涉及厂商/主题)——
|
# —— 相关实体(涉及装备/涉及技术/涉及厂商/主题/相关装备/应用领域)——
|
||||||
|
import re as _re
|
||||||
related_entities = []
|
related_entities = []
|
||||||
for key in ("涉及装备", "涉及技术", "涉及厂商", "主题"):
|
for key in ("涉及装备", "涉及技术", "涉及厂商", "主题", "相关装备", "应用领域"):
|
||||||
val = fm.get(key)
|
val = fm.get(key)
|
||||||
if val:
|
if val:
|
||||||
if isinstance(val, list):
|
if isinstance(val, list):
|
||||||
related_entities.extend(val)
|
for item in val:
|
||||||
|
# 去掉 Obsidian 双链格式 [[xxx]]
|
||||||
|
cleaned = _re.sub(r"\[\[|\]\]", "", str(item)).strip()
|
||||||
|
if cleaned:
|
||||||
|
related_entities.append(cleaned)
|
||||||
elif isinstance(val, str):
|
elif isinstance(val, str):
|
||||||
# 可能是 "山东舰, 福建舰" 这样的逗号分隔字符串
|
# 可能是 "山东舰, 福建舰" 这样的逗号分隔字符串
|
||||||
for item in val.replace(",", ",").split(","):
|
for item in val.replace(",", ",").split(","):
|
||||||
item = item.strip()
|
item = _re.sub(r"\[\[|\]\]", "", item).strip()
|
||||||
if item:
|
if item:
|
||||||
related_entities.append(item)
|
related_entities.append(item)
|
||||||
|
|
||||||
@@ -130,6 +148,26 @@ class KnowledgeService:
|
|||||||
# related_concepts 字段预留给双链解析(Phase 4),本 Task 原样存入
|
# related_concepts 字段预留给双链解析(Phase 4),本 Task 原样存入
|
||||||
metadata["double_bracket_links"] = self._extract_double_brackets(parsed.content)
|
metadata["double_bracket_links"] = self._extract_double_brackets(parsed.content)
|
||||||
|
|
||||||
|
# 保留原始 Obsidian 类别(映射到 manual 的类型,如装备/技术/动态/术语/厂商等)
|
||||||
|
if raw_type and source_type == "manual":
|
||||||
|
metadata["obsidian_category"] = raw_type
|
||||||
|
|
||||||
|
# 源文件路径保留(预埋 PDF 原文链接)
|
||||||
|
raw_source_files = fm.get("源文件")
|
||||||
|
if raw_source_files:
|
||||||
|
cleaned_files = []
|
||||||
|
if isinstance(raw_source_files, list):
|
||||||
|
for sf in raw_source_files:
|
||||||
|
cleaned = _re.sub(r'[\[\]"]', "", str(sf)).strip()
|
||||||
|
if cleaned:
|
||||||
|
cleaned_files.append(cleaned)
|
||||||
|
elif isinstance(raw_source_files, str):
|
||||||
|
cleaned = _re.sub(r'[\[\]"]', "", raw_source_files).strip()
|
||||||
|
if cleaned:
|
||||||
|
cleaned_files.append(cleaned)
|
||||||
|
if cleaned_files:
|
||||||
|
metadata["source_files"] = cleaned_files
|
||||||
|
|
||||||
# —— 正文(去掉 frontmatter 的纯内容)——
|
# —— 正文(去掉 frontmatter 的纯内容)——
|
||||||
content_md = parsed.content
|
content_md = parsed.content
|
||||||
|
|
||||||
@@ -229,13 +267,11 @@ class KnowledgeService:
|
|||||||
sources.add(tags["source_detail"])
|
sources.add(tags["source_detail"])
|
||||||
return sorted(list(sources))
|
return sorted(list(sources))
|
||||||
|
|
||||||
def search_similar(self, query_text: str, top_k: int = 5) -> list[dict]:
|
def search_similar(self, query_text: str, top_k: int = 10, min_similarity: float = 0.3) -> list[dict]:
|
||||||
"""
|
"""
|
||||||
语义检索:查询句转为向量,用 SQL 余弦距离(<=>)在数据库层检索
|
语义检索:查询句转为向量,用 SQL 余弦距离(<=>)在数据库层检索
|
||||||
返回 top_k 条相似笔记,含相似度分数 + 原文片段(SQL 端截断前 200 字)。
|
返回 top_k 条相似笔记,含相似度分数 + 智能摘要片段。
|
||||||
|
过滤掉 similarity < min_similarity 的条目。
|
||||||
注意:当前取前 200 字是已知妥协(整篇向量检索无法定位中段命中点),
|
|
||||||
Phase 4a 做切块检索(chunk)时可优化为取最相关片段。
|
|
||||||
"""
|
"""
|
||||||
query_vector = self.embedder.embed_single(query_text, embed_type="query")
|
query_vector = self.embedder.embed_single(query_text, embed_type="query")
|
||||||
vec_str = "[" + ",".join(str(v) for v in query_vector) + "]"
|
vec_str = "[" + ",".join(str(v) for v in query_vector) + "]"
|
||||||
@@ -248,7 +284,7 @@ class KnowledgeService:
|
|||||||
ki.source_type,
|
ki.source_type,
|
||||||
ki.author,
|
ki.author,
|
||||||
ki.tags,
|
ki.tags,
|
||||||
SUBSTRING(ki.content_md, 1, 200) AS snippet,
|
ki.content_md,
|
||||||
1 - (ke.embedding <=> '{vec_str}'::vector) AS similarity
|
1 - (ke.embedding <=> '{vec_str}'::vector) AS similarity
|
||||||
FROM knowledge_embeddings ke
|
FROM knowledge_embeddings ke
|
||||||
JOIN knowledge_items ki ON ke.knowledge_id = ki.id
|
JOIN knowledge_items ki ON ke.knowledge_id = ki.id
|
||||||
@@ -260,19 +296,88 @@ class KnowledgeService:
|
|||||||
rows = session.execute(stmt).all()
|
rows = session.execute(stmt).all()
|
||||||
results = []
|
results = []
|
||||||
for r in rows:
|
for r in rows:
|
||||||
|
similarity = round(r.similarity, 4)
|
||||||
|
# 过滤低于阈值的条目
|
||||||
|
if similarity < min_similarity:
|
||||||
|
continue
|
||||||
tags = r.tags or {}
|
tags = r.tags or {}
|
||||||
source_detail = tags.get("source_detail") if isinstance(tags, dict) else None
|
source_detail = tags.get("source_detail") if isinstance(tags, dict) else None
|
||||||
|
snippet = self._extract_smart_snippet(r.content_md, query_text)
|
||||||
results.append({
|
results.append({
|
||||||
"id": r.id,
|
"id": r.id,
|
||||||
"title": r.title,
|
"title": r.title,
|
||||||
"source_type": r.source_type,
|
"source_type": r.source_type,
|
||||||
"author": r.author,
|
"author": r.author,
|
||||||
"source_detail": source_detail,
|
"source_detail": source_detail,
|
||||||
"snippet": r.snippet,
|
"snippet": snippet,
|
||||||
"similarity": round(r.similarity, 4),
|
"content_md": r.content_md,
|
||||||
|
"similarity": similarity,
|
||||||
})
|
})
|
||||||
return results
|
return results
|
||||||
|
|
||||||
|
def _extract_smart_snippet(self, content_md: str, query_text: str, max_len: int = 300) -> str:
|
||||||
|
"""
|
||||||
|
智能摘要提取:根据搜索词定位最相关段落,截取摘要并加粗关键词。
|
||||||
|
中文连续字符串会被拆成 2-gram 子串以提高段落匹配命中率。
|
||||||
|
"""
|
||||||
|
import re as _re
|
||||||
|
|
||||||
|
if not content_md:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
# 1. 切分关键词:先按空格/标点拆,再把长中文词拆成 2-gram
|
||||||
|
raw_parts = _re.split(r'[\s,。!?、;:""''()\(\)\[\]\-]+', query_text)
|
||||||
|
raw_parts = [p.strip() for p in raw_parts if len(p.strip()) > 1]
|
||||||
|
|
||||||
|
keywords = []
|
||||||
|
for part in raw_parts:
|
||||||
|
keywords.append(part)
|
||||||
|
if len(part) > 2 and _re.fullmatch(r'[一-鿿]+', part):
|
||||||
|
for i in range(len(part) - 1):
|
||||||
|
bigram = part[i:i+2]
|
||||||
|
if bigram not in keywords:
|
||||||
|
keywords.append(bigram)
|
||||||
|
|
||||||
|
if not keywords:
|
||||||
|
return content_md[:max_len]
|
||||||
|
|
||||||
|
# 2. 按段落分割(跳过纯 markdown 标记行如 # ## --- 等)
|
||||||
|
paragraphs = content_md.split("\n\n")
|
||||||
|
if len(paragraphs) < 3:
|
||||||
|
paragraphs = content_md.split("\n")
|
||||||
|
paragraphs = [p.strip() for p in paragraphs
|
||||||
|
if p.strip() and not _re.fullmatch(r'[#\-=\s>]+', p.strip())]
|
||||||
|
|
||||||
|
# 3. 计算每段关键词命中数(完整词权重 3,bigram 权重 1)
|
||||||
|
best_para = ""
|
||||||
|
best_score = 0
|
||||||
|
for para in paragraphs:
|
||||||
|
score = 0
|
||||||
|
for kw in keywords:
|
||||||
|
if kw in para:
|
||||||
|
score += 3 if kw in raw_parts else 1
|
||||||
|
if score > best_score:
|
||||||
|
best_score = score
|
||||||
|
best_para = para
|
||||||
|
|
||||||
|
# 4. fallback:无关键词命中时用正文前 max_len 字
|
||||||
|
if best_score == 0:
|
||||||
|
snippet = content_md[:max_len]
|
||||||
|
else:
|
||||||
|
snippet = best_para[:max_len]
|
||||||
|
|
||||||
|
# 5. 加粗命中关键词(优先标记完整词,避免 bigram 重复标记)
|
||||||
|
marked = set()
|
||||||
|
for kw in sorted(keywords, key=len, reverse=True):
|
||||||
|
if kw in snippet and kw not in marked:
|
||||||
|
snippet = snippet.replace(kw, f"**{kw}**", 1)
|
||||||
|
marked.add(kw)
|
||||||
|
for sub_kw in keywords:
|
||||||
|
if sub_kw != kw and sub_kw in kw:
|
||||||
|
marked.add(sub_kw)
|
||||||
|
|
||||||
|
return snippet
|
||||||
|
|
||||||
def get_item_count(self) -> int:
|
def get_item_count(self) -> int:
|
||||||
with Session(engine) as session:
|
with Session(engine) as session:
|
||||||
return len(session.exec(select(KnowledgeItem)).all())
|
return len(session.exec(select(KnowledgeItem)).all())
|
||||||
|
|||||||
@@ -0,0 +1,169 @@
|
|||||||
|
"""
|
||||||
|
批量导入脚本:将 Obsidian 知识库目录下的 .md 文件
|
||||||
|
导入 TPS 知识库(knowledge_items + knowledge_embeddings)。
|
||||||
|
|
||||||
|
运行方式:
|
||||||
|
cd backend && python -m scripts.import_obsidian_kb --limit 10
|
||||||
|
cd backend && python -m scripts.import_obsidian_kb # 全量
|
||||||
|
cd backend && python -m scripts.import_obsidian_kb --dry-run # 只预览不写库
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from sqlmodel import Session, select
|
||||||
|
|
||||||
|
from app.db.session import engine
|
||||||
|
from app.models.knowledge import KnowledgeItem
|
||||||
|
from app.services.knowledge_service import KnowledgeService
|
||||||
|
|
||||||
|
# ── 默认 Obsidian 知识库根目录 ────────────────────────────────
|
||||||
|
DEFAULT_OBSIDIAN_DIR = r"E:\AIworks\obsidian\MSTknowledge_base\知识库"
|
||||||
|
|
||||||
|
# ── 跳过的目录名 ────────────────────────────────────────────────
|
||||||
|
SKIP_DIRS = {".obsidian", "99-原始素材"}
|
||||||
|
|
||||||
|
|
||||||
|
def collect_md_files(root: Path) -> list[Path]:
|
||||||
|
"""递归收集 root 下所有 .md 文件,跳过 SKIP_DIRS 中的目录。"""
|
||||||
|
md_files = []
|
||||||
|
for p in root.rglob("*.md"):
|
||||||
|
# 检查路径中是否包含跳过目录
|
||||||
|
parts = p.relative_to(root).parts
|
||||||
|
if any(skip in parts for skip in SKIP_DIRS):
|
||||||
|
continue
|
||||||
|
md_files.append(p)
|
||||||
|
return sorted(md_files, key=lambda p: str(p.relative_to(root)))
|
||||||
|
|
||||||
|
|
||||||
|
def check_exists(source_file_name: str) -> bool:
|
||||||
|
"""查 knowledge_items 表,判断 source_file_name 是否已存在。"""
|
||||||
|
with Session(engine) as session:
|
||||||
|
stmt = select(KnowledgeItem).where(
|
||||||
|
KnowledgeItem.source_file_name == source_file_name
|
||||||
|
)
|
||||||
|
existing = session.exec(stmt).first()
|
||||||
|
return existing is not None
|
||||||
|
|
||||||
|
|
||||||
|
def has_frontmatter(content: str) -> bool:
|
||||||
|
"""检查文件内容是否包含 YAML frontmatter(--- 包裹)。"""
|
||||||
|
stripped = content.strip()
|
||||||
|
if not stripped.startswith("---"):
|
||||||
|
return False
|
||||||
|
# 找到第二个 ---
|
||||||
|
second = stripped.find("---", 3)
|
||||||
|
return second > 0
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description="批量导入 Obsidian 知识库 .md 文件到 TPS 知识库")
|
||||||
|
parser.add_argument("--dir", type=str, default=DEFAULT_OBSIDIAN_DIR,
|
||||||
|
help="Obsidian 知识库根目录")
|
||||||
|
parser.add_argument("--limit", type=int, default=0,
|
||||||
|
help="只处理前 N 个文件(测试用,0=全量)")
|
||||||
|
parser.add_argument("--dry-run", action="store_true",
|
||||||
|
help="只打印解析结果,不真正调 API 写库")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
root = Path(args.dir)
|
||||||
|
if not root.is_dir():
|
||||||
|
print(f"[ERROR] 目录不存在: {root}")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
# 收集文件
|
||||||
|
md_files = collect_md_files(root)
|
||||||
|
if not md_files:
|
||||||
|
print("[ERROR] 未找到任何 .md 文件")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
total = len(md_files)
|
||||||
|
if args.limit > 0:
|
||||||
|
md_files = md_files[:args.limit]
|
||||||
|
print(f"共发现 {total} 个 .md 文件,本次处理前 {len(md_files)} 个\n")
|
||||||
|
else:
|
||||||
|
print(f"共发现 {total} 个 .md 文件,开始全量导入...\n")
|
||||||
|
|
||||||
|
if args.dry_run:
|
||||||
|
print("【DRY-RUN 模式】只预览解析结果,不写入数据库\n")
|
||||||
|
|
||||||
|
service = KnowledgeService()
|
||||||
|
success_count = 0
|
||||||
|
skip_exists_count = 0
|
||||||
|
skip_empty_count = 0
|
||||||
|
skip_no_fm_count = 0
|
||||||
|
fail_count = 0
|
||||||
|
processed = len(md_files)
|
||||||
|
|
||||||
|
for idx, file_path in enumerate(md_files, 1):
|
||||||
|
rel_path = str(file_path.relative_to(root)).replace("\\", "/")
|
||||||
|
|
||||||
|
# 跳过空文件
|
||||||
|
if file_path.stat().st_size == 0:
|
||||||
|
print(f"[{idx}/{processed}] [SKIP] {rel_path} -- 空文件,跳过")
|
||||||
|
skip_empty_count += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 读取内容
|
||||||
|
try:
|
||||||
|
content_bytes = file_path.read_bytes()
|
||||||
|
except Exception as e:
|
||||||
|
print(f"[{idx}/{processed}] [FAIL] {rel_path} -- 读取失败:{e}")
|
||||||
|
fail_count += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 检查 frontmatter
|
||||||
|
content_str = content_bytes.decode("utf-8", errors="replace")
|
||||||
|
if not has_frontmatter(content_str):
|
||||||
|
print(f"[{idx}/{processed}] [SKIP] {rel_path} -- 无 frontmatter,跳过")
|
||||||
|
skip_no_fm_count += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 查重(用 Obsidian 相对路径作为 source_file_name)
|
||||||
|
source_file_name = rel_path
|
||||||
|
if check_exists(source_file_name):
|
||||||
|
print(f"[{idx}/{processed}] [SKIP] {rel_path} -- 已存在,跳过")
|
||||||
|
skip_exists_count += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
# dry-run 模式:只解析不入库
|
||||||
|
if args.dry_run:
|
||||||
|
try:
|
||||||
|
parsed = service.parse_md_file(content_bytes, source_file_name)
|
||||||
|
char_count = len(parsed["content_md"])
|
||||||
|
print(f"[{idx}/{processed}] [PREVIEW] {rel_path}")
|
||||||
|
print(f" title={parsed['title']}")
|
||||||
|
print(f" source_type={parsed['source_type']}, author={parsed['author']}")
|
||||||
|
print(f" metadata={parsed['metadata']}")
|
||||||
|
print(f" entities={len(parsed['related_entities'] or [])}条, 正文{char_count}字")
|
||||||
|
success_count += 1
|
||||||
|
except Exception as e:
|
||||||
|
print(f"[{idx}/{processed}] [FAIL] {rel_path} -- 解析失败:{e}")
|
||||||
|
fail_count += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
# 正式入库
|
||||||
|
try:
|
||||||
|
parsed = service.parse_md_file(content_bytes, source_file_name)
|
||||||
|
char_count = len(parsed["content_md"])
|
||||||
|
service.store_md_file(file_content=content_bytes, file_name=source_file_name)
|
||||||
|
print(f"[{idx}/{processed}] [OK] {rel_path} -- 入库成功({char_count}字,类型={parsed['source_type']})")
|
||||||
|
success_count += 1
|
||||||
|
except Exception as e:
|
||||||
|
print(f"[{idx}/{processed}] [FAIL] {rel_path} -- 入库失败:{e}")
|
||||||
|
fail_count += 1
|
||||||
|
|
||||||
|
# 汇总
|
||||||
|
print(f"\n{'='*60}")
|
||||||
|
mode_label = "预览" if args.dry_run else "导入"
|
||||||
|
print(f"{mode_label}完成:成功 {success_count} 篇"
|
||||||
|
f" / 跳过(已存在) {skip_exists_count} 篇"
|
||||||
|
f" / 跳过(空文件) {skip_empty_count} 篇"
|
||||||
|
f" / 跳过(无frontmatter) {skip_no_fm_count} 篇"
|
||||||
|
f" / 失败 {fail_count} 篇")
|
||||||
|
print(f"{'='*60}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -5,6 +5,19 @@ import useAuthStore from '../../stores/authStore'
|
|||||||
import knowledgeService from '../../services/knowledgeService'
|
import knowledgeService from '../../services/knowledgeService'
|
||||||
import KnowledgeTree from '../../components/KnowledgeTree/KnowledgeTree'
|
import KnowledgeTree from '../../components/KnowledgeTree/KnowledgeTree'
|
||||||
|
|
||||||
|
// 渲染 snippet 中的 **keyword** 加粗标记(不使用 dangerouslySetInnerHTML)
|
||||||
|
function renderSnippet(text) {
|
||||||
|
if (!text) return null
|
||||||
|
const parts = text.split(/\*\*(.+?)\*\*/g)
|
||||||
|
return parts.map((part, i) => {
|
||||||
|
// 奇数索引是被 ** 包裹的关键词
|
||||||
|
if (i % 2 === 1) {
|
||||||
|
return <strong key={i} style={{ color: '#1677ff' }}>{part}</strong>
|
||||||
|
}
|
||||||
|
return <span key={i}>{part}</span>
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
const { Dragger } = Upload
|
const { Dragger } = Upload
|
||||||
|
|
||||||
// source_type 枚举值(固定五类,不写死但供 Select 用)
|
// source_type 枚举值(固定五类,不写死但供 Select 用)
|
||||||
@@ -38,6 +51,20 @@ export default function KnowledgeBase() {
|
|||||||
const [searchQuery, setSearchQuery] = useState('')
|
const [searchQuery, setSearchQuery] = useState('')
|
||||||
const [searchResults, setSearchResults] = useState([])
|
const [searchResults, setSearchResults] = useState([])
|
||||||
const [searchLoading, setSearchLoading] = useState(false)
|
const [searchLoading, setSearchLoading] = useState(false)
|
||||||
|
const [expandedIds, setExpandedIds] = useState(new Set()) // 展开全文的卡片 id 集合
|
||||||
|
|
||||||
|
// 切换卡片展开/收起
|
||||||
|
const toggleExpand = (id) => {
|
||||||
|
setExpandedIds(prev => {
|
||||||
|
const next = new Set(prev)
|
||||||
|
if (next.has(id)) {
|
||||||
|
next.delete(id)
|
||||||
|
} else {
|
||||||
|
next.add(id)
|
||||||
|
}
|
||||||
|
return next
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
const fetchItems = async () => {
|
const fetchItems = async () => {
|
||||||
setLoading(true)
|
setLoading(true)
|
||||||
@@ -287,8 +314,8 @@ export default function KnowledgeBase() {
|
|||||||
{searchQuery && (
|
{searchQuery && (
|
||||||
<span style={{ fontSize: 12, color: '#888' }}>
|
<span style={{ fontSize: 12, color: '#888' }}>
|
||||||
{searchResults.length > 0
|
{searchResults.length > 0
|
||||||
? `找到 ${searchResults.length} 条相关结果`
|
? `找到 ${searchResults.length} 条相关结果(相关度 > 30%)`
|
||||||
: '无相关结果'}
|
: '未找到相关内容,试试换个说法?'}
|
||||||
</span>
|
</span>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
@@ -299,11 +326,21 @@ export default function KnowledgeBase() {
|
|||||||
<div style={{ fontSize: 13, color: '#666', marginBottom: 8, fontWeight: 500 }}>
|
<div style={{ fontSize: 13, color: '#666', marginBottom: 8, fontWeight: 500 }}>
|
||||||
搜索结果
|
搜索结果
|
||||||
</div>
|
</div>
|
||||||
{searchResults.map(item => (
|
{searchResults.map(item => {
|
||||||
|
const isHigh = item.similarity >= 0.7
|
||||||
|
const isLow = item.similarity < 0.5
|
||||||
|
const isExpanded = expandedIds.has(item.id)
|
||||||
|
return (
|
||||||
<Card
|
<Card
|
||||||
key={item.id}
|
key={item.id}
|
||||||
size="small"
|
size="small"
|
||||||
style={{ borderRadius: 10, marginBottom: 8 }}
|
style={{
|
||||||
|
borderRadius: 10,
|
||||||
|
marginBottom: 8,
|
||||||
|
opacity: isLow ? 0.65 : 1,
|
||||||
|
background: isLow ? '#fafafa' : undefined,
|
||||||
|
borderLeft: isHigh ? '3px solid #1677ff' : undefined,
|
||||||
|
}}
|
||||||
bodyStyle={{ padding: '12px 16px' }}
|
bodyStyle={{ padding: '12px 16px' }}
|
||||||
>
|
>
|
||||||
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'flex-start' }}>
|
<div style={{ display: 'flex', justifyContent: 'space-between', alignItems: 'flex-start' }}>
|
||||||
@@ -337,25 +374,62 @@ export default function KnowledgeBase() {
|
|||||||
WebkitBoxOrient: 'vertical',
|
WebkitBoxOrient: 'vertical',
|
||||||
}}
|
}}
|
||||||
>
|
>
|
||||||
{item.snippet}
|
{renderSnippet(item.snippet)}
|
||||||
</div>
|
</div>
|
||||||
|
{/* 展开全文区域 */}
|
||||||
|
{isExpanded && item.content_md && (
|
||||||
|
<div
|
||||||
|
style={{
|
||||||
|
fontSize: 12,
|
||||||
|
color: '#444',
|
||||||
|
background: '#fafafa',
|
||||||
|
borderRadius: 6,
|
||||||
|
padding: '10px 12px',
|
||||||
|
marginTop: 8,
|
||||||
|
lineHeight: 1.7,
|
||||||
|
whiteSpace: 'pre-wrap',
|
||||||
|
maxHeight: 400,
|
||||||
|
overflowY: 'auto',
|
||||||
|
border: '1px solid #e8e8e8',
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
{item.content_md}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{/* 展开/收起按钮 */}
|
||||||
|
{item.content_md && (
|
||||||
|
<div style={{ marginTop: 6 }}>
|
||||||
|
<span
|
||||||
|
onClick={() => toggleExpand(item.id)}
|
||||||
|
style={{
|
||||||
|
fontSize: 12,
|
||||||
|
color: '#1677ff',
|
||||||
|
cursor: 'pointer',
|
||||||
|
userSelect: 'none',
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
{isExpanded ? '收起全文 ▲' : '展开全文 ▼'}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
<div style={{
|
<div style={{
|
||||||
marginLeft: 16,
|
marginLeft: 16,
|
||||||
minWidth: 56,
|
minWidth: 56,
|
||||||
textAlign: 'center',
|
textAlign: 'center',
|
||||||
background: item.similarity >= 0.7 ? '#e6f4ff' : '#f5f5f5',
|
background: isHigh ? '#e6f4ff' : '#f5f5f5',
|
||||||
borderRadius: 8,
|
borderRadius: 8,
|
||||||
padding: '4px 8px',
|
padding: '4px 8px',
|
||||||
}}>
|
}}>
|
||||||
<div style={{ fontSize: 18, fontWeight: 700, color: item.similarity >= 0.7 ? '#1677ff' : '#666' }}>
|
<div style={{ fontSize: 18, fontWeight: 700, color: isHigh ? '#1677ff' : '#666' }}>
|
||||||
{Math.max(0, Math.round(item.similarity * 100))}%
|
{Math.max(0, Math.round(item.similarity * 100))}%
|
||||||
</div>
|
</div>
|
||||||
<div style={{ fontSize: 11, color: '#888' }}>相关度</div>
|
<div style={{ fontSize: 11, color: '#888' }}>相关度</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</Card>
|
</Card>
|
||||||
))}
|
)
|
||||||
|
})}
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
|||||||
@@ -56,9 +56,9 @@ const knowledgeService = {
|
|||||||
/**
|
/**
|
||||||
* 语义检索:输入一段文字,返回最相关的知识库条目
|
* 语义检索:输入一段文字,返回最相关的知识库条目
|
||||||
* @param {string} queryText - 查询文字
|
* @param {string} queryText - 查询文字
|
||||||
* @param {number} topK - 返回条数,默认 5
|
* @param {number} topK - 返回条数,默认 10
|
||||||
*/
|
*/
|
||||||
async searchItems(queryText, topK = 5) {
|
async searchItems(queryText, topK = 10) {
|
||||||
const resp = await http.post('/knowledge/search', {
|
const resp = await http.post('/knowledge/search', {
|
||||||
query: queryText,
|
query: queryText,
|
||||||
top_k: topK,
|
top_k: topK,
|
||||||
|
|||||||
Reference in New Issue
Block a user