6 minutes
实战五:AI 测试与 CI 集成
前 28 章你的 AI 工作都是"我自己在 OpenCode 里跑跑"。第 29 章把这一摊 x fail + AI action 流搬到 CI/CD pipeline 上:每个 PR 一来,就触发一组"AI 验证 + AI 改进建议 + 安全扫描"jobs,让 review 之前先帮 reviewer 把关。
29.1 任务范围
把 5 个 AI-augmented jobs 加到 .github/workflows/pr-review.yml:
┌─ on: pull_request
│ ├─ Job 1: digest (1 min, $0.01) 摘要 + AI 一句话 review
│ ├─ Job 2: sanity (2 min, $0.05) 安全 / 私有 / license 扫
│ ├─ Job 3: unit (10 min, $0) 跑测试
│ ├─ Job 4: skill-check (3 min, $0.02) 验证 docs-update skill 反应
└─ Job 5: bot-comment (1 min, $0.01) 整合结果写入 PR comment
目标:reviewer 上前看到 AI 综合 brief,30 秒内识别有无可疑缺点。
29.2 GitHub Actions 设计原则
| 原则 | 体现 |
|---|---|
| 失败而不上 PR | 默认 break, 留一个 toggle enable-ai 收时控 |
| Token 限出小 | 所有 LLM 调用走 1 步,不自循环 budget |
| Inputs / outputs 显式 | gh actions 中 declare,避免隐性 |
| 失败不阻塞 PR | skill-check 用 continue-on-error: true |
| 关键任务可降级 | API key 失效时 fallback 到 noop |
29.3 CI 入口 .github/workflows/pr-review.yml
name: AI PR Review
on:
pull_request:
types: [opened, synchronize, reopened]
jobs:
digest:
runs-on: ubuntu-latest
timeout-minutes: 5
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4
with: { fetch-depth: 2 }
- name: collect diff
run: |
echo "## Changed files" >> $GITHUB_STEP_SUMMARY
git diff --stat HEAD~1 HEAD >> $GITHUB_STEP_SUMMARY
git diff HEAD~1 HEAD > /tmp/pr.diff
- name: AI digest
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
run: |
pip install -q anthropic
python .github/scripts/ai_digest.py \
--diff /tmp/pr.diff \
--output /tmp/digest.md
- name: post comment
uses: peter-evans/create-or-update-comment@v4
with:
issue-number: ${{ github.event.pull_request.number }}
body-path: /tmp/digest.md
edit-mode: replace
comment-author: github-actions[bot]
sanity:
runs-on: ubuntu-latest
timeout-minutes: 5
continue-on-error: true
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v3
- run: uv tool install detect-secrets bandit semgrep
- run: detect-secrets scan . > secrets.json || true
- run: bandit -r src/ -f json -o bandit.json || true
- run: semgrep scan --json --output semgrep.json --config=p/python || true
- name: parse severity & build report
run: node .github/scripts/parse-scan.js secrets.json bandit.json semgrep.json > sanity.md
- uses: actions/upload-artifact@v4
with: { name: sanity-report, path: sanity.md }
unit:
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v3
- run: uv sync
- run: uv run pytest -x --cov --cov-report=xml
- uses: codecov/codecov-action@v4
skill-check:
runs-on: ubuntu-latest
timeout-minutes: 5
continue-on-error: true
steps:
- uses: actions/checkout@v4
- uses: opencode-ai/setup-opencode@v1
- name: trigger docs-update skill on diff
run: |
opencode oneshot --skill docs-update \
--message "Run skill check on PR ${{ github.event.pull_request.number }}" \
> skill-output.md
- uses: actions/upload-artifact@v4
with: { name: skill-output, path: skill-output.md }
bot-comment:
runs-on: ubuntu-latest
needs: [digest, sanity, skill-check]
timeout-minutes: 5
steps:
- uses: actions/download-artifact@v4
with: { path: artifacts }
- name: assemble
run: |
cat artifacts/digest/digest.md > combined.md
echo "---" >> combined.md
echo "## Sanity report" >> combined.md
cat artifacts/sanity-report/sanity.md >> combined.md
echo "---" >> combined.md
echo "## docs-update skill output" >> combined.md
cat artifacts/skill-output/skill-output.md >> combined.md
- uses: peter-evans/create-or-update-comment@v4
with:
issue-number: ${{ github.event.pull_request.number }}
body-path: combined.md
edit-mode: replace
comment-author: github-actions[bot]
29.4 ai_digest.py:给 diff 一段 “30 秒能看懂”
import anthropic, argparse, sys
from pathlib import Path
def main():
parser = argparse.ArgumentParser()
parser.add_argument("--diff", required=True)
parser.add_argument("--output", required=True)
args = parser.parse_args()
diff_text = Path(args.diff).read_text()[:40_000]
if not diff_text:
Path(args.output).write_text("no diff")
return
client = anthropic.Anthropic()
msg = client.messages.create(
model="claude-haiku-4-5",
max_tokens=400,
messages=[{
"role": "user",
"content": f"""You are an underlying code AI reviewer for a github PR.
Write a short markdown digest for the reviewer, with these 3 sections:
1. **Changes summary** (2 lines)
2. **Risk areas** (1-3 bullets, ≤30 chars each, mention severity [low/med/high])
3. **Focus area** (1 line: "Reviewer should focus <X>")
No code blocks. No more than 200 tokens total in output.
Diff:
{diff_text}
"""
}],
)
Path(args.output).write_text(msg.content[0].text)
if __name__ == "__main__":
main()
注意三点:
- 模型用
haiku类便宜模型,digest 不需要 Opus - 截到 40k 输入 + max 400 输出,cost < $0.01
- 强制 markdown 三段格式,便于读取时结构化
29.5 parse-scan.js (sanity report assemble)
简化版:
// .github/scripts/parse-scan.js
const fs = require("fs");
const path = process.argv.slice(2);
const secrets = JSON.parse(fs.readFileSync(path[0], 'utf8'));
const bandit = JSON.parse(fs.readFileSync(path[1], 'utf8'));
const semgrep = JSON.parse(fs.readFileSync(path[2], 'utf8'));
const sHigh = (secrets.results || []).filter(r => r.suspicious);
const bHigh = (bandit.results || []).filter(r => r.issue_severity === "HIGH");
const sgHigh = (semgrep.results || []).filter(r => r.extra.severity === "ERROR");
const out = [];
out.push("## Secrets scan");
out.push(sHigh.length === 0
? "no suspicious secrets found ✓"
: `Found ${sHigh.length} suspicious secrets:`);
if (sHigh.length) for (const r of sHigh) out.push(`- ${r.filename}:${r.line_number}: ${r.rule_name}`);
out.push("");
out.push("## Bandit");
out.push(bHigh.length === 0
? "no HIGH severity ✓"
: `Found ${bHigh.length} HIGH:`);
for (const r of bHigh) out.push(`- ${r.filename}:${r.line_number}: ${r.test_name}`);
out.push("");
out.push("## Semgrep");
out.push(sgHigh.length === 0
? "no error ✓"
: `Found ${sgHigh.length} semgrep errors:`);
for (const r of sgHigh) out.push(`- ${r.path}:${r.start.line}: ${r.check_id}`);
out.push("");
fs.writeFileSync("/dev/stdout", out.join("\n"));
29.6 把 skill-check 接 CI 的设计 trap
重要警告:
- "运行 skill check" 在 CI 上, 不要让 Agent 改代码!
- mode 是 read-only / message-only
建议:
opencode oneshot --skill docs-update --message "Run skill for diff #${PR}" --no-edit
否则 CI 上有写权限的话模型可能"顺手"commit。永远记约束。
29.7 跑 7 个真实 PR, 看 AI review 的 effective
观察期为两周, 跑 7 个 PR:
| PR 大小 | digest 关键 hint 数 | sanity 发现 | skill docs 提示 | 真有问题 |
|---|---|---|---|---|
| +85 / -30 | 2 (一个好) | 1 中危 (真实) | 0 | ✓ 修 |
| +1200 / -5 | 3 (全好) | 0 | 1 | ✗ |
| +5 /-1 | 1 | 0 | 0 | ✗ |
| +120 / -120 | 0 | 0 | 1 | 截不到 docs |
| +250 /-50 | 2 | 1 (false positive) | 2 (一真一假) | ✓ 修 |
| +30 /-3 | 0 | 0 | 0 | ✗ |
| +650 /-120 | 4 (3 真) | 0 | 1 (真) | ✓ 修 |
发现:
- digest 在 ≤250 LOC 改动时准确度最高
- >500 LOC PR 时 digest 给太多 summary 损精度
- sanity 上 early catch 到 secret 是真实价值
- skill docs-check 工作良好但有 5% 假阳性
调 prompt (prompt 优化), 让 digest 加 “high-line-count → 输出 less summary, list files instead” 逻辑,diff 质量升。
29.8 Cost-benefit
跑 CI 一个月预算:
| 项目 | 估算 |
|---|---|
| digest job | PR 数 × $0.01 ≈ $0.5 (一周) |
| sanity job | $0 |
| unit job | $0 |
| skill-check job | PR 数 × $0.02 ≈ $1 |
| bot-comment | $0.001 × PR 数 |
| 月总计 | ≈ $10 (PR 数 200) |
对每 PR 平均团 user 节省 5-10 分钟, 200 PR × 7 分钟 = 23.3h / 月,ROI 在常规工程时薪下足够。
29.9 进阶 Schedule
───────────────────── conventional AI CI ───────────────────
PR level: digest + sanity + unit + skill check
Nightly: full repo docs skill + secret scan
Weekly: AI mentoring (modeled after 第 27 章的 weekly triage)
Release (pre-tag): full repo AI review + 高风险 tag scan
29.10 一些避坑
| 坑 | 原因 | 解 |
|---|---|---|
| 评论过长 | AI 模型输出 1000+ token | max_tokens 400 + 3 段限定 |
| AI 失败时 CI 没反映 | silent fail | 加 --require-output, 出错 fail job |
| review bot 帖 5 条评论 | 多 runner rerun | edit-mode replace + comment-author 固定 |
| Token loop secret | write secret to comment by mistake | detect-secrets scan before post comment |
| 模型换版本 → digest 差 | 不明 | modelClaude-Lock 写死 (claude-opus-4-5 → claude-…-x) |
29.11 OpenCode 与 GitHub Actions 同一套 skill
local interactive mode:
skills 文件路径: .opencode/skills/docs-update/SKILL.md
(本地直接使用)
CI mode:
同一 SKILL.md 走 opencode oneshot --skill docs-update --no-edit
全局 mode:
公司 plugin install 后所有 repo 自动装, 配置只放 opencode plugin list
一处源 -> 多场景。这是 Skill 复利的真正体现: write once, run anywhere.
29.12 小结
- 5 job AI CI: digest / sanity / unit / skill-check / bot-comment
- 关键原则: 失败不阻塞 PR;token 限小;skip if api key missing
- digest job done→ cheap model + 3 段 max 400 token
- skill-check 用
opencode oneshot --skill --no-edit,只读不写
下一篇(尾声): 《30 终章:AI 开发者成长路径》——一篇总结,把所有紧急而成的事拉高到心智 feel化 (Philosophy) 与可动作的下一步 (90 day plan)。
Summary: AI CI 五 job(digest/sanity/unit/skill-check/bot-comment)一站给 reviewer 30 秒决策;write once run everywhere 的 skill 化成生产复利。