Issue #496 step 2 — security_review verification backfill (assessed 2026-11-15)
Assessed commit: 7b236db9fb3eceaa8a24ec1d419724c9c8a2eeb4 (origin/main after the
review-driven rebase; the first pass was run at 13079ab4). The evidence is pinned per skill in
the verification baseline overrides in
docs/dev/production-verification-baseline.yaml: the mechanical scan and
reconciliation are reproduced by scripts/backfill_security_review.py, and each
skill-override evidence carries the assessment date, the process used, and the
assessed commit. The evidence is an automated heuristic scan; no named human
reviewer is recorded, and a reviewer should be added only after that person has
reviewed the skill.
Scope
This is the second slice of issue #496. Step 1 (#588) backfilled the
unit_tests axis. This step covers the security_review axis for production
skills that read an API key, following the lightweight checklist in
docs/dev/security-review-checklist.md.
The issue acceptance criterion for this axis reads “security_review is passed
for at least all production skills that read API keys”. This PR records the
first pass against that criterion. Acceptance criteria not covered here —
instruction_contract for every canonical-workflow skill, and the
“stale-evidence reset rule enforced by validate_skills_index.py” — remain open
for follow-ups and are called out below.
Method
- Enumeration:
scripts/backfill_security_review.pyscansskills/<id>/scripts/**/*.py(excludingtests, the same rule as #588) forFMP_API_KEY|FINVIZ_API_KEY|ALPACA_API_KEY|ALPACA_SECRET_KEY|--api-key|X-API-Keyand reconciles that set against the SSoTintegrations[]declarations (idinfmp/finviz/alpaca). Integrationnotetext (for example “Alpaca order templates” onbreakout-trade-planner) is deliberately not treated as a key declaration. - Check: per-skill mechanical checks — hard-coded credential literal, shell
invocation, a key-bearing request combined with an unredacted exception
print/log (
requestsexception text embeds the full URL including the query string), explicit network timeout, dependency manifest — plus the repo-widescripts/check_skill_deps.py check(exit 0). - Fail-closed gate:
--reviewer-confirmed <id>starts empty; a skill is only proposedpassedwhen it is mechanically clean and explicitly reviewer-confirmed. The raw report (reports/security_review_evidence.json) records every per-skill finding so the pass can be independently re-checked.reports/is gitignored, so the report is not a committed artifact — the recorded evidence pins the reviewed source directories, the reviewer/date/ commit, and the regenerable commandpython3 scripts/backfill_security_review.py --reviewer-confirmed <ids> ... --output-dir reports/, mirroring the #588 precedent of citing reproducible evidence rather than committing report files.
Enumeration result (28 target skills)
| Group | Count | Skills |
|---|---|---|
| Key readers via the generated FMP client | 17 | canslim-screener, cot-contrarian-detector, dividend-growth-pullback-screener, earnings-calendar, earnings-trade-analyzer, economic-calendar-fetcher, ftd-detector, ibd-distribution-day-monitor, institutional-flow-tracker, kanchi-dividend-sop, macro-regime-detector, market-top-detector, pair-trade-screener, parabolic-short-trade-planner, pead-screener, value-dividend-screener, vcp-screener |
| Direct HTTP with explicit timeout | 8 | downtrend-duration-analyzer, news-reaction-failure-analyzer, options-strategy-advisor, portfolio-manager, signal-postmortem, technical-analyst, theme-detector, trader-memory-core |
| No HTTP data fetch in scripts | 3 | edge-candidate-agent, finviz-screener, kanchi-dividend-review-monitor |
Reconciliation against integrations[]:
declared_but_no_script_ref(integration declares a key source but scripts do not reference the credential):edge-candidate-agent,kanchi-dividend-review-monitor,options-strategy-advisor. These are alloptional/recommendedenrichments where the credential is consumed through a caller path rather than a literal in the skill’s scripts; each is still reviewed.script_ref_but_not_declared(scripts reference a credential theintegrations[]do not declare): only the two scanner-only meta skills,dual-axis-skill-reviewerandskill-integration-tester, which contain the credential pattern because they audit other skills. They are excluded from the reviewed set by an explicit allowlist and are not key readers.
Excluded from the reviewed set (with reason)
dual-axis-skill-reviewer,skill-integration-tester— meta skills that scan other skills for credential patterns; their own scripts do not read a key.breakout-trade-planner—integrations[]islocal_calculation(“Alpaca order templates”); its scripts contain no key reference, so it does not read an API key.
Outcome
22 of the 28 target skills are recorded security_review: passed (index value +
baseline override with per-skill rationale and evidence). Six skills are recorded
not_verified because they send a key in the request and print or log the
requests exception, whose text contains the request URL with the key:
| Skill | Leak path |
|---|---|
| downtrend-duration-analyzer | analyze_downtrends.py: apikey in params, {e} to stderr |
| value-dividend-screener | screen_dividend_stocks.py: FINVIZ auth in params, {e} to stderr |
| dividend-growth-pullback-screener | screen_dividend_growth_rsi.py: FINVIZ auth in params, {e} to stderr |
| parabolic-short-trade-planner | check_live_apis.py: apikey in params, {e} to stdout |
| canslim-screener | check_institutional_endpoint.py: apikey in params, {e} to stderr |
| theme-detector | representative_stock_selector.py (auth= in URL, logger.exception) and finviz_performance_client.py |
These skills need redaction (the _redact pattern in technical-analyst) in a
follow-up before the axis can pass; their code is unchanged here. The collector
now flags this pattern (exception_key_leak_findings) and counts it against
clean. The scan is a file-level heuristic, not a data-flow proof, and it does
not check logging of the key value in general.
Policy for the three skills with no key read in their scripts
(edge-candidate-agent, kanchi-dividend-review-monitor) or a caller-provided
key only (options-strategy-advisor): they are recorded passed because their
integrations[] declare a key source and the scripts make no key-bearing
requests; the rationale says so explicitly. The global audit.baseline_commit
is intentionally left untouched (the #588 precedent pins per-axis evidence at
per-override assessed_commit values instead).
Not covered by this PR (tracked under #496)
instruction_contractfor every canonical-workflow skill — follow-up.- The “stale-evidence reset rule enforced by
validate_skills_index.py” — this rule currently exists only as prose indocs/dev/production-verification.md(“Keeping evidence current”); no mechanical staleness reset exists in the validator. This PR does not claim to satisfy that criterion and leaves it open. unit_testsre-backfill of the skills leftnot_verifiedin step 1 (#588, 47 skills) — follow-up.