blind_eval.py runs PoC #1's blind-eval methodology over N held-out
corpus dialogues automatically (generate_draft.py refactored to
expose draft_reply() so both share the same drafting logic).
retrieve_style.py implements the keyword/recency search from
tech-design.md §2-1 and wires into generate_draft.py as --history,
replacing hand-curated --style files.
Bash access was intermittently restricted for part of this session
(auto-mode safety classifier), so these were initially written and
committed-pending without live execution. Now verified for real:
generate_draft.py's existing behavior still holds after the
draft_reply() refactor, blind_eval.py runs cleanly against val.jsonl,
and retrieve_style.py's original weighted-sum scoring had a real bug
-- recency drowned out keyword overlap for short Korean messages
(particle attachment means "핀란드" and "핀란드는" don't share a
token), so it was effectively returning the most recent messages
regardless of topic. Fixed by ranking on (overlap, recency) instead
of a weighted sum, confirmed the Finland-related exemplar now ranks
first for a matching query.
escalation_filter.py implements tech-design.md §3's first step as an
actual hard gate, not just a system-prompt instruction: money,
appointment-confirmation, and emotional content stop generate_draft.py
before it ever calls Gemini. Self-test 10/10; measured a 0.93% trigger
rate against 82,305 real corpus utterances (mostly factual price
mentions, not personal money requests -- noted as an upper bound, not
a real-usage estimate).
Provide .env.example, ignore local .env, and teach generate_draft.py to
load the repo-root .env so PoC #1 can use Gemini without exporting keys
in the shell every time.
Co-authored-by: okuma <o0kuma@users.noreply.github.com>
Uses google-genai + GEMINI_API_KEY instead of the Anthropic SDK, per
product decision to run the server-fallback LLM on Gemini. Same
prompt contract and escalation behavior; only the client/env var
name changed. README documents where to actually set the key.
generate_draft.py takes a few style-exemplar messages plus recent
conversation context and drafts a reply via LLM call (the server
fallback path from tech-design.md §2), with escalation baked into
the system prompt for money/appointment/emotional content. Verified
prompt construction against a real corpus dialogue and hand-compared
a generated draft to the withheld real reply (README "샘플 검증") --
no API key in this session, so the live call itself is untested.
Claude
Cursor Agent