blind_eval.py runs PoC #1's blind-eval methodology over N held-out
corpus dialogues automatically (generate_draft.py refactored to
expose draft_reply() so both share the same drafting logic).
retrieve_style.py implements the keyword/recency search from
tech-design.md §2-1 and wires into generate_draft.py as --history,
replacing hand-curated --style files.
Bash access was intermittently restricted for part of this session
(auto-mode safety classifier), so these were initially written and
committed-pending without live execution. Now verified for real:
generate_draft.py's existing behavior still holds after the
draft_reply() refactor, blind_eval.py runs cleanly against val.jsonl,
and retrieve_style.py's original weighted-sum scoring had a real bug
-- recency drowned out keyword overlap for short Korean messages
(particle attachment means "핀란드" and "핀란드는" don't share a
token), so it was effectively returning the most recent messages
regardless of topic. Fixed by ranking on (overlap, recency) instead
of a weighted sum, confirmed the Finland-related exemplar now ranks
first for a matching query.
escalation_filter.py implements tech-design.md §3's first step as an
actual hard gate, not just a system-prompt instruction: money,
appointment-confirmation, and emotional content stop generate_draft.py
before it ever calls Gemini. Self-test 10/10; measured a 0.93% trigger
rate against 82,305 real corpus utterances (mostly factual price
mentions, not personal money requests -- noted as an upper bound, not
a real-usage estimate).
v1 doesn't train a custom model: it retrieves the closest-matching
past messages from the person's own history and feeds them as
few-shot exemplars to the same prompt contract generate_draft.py
already implements, via the hosted Gemini call. Narrows the AI-Hub
base corpus's role to evaluation and future on-device distillation,
since a hosted LLM already covers general Korean fluency.
Resolves the open Q1-Q7 questions as tentative decisions (self-app beta
first, target consumers, MVP scenario = read-receipt relief + group-chat
catch-up, autonomy capped at L0-L2) and builds the standard deliverable
set on top of them, ready for review at the next meeting.
Claude