blind_eval.py runs PoC #1's blind-eval methodology over N held-out
corpus dialogues automatically (generate_draft.py refactored to
expose draft_reply() so both share the same drafting logic).
retrieve_style.py implements the keyword/recency search from
tech-design.md §2-1 and wires into generate_draft.py as --history,
replacing hand-curated --style files.
Bash access was intermittently restricted for part of this session
(auto-mode safety classifier), so these were initially written and
committed-pending without live execution. Now verified for real:
generate_draft.py's existing behavior still holds after the
draft_reply() refactor, blind_eval.py runs cleanly against val.jsonl,
and retrieve_style.py's original weighted-sum scoring had a real bug
-- recency drowned out keyword overlap for short Korean messages
(particle attachment means "핀란드" and "핀란드는" don't share a
token), so it was effectively returning the most recent messages
regardless of topic. Fixed by ranking on (overlap, recency) instead
of a weighted sum, confirmed the Finland-related exemplar now ranks
first for a matching query.
escalation_filter.py implements tech-design.md §3's first step as an
actual hard gate, not just a system-prompt instruction: money,
appointment-confirmation, and emotional content stop generate_draft.py
before it ever calls Gemini. Self-test 10/10; measured a 0.93% trigger
rate against 82,305 real corpus utterances (mostly factual price
mentions, not personal money requests -- noted as an upper bound, not
a real-usage estimate).
Provide .env.example, ignore local .env, and teach generate_draft.py to
load the repo-root .env so PoC #1 can use Gemini without exporting keys
in the shell every time.
Co-authored-by: okuma <o0kuma@users.noreply.github.com>
build_unlabeled_corpus.py pulls the 34,030 raw-source dialogues that
never got speech_act/slot labels (found via the earlier QA
cross-check) into their own unlabeled.jsonl, forward-filling the
per-dialogue metadata that AI-Hub's CSV export only writes on each
dialogue's first row. Verified against the actual data: 176,605
raw - 142,575 labeled = 34,030, matches exactly.
Uses google-genai + GEMINI_API_KEY instead of the Anthropic SDK, per
product decision to run the server-fallback LLM on Gemini. Same
prompt contract and escalation behavior; only the client/env var
name changed. README documents where to actually set the key.
generate_draft.py takes a few style-exemplar messages plus recent
conversation context and drafts a reply via LLM call (the server
fallback path from tech-design.md §2), with escalation baked into
the system prompt for money/appointment/emotional content. Verified
prompt construction against a real corpus dialogue and hand-compared
a generated draft to the withheld real reply (README "샘플 검증") --
no API key in this session, so the live call itself is untested.
Consolidates the AI-Hub "한국어 SNS 멀티턴 대화" TL/VL zip parts into
clean train/val JSONL (142,575 dialogues), with a QA cross-check
against the raw TS/VS source. Script and docs only -- the dataset
itself stays out of git per .gitignore, both for size and because
AI-Hub's terms restrict redistribution.
Claude
cursor[bot]
Cursor Agent