From b4778b03d22c604e3b096c3dc387b4542662a7d2 Mon Sep 17 00:00:00 2001 From: Woohyeok Choi Date: Wed, 16 Sep 2026 16:05:48 +0900 Subject: [PATCH] =?UTF-8?q?[Feat]=20=EB=8F=99=EC=A0=81=20Few-shot=20?= =?UTF-8?q?=EC=9A=B4=EC=98=81=20=EB=AA=A8=EB=8B=88=ED=84=B0=EB=A7=81=20?= =?UTF-8?q?=EC=A7=80=ED=91=9C=20=EC=B6=94=EA=B0=80?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Few-shot 선택 모드별 실행 횟수 기록 - embedding 및 fallback 선택 비율 관측 지원 - 검색 지연 histogram과 선택 후보 수 분포 추가 - Few-shot 전용 Cohere 논리 호출량 및 실패 유형 기록 - 메트릭 태그 cardinality 제한 - 단계적 운영 활성화와 중단 판단 기준 문서화 - feature flag 기반 즉시 정적 Few-shot 복귀 절차 추가 - fallback 및 Cohere 실패 메트릭 회귀 테스트 추가 --- docs/fewshot-operations.md | 94 +++++++++++++++++++ .../fewshot/DefaultFewShotSearchService.java | 44 +++++++-- .../ai/fewshot/FewShotMetricsRecorder.java | 67 +++++++++++++ .../DefaultFewShotSearchServiceTest.java | 9 +- .../fewshot/FewShotMetricsRecorderTest.java | 29 ++++++ .../fewshot/ReviewedFewShotDatasetTest.java | 2 +- 6 files changed, 237 insertions(+), 8 deletions(-) create mode 100644 docs/fewshot-operations.md create mode 100644 src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotMetricsRecorder.java create mode 100644 src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotMetricsRecorderTest.java diff --git a/docs/fewshot-operations.md b/docs/fewshot-operations.md new file mode 100644 index 0000000..f3711b0 --- /dev/null +++ b/docs/fewshot-operations.md @@ -0,0 +1,94 @@ +# 동적 Few-shot 단계적 운영 가이드 + +## 운영 전제 + +- 기본값 `ANALYSIS_FEW_SHOT_DYNAMIC_SELECTION_ENABLED=false`를 유지합니다. +- 권장 검색값은 `top-k=5`, `min-similarity=0.40`, `minimum-selected-count=2`입니다. +- 승인 데이터셋 버전을 명시하고, 후보 변경 시 새 버전으로 배포합니다. +- 개인정보 정책은 `docs/fewshot-privacy-masking.md`를 따릅니다. + +## 관측 지표 + +Prometheus endpoint는 관리 포트의 `/actuator/prometheus`입니다. + +| 지표 | 태그 | 용도 | +| --- | --- | --- | +| `fewshot_selection_count_total` | `mode`, `cache_hit` | EMBEDDING·LOCAL_FALLBACK·STATIC_FALLBACK 비율 | +| `fewshot_selection_duration_seconds` | `mode`, `cache_hit` | 검색 평균·P95 지연 | +| `fewshot_selection_selected_candidates_count/_sum` | `mode`, `cache_hit` | 선택 후보 수 분포 | +| `fewshot_cohere_logical_calls_total` | 없음 | Few-shot이 발생시킨 Cohere 논리 호출량 | +| `fewshot_cohere_failure_count_total` | `reason` | Cohere 검색 실패 유형 | + +원문 JD·답변·검색 텍스트·embedding은 지표 태그나 로그에 넣지 않습니다. `reason`은 정해진 예외 +분류만 허용하고 그 외 값은 `Other`로 묶어 tag cardinality 증가를 방지합니다. + +대표 PromQL: + +```promql +sum(rate(fewshot_selection_count_total[10m])) by (mode) + +sum(rate(fewshot_selection_count_total{mode=~"LOCAL_FALLBACK|STATIC_FALLBACK"}[10m])) +/ +sum(rate(fewshot_selection_count_total[10m])) + +histogram_quantile( + 0.95, + sum(rate(fewshot_selection_duration_seconds_bucket[10m])) by (le) +) + +sum(rate(fewshot_cohere_failure_count_total[10m])) by (reason) + +sum(increase(fewshot_cohere_logical_calls_total[1h])) +``` + +## 단계적 활성화 + +애플리케이션 feature flag는 boolean이므로 트래픽 비율은 배포 플랫폼의 인스턴스 또는 라우팅 +단위로 나눕니다. + +1. 평가·내부 인스턴스에서만 활성화하고 최소 1일 관측합니다. +2. 운영 canary 인스턴스 5%에서 활성화합니다. +3. 이상이 없으면 25% → 50% → 100% 순으로 확대합니다. +4. 각 단계에서 최소 하나의 일간 피크 구간을 포함해 관측합니다. +5. 단계 변경 시 datasetVersion·설정값·배포 시각을 운영 기록에 남깁니다. + +활성화 환경변수: + +```text +ANALYSIS_FEW_SHOT_DYNAMIC_SELECTION_ENABLED=true +ANALYSIS_FEW_SHOT_DATASET_VERSION=fewshot-pm-reviewed-20260914-v2 +ANALYSIS_FEW_SHOT_TOP_K=5 +ANALYSIS_FEW_SHOT_MIN_SIMILARITY=0.40 +ANALYSIS_FEW_SHOT_MINIMUM_SELECTED_COUNT=2 +``` + +## 중단 판단 기준 + +초기 canary에서는 다음 중 하나면 확대를 멈추고 원인을 확인합니다. + +- 10분간 LOCAL_FALLBACK + STATIC_FALLBACK 비율이 10% 초과 +- 10분간 Cohere 실패가 Few-shot 선택 요청의 5% 초과 +- Few-shot 선택 P95가 2초 초과 또는 기존 기준 대비 30% 이상 증가 +- 분석 전체 P95가 기존 기준 대비 20% 이상 증가 +- 시간당 Cohere 호출량이나 OpenAI 입력 토큰이 예상 범위를 20% 이상 초과 +- 회귀 샘플에서 unsupported fact 또는 false positive 증가 + +트래픽이 적으면 짧은 비율만으로 판단하지 않고 최소 20건 이상의 표본을 함께 확인합니다. + +## 즉시 복귀 + +1. 활성 인스턴스의 `ANALYSIS_FEW_SHOT_DYNAMIC_SELECTION_ENABLED=false`로 재배포합니다. +2. 선택 모드가 STATIC으로 돌아왔는지 로그와 지표로 확인합니다. +3. fallback이 아니라 feature flag 비활성으로 복귀했는지 확인합니다. +4. datasetVersion, 오류 시각, 실패 reason, 지연과 호출량을 장애 기록에 남깁니다. +5. 승인 데이터나 캐시를 삭제하지 않습니다. 원인 수정 후 평가 환경에서 재검증합니다. + +flag를 끄면 기존 정적 Few-shot 프롬프트로 돌아가며 API 응답·DB 스키마는 바뀌지 않습니다. + +## 활성화 승인 조건 + +- 승인 후보 데이터와 개인정보 검수 완료 +- STATIC/DYNAMIC 품질 비교와 튜닝 회귀 평가 완료 +- fallback·Cohere 실패·검색 P95·호출량 대시보드 확인 가능 +- canary 단계에서 중단 기준 미충족 +- 장애 시 flag 비활성 재배포 절차를 담당자가 실행 가능 diff --git a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchService.java b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchService.java index 9ab9ddf..21dee7a 100644 --- a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchService.java +++ b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchService.java @@ -2,6 +2,7 @@ import com.jobdri.jobdri_api.global.cohere.CohereEmbeddingClient; import lombok.extern.slf4j.Slf4j; +import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import org.springframework.util.StringUtils; @@ -37,6 +38,7 @@ public class DefaultFewShotSearchService implements FewShotSearchService { private final FewShotSearchTextBuilder textBuilder; private final CohereEmbeddingClient cohereEmbeddingClient; private final FewShotProperties properties; + private final FewShotMetricsRecorder metricsRecorder; private final Map selectionCache = new ConcurrentHashMap<>(); private final Map queryEmbeddingCache = new ConcurrentHashMap<>(); private final Map> queryEmbeddingInFlight = @@ -47,16 +49,19 @@ public class DefaultFewShotSearchService implements FewShotSearchService { private final Map> documentEmbeddingInFlight = new ConcurrentHashMap<>(); + @Autowired public DefaultFewShotSearchService( FewShotCaseStore caseStore, FewShotSearchTextBuilder textBuilder, CohereEmbeddingClient cohereEmbeddingClient, - FewShotProperties properties + FewShotProperties properties, + FewShotMetricsRecorder metricsRecorder ) { this.caseStore = caseStore; this.textBuilder = textBuilder; this.cohereEmbeddingClient = cohereEmbeddingClient; this.properties = properties; + this.metricsRecorder = metricsRecorder; } @Override @@ -66,11 +71,13 @@ public List searchRelevantFewShots(FewShotSearchQuery query return List.of(); } int requestedTopK = topK > 0 ? topK : properties.getSearch().getTopK(); + long startedAt = System.nanoTime(); List activeCases = caseStore.loadActiveCases(); String datasetFingerprint = datasetFingerprint(activeCases); String cacheKey = selectionCacheKey(query, requestedTopK, datasetFingerprint); SelectionCacheEntry cached = readSelectionCache(cacheKey); if (cached != null) { + recordMetrics(cached.selectionMode(), true, cached.selectedCases().size(), startedAt); log.debug( "few-shot selection cache hit. selectionMode={}, selectedCount={}, datasetVersion={}", cached.selectionMode(), @@ -80,7 +87,6 @@ public List searchRelevantFewShots(FewShotSearchQuery query return cached.selectedCases(); } - long startedAt = System.nanoTime(); List candidates = localPrefilter(activeCases, query); List selected = selectWithCohere(query, candidates, requestedTopK); FewShotSelectionMode selectionMode = selected.isEmpty() @@ -99,6 +105,7 @@ public List searchRelevantFewShots(FewShotSearchQuery query if (properties.isCacheEnabled()) { selectionCache.put(cacheKey, new SelectionCacheEntry(selected, selectionMode, expiresAt())); } + recordMetrics(selectionMode, false, selected.size(), startedAt); log.info( "dynamic few-shot selection completed. enabled=true, selectionMode={}, totalCandidates={}, filteredCandidates={}, selectedIds={}, sources={}, scores={}, latencyMs={}", selectionMode, @@ -112,6 +119,20 @@ public List searchRelevantFewShots(FewShotSearchQuery query return selected; } + private void recordMetrics( + FewShotSelectionMode selectionMode, + boolean cacheHit, + int selectedCount, + long startedAt + ) { + metricsRecorder.recordSelection( + selectionMode, + cacheHit, + selectedCount, + (System.nanoTime() - startedAt) / 1_000_000 + ); + } + private List selectWithCohere( FewShotSearchQuery query, List candidates, @@ -143,6 +164,7 @@ private List selectWithCohere( .thenComparing(item -> item.fewShotCase().id())); return diversify(ranked, topK); } catch (Exception e) { + metricsRecorder.recordCohereFailure(e.getClass().getSimpleName()); log.warn("dynamic few-shot Cohere selection failed. fallback=local, reason={}, message={}", e.getClass().getSimpleName(), e.getMessage()); log.debug("dynamic few-shot Cohere exception", e); return List.of(); @@ -173,7 +195,7 @@ private static String formatScore(double score) { private float[] resolveQueryEmbedding(String queryText) { if (!properties.isCacheEnabled()) { - return cohereEmbeddingClient.embedQuery(queryText); + return embedQuery(queryText); } Instant now = Instant.now(); maintainQueryEmbeddingCache(now); @@ -201,7 +223,7 @@ private float[] resolveQueryEmbedding(String queryText) { try { QueryEmbeddingCacheEntry initialized = new QueryEmbeddingCacheEntry( - cohereEmbeddingClient.embedQuery(queryText), + embedQuery(queryText), expiresAt() ); queryEmbeddingCache.put(key, initialized); @@ -288,7 +310,7 @@ private List resolveDocumentEmbeddings( List documents ) { if (!properties.isCacheEnabled()) { - return cohereEmbeddingClient.embedDocuments(documents); + return embedDocuments(documents); } Instant now = Instant.now(); documentEmbeddingCache.entrySet().removeIf(entry -> entry.getValue().expiresAt().isBefore(now)); @@ -355,7 +377,7 @@ private void initializeMissingDocumentEmbeddings(List return; } try { - List embeddedDocuments = cohereEmbeddingClient.embedDocuments( + List embeddedDocuments = embedDocuments( owned.stream().map(PendingDocumentEmbedding::document).toList() ); if (embeddedDocuments.size() != owned.size()) { @@ -622,4 +644,14 @@ public float[] embedding() { public long cohereApiCallCount() { return cohereEmbeddingClient.apiCallCount(); } + + private float[] embedQuery(String queryText) { + metricsRecorder.recordCohereLogicalCalls(1L); + return cohereEmbeddingClient.embedQuery(queryText); + } + + private List embedDocuments(List documents) { + metricsRecorder.recordCohereLogicalCalls(1L); + return cohereEmbeddingClient.embedDocuments(documents); + } } diff --git a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotMetricsRecorder.java b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotMetricsRecorder.java new file mode 100644 index 0000000..c6d48ab --- /dev/null +++ b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotMetricsRecorder.java @@ -0,0 +1,67 @@ +package com.jobdri.jobdri_api.domain.analysis.service.ai.fewshot; + +import io.micrometer.core.instrument.Counter; +import io.micrometer.core.instrument.DistributionSummary; +import io.micrometer.core.instrument.MeterRegistry; +import io.micrometer.core.instrument.Tags; +import io.micrometer.core.instrument.Timer; +import org.springframework.stereotype.Component; + +import java.time.Duration; +import java.util.concurrent.TimeUnit; + +@Component +public class FewShotMetricsRecorder { + private static final Duration[] SELECTION_SLOS = { + Duration.ofMillis(10), Duration.ofMillis(50), Duration.ofMillis(100), + Duration.ofMillis(250), Duration.ofMillis(500), Duration.ofSeconds(1), + Duration.ofSeconds(3), Duration.ofSeconds(5), Duration.ofSeconds(10) + }; + + private final MeterRegistry meterRegistry; + + public FewShotMetricsRecorder(MeterRegistry meterRegistry) { + this.meterRegistry = meterRegistry; + } + + public void recordSelection(FewShotSelectionMode mode, boolean cacheHit, int selectedCount, long durationMillis) { + Tags tags = Tags.of("mode", mode.name(), "cache_hit", Boolean.toString(cacheHit)); + Counter.builder("fewshot.selection.count").tags(tags).register(meterRegistry).increment(); + Timer.builder("fewshot.selection.duration") + .tags(tags) + .publishPercentileHistogram() + .serviceLevelObjectives(SELECTION_SLOS) + .register(meterRegistry) + .record(durationMillis, TimeUnit.MILLISECONDS); + DistributionSummary.builder("fewshot.selection.selected.candidates") + .tags(tags) + .register(meterRegistry) + .record(selectedCount); + } + + public void recordCohereLogicalCalls(long count) { + if (count > 0) { + Counter.builder("fewshot.cohere.logical.calls") + .register(meterRegistry) + .increment(count); + } + } + + public void recordCohereFailure(String reason) { + Counter.builder("fewshot.cohere.failure.count") + .tag("reason", normalizeReason(reason)) + .register(meterRegistry) + .increment(); + } + + private String normalizeReason(String reason) { + if (reason == null || reason.isBlank()) { + return "Unknown"; + } + return switch (reason) { + case "GeneralException", "IllegalArgumentException", "IllegalStateException", + "TimeoutException", "ExecutionException" -> reason; + default -> "Other"; + }; + } +} diff --git a/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchServiceTest.java b/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchServiceTest.java index 0c01cf0..2411df2 100644 --- a/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchServiceTest.java +++ b/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchServiceTest.java @@ -18,6 +18,8 @@ import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyLong; +import static org.mockito.ArgumentMatchers.eq; import static org.mockito.Mockito.mock; import static org.mockito.Mockito.never; import static org.mockito.Mockito.times; @@ -28,12 +30,14 @@ class DefaultFewShotSearchServiceTest { private final FewShotCaseStore caseStore = mock(FewShotCaseStore.class); private final FewShotSearchTextBuilder textBuilder = new FewShotSearchTextBuilder(); private final CohereEmbeddingClient cohereEmbeddingClient = mock(CohereEmbeddingClient.class); + private final FewShotMetricsRecorder metricsRecorder = mock(FewShotMetricsRecorder.class); private final FewShotProperties properties = new FewShotProperties(); private final DefaultFewShotSearchService service = new DefaultFewShotSearchService( caseStore, textBuilder, cohereEmbeddingClient, - properties + properties, + metricsRecorder ); @Test @@ -96,6 +100,9 @@ void fallsBackToLocalSelectionWhenCohereFails() { assertThat(result).hasSize(1); assertThat(result.getFirst().selectionMethod()).isEqualTo("local-fallback"); + verify(metricsRecorder).recordCohereLogicalCalls(1L); + verify(metricsRecorder).recordCohereFailure("RuntimeException"); + verify(metricsRecorder).recordSelection(eq(FewShotSelectionMode.LOCAL_FALLBACK), eq(false), eq(1), anyLong()); } @Test diff --git a/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotMetricsRecorderTest.java b/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotMetricsRecorderTest.java new file mode 100644 index 0000000..19ba344 --- /dev/null +++ b/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotMetricsRecorderTest.java @@ -0,0 +1,29 @@ +package com.jobdri.jobdri_api.domain.analysis.service.ai.fewshot; + +import io.micrometer.core.instrument.simple.SimpleMeterRegistry; +import org.junit.jupiter.api.Test; + +import static org.assertj.core.api.Assertions.assertThat; + +class FewShotMetricsRecorderTest { + + @Test + void recordsSelectionCallsAndBoundedFailureReason() { + SimpleMeterRegistry registry = new SimpleMeterRegistry(); + FewShotMetricsRecorder recorder = new FewShotMetricsRecorder(registry); + + recorder.recordSelection(FewShotSelectionMode.EMBEDDING, false, 3, 42); + recorder.recordCohereLogicalCalls(2); + recorder.recordCohereFailure("UnexpectedVendorException"); + + assertThat(registry.get("fewshot.selection.count") + .tags("mode", "EMBEDDING", "cache_hit", "false").counter().count()).isEqualTo(1.0); + assertThat(registry.get("fewshot.selection.duration") + .tags("mode", "EMBEDDING", "cache_hit", "false").timer().count()).isEqualTo(1L); + assertThat(registry.get("fewshot.selection.selected.candidates") + .tags("mode", "EMBEDDING", "cache_hit", "false").summary().totalAmount()).isEqualTo(3.0); + assertThat(registry.get("fewshot.cohere.logical.calls").counter().count()).isEqualTo(2.0); + assertThat(registry.get("fewshot.cohere.failure.count") + .tag("reason", "Other").counter().count()).isEqualTo(1.0); + } +} diff --git a/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/ReviewedFewShotDatasetTest.java b/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/ReviewedFewShotDatasetTest.java index 9904d4d..2ea12fb 100644 --- a/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/ReviewedFewShotDatasetTest.java +++ b/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/ReviewedFewShotDatasetTest.java @@ -46,7 +46,7 @@ void excludesActualReviewedInputEvenWithDifferentIdAndWhitespace() { var store = store(); var candidate = store.loadActiveCases().getFirst(); var service = new DefaultFewShotSearchService(store, new FewShotSearchTextBuilder(), - mock(CohereEmbeddingClient.class), properties); + mock(CohereEmbeddingClient.class), properties, mock(FewShotMetricsRecorder.class)); var query = new FewShotSearchQuery("HOLDOUT-COPY", candidate.jobCategory(), candidate.jobTitle(), candidate.mainTasks(), candidate.qualifications(), candidate.question(), " " + candidate.sanitizedAnswer().replace("\n", " ") + " ");