[Feat] 동적 Few-shot 운영 모니터링 지표 추가

docs/fewshot-operations.md

@@ -0,0 +1,94 @@
+# 동적 Few-shot 단계적 운영 가이드
+
+## 운영 전제
+
+- 기본값 `ANALYSIS_FEW_SHOT_DYNAMIC_SELECTION_ENABLED=false`를 유지합니다.
+- 권장 검색값은 `top-k=5`, `min-similarity=0.40`, `minimum-selected-count=2`입니다.
+- 승인 데이터셋 버전을 명시하고, 후보 변경 시 새 버전으로 배포합니다.
+- 개인정보 정책은 `docs/fewshot-privacy-masking.md`를 따릅니다.
+
+## 관측 지표
+
+Prometheus endpoint는 관리 포트의 `/actuator/prometheus`입니다.
+
+| 지표 | 태그 | 용도 |
+| --- | --- | --- |
+| `fewshot_selection_count_total` | `mode`, `cache_hit` | EMBEDDING·LOCAL_FALLBACK·STATIC_FALLBACK 비율 |
+| `fewshot_selection_duration_seconds` | `mode`, `cache_hit` | 검색 평균·P95 지연 |
+| `fewshot_selection_selected_candidates_count/_sum` | `mode`, `cache_hit` | 선택 후보 수 분포 |
+| `fewshot_cohere_logical_calls_total` | 없음 | Few-shot이 발생시킨 Cohere 논리 호출량 |
+| `fewshot_cohere_failure_count_total` | `reason` | Cohere 검색 실패 유형 |
+
+원문 JD·답변·검색 텍스트·embedding은 지표 태그나 로그에 넣지 않습니다. `reason`은 정해진 예외
+분류만 허용하고 그 외 값은 `Other`로 묶어 tag cardinality 증가를 방지합니다.
+
+대표 PromQL:
+
+```promql
+sum(rate(fewshot_selection_count_total[10m])) by (mode)
+
+sum(rate(fewshot_selection_count_total{mode=~"LOCAL_FALLBACK|STATIC_FALLBACK"}[10m]))
+/
+sum(rate(fewshot_selection_count_total[10m]))
+
+histogram_quantile(
+  0.95,
+  sum(rate(fewshot_selection_duration_seconds_bucket[10m])) by (le)
+)
+
+sum(rate(fewshot_cohere_failure_count_total[10m])) by (reason)
+
+sum(increase(fewshot_cohere_logical_calls_total[1h]))
+```
+
+## 단계적 활성화
+
+애플리케이션 feature flag는 boolean이므로 트래픽 비율은 배포 플랫폼의 인스턴스 또는 라우팅
+단위로 나눕니다.
+
+1. 평가·내부 인스턴스에서만 활성화하고 최소 1일 관측합니다.
+2. 운영 canary 인스턴스 5%에서 활성화합니다.
+3. 이상이 없으면 25% → 50% → 100% 순으로 확대합니다.
+4. 각 단계에서 최소 하나의 일간 피크 구간을 포함해 관측합니다.
+5. 단계 변경 시 datasetVersion·설정값·배포 시각을 운영 기록에 남깁니다.
+
+활성화 환경변수:
+
+```text
+ANALYSIS_FEW_SHOT_DYNAMIC_SELECTION_ENABLED=true
+ANALYSIS_FEW_SHOT_DATASET_VERSION=fewshot-pm-reviewed-20260914-v2
+ANALYSIS_FEW_SHOT_TOP_K=5
+ANALYSIS_FEW_SHOT_MIN_SIMILARITY=0.40
+ANALYSIS_FEW_SHOT_MINIMUM_SELECTED_COUNT=2
+```
+
+## 중단 판단 기준
+
+초기 canary에서는 다음 중 하나면 확대를 멈추고 원인을 확인합니다.
+
+- 10분간 LOCAL_FALLBACK + STATIC_FALLBACK 비율이 10% 초과
+- 10분간 Cohere 실패가 Few-shot 선택 요청의 5% 초과
+- Few-shot 선택 P95가 2초 초과 또는 기존 기준 대비 30% 이상 증가
+- 분석 전체 P95가 기존 기준 대비 20% 이상 증가
+- 시간당 Cohere 호출량이나 OpenAI 입력 토큰이 예상 범위를 20% 이상 초과
+- 회귀 샘플에서 unsupported fact 또는 false positive 증가
+
+트래픽이 적으면 짧은 비율만으로 판단하지 않고 최소 20건 이상의 표본을 함께 확인합니다.
+
+## 즉시 복귀
+
+1. 활성 인스턴스의 `ANALYSIS_FEW_SHOT_DYNAMIC_SELECTION_ENABLED=false`로 재배포합니다.
+2. 선택 모드가 STATIC으로 돌아왔는지 로그와 지표로 확인합니다.
+3. fallback이 아니라 feature flag 비활성으로 복귀했는지 확인합니다.
+4. datasetVersion, 오류 시각, 실패 reason, 지연과 호출량을 장애 기록에 남깁니다.
+5. 승인 데이터나 캐시를 삭제하지 않습니다. 원인 수정 후 평가 환경에서 재검증합니다.
+
+flag를 끄면 기존 정적 Few-shot 프롬프트로 돌아가며 API 응답·DB 스키마는 바뀌지 않습니다.
+
+## 활성화 승인 조건
+
+- 승인 후보 데이터와 개인정보 검수 완료
+- STATIC/DYNAMIC 품질 비교와 튜닝 회귀 평가 완료
+- fallback·Cohere 실패·검색 P95·호출량 대시보드 확인 가능
+- canary 단계에서 중단 기준 미충족
+- 장애 시 flag 비활성 재배포 절차를 담당자가 실행 가능

src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchService.java

@@ -2,6 +2,7 @@
 
 import com.jobdri.jobdri_api.global.cohere.CohereEmbeddingClient;
 import lombok.extern.slf4j.Slf4j;
+import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.stereotype.Service;
 import org.springframework.util.StringUtils;
 
@@ -37,6 +38,7 @@ public class DefaultFewShotSearchService implements FewShotSearchService {
     private final FewShotSearchTextBuilder textBuilder;
     private final CohereEmbeddingClient cohereEmbeddingClient;
     private final FewShotProperties properties;
+    private final FewShotMetricsRecorder metricsRecorder;
     private final Map<String, SelectionCacheEntry> selectionCache = new ConcurrentHashMap<>();
     private final Map<String, QueryEmbeddingCacheEntry> queryEmbeddingCache = new ConcurrentHashMap<>();
     private final Map<String, CompletableFuture<QueryEmbeddingCacheEntry>> queryEmbeddingInFlight =
@@ -47,16 +49,19 @@ public class DefaultFewShotSearchService implements FewShotSearchService {
     private final Map<String, CompletableFuture<DocumentEmbeddingCacheEntry>> documentEmbeddingInFlight =
             new ConcurrentHashMap<>();
 
+    @Autowired
     public DefaultFewShotSearchService(
             FewShotCaseStore caseStore,
             FewShotSearchTextBuilder textBuilder,
             CohereEmbeddingClient cohereEmbeddingClient,
-            FewShotProperties properties
+            FewShotProperties properties,
+            FewShotMetricsRecorder metricsRecorder
     ) {
         this.caseStore = caseStore;
         this.textBuilder = textBuilder;
         this.cohereEmbeddingClient = cohereEmbeddingClient;
         this.properties = properties;
+        this.metricsRecorder = metricsRecorder;
     }
 
     @Override
@@ -66,11 +71,13 @@ public List<SelectedFewShotCase> searchRelevantFewShots(FewShotSearchQuery query
             return List.of();
         }
         int requestedTopK = topK > 0 ? topK : properties.getSearch().getTopK();
+        long startedAt = System.nanoTime();
         List<FewShotCase> activeCases = caseStore.loadActiveCases();
         String datasetFingerprint = datasetFingerprint(activeCases);
         String cacheKey = selectionCacheKey(query, requestedTopK, datasetFingerprint);
         SelectionCacheEntry cached = readSelectionCache(cacheKey);
         if (cached != null) {
+            recordMetrics(cached.selectionMode(), true, cached.selectedCases().size(), startedAt);
             log.debug(
                     "few-shot selection cache hit. selectionMode={}, selectedCount={}, datasetVersion={}",
                     cached.selectionMode(),
@@ -80,7 +87,6 @@ public List<SelectedFewShotCase> searchRelevantFewShots(FewShotSearchQuery query
             return cached.selectedCases();
         }
 
-        long startedAt = System.nanoTime();
         List<FewShotCase> candidates = localPrefilter(activeCases, query);
         List<SelectedFewShotCase> selected = selectWithCohere(query, candidates, requestedTopK);
         FewShotSelectionMode selectionMode = selected.isEmpty()
@@ -99,6 +105,7 @@ public List<SelectedFewShotCase> searchRelevantFewShots(FewShotSearchQuery query
         if (properties.isCacheEnabled()) {
             selectionCache.put(cacheKey, new SelectionCacheEntry(selected, selectionMode, expiresAt()));
         }
+        recordMetrics(selectionMode, false, selected.size(), startedAt);
         log.info(
                 "dynamic few-shot selection completed. enabled=true, selectionMode={}, totalCandidates={}, filteredCandidates={}, selectedIds={}, sources={}, scores={}, latencyMs={}",
                 selectionMode,
@@ -112,6 +119,20 @@ public List<SelectedFewShotCase> searchRelevantFewShots(FewShotSearchQuery query
         return selected;
     }
 
+    private void recordMetrics(
+            FewShotSelectionMode selectionMode,
+            boolean cacheHit,
+            int selectedCount,
+            long startedAt
+    ) {
+        metricsRecorder.recordSelection(
+                selectionMode,
+                cacheHit,
+                selectedCount,
+                (System.nanoTime() - startedAt) / 1_000_000
+        );
+    }
+
     private List<SelectedFewShotCase> selectWithCohere(
             FewShotSearchQuery query,
             List<FewShotCase> candidates,
@@ -143,6 +164,7 @@ private List<SelectedFewShotCase> selectWithCohere(
                     .thenComparing(item -> item.fewShotCase().id()));
             return diversify(ranked, topK);
         } catch (Exception e) {
+            metricsRecorder.recordCohereFailure(e.getClass().getSimpleName());
             log.warn("dynamic few-shot Cohere selection failed. fallback=local, reason={}, message={}", e.getClass().getSimpleName(), e.getMessage());
             log.debug("dynamic few-shot Cohere exception", e);
             return List.of();
@@ -173,7 +195,7 @@ private static String formatScore(double score) {
 
     private float[] resolveQueryEmbedding(String queryText) {
         if (!properties.isCacheEnabled()) {
-            return cohereEmbeddingClient.embedQuery(queryText);
+            return embedQuery(queryText);
         }
         Instant now = Instant.now();
         maintainQueryEmbeddingCache(now);
@@ -201,7 +223,7 @@ private float[] resolveQueryEmbedding(String queryText) {
 
         try {
             QueryEmbeddingCacheEntry initialized = new QueryEmbeddingCacheEntry(
-                    cohereEmbeddingClient.embedQuery(queryText),
+                    embedQuery(queryText),
                     expiresAt()
             );
             queryEmbeddingCache.put(key, initialized);
@@ -288,7 +310,7 @@ private List<float[]> resolveDocumentEmbeddings(
             List<String> documents
     ) {
         if (!properties.isCacheEnabled()) {
-            return cohereEmbeddingClient.embedDocuments(documents);
+            return embedDocuments(documents);
         }
         Instant now = Instant.now();
         documentEmbeddingCache.entrySet().removeIf(entry -> entry.getValue().expiresAt().isBefore(now));
@@ -355,7 +377,7 @@ private void initializeMissingDocumentEmbeddings(List<PendingDocumentEmbedding>
             return;
         }
         try {
-            List<float[]> embeddedDocuments = cohereEmbeddingClient.embedDocuments(
+            List<float[]> embeddedDocuments = embedDocuments(
                     owned.stream().map(PendingDocumentEmbedding::document).toList()
             );
             if (embeddedDocuments.size() != owned.size()) {
@@ -622,4 +644,14 @@ public float[] embedding() {
     public long cohereApiCallCount() {
         return cohereEmbeddingClient.apiCallCount();
     }
+
+    private float[] embedQuery(String queryText) {
+        metricsRecorder.recordCohereLogicalCalls(1L);
+        return cohereEmbeddingClient.embedQuery(queryText);
+    }
+
+    private List<float[]> embedDocuments(List<String> documents) {
+        metricsRecorder.recordCohereLogicalCalls(1L);
+        return cohereEmbeddingClient.embedDocuments(documents);
+    }
 }

src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotMetricsRecorder.java

@@ -0,0 +1,67 @@
+package com.jobdri.jobdri_api.domain.analysis.service.ai.fewshot;
+
+import io.micrometer.core.instrument.Counter;
+import io.micrometer.core.instrument.DistributionSummary;
+import io.micrometer.core.instrument.MeterRegistry;
+import io.micrometer.core.instrument.Tags;
+import io.micrometer.core.instrument.Timer;
+import org.springframework.stereotype.Component;
+
+import java.time.Duration;
+import java.util.concurrent.TimeUnit;
+
+@Component
+public class FewShotMetricsRecorder {
+    private static final Duration[] SELECTION_SLOS = {
+            Duration.ofMillis(10), Duration.ofMillis(50), Duration.ofMillis(100),
+            Duration.ofMillis(250), Duration.ofMillis(500), Duration.ofSeconds(1),
+            Duration.ofSeconds(3), Duration.ofSeconds(5), Duration.ofSeconds(10)
+    };
+
+    private final MeterRegistry meterRegistry;
+
+    public FewShotMetricsRecorder(MeterRegistry meterRegistry) {
+        this.meterRegistry = meterRegistry;
+    }
+
+    public void recordSelection(FewShotSelectionMode mode, boolean cacheHit, int selectedCount, long durationMillis) {
+        Tags tags = Tags.of("mode", mode.name(), "cache_hit", Boolean.toString(cacheHit));
+        Counter.builder("fewshot.selection.count").tags(tags).register(meterRegistry).increment();
+        Timer.builder("fewshot.selection.duration")
+                .tags(tags)
+                .publishPercentileHistogram()
+                .serviceLevelObjectives(SELECTION_SLOS)
+                .register(meterRegistry)
+                .record(durationMillis, TimeUnit.MILLISECONDS);
+        DistributionSummary.builder("fewshot.selection.selected.candidates")
+                .tags(tags)
+                .register(meterRegistry)
+                .record(selectedCount);
+    }
+
+    public void recordCohereLogicalCalls(long count) {
+        if (count > 0) {
+            Counter.builder("fewshot.cohere.logical.calls")
+                    .register(meterRegistry)
+                    .increment(count);
+        }
+    }
+
+    public void recordCohereFailure(String reason) {
+        Counter.builder("fewshot.cohere.failure.count")
+                .tag("reason", normalizeReason(reason))
+                .register(meterRegistry)
+                .increment();
+    }
+
+    private String normalizeReason(String reason) {
+        if (reason == null || reason.isBlank()) {
+            return "Unknown";
+        }
+        return switch (reason) {
+            case "GeneralException", "IllegalArgumentException", "IllegalStateException",
+                 "TimeoutException", "ExecutionException" -> reason;
+            default -> "Other";
+        };
+    }
+}

src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchServiceTest.java

@@ -18,6 +18,8 @@
 import static org.assertj.core.api.Assertions.assertThat;
 import static org.assertj.core.api.Assertions.assertThatThrownBy;
 import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyLong;
+import static org.mockito.ArgumentMatchers.eq;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.never;
 import static org.mockito.Mockito.times;
@@ -28,12 +30,14 @@ class DefaultFewShotSearchServiceTest {
     private final FewShotCaseStore caseStore = mock(FewShotCaseStore.class);
     private final FewShotSearchTextBuilder textBuilder = new FewShotSearchTextBuilder();
     private final CohereEmbeddingClient cohereEmbeddingClient = mock(CohereEmbeddingClient.class);
+    private final FewShotMetricsRecorder metricsRecorder = mock(FewShotMetricsRecorder.class);
     private final FewShotProperties properties = new FewShotProperties();
     private final DefaultFewShotSearchService service = new DefaultFewShotSearchService(
             caseStore,
             textBuilder,
             cohereEmbeddingClient,
-            properties
+            properties,
+            metricsRecorder
     );
 
     @Test
@@ -96,6 +100,9 @@ void fallsBackToLocalSelectionWhenCohereFails() {
 
         assertThat(result).hasSize(1);
         assertThat(result.getFirst().selectionMethod()).isEqualTo("local-fallback");
+        verify(metricsRecorder).recordCohereLogicalCalls(1L);
+        verify(metricsRecorder).recordCohereFailure("RuntimeException");
+        verify(metricsRecorder).recordSelection(eq(FewShotSelectionMode.LOCAL_FALLBACK), eq(false), eq(1), anyLong());
     }
 
     @Test

src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotMetricsRecorderTest.java

@@ -0,0 +1,29 @@
+package com.jobdri.jobdri_api.domain.analysis.service.ai.fewshot;
+
+import io.micrometer.core.instrument.simple.SimpleMeterRegistry;
+import org.junit.jupiter.api.Test;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+class FewShotMetricsRecorderTest {
+
+    @Test
+    void recordsSelectionCallsAndBoundedFailureReason() {
+        SimpleMeterRegistry registry = new SimpleMeterRegistry();
+        FewShotMetricsRecorder recorder = new FewShotMetricsRecorder(registry);
+
+        recorder.recordSelection(FewShotSelectionMode.EMBEDDING, false, 3, 42);
+        recorder.recordCohereLogicalCalls(2);
+        recorder.recordCohereFailure("UnexpectedVendorException");
+
+        assertThat(registry.get("fewshot.selection.count")
+                .tags("mode", "EMBEDDING", "cache_hit", "false").counter().count()).isEqualTo(1.0);
+        assertThat(registry.get("fewshot.selection.duration")
+                .tags("mode", "EMBEDDING", "cache_hit", "false").timer().count()).isEqualTo(1L);
+        assertThat(registry.get("fewshot.selection.selected.candidates")
+                .tags("mode", "EMBEDDING", "cache_hit", "false").summary().totalAmount()).isEqualTo(3.0);
+        assertThat(registry.get("fewshot.cohere.logical.calls").counter().count()).isEqualTo(2.0);
+        assertThat(registry.get("fewshot.cohere.failure.count")
+                .tag("reason", "Other").counter().count()).isEqualTo(1.0);
+    }
+}

src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/ReviewedFewShotDatasetTest.java

@@ -46,7 +46,7 @@ void excludesActualReviewedInputEvenWithDifferentIdAndWhitespace() {
         var store = store();
         var candidate = store.loadActiveCases().getFirst();
         var service = new DefaultFewShotSearchService(store, new FewShotSearchTextBuilder(),
-                mock(CohereEmbeddingClient.class), properties);
+                mock(CohereEmbeddingClient.class), properties, mock(FewShotMetricsRecorder.class));
         var query = new FewShotSearchQuery("HOLDOUT-COPY", candidate.jobCategory(), candidate.jobTitle(),
                 candidate.mainTasks(), candidate.qualifications(), candidate.question(),
                 "  " + candidate.sanitizedAnswer().replace("\n", "  ") + " ");