diff --git a/docs/evaluation/fewshot-static-dynamic-20260916.md b/docs/evaluation/fewshot-static-dynamic-20260916.md index 0977233b..386cb4c7 100644 --- a/docs/evaluation/fewshot-static-dynamic-20260916.md +++ b/docs/evaluation/fewshot-static-dynamic-20260916.md @@ -73,15 +73,26 @@ DYNAMIC의 overall usefulness가 2점으로 내려갔습니다. 4번 튜닝에 - 케이스별 top similarity 평균: 0.4778 - 케이스별 bottom similarity 평균: 0.3503 - minSimilarity=-1.0, topK=5라서 모든 케이스에 후보 5건이 들어갔습니다. -- Cohere HTTP 호출 수: STATIC 0회, DYNAMIC 21회로 계산됩니다. - 첫 케이스의 query 1회+document batch 1회, 이후 query 19회이며 document embedding은 - dataset cache를 재사용했습니다. 현재 sidecar가 직접 센 값은 아니므로 4번 전에 호출 카운터 - 계측을 추가하는 것이 안전합니다. +- 최초 품질 실행에서 예상한 Cohere 호출 수는 STATIC 0회, DYNAMIC 21회였습니다. 아래 계측 + 재실행에서 sidecar의 논리 호출 수가 같은 값임을 확인했습니다. - Judge 토큰: STATIC input 116,728 / output 9,226, DYNAMIC input 115,906 / output 8,278. -- 분석 OpenAI 토큰은 기존 분석 adapter가 usage를 결과에 전달하지 않아 이번 실행에서 - 수집하지 못했습니다. 기존 CSV의 token 컬럼도 비어 있습니다. 비용 비교 완료 전 - 분석 usage 계측을 추가해야 합니다. +- 분석 usage와 케이스별 논리적 Cohere 호출 수 계측을 추가한 뒤 동일 입력·설정으로 분석만 + 재실행했습니다(양쪽 20/20 성공). + +| 계측 재실행 | STATIC | DYNAMIC | 변화 | +| --- | ---: | ---: | ---: | +| 분석 input tokens | 160,941 | 311,081 | +150,140 (+93.3%) | +| 분석 output tokens | 10,438 | 9,903 | -535 (-5.1%) | +| 분석 total tokens | 171,379 | 320,984 | +149,605 (+87.3%) | +| Cohere 논리 호출 | 0 | 21 | +21 | +| 평균 분석 지연 | 5,511ms | 5,880ms | +369ms (+6.7%) | +| P95 분석 지연 | 8,344ms | 10,350ms | +2,006ms (+24.0%) | + +논리적 Cohere 호출은 애플리케이션의 embedding 요청 횟수이며 HTTP 계층의 내부 재전송은 +포함하지 않습니다. DYNAMIC의 첫 케이스는 query 1회와 document batch 1회, 이후 19건은 +query 1회씩 호출됐습니다. 재실행은 사용량 계측 목적이라 Judge를 다시 실행하지 않았으며, +품질 수치는 앞선 동일 조건 20건 비교 결과를 사용합니다. ## 실행 중 발견하고 수정한 문제 @@ -102,12 +113,15 @@ DYNAMIC의 overall usefulness가 2점으로 내려갔습니다. 4번 튜닝에 - `judge-comparison.csv` - 각 실행 로그 +사용량 계측 재실행 결과는 `build/evaluation/fewshot-comparison-20260916/usage-metrics/`의 +`static-analysis.csv`, `dynamic-analysis.csv`, 각 sidecar와 로그에 있습니다. + `build/`는 Git 추적 대상이 아닙니다. 결과 재현이 필요하면 동일 holdout과 설정으로 다시 실행하거나, 개인정보 보관 정책을 확인한 후 별도 안전한 저장소에 보관해야 합니다. ## 다음 결정 -3번의 품질 비교는 완료됐고 결과는 DYNAMIC의 튜닝 진행을 지지합니다. 다만 완료 조건의 비용 -항목 중 분석 OpenAI 토큰 실측이 남아 있으므로, 이 계측을 추가하기 전에는 3번을 완전 완료로 -닫지 않습니다. 다음 작업은 분석 usage와 Cohere 호출 수 계측을 추가한 뒤 4번 임계값·top-k -튜닝 실험을 설계하는 것입니다. +3번의 품질·지연·토큰·Cohere 호출량 비교와 계측을 완료했습니다. DYNAMIC은 품질이 좋아졌지만 +분석 input token이 93.3% 증가했으므로 그대로 운영 활성화하지 않습니다. 다음은 4번 +`min-similarity`, `top-k`, `minimum-selected-count` 튜닝으로 품질 이득을 유지하면서 낮은 유사도 +후보와 토큰 비용을 줄이는 실험입니다. diff --git a/docs/fewshot-evaluation-metadata.md b/docs/fewshot-evaluation-metadata.md index 3bb33dd0..7198f4cb 100644 --- a/docs/fewshot-evaluation-metadata.md +++ b/docs/fewshot-evaluation-metadata.md @@ -39,6 +39,8 @@ sidecar 저장 실패는 평가 실행 실패로 전파됩니다. 프롬프트 순서와 최대·최소 점수 순서는 다를 수 있습니다. - `datasetVersion/minSimilarity/topK/minimumSelectedCount`: 실행의 선택 설정. topK는 설정한 요청 개수이며 실제 개수는 selectedCases 배열 길이입니다. +- `cohereApiCallCount`: 해당 프롬프트 선택 중 발생한 논리적 Cohere embedding 호출 수. + 캐시 적중 시 0이며, SDK/HTTP 계층의 내부 재전송 횟수와는 구분합니다. - `reason`: 정적 선택·fallback·미적용 사유 코드. 예외 메시지 원문은 넣지 않습니다. 정적 예시 ID는 기존 로더와 같은 `FS-FIXED-1..N`입니다. @@ -61,11 +63,12 @@ STATIC으로 추정하지 않습니다. 선택 스냅샷에는 자소서 원문, JD, 프롬프트 본문, 임베딩 벡터를 넣지 않습니다. 기존 평가 CSV의 원문 보존 동작은 바꾸지 않았으므로 CSV 접근 권한은 기존대로 관리해야 합니다. -## 다음 작업 +## 분석 사용량 -3번 STATIC/DYNAMIC 비교 평가에서는 동일 모델·temperature·holdout·Judge 조건을 유지하고 -각 실행의 CSV와 sidecar를 함께 보관합니다. -토큰 사용량·Cohere 호출 수·전체 비용 수집 및 실제 품질 비교는 이번 변경에 포함하지 않습니다. +평가 CSV의 `candidateInputTokens`, `candidateOutputTokens`, `finalInputTokens`, +`finalOutputTokens`, `totalInputTokens`, `totalOutputTokens`에는 OpenAI 응답 usage를 기록합니다. +single-pass 사용량은 final 컬럼에 기록되고 candidate 컬럼은 비어 있습니다. two-pass는 후보와 +최종 검토를 나눠 기록하며, 선택적으로 실행되는 recheck 사용량은 final에 합산합니다. ## 검증 diff --git a/src/main/java/com/jobdri/jobdri_api/domain/analysis/infrastructure/ai/OpenAiAnalysisAdapter.java b/src/main/java/com/jobdri/jobdri_api/domain/analysis/infrastructure/ai/OpenAiAnalysisAdapter.java index 49ba7b6f..697f5104 100644 --- a/src/main/java/com/jobdri/jobdri_api/domain/analysis/infrastructure/ai/OpenAiAnalysisAdapter.java +++ b/src/main/java/com/jobdri/jobdri_api/domain/analysis/infrastructure/ai/OpenAiAnalysisAdapter.java @@ -7,6 +7,7 @@ import com.openai.core.RequestOptions; import com.openai.models.responses.ResponseCreateParams; import com.openai.models.responses.StructuredResponse; +import com.openai.models.responses.ResponseUsage; import lombok.RequiredArgsConstructor; import org.springframework.beans.factory.annotation.Value; import org.springframework.stereotype.Component; @@ -33,6 +34,15 @@ public T createStructuredResponse( String prompt, Class responseType, Duration timeout + ) { + return createStructuredResponseWithUsage(operationName, prompt, responseType, timeout).content(); + } + + public StructuredCallResult createStructuredResponseWithUsage( + String operationName, + String prompt, + Class responseType, + Duration timeout ) { var params = ResponseCreateParams.builder() .model(analysisModel) @@ -53,8 +63,13 @@ public T createStructuredResponse( : openAIClient.responses().create(params, requestOptions) ); T structuredContent = analysisResponseParser.extractStructuredContent(response); + ResponseUsage usage = response.usage().orElse(null); success = true; - return structuredContent; + return new StructuredCallResult<>( + structuredContent, + toIntegerTokenCount(usage == null ? null : usage.inputTokens()), + toIntegerTokenCount(usage == null ? null : usage.outputTokens()) + ); } finally { asyncMetricsRecorder.recordLlmRequest( operationName, @@ -64,6 +79,16 @@ public T createStructuredResponse( } } + private Integer toIntegerTokenCount(Long tokens) { + if (tokens == null) { + return null; + } + return tokens > Integer.MAX_VALUE ? Integer.MAX_VALUE : tokens.intValue(); + } + + public record StructuredCallResult(T content, Integer inputTokens, Integer outputTokens) { + } + private long elapsedMillis(long startedAt) { return (System.nanoTime() - startedAt) / 1_000_000; } diff --git a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisAiClient.java b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisAiClient.java index ab5ffe8b..0afcad0c 100644 --- a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisAiClient.java +++ b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisAiClient.java @@ -240,14 +240,20 @@ private AnalysisAiCallResult analyzeSinglePass( Consumer recorder ) { long startedAt = System.nanoTime(); - AnalysisLlmResponse response = createStructuredResponse( + var callResult = createStructuredResponseWithUsage( operationName, analysisPromptBuilder.buildSinglePassPrompt(promptInput, referenceContext, jobCategoryEvaluationCriteria, recorder), AnalysisLlmResponse.class, deadline ); + AnalysisLlmResponse response = callResult.content(); response = analysisResponseParser.sanitizeSinglePassSubheadings(promptInput, response); - return AnalysisAiCallResult.singlePass(response, elapsedMillis(startedAt)); + return AnalysisAiCallResult.singlePass( + response, + elapsedMillis(startedAt), + callResult.inputTokens(), + callResult.outputTokens() + ); } private AnalysisAiCallResult analyzeTwoPass( @@ -258,12 +264,13 @@ private AnalysisAiCallResult analyzeTwoPass( Instant deadline ) { long candidateStartedAt = System.nanoTime(); - AnalysisCandidateResponse rawCandidates = createStructuredResponse( + var candidateCallResult = createStructuredResponseWithUsage( operationName + "-candidates", buildCandidatePrompt(promptInput, referenceContext, jobCategoryEvaluationCriteria), AnalysisCandidateResponse.class, deadline ); + AnalysisCandidateResponse rawCandidates = candidateCallResult.content(); long candidateLatencyMs = elapsedMillis(candidateStartedAt); AnalysisCandidateResponse sanitizedCandidates = sanitizeCandidates(promptInput, rawCandidates); log.debug( @@ -280,12 +287,13 @@ private AnalysisAiCallResult analyzeTwoPass( ); long finalStartedAt = System.nanoTime(); - CandidateReviewResponse reviewResponse = createStructuredResponse( + var finalCallResult = createStructuredResponseWithUsage( operationName + "-final", buildFinalPrompt(promptInput, referenceContext, jobCategoryEvaluationCriteria, sanitizedCandidates), CandidateReviewResponse.class, deadline ); + CandidateReviewResponse reviewResponse = finalCallResult.content(); CandidateReviewResponse validatedReviewResponse = validateCandidateReview( promptInput, sanitizedCandidates, @@ -300,7 +308,7 @@ private AnalysisAiCallResult analyzeTwoPass( rejectedDecisionCount(validatedReviewResponse), rejectionCodeCounts(validatedReviewResponse) ); - CandidateReviewResponse recheckedReviewResponse = recheckWhenAllCandidatesRejected( + RecheckCallResult recheckCallResult = recheckWhenAllCandidatesRejected( promptInput, referenceContext, jobCategoryEvaluationCriteria, @@ -309,6 +317,7 @@ private AnalysisAiCallResult analyzeTwoPass( operationName, deadline ); + CandidateReviewResponse recheckedReviewResponse = recheckCallResult.reviewResponse(); AnalysisLlmResponse response = buildFinalResponse(promptInput, sanitizedCandidates, recheckedReviewResponse); long finalLatencyMs = elapsedMillis(finalStartedAt); logQuestionFlowStats(sanitizedCandidates, recheckedReviewResponse, response); @@ -336,7 +345,11 @@ private AnalysisAiCallResult analyzeTwoPass( sanitizedCandidates, recheckedReviewResponse, candidateLatencyMs, - finalLatencyMs + finalLatencyMs, + candidateCallResult.inputTokens(), + candidateCallResult.outputTokens(), + sumTokens(finalCallResult.inputTokens(), recheckCallResult.inputTokens()), + sumTokens(finalCallResult.outputTokens(), recheckCallResult.outputTokens()) ); } @@ -392,22 +405,22 @@ private AnalysisAiCallResult analyzeHybridExact( twoPassResult.sanitizedCandidateResponse(), twoPassResult.candidateReviewResponse(), twoPassResult.candidateCallLatencyMs(), - singlePassResult.finalCallLatencyMs() + twoPassResult.finalCallLatencyMs() + singlePassResult.finalCallLatencyMs() + twoPassResult.finalCallLatencyMs(), + twoPassResult.candidateInputTokens(), + twoPassResult.candidateOutputTokens(), + sumTokens(singlePassResult.finalInputTokens(), twoPassResult.finalInputTokens()), + sumTokens(singlePassResult.finalOutputTokens(), twoPassResult.finalOutputTokens()) ); } - private T createStructuredResponse(String operationName, String prompt, Class responseType) { - return openAiAnalysisAdapter.createStructuredResponse(operationName, prompt, responseType); - } - - private T createStructuredResponse( + private OpenAiAnalysisAdapter.StructuredCallResult createStructuredResponseWithUsage( String operationName, String prompt, Class responseType, Instant deadline ) { if (deadline == null) { - return openAiAnalysisAdapter.createStructuredResponse(operationName, prompt, responseType); + return openAiAnalysisAdapter.createStructuredResponseWithUsage(operationName, prompt, responseType, null); } Duration remaining = Duration.between(Instant.now(), deadline); if (remaining.isZero() || remaining.isNegative()) { @@ -416,11 +429,15 @@ private T createStructuredResponse( "평가 사례 처리 시간이 제한을 초과했습니다." ); } - return openAiAnalysisAdapter.createStructuredResponse(operationName, prompt, responseType, remaining); + return openAiAnalysisAdapter.createStructuredResponseWithUsage(operationName, prompt, responseType, remaining); } - private T createStructuredResponse(String operationName, String prompt, Class responseType, Duration timeout) { - return openAiAnalysisAdapter.createStructuredResponse(operationName, prompt, responseType); + private Integer sumTokens(Integer left, Integer right) { + if (left == null && right == null) { + return null; + } + long sum = (left == null ? 0L : left.longValue()) + (right == null ? 0L : right.longValue()); + return sum > Integer.MAX_VALUE ? Integer.MAX_VALUE : (int) sum; } String buildPrompt( @@ -635,7 +652,7 @@ AnalysisLlmResponse buildFinalResponse( ); } - CandidateReviewResponse recheckWhenAllCandidatesRejected( + private RecheckCallResult recheckWhenAllCandidatesRejected( AnalysisPromptInput promptInput, RetrievalContext referenceContext, JobCategoryEvaluationCriteria jobCategoryEvaluationCriteria, @@ -649,7 +666,7 @@ CandidateReviewResponse recheckWhenAllCandidatesRejected( : sanitizedCandidates.analysisCandidates().size(); int acceptedCandidates = acceptedDecisionCount(reviewResponse); if (firstPassCandidates == 0 || acceptedCandidates > 0) { - return reviewResponse; + return new RecheckCallResult(reviewResponse, null, null); } log.debug( @@ -659,12 +676,13 @@ CandidateReviewResponse recheckWhenAllCandidatesRejected( rejectedDecisionCount(reviewResponse), rejectionCodeCounts(reviewResponse) ); - CandidateRecheckResponse recheckResponse = createStructuredResponse( + var recheckCallResult = createStructuredResponseWithUsage( operationName + "-recheck", buildRecheckPrompt(promptInput, referenceContext, jobCategoryEvaluationCriteria, sanitizedCandidates, reviewResponse), CandidateRecheckResponse.class, deadline ); + CandidateRecheckResponse recheckResponse = recheckCallResult.content(); CandidateReviewResponse rechecked = applyRecheckResponse(promptInput, sanitizedCandidates, reviewResponse, recheckResponse); int recoveredMentionedCount = recoveredDecisionCount(rechecked, QuestionAnalysisStatus.MENTIONED); int recoveredFabricatedCount = recoveredDecisionCount(rechecked, QuestionAnalysisStatus.FABRICATED); @@ -682,7 +700,11 @@ CandidateReviewResponse recheckWhenAllCandidatesRejected( recheckResponse != null && StringUtils.hasText(recheckResponse.candidateId()), rejectedDecisionCount(rechecked) ); - return rechecked; + return new RecheckCallResult( + rechecked, + recheckCallResult.inputTokens(), + recheckCallResult.outputTokens() + ); } CandidateReviewResponse applyRecheckResponse( @@ -1759,6 +1781,13 @@ private enum RecheckValidationFailureReason { INVALID_FABRICATED } + private record RecheckCallResult( + CandidateReviewResponse reviewResponse, + Integer inputTokens, + Integer outputTokens + ) { + } + public record AnalysisAiCallResult( AnalysisLlmResponse response, AnalysisCandidateResponse rawCandidateResponse, @@ -1766,10 +1795,22 @@ public record AnalysisAiCallResult( CandidateReviewResponse candidateReviewResponse, boolean twoPassEnabled, long candidateCallLatencyMs, - long finalCallLatencyMs + long finalCallLatencyMs, + Integer candidateInputTokens, + Integer candidateOutputTokens, + Integer finalInputTokens, + Integer finalOutputTokens ) { - static AnalysisAiCallResult singlePass(AnalysisLlmResponse response, long latencyMs) { - return new AnalysisAiCallResult(response, null, null, null, false, 0, latencyMs); + static AnalysisAiCallResult singlePass( + AnalysisLlmResponse response, + long latencyMs, + Integer inputTokens, + Integer outputTokens + ) { + return new AnalysisAiCallResult( + response, null, null, null, false, 0, latencyMs, + null, null, inputTokens, outputTokens + ); } static AnalysisAiCallResult twoPass( @@ -1778,7 +1819,11 @@ static AnalysisAiCallResult twoPass( AnalysisCandidateResponse sanitizedCandidateResponse, CandidateReviewResponse candidateReviewResponse, long candidateCallLatencyMs, - long finalCallLatencyMs + long finalCallLatencyMs, + Integer candidateInputTokens, + Integer candidateOutputTokens, + Integer finalInputTokens, + Integer finalOutputTokens ) { return new AnalysisAiCallResult( response, @@ -1787,7 +1832,11 @@ static AnalysisAiCallResult twoPass( candidateReviewResponse, true, candidateCallLatencyMs, - finalCallLatencyMs + finalCallLatencyMs, + candidateInputTokens, + candidateOutputTokens, + finalInputTokens, + finalOutputTokens ); } @@ -1797,7 +1846,11 @@ static AnalysisAiCallResult hybridExact( AnalysisCandidateResponse sanitizedCandidateResponse, CandidateReviewResponse candidateReviewResponse, long candidateCallLatencyMs, - long finalCallLatencyMs + long finalCallLatencyMs, + Integer candidateInputTokens, + Integer candidateOutputTokens, + Integer finalInputTokens, + Integer finalOutputTokens ) { return new AnalysisAiCallResult( response, @@ -1806,7 +1859,11 @@ static AnalysisAiCallResult hybridExact( candidateReviewResponse, true, candidateCallLatencyMs, - finalCallLatencyMs + finalCallLatencyMs, + candidateInputTokens, + candidateOutputTokens, + finalInputTokens, + finalOutputTokens ); } } diff --git a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisPromptBuilder.java b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisPromptBuilder.java index af17861f..5f46c8ba 100644 --- a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisPromptBuilder.java +++ b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/AnalysisPromptBuilder.java @@ -621,6 +621,7 @@ private FewShotPromptSelection resolveFewShotPromptBlock(AnalysisPromptInput pro if (fewShotSearchService == null || fewShotProperties == null || !fewShotProperties.isDynamicSelectionEnabled()) { return staticSelection("STATIC", "dynamic_disabled"); } + long cohereCallsBefore = fewShotSearchService.cohereApiCallCount(); try { List selectedFewShots = fewShotSearchService.searchRelevantFewShots( FewShotSearchQuery.from(promptInput), @@ -633,7 +634,8 @@ private FewShotPromptSelection resolveFewShotPromptBlock(AnalysisPromptInput pro promptInput.caseId(), fewShotProperties.getDatasetVersion() ); - return staticSelection("STATIC_FALLBACK", "empty_selection"); + return staticSelection("STATIC_FALLBACK", "empty_selection", + fewShotSearchService.cohereApiCallCount() - cohereCallsBefore); } log.debug( "dynamic few-shot prompt selected. caseId={}, selectedIds={}, sources={}, scores={}, datasetVersion={}", @@ -646,10 +648,12 @@ private FewShotPromptSelection resolveFewShotPromptBlock(AnalysisPromptInput pro if (selectedFewShots.stream().anyMatch(item -> item == null || item.fewShotCase() == null || !org.springframework.util.StringUtils.hasText(item.fewShotCase().promptBlock()) || !Double.isFinite(item.score()))) { - return staticSelection("STATIC_FALLBACK", "invalid_selection"); + return staticSelection("STATIC_FALLBACK", "invalid_selection", + fewShotSearchService.cohereApiCallCount() - cohereCallsBefore); } return new FewShotPromptSelection(fewShotPromptProvider.buildPromptBlock(selectedFewShots), - FewShotSelectionMetadata.selected(selectedFewShots, fewShotProperties)); + FewShotSelectionMetadata.selected(selectedFewShots, fewShotProperties, + fewShotSearchService.cohereApiCallCount() - cohereCallsBefore)); } catch (Exception e) { log.warn( "dynamic few-shot selection failed. selectionMode={}, caseId={}, datasetVersion={}, reason={}, message={}", @@ -660,18 +664,23 @@ private FewShotPromptSelection resolveFewShotPromptBlock(AnalysisPromptInput pro e.getMessage() ); log.debug("dynamic few-shot selection exception", e); - return staticSelection("STATIC_FALLBACK", "selection_exception"); + return staticSelection("STATIC_FALLBACK", "selection_exception", + fewShotSearchService.cohereApiCallCount() - cohereCallsBefore); } } private FewShotPromptSelection staticSelection(String mode, String reason) { + return staticSelection(mode, reason, 0L); + } + + private FewShotPromptSelection staticSelection(String mode, String reason, long cohereApiCallCount) { return new FewShotPromptSelection(fewShotPromptProvider.getPrompt(), FewShotSelectionMetadata.staticSelection(mode, reason, - fewShotPromptProvider.getFixedExampleBlocks().size(), fewShotProperties)); + fewShotPromptProvider.getFixedExampleBlocks().size(), fewShotProperties, cohereApiCallCount)); } FewShotSelectionMetadata fewShotNotApplied() { - return FewShotSelectionMetadata.staticSelection("NOT_APPLIED", "two_pass", 0, fewShotProperties); + return FewShotSelectionMetadata.staticSelection("NOT_APPLIED", "two_pass", 0, fewShotProperties, 0L); } private record FewShotPromptSelection(String prompt, FewShotSelectionMetadata metadata) { diff --git a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchService.java b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchService.java index 13a5a441..9ab9ddf9 100644 --- a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchService.java +++ b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/DefaultFewShotSearchService.java @@ -617,4 +617,9 @@ public float[] embedding() { return embedding.clone(); } } + + @Override + public long cohereApiCallCount() { + return cohereEmbeddingClient.apiCallCount(); + } } diff --git a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotSearchService.java b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotSearchService.java index 86496179..5a9e19bb 100644 --- a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotSearchService.java +++ b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotSearchService.java @@ -4,4 +4,8 @@ public interface FewShotSearchService { List searchRelevantFewShots(FewShotSearchQuery query, int topK); + + default long cohereApiCallCount() { + return 0L; + } } diff --git a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotSelectionMetadata.java b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotSelectionMetadata.java index 0a85e59b..65224910 100644 --- a/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotSelectionMetadata.java +++ b/src/main/java/com/jobdri/jobdri_api/domain/analysis/service/ai/fewshot/FewShotSelectionMetadata.java @@ -12,6 +12,7 @@ public record FewShotSelectionMetadata( int topK, int minimumSelectedCount, String scoreType, + long cohereApiCallCount, List selectedCases, Double topScore, Double bottomScore, @@ -24,24 +25,29 @@ public record FewShotSelectionMetadata( public record Candidate(String id, String source, Double score, String datasetVersion) { } - public static FewShotSelectionMetadata selected(List selected, FewShotProperties properties) { + public static FewShotSelectionMetadata selected( + List selected, + FewShotProperties properties, + long cohereApiCallCount + ) { boolean embedding = selected.stream().allMatch(item -> "cohere-embedding".equals(item.selectionMethod())); var scores = selected.stream().mapToDouble(SelectedFewShotCase::score).summaryStatistics(); return new FewShotSelectionMetadata( embedding ? "EMBEDDING" : "LOCAL_FALLBACK", "", properties.getDatasetVersion(), properties.getSearch().getMinSimilarity(), properties.getSearch().getTopK(), properties.getSearch().getMinimumSelectedCount(), embedding ? "COSINE_SIMILARITY" : "LOCAL_HEURISTIC", + cohereApiCallCount, selected.stream().map(item -> new Candidate(item.fewShotCase().id(), item.fewShotCase().source().name(), item.score(), item.fewShotCase().datasetVersion())).toList(), scores.getMax(), scores.getMin(), scores.getAverage()); } public static FewShotSelectionMetadata staticSelection(String mode, String reason, int exampleCount, - FewShotProperties configured) { + FewShotProperties configured, long cohereApiCallCount) { var properties = configured == null ? new FewShotProperties() : configured; return new FewShotSelectionMetadata(mode, reason, properties.getDatasetVersion(), properties.getSearch().getMinSimilarity(), properties.getSearch().getTopK(), - properties.getSearch().getMinimumSelectedCount(), "NONE", + properties.getSearch().getMinimumSelectedCount(), "NONE", cohereApiCallCount, IntStream.rangeClosed(1, exampleCount) .mapToObj(i -> new Candidate("FS-FIXED-" + i, "FIXED", null, "static-resource")).toList(), null, null, null); diff --git a/src/main/java/com/jobdri/jobdri_api/domain/evaluation/analysis/EvaluationAnalysisBatchService.java b/src/main/java/com/jobdri/jobdri_api/domain/evaluation/analysis/EvaluationAnalysisBatchService.java index e85388bf..c2476e91 100644 --- a/src/main/java/com/jobdri/jobdri_api/domain/evaluation/analysis/EvaluationAnalysisBatchService.java +++ b/src/main/java/com/jobdri/jobdri_api/domain/evaluation/analysis/EvaluationAnalysisBatchService.java @@ -202,18 +202,26 @@ private EvaluationAnalysisResult analyzeCase(EvaluationAnalysisCase evaluationCa generatedResult.finalCallLatencyMs(), generatedResult.candidateCallLatencyMs(), generatedResult.finalCallLatencyMs(), - null, - null, - null, - null, - null, - null, + generatedResult.candidateInputTokens(), + generatedResult.candidateOutputTokens(), + generatedResult.finalInputTokens(), + generatedResult.finalOutputTokens(), + sumTokens(generatedResult.candidateInputTokens(), generatedResult.finalInputTokens()), + sumTokens(generatedResult.candidateOutputTokens(), generatedResult.finalOutputTokens()), "", "", createdAt() ); } + private Integer sumTokens(Integer left, Integer right) { + if (left == null && right == null) { + return null; + } + long sum = (left == null ? 0L : left.longValue()) + (right == null ? 0L : right.longValue()); + return sum > Integer.MAX_VALUE ? Integer.MAX_VALUE : (int) sum; + } + private List buildMissingKeywords( EvaluationAnalysisCase evaluationCase, EvaluationLlmSnapshot llmResponse diff --git a/src/main/java/com/jobdri/jobdri_api/domain/evaluation/analysis/model/EvaluationGeneratedResult.java b/src/main/java/com/jobdri/jobdri_api/domain/evaluation/analysis/model/EvaluationGeneratedResult.java index 4e761130..ccbd51ba 100644 --- a/src/main/java/com/jobdri/jobdri_api/domain/evaluation/analysis/model/EvaluationGeneratedResult.java +++ b/src/main/java/com/jobdri/jobdri_api/domain/evaluation/analysis/model/EvaluationGeneratedResult.java @@ -9,6 +9,29 @@ public record EvaluationGeneratedResult( String candidateReviewResponseJson, EvaluationCandidateReviewSnapshot candidateReviewSnapshot, long candidateCallLatencyMs, - long finalCallLatencyMs + long finalCallLatencyMs, + Integer candidateInputTokens, + Integer candidateOutputTokens, + Integer finalInputTokens, + Integer finalOutputTokens ) { + public EvaluationGeneratedResult( + EvaluationLlmSnapshot responseSnapshot, + String rawLlmResponseJson, + String rawCandidateResponseJson, + String sanitizedCandidateResponseJson, + EvaluationCandidateSnapshot sanitizedCandidateSnapshot, + String candidateReviewResponseJson, + EvaluationCandidateReviewSnapshot candidateReviewSnapshot, + long candidateCallLatencyMs, + long finalCallLatencyMs + ) { + this( + responseSnapshot, rawLlmResponseJson, rawCandidateResponseJson, + sanitizedCandidateResponseJson, sanitizedCandidateSnapshot, + candidateReviewResponseJson, candidateReviewSnapshot, + candidateCallLatencyMs, finalCallLatencyMs, + null, null, null, null + ); + } } diff --git a/src/main/java/com/jobdri/jobdri_api/domain/evaluation/infrastructure/analysis/AnalysisAiEvaluationAnalysisGenerator.java b/src/main/java/com/jobdri/jobdri_api/domain/evaluation/infrastructure/analysis/AnalysisAiEvaluationAnalysisGenerator.java index 54be3cb5..68449bf1 100644 --- a/src/main/java/com/jobdri/jobdri_api/domain/evaluation/infrastructure/analysis/AnalysisAiEvaluationAnalysisGenerator.java +++ b/src/main/java/com/jobdri/jobdri_api/domain/evaluation/infrastructure/analysis/AnalysisAiEvaluationAnalysisGenerator.java @@ -94,7 +94,11 @@ public EvaluationGeneratedResult generate(EvaluationAnalysisCommand command) { candidateReviewResponseJson, reviewSnapshotParser.parse(candidateReviewResponseJson), aiCallResult.candidateCallLatencyMs(), - aiCallResult.finalCallLatencyMs() + aiCallResult.finalCallLatencyMs(), + aiCallResult.candidateInputTokens(), + aiCallResult.candidateOutputTokens(), + aiCallResult.finalInputTokens(), + aiCallResult.finalOutputTokens() ); } diff --git a/src/main/java/com/jobdri/jobdri_api/global/cohere/CohereEmbeddingClient.java b/src/main/java/com/jobdri/jobdri_api/global/cohere/CohereEmbeddingClient.java index 86662aa3..1545b26f 100644 --- a/src/main/java/com/jobdri/jobdri_api/global/cohere/CohereEmbeddingClient.java +++ b/src/main/java/com/jobdri/jobdri_api/global/cohere/CohereEmbeddingClient.java @@ -29,6 +29,7 @@ import java.time.format.DateTimeParseException; import java.util.ArrayList; import java.util.List; +import java.util.concurrent.atomic.AtomicLong; @Component @Slf4j @@ -45,6 +46,7 @@ public class CohereEmbeddingClient { private final CohereProperties properties; private final RestClient restClient; + private final AtomicLong apiCallCount = new AtomicLong(); public CohereEmbeddingClient(CohereProperties properties, RestClient.Builder restClientBuilder) { this.properties = properties; @@ -110,6 +112,7 @@ private CohereEmbeddingResponse callCohere(CohereEmbeddingRequest request) { } private CohereEmbeddingResponse callCohereOnce(CohereEmbeddingRequest request) { + apiCallCount.incrementAndGet(); try { return restClient.post() .uri("/v2/embed") @@ -150,6 +153,10 @@ private CohereEmbeddingResponse callCohereOnce(CohereEmbeddingRequest request) { } } + public long apiCallCount() { + return apiCallCount.get(); + } + private List validateResponse(CohereEmbeddingResponse response, int expectedCount) { if (response == null || response.embeddings() == null || response.embeddings().floatValues() == null || response.embeddings().floatValues().isEmpty()) { diff --git a/src/test/java/com/jobdri/jobdri_api/domain/analysis/infrastructure/ai/OpenAiAnalysisAdapterTest.java b/src/test/java/com/jobdri/jobdri_api/domain/analysis/infrastructure/ai/OpenAiAnalysisAdapterTest.java index e4905c2d..1fb04ea5 100644 --- a/src/test/java/com/jobdri/jobdri_api/domain/analysis/infrastructure/ai/OpenAiAnalysisAdapterTest.java +++ b/src/test/java/com/jobdri/jobdri_api/domain/analysis/infrastructure/ai/OpenAiAnalysisAdapterTest.java @@ -6,12 +6,14 @@ import com.jobdri.jobdri_api.global.metrics.AsyncMetricsRecorder; import com.openai.client.OpenAIClient; import com.openai.models.responses.StructuredResponse; +import com.openai.models.responses.ResponseUsage; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.DisplayName; import org.junit.jupiter.api.Test; import org.springframework.test.util.ReflectionTestUtils; import java.util.List; +import java.util.Optional; import static org.assertj.core.api.Assertions.assertThat; import static org.assertj.core.api.Assertions.assertThatThrownBy; @@ -57,6 +59,26 @@ void createStructuredResponseRecordsSuccessMetric() throws Exception { verify(asyncMetricsRecorder, never()).recordLlmRequest(eq("analysis"), eq("error"), any(Long.class)); } + @Test + @DisplayName("구조화 응답의 입력·출력 토큰 사용량을 함께 반환한다") + void createStructuredResponseWithUsageReturnsTokenCounts() throws Exception { + StructuredResponse response = mock(StructuredResponse.class); + ResponseUsage usage = mock(ResponseUsage.class); + when(usage.inputTokens()).thenReturn(123L); + when(usage.outputTokens()).thenReturn(45L); + when(response.usage()).thenReturn(Optional.of(usage)); + when(llmConcurrencyLimiter.execute(eq("analysis"), any())).thenReturn(response); + when(analysisResponseParser.extractStructuredContent(response)).thenReturn("ok"); + + var result = openAiAnalysisAdapter.createStructuredResponseWithUsage( + "analysis", "prompt", String.class, null + ); + + assertThat(result.content()).isEqualTo("ok"); + assertThat(result.inputTokens()).isEqualTo(123); + assertThat(result.outputTokens()).isEqualTo(45); + } + @Test @DisplayName("limiter 실행 중 RuntimeException이 나면 그대로 전파하고 error 메트릭을 기록한다") void createStructuredResponsePropagatesLimiterFailure() { diff --git a/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/FewShotMetadataPromptTest.java b/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/FewShotMetadataPromptTest.java index afa943e6..e0fde988 100644 --- a/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/FewShotMetadataPromptTest.java +++ b/src/test/java/com/jobdri/jobdri_api/domain/analysis/service/ai/FewShotMetadataPromptTest.java @@ -44,6 +44,7 @@ void embeddingSnapshotMatchesPromptAndIsRecordedOnce() throws Exception { properties.setDatasetVersion("version-2"); properties.getSearch().setTopK(2); properties.getSearch().setMinSimilarity(0.3); + when(search.cohereApiCallCount()).thenReturn(10L, 12L); when(search.searchRelevantFewShots(any(), anyInt())).thenReturn(List.of( selected("A", 0.4, "cohere-embedding"), selected("B", 0.8, "cohere-embedding"))); assertThat(build()).contains("BLOCK-A", "BLOCK-B").doesNotContain(provider.getPrompt()); @@ -56,6 +57,7 @@ void embeddingSnapshotMatchesPromptAndIsRecordedOnce() throws Exception { assertThat(meta.datasetVersion()).isEqualTo("version-2"); assertThat(meta.minSimilarity()).isEqualTo(0.3); assertThat(meta.topK()).isEqualTo(2); + assertThat(meta.cohereApiCallCount()).isEqualTo(2); assertThat(meta.selectedCases()).extracting(FewShotSelectionMetadata.Candidate::id).containsExactly("A", "B"); assertThat(new ObjectMapper().writeValueAsString(meta)).doesNotContain("비밀 원문", "BLOCK-", "sanitizedAnswer", "promptBlock"); verify(search, times(1)).searchRelevantFewShots(any(), anyInt()); diff --git a/src/test/java/com/jobdri/jobdri_api/domain/evaluation/analysis/EvaluationAnalysisBatchServiceTest.java b/src/test/java/com/jobdri/jobdri_api/domain/evaluation/analysis/EvaluationAnalysisBatchServiceTest.java index c5759d33..c4b02731 100644 --- a/src/test/java/com/jobdri/jobdri_api/domain/evaluation/analysis/EvaluationAnalysisBatchServiceTest.java +++ b/src/test/java/com/jobdri/jobdri_api/domain/evaluation/analysis/EvaluationAnalysisBatchServiceTest.java @@ -156,6 +156,31 @@ void runWritesSanitizedEvaluationResults() throws Exception { assertThat(commandCaptor.getValue().answer()).isEqualTo("데이터 처리 경험이 있습니다."); } + @Test + @DisplayName("분석 호출 토큰 사용량과 합계를 평가 CSV에 기록한다") + void runWritesAnalysisTokenUsage() throws Exception { + EvaluationGeneratedResult generatedResult = result( + new AnalysisLlmResponse(80, 70, 60, "피드백", List.of(), List.of()), + null, null, 10, 20, 100, 30, 200, 40 + ); + EvaluationAnalysisGenerator generator = command -> generatedResult; + Path input = tempDir.resolve("token-input.csv"); + Path output = tempDir.resolve("token-output.csv"); + Files.writeString(input, + "caseId,jobCategoryMiddle,jobCategorySmall,mainTasks,qualifications,preferences,question,answer\n" + + "EV-01,개발,백엔드,API,Java,,질문,답변\n"); + + new EvaluationAnalysisBatchService(generator, objectMapper).run(input, output); + + Map row = EvaluationCsvSupport.read(output).getFirst(); + assertThat(row.get("candidateInputTokens")).isEqualTo("100"); + assertThat(row.get("candidateOutputTokens")).isEqualTo("30"); + assertThat(row.get("finalInputTokens")).isEqualTo("200"); + assertThat(row.get("finalOutputTokens")).isEqualTo("40"); + assertThat(row.get("totalInputTokens")).isEqualTo("300"); + assertThat(row.get("totalOutputTokens")).isEqualTo("70"); + } + @Test @DisplayName("CSV 입력 순서대로 evaluation command를 generator에 전달한다") void runPassesEvaluationCommandsInCsvOrder() throws Exception { @@ -622,6 +647,21 @@ private EvaluationGeneratedResult result( CandidateReviewResponse candidateReviewResponse, long candidateCallLatencyMs, long finalCallLatencyMs + ) throws Exception { + return result(response, sanitizedCandidateResponse, candidateReviewResponse, + candidateCallLatencyMs, finalCallLatencyMs, null, null, null, null); + } + + private EvaluationGeneratedResult result( + AnalysisLlmResponse response, + AnalysisCandidateResponse sanitizedCandidateResponse, + CandidateReviewResponse candidateReviewResponse, + long candidateCallLatencyMs, + long finalCallLatencyMs, + Integer candidateInputTokens, + Integer candidateOutputTokens, + Integer finalInputTokens, + Integer finalOutputTokens ) throws Exception { String rawLlmResponseJson = objectMapper.writeValueAsString(response == null ? List.of() : response); String rawCandidateResponseJson = objectMapper.writeValueAsString(List.of()); @@ -644,7 +684,11 @@ private EvaluationGeneratedResult result( candidateReviewResponseJson, new EvaluationCandidateReviewSnapshotParser(objectMapper).parse(candidateReviewResponseJson), candidateCallLatencyMs, - finalCallLatencyMs + finalCallLatencyMs, + candidateInputTokens, + candidateOutputTokens, + finalInputTokens, + finalOutputTokens ); } } diff --git a/src/test/java/com/jobdri/jobdri_api/global/cohere/CohereEmbeddingClientTest.java b/src/test/java/com/jobdri/jobdri_api/global/cohere/CohereEmbeddingClientTest.java index dcc3c6fe..00bf2bdf 100644 --- a/src/test/java/com/jobdri/jobdri_api/global/cohere/CohereEmbeddingClientTest.java +++ b/src/test/java/com/jobdri/jobdri_api/global/cohere/CohereEmbeddingClientTest.java @@ -45,6 +45,7 @@ void embedDocuments() throws Exception { assertThat(requestJson.get().get("embedding_types").get(0).asText()).isEqualTo("float"); assertThat(requestJson.get().get("texts")).hasSize(2); assertThat(server.authorizationHeader()).isEqualTo("Bearer test-api-key"); + assertThat(client.apiCallCount()).isEqualTo(1); } } @@ -60,6 +61,7 @@ void embedQuery() throws Exception { assertThat(embedding).hasSize(3); assertThat(requestJson.get().get("input_type").asText()).isEqualTo("search_query"); assertThat(requestJson.get().get("texts")).hasSize(1); + assertThat(client.apiCallCount()).isEqualTo(1); } } @@ -134,6 +136,7 @@ void retryTransientCohereError() throws Exception { assertThat(embedding).hasSize(3); assertThat(server.requestCount()).isEqualTo(2); + assertThat(client.apiCallCount()).isEqualTo(1); assertThat(requestJson.get().get("input_type").asText()).isEqualTo("search_query"); } }