Propagate global --cache_dir to continuous batching pipeline (#4230)

[exzile]

Summary

Fixes #4230.

The continuous batching servable initializer constructs the GenAI ContinuousBatchingPipeline directly and never applied the server-level --cache_dir (ServerSettings.cacheDir). Unlike the non-CB path — which applies it via ModelInstance::setCacheOptions (ieCore.set_property(ov::cache_dir(...))) — the CB path left model-compilation caching disabled unless the user manually duplicated the value into the node's plugin_config as CACHE_DIR. As a result, .blob/.cl_cache artifacts were never persisted and every server restart fully recompiled the model.

Fix

In ContinuousBatchingServableInitializer::initialize, inject the global cache_dir into the pipeline pluginConfig (keyed by ov::cache_dir.name() == CACHE_DIR) right after parsing the node plugin_config and before constructing the pipeline. An explicit CACHE_DIR in the node's plugin_config remains authoritative.

const std::string& globalCacheDir = Config::instance().cacheDir();
if (!globalCacheDir.empty()) {
    if (properties->pluginConfig.find(ov::cache_dir.name()) == properties->pluginConfig.end()) {
        properties->pluginConfig[ov::cache_dir.name()] = globalCacheDir;
    } // else: explicit node CACHE_DIR wins
}

This also applies to the VLM continuous-batching servable, which reuses this initializer.

Testing

Added a regression test LLMNodeOptionsCacheDirPropagation (run for both the CB and VLM fixtures) covering:

1. The global --cache_dir is propagated into pluginConfig["CACHE_DIR"] when the node does not set it.
2. An explicit CACHE_DIR in the node plugin_config takes precedence over the global value.

Built and ran locally on Windows (MSVC) against a real facebook/opt-125m continuous-batching pipeline:

[       OK ] LLMOptionsHttpTest.LLMNodeOptionsCheckPluginConfig
[       OK ] LLMOptionsHttpTest.LLMNodeOptionsCacheDirPropagation
[  PASSED  ] 2 tests.

🤖 Generated with Claude Code

[exzile]

End-to-end verification on GPU

Beyond the unit test, I verified the actual caching behavior on an Intel Arc Pro B70 dGPU (EXPORT_IMPORT capable) serving facebook/opt-125m via continuous batching with device: "GPU" and a --cache_dir pointing at an initially-empty directory:

Run 1 (empty cache):

• Log confirms the fix path: servable_initializer.cpp] Applying global cache_dir to continuous batching pipeline: .../cachedir
• Cache dir went from empty → *.cl_cache files + a 130 MB .blob (compiled model)
• Pipeline init → AVAILABLE: ~1.6 s

Run 2 (restart, populated cache):

• Same fix path fires
• No recompile: blob is reused (file unchanged), no new artifacts written
• Pipeline init → AVAILABLE: ~0.24 s (~6.7× faster cold start)

Without the change the cache directory stays empty on the CB path and every restart recompiles. With it, blobs persist and are reused across restarts as intended.

[dkalinowski]

we probably need this restore also after any ASSERT in this suite. check how other tests do that via guard patter or SetUp/TearDown mechanic (some tests recover env variables this way)

[exzile]

@dkalinowski Added a GlobalCacheDirGuard RAII struct that restores the Config singleton (and removes the temp cache dir) on scope exit, so a failed ASSERT_* mid-test can no longer leak the modified --cache_dir into later tests. Used it in both cache tests; removed the manual end-of-test restores.
Let me know if you want to change this or go a different route.

[dkalinowski]

This fixes it for language_model + continuous_batching

what about:

• visual_language_model + legacy
• visual_language_model + continuous_batching
• language_model + legacy
  ?

[exzile]

Investigated all four paths. VLM_CB already shares the same ContinuousBatchingServableInitializer, so it was already covered.
**The two legacy paths (LM and VLM) construct GenAI pipelines directly and had the same bug — global --cache_dir was never applied.

Extracted the propagation into a shared GenAiServableInitializer::applyGlobalCacheDir() helper (matching the existing loadChatTemplate static pattern), now called by all three initializers (base, CB, LM-legacy, VLM-legacy). This eliminates the duplication and covers every path through one tested code path.

This should suffice, if not we can rewire it a different way. @dkalinowski