Add a smart storage logic to enable configuration and training to use the same data

[viktorbeck98]

Configuration and training are not able to use the same data currently in case of a data stream, which would be preferred. Data that is used in the configuration phase will be lost otherwise.

[ernstleierzopf]

I think this issue from February is deprecated. Looking at the following test, we use the same data for the configuration phase as for the training phase:

class TestNewValueDetectorEndToEnd:
    """Regression test: full configure/train/detect pipeline on audit.log."""

    def test_audit_log_anomalies(self):
        parser = MatcherParser(config=_PARSER_CONFIG)
        detector = NewValueDetector()

        logs = list(From.log(parser, in_path=AUDIT_LOG, do_process=True))

        for log in logs[:TRAIN_UNTIL]:
            detector.configure(log)
        detector.set_configuration()

        for log in logs[:TRAIN_UNTIL]:
            detector.train(log)

        detected_ids: set[str] = set()
        for log in logs[TRAIN_UNTIL:]:
            output = schemas.DetectorSchema()
            if detector.detect(log, output_=output):
                detected_ids.add(log["logID"])

        assert detected_ids == {'1859', '1860', '1861', '1862', '1864', '1865', '1866', '1867'}

Please reopen if this issue is still relevant.

[viktorbeck98]

@ernstleierzopf you misunderstand this issue. In a stream it is not possible to use the same training data because it is not saved anywhere

[ernstleierzopf]

@viktorbeck98 okay, if I understand it correctly, we are not able to train until the configuration phase is finished (we need the config to be set in train). Then our only option is to store the streamed data and reuse it later? I see no easy way to get the data out of auto_conf_persistency.

I have a few questions and points to clarify the issue, as I looked into the codebase.

• The current implementation does not offer functionality for streams yet. At least I do not see it and there are no unittests for this. If there is a way please reference the important classes and we should create unittests for this.

• The data needs to be preserved only when using auto_config: True, otherwise the user can easily create his own buffer, because he sees that the data is lost.

• Even though you are allowed to keep data_use_configure and data_use_training None, this brings the FitLogic into a NOTHING state. So it is expected to set these values. With data_use_configure there is no configuration stage -> no data needs to be kept. If both variables are set, you know exactly how much data needs to be kept (data_use_configure).

  As I am writing this, I see that in the _fit_logic.py file there is a finish_config method after which the previously stored config data could be immediately used for training. This method is only used in the core.py file. There the self.buffer_train variable (TrainBuffer()) is used to keep the data used for configuration.

I think this problem does not exist, it is already solved in code.
Only when you do not use auto_config: True, then you need to keep you data buffered on your own. But I think that is the point of not using auto configuration. In that case you need to write code anyways and can easily keep track of the data on your own.

@ipmach I think you wrote the code, do you agree on my analysis of this issue? I figured it literally out while thinking about solutions for this issue.

[viktorbeck98]

The current implementation does not offer functionality for streams yet. At least I do not see it and there are no unittests for this. If there is a way please reference the important classes and we should create unittests for this.

• Well, kind of it does but we have not tested the whole pipeline with streaming, that is correct. One thing that is open e.g. is the dependency of parsing and anomaly detection -> what happens when a pared template changed, how will this affect the detector during operation/training?

The data needs to be preserved only when using auto_config: True, otherwise the user can easily create his own buffer, because he sees that the data is lost.

• Correct. We have planned a buffer component, that @whotwagner is working on

Even though you are allowed to keep data_use_configure and data_use_training None, this brings the FitLogic into a NOTHING state. So it is expected to set these values. With data_use_configure there is no configuration stage -> no data needs to be kept. If both variables are set, you know exactly how much data needs to be kept (data_use_configure).

• I will leave it to @ipmach to answer this

[ernstleierzopf]

As described in my last comment, I think the buffer functionality is already implemented. Please see use_config_data_as_training in cory.py. It is also already tested in test_core.py. I think this issue is already solved.

I would argue that changing existing templates and reuse of trained detectors should not be supported. As the detectors reference the templates with their position, they should not be changed in any circumstance. It should not be problematic to add new templates, however, changing existing ones could lead to unexpected results and data corruption. Without also adapting the persisted data, in which case this whole endeavor is already a manual process, it cannot be fully tested and correct detection verified.

Please verify again. I suggest closing this issue as the functionality already exists and does not need new components.

[viktorbeck98]

@ernstleierzopf the current functionality regarding use_config_data_as_training is only a temporal solution or at least there will be another option implemented that will outsource storing templates to a buffer component because the current functionality stores every log in RAM so it can be reused for training.

The support of changing templates while operations is, while complex, also necessary to integrate parsers of the literature that modify their templates during processing which is relevant for our research work.

These topics have been discussed frequently already and are fixed steps in our DetectMate roadmap.

[ernstleierzopf]

Sorry for the late response.
The buffer component makes sense, however maybe a hybrid solution between RAM and storage could be used for smaller setups or when log data is not processed in large batches. I will add a generated code section with this idea below.

Change of templates is another separate task that can be extracted into a new issue.
To achieve this, each parser needs to keep track of the current templates and check if any of them changed. Could you please clarify for me if changes of existing template lines are planned specifically or if only new templates can be added and old removed?
What happens if a template line with the variables a,b is changed to track a,c? Should the already trained data be kept - that does not make too much sense to me. If the trained data should be forgotten, then it is the same behavior as adding a new template line and removing an old one.
Changes such as a,b,c -> a,b could work seamlessly. But what should be done if you go from a,b to a,b,c? You have missing information, what should be done? If you define the rules, I could try to figure this out :)

We could also make this a multipart task based the difficulty.

What do you think?

# 1. Create your directories
sudo mkdir -p /mnt/ram_speed      # The 4GB RAM section
sudo mkdir -p /mnt/disk_overflow   # The HDD/SSD section
sudo mkdir -p /mnt/disk_work       # Required metadata folder for overlayfs
sudo mkdir -p /mnt/unified_output  # THE TARGET FOLDER (where you actually write data)

# 2. Mount your 4GB tmpfs
sudo mount -t tmpfs -o size=4G none /mnt/ram_speed

# 3. Create the unified overlay layer
sudo mount -t overlay overlay \
  -o lowerdir=/mnt/ram_speed,upperdir=/mnt/disk_overflow,workdir=/mnt/disk_work \
  /mnt/unified_output 

import os
import subprocess
import sys

# Define our directory paths
BASE_DIR = "/tmp/hybrid_setup"
RAM_DIR = f"{BASE_DIR}/ram_speed"        # The 4GB RAM tier
DISK_DIR = f"{BASE_DIR}/disk_overflow"    # The HDD/SSD tier
WORK_DIR = f"{BASE_DIR}/disk_work"        # Required OverlayFS metadata folder
UNIFIED_DIR = f"{BASE_DIR}/unified"       # THE FOLDER YOU WRITE TO

def setup_hybrid_storage():
    # 1. Ensure all directories exist
    print("Creating layout directories...")
    for directory in [RAM_DIR, DISK_DIR, WORK_DIR, UNIFIED_DIR]:
        os.makedirs(directory, exist_ok=True)

    try:
        # 2. Mount the 4GB tmpfs (RAM)
        print("Mounting 4GB RAM storage (tmpfs)...")
        subprocess.run(
            ["mount", "-t", "tmpfs", "-o", "size=4G", "none", RAM_DIR], 
            check=True
        )

        # 3. Mount the OverlayFS combining RAM and Disk
        print("Creating the unified OverlayFS layer...")
        overlay_options = f"lowerdir={RAM_DIR},upperdir={DISK_DIR},workdir={WORK_DIR}"
        subprocess.run(
            ["mount", "-t", "overlay", "overlay", "-o", overlay_options, UNIFIED_DIR], 
            check=True
        )

        print("\n🎉 Success! Setup complete.")
        print(f"👉 Always read/write your data to: {UNIFIED_DIR}")
        print("When the 4GB RAM limit is reached, it will automatically overflow to disk.")

    except subprocess.CalledProcessError as e:
        print(f"\n❌ Error during mounting: {e}", file=sys.stderr)
        print("Make sure you are running this script with 'sudo'!", file=sys.stderr)
        sys.exit(1)

def cleanup_hybrid_storage():
    """Call this function to safely tear down the mounts when done."""
    print("\nCleaning up mounts...")
    # Must unmount unified overlay layer first, then the ram tier
    for directory in [UNIFIED_DIR, RAM_DIR]:
        if os.path.ismount(directory):
            subprocess.run(["umount", directory], check=False)
    print("Cleanup complete.")

if __name__ == "__main__":
    # Check if running as root
    if os.geteuid() != 0:
        print("This script must be run as root. Try: sudo python script.py")
        sys.exit(1)

    try:
        setup_hybrid_storage()
        # Keep the script alive so the mount stays active while you work
        input("\nPress Enter to safely unmount and exit...")
    finally:
        cleanup_hybrid_storage()

[viktorbeck98]

The hybrid solution seems like a good idea. If you want you can implement that.

About changing templates. This is a very difficult topic and we dont have a solution yet. This is a topic for research but we have not yet done a proper investigation of what is possible and what isn't. I think for now it's ok to assume that a changed template == a new template. At the same time we could also utilize the buffer component here and save everything happening in training so that it can be reparsed once training finishes. This would be a prozess that the detectors' auto configuration in online mode needs anyways, so we could use this procedure in two different situations.

[ipmach]

Hi, sorry late reply. But what you are proposing have multiple issues.

• it requires to run in a Linux system only.
• it requires to run in sudo mode making a security vulnerability in the system.

Also, currently we have two independent methods. This will be a third.

While we currently not have memory issues related to this, this will add a bunch of restrictions of where detectmate can run.

Best,
Andre