contrast_curve() with nbranch > 1 crashes on long runs

[Will-lab418]

I have been running contrast_curve() on a large cube (9.21 GB, 28800 frames, 293x293 pixels) with nproc = 5 and have noticed the function crashing depending on the value of nbranch. when set to 3 the run will consistently crash whereas it usually is able to finish when set to 1.

vip_hci version: 2.1.0

Other settings used are:
sigma = 5
algo = median_sub
student = True
imlib = 'opencv'

I have kept track of the memory usage while running the function and noticed a difference depending on the value of nbranch.

When nbranch = 1, the memory will usually increase but level out after some time, and the run will usually finish after ~2-3 hours.

When nbranch = 3, the memory will not level out and will continue to slowly increase with occasional sharp spikes. The run will stop without an error message and without creating any output files, anywhere from 11-15 hours runtime.

This is on a shared machine but I've seen the same behaviour both when others were actively using it and when I was the only one running anything so I don't think it's just contention from other users.

I'm hoping that someone who is more familiar with VIP will have a better idea of what's causing this, and a potential fix. I have had nbranch = 3 complete once after ~12 hours, and have also had nbranch = 1 crash after running (but these are exceptions).

Happy to share my script for calling the function if that helps.

[IainHammond]

Hi Will, thanks for opening the issue! I ran the contrast curve function myself last week with nbranch = 3 on a smaller cube (~100MB), with algo = pca and didn't have any problems. This might be a memory limitation of python somewhere or some inefficient code that's causing it to stall. There are a few good ways to help us isolate the problem:

• Use the cube_subsample function (from vip_hci.preproc import cube_subsample) to reduce the number of frames. I've been doing this for the METIS cubes since there are so many frames. Something like cube, angs = cube_subsample(cube, n=10, parallactic=angs) would mean combine every 10 frames and make the necessary adjustment to the parallactic angles - making the cube 2880 frames instead. This gives a big speed boost and generally does not impact the achieved contrast, but higher amounts of subsampling will reduce the number of frames significantly and therefore how well the speckle pattern is estimated later by PCA or median ADI (median_sub)
• Try algo = pca as this would help us work out if it's a limitation of the median_sub function
• Try nproc=1. This completely disables the multiprocessing, but would isolate if it's a problem with python multiprocessing as we can run into in some cases. Since each CPU needs its own instance of python and the variables, this could easily inflate the memory use and crash the code. Being on a shared machine could complicate things too

I would try the subsampling first since you may as well use that any way to get faster results. If there is no improvement to the problem, I'd move to the second and third options

Bonus: Let me know if you are using AMD or Intel CPUs. We currently don't disable multithreading on AMD CPUs (but we should be) which could be causing an issue, but I know how to fix it

[Will-lab418]

Hi Iain, thanks again for your suggestions, I've had a chance to give each of them a go and here's how they went:

• I sampled the cube from 28,800 frames to 2,880 which did work, finishing with ~1.5 hours runtime. The contrast curve did have small variations compared to the contrast curve from a previous very lucky nbranch = 3 run. But these differences were small and had no meaningful impact on the resulting DPM.

• When using algo = pca on the full unsampled cube the memory usage was worse than using median_sub at one point in particular spiking to 476 GB. It stalled and crashed after around 20 hours.

• Disabling multiprocessing with nproc = 1 on the full cube turned out to be the best solution. The memory usage stayed completely flat throughout and finished after 1 hour with a contrast curve identical to my nbranch = 3 run that had finished.

I think the nbranch isn't the cause of the issue at all but rather it is some kind of issue with multiprocessing on large cubes. For now I'm happy to use contrast_curve without multiprocessing and using cube_subsample as a backup in case that doesn't work.

Also, the CPUs shouldn't be an issue as the machine I've been running VIP on is using Intel CPUs.

[VChristiaens]

Hi @Will-lab418 (and thanks @IainHammond for the previous reply),
The symptoms you describe, such as the fact that even for a 10x subsampled cube it takes longer to calculate the contrast curve (~1.5h) in multiprocessing with nproc=5 than calculating it on the full cube withnproc=1 (~1h), suggest a problem of oversubscription to me, such as induced by multiprocessing+multithreading. Requesting many more threads than available cores would significantly hamper performance.

Before investigating other options, can you clarify whether you disabled multithreading?
Have you tried to either:

• add the following as preamble of your script:

import os

os.environ["OMP_NUM_THREADS"] = "1"
os.environ["MKL_NUM_THREADS"] = "1"
os.environ["OPENBLAS_NUM_THREADS"] = "1"
os.environ["BLIS_NUM_THREADS"] = "1"

import numpy as np
import scipy
import sklearn

• or run the following in your terminal before your script:

export OMP_NUM_THREADS=1
export MKL_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1
export BLIS_NUM_THREADS=1
python your_script.py

?