MIDAS — Microstructural Imaging using Diffraction Analysis Software

MIDAS is an open-source suite for reconstructing three-dimensional microstructures from High-Energy Diffraction Microscopy (HEDM) data. Developed at the Advanced Photon Source at Argonne National Laboratory, it supports the complete data-reduction pipeline — from raw detector frames to grain maps, strain tensors, spatially resolved orientation fields, and tomographic reconstructions.

Version: 11.0 Contact: Hemant Sharma (hsharma@anl.gov)

---

Key Capabilities

Technique	What It Produces	Detector Distance
Far-Field HEDM (FF-HEDM)	Grain centroids, average orientations, full elastic strain tensors	≈ 1 m
Near-Field HEDM (NF-HEDM)	Spatially resolved 3D orientation maps, grain morphology, grain boundary networks	≈ 5–10 mm
Point-Focus HEDM (PF-HEDM)	High-resolution grain orientations from focused beam	≈ 1 m
Radial Integration (Caking)	1D intensity vs. 2θ profiles for Rietveld refinement (GSAS-II)	—
Grain Matching & Stitching	Track grains across load states; combine multi-layer scans	—
Tomography (CT)	Absorption-contrast cross-sections via gridrec algorithm	—

Version 11 Highlights

• GPU acceleration — CUDA-accelerated indexing (IndexerGPU), strain fitting (FitPosOrStrainsGPU), NF orientation fitting (FitOrientationGPU), scanning HEDM (IndexerScanningGPU, FitOrStrainsScanningGPU), radial integration (IntegratorFitPeaksGPUStream), and tomographic reconstruction (tomo_gpu). See GPU_Acceleration
• Consolidated binary I/O — PF/scanning HEDM replaces ~30K+ per-voxel files with 3 consolidated binary files per scan via IndexerConsolidatedIO.h
• CalibrantIntegratorOMP — new primary calibration executable (replaces archived CalibrantPanelShiftsOMP)
• Switchable peak fitting — unified pseudo-Voigt (pV) and Thompson-Cox-Hastings (TCH) modes across calibration, integration, and auto-calibration
• Physical corrections — parallax, solid-angle, and polarization corrections in DetectorMapper (matching pyFAI convention)
• Q-spacing integration — radial bins equally spaced in Q (Å⁻¹) via QBinSize/QMin/QMax parameters
• CBF file format — new ReadCBFFrame() reader for CBF detector data
• Streaming median — histogram-based streaming median for NF image processing (~500 MB vs 11.5 GB)
• Stripe artifact removal — Vo et al. (2018) algorithms for tomography ring/stripe correction
• Shared integration library — extracted IntegrationCore, MapperCore, CalibrationCore, CalibPeakFit modules
• Centralized MIDAS_ParamParser — unified parameter file parsing across all executables
• Gradient-aware pixel splitting — sub-pixel radial resampling for improved integration accuracy

Recent GUI Update (May 2026)

• FF viewer caking overlay workflow — added load/save/edit support for cake_parameters*.csv, per-detector caking metadata, and composite-sector overlay drawing.
• Plot/Clear + keyboard toggle — caking overlay can now be toggled directly from the GUI and with the C shortcut.
• Multi-detector caking controls — HYDRA mode now includes per-GE cake rows and a dedicated cake parameter editor panel.
• Path precedence hardening — explicit user selections (first file, dark file, HDF5 data/dark dataset paths) are preserved and no longer overwritten by later parameter-file loads.
• Dark subtraction fallback — when no separate dark file is configured, dark data can be read from the current HDF5 data file path.
• Overlay readability improvements — lab-frame axes were restyled for clearer on-image visibility.

Version 10 Highlights

• Consolidated HDF5 output — all FF-HEDM results (grains, spots, strains, peak provenance) in a single .h5 file
• Pipeline restart — --resume and --restart-from flags on all workflows (FF, PF, NF, dual-dataset, multi-resolution) allow resuming from any completed stage using the HDF5 checkpoint
• Version & provenance tracking — every C binary and Python workflow embeds MIDAS v11.0 (<git-hash>) into output files, HDF5 attributes, and Zarr metadata for full reproducibility
• Pseudo-Voigt peak fitting — two-stage decomposed fitting (Lorentzian + Gaussian) with mu-weighted effective widths
• Grain matching & stitching — Python-native optimal (Hungarian) matching across load states with affine deformation support
• Reprocess mode — regenerate MergeMap.csv and consolidated HDF5 on old datasets (-reprocess 1)
• Dynamic detector sizes — auto-detected from data, no hardcoded dimensions
• Zarr-ZIP data format — compressed, portable, self-contained analysis archives
• Forward simulation engine — compressed output with OpenMP parallelism
• Multi-resolution NF-HEDM — iterative reconstruction at increasing grid resolution
• Interactive GUI — browser-based visualization of NF calibration, microstructure, and FF results
• Multi-stage auto-calibration — geometry-first calibration strategy with panel auto-detection from masks

---

Repository Layout

MIDAS/
├── FF_HEDM/          # Far-field HEDM (calibration, indexing, fitting, integration)
│   ├── src/          #   C source code (peak search, fitting, merging, indexing, grains)
│   ├── workflows/    #   Python workflow drivers (ff_MIDAS.py)
│   ├── bin/          #   Compiled binaries (auto-generated by build)
│   └── Example/      #   Example dataset for testing
├── NF_HEDM/          # Near-field HEDM reconstruction
│   ├── src/          #   C source code (forward model, grid reconstruction)
│   ├── workflows/    #   Python workflow drivers (nf_MIDAS.py)
│   ├── bin/          #   Compiled binaries (auto-generated by build)
│   ├── v7/           #   Workflow templates
│   ├── seedOrientations/  # Orientation seed files (downloaded by build)
│   └── Example/      #   Example dataset for testing
├── DT/               # Diffraction tomography (peak-fit integrator)
├── TOMO/             # Tomographic reconstruction (gridrec CT engine)
├── utils/            # Python utilities
│   ├── AutoCalibrateZarr.py   # FF-HEDM auto-calibration
│   ├── match_grains.py        # Grain matching and layer stitching
│   ├── calcMiso.py            # Crystallographic misorientation calculations
│   ├── gsas_ii_refine.py      # GSAS-II integration
│   └── ...
├── gui/              # Interactive visualization GUI
├── manuals/          # Comprehensive documentation (see below)
├── cmake/            # CMake build configuration and dependency management
├── build.sh          # Build script (Linux / macOS)
├── build_wsl_windows.sh  # Build script (Windows via WSL)
├── environment.yml   # Conda environment specification
├── CMakeLists.txt    # Top-level CMake configuration
└── LICENSE           # UChicago Argonne open-source license

---

Documentation

Full manuals are in the manuals/ directory. Start with the manuals README for an overview of all HEDM techniques, coordinate systems, and a getting-started checklist.

Manual	Topic
GPU_Acceleration	GPU-accelerated computation (CUDA)
FF_Calibration	FF-HEDM geometry calibration
FF_Analysis	FF-HEDM grain indexing and fitting
FF_Match_Stack_Reconstructions	Grain matching across load states and layer stitching
FF_Radial_Integration	Radial integration / caking
FF_Interactive_Plotting	Interactive FF-HEDM visualization
FF_Visualization	FF-HEDM result visualization
FF_Dual_Datasets	Dual-dataset FF-HEDM analysis
PF_Analysis	Point-Focus HEDM analysis
NF_Calibration	NF-HEDM detector calibration
NF_Analysis	NF-HEDM reconstruction workflow
NF_MultiResolution_Analysis	Multi-resolution NF-HEDM
NF_GUI	NF-HEDM interactive GUI
Forward_Simulation	Forward simulation for validation
GSAS-II_Integration	Importing MIDAS output into GSAS-II
Tomography_Reconstruction	Absorption-contrast CT reconstruction
FF_Benchmark	FF-HEDM benchmark testing
NF_Benchmark	NF-HEDM benchmark testing

---

Installation

Python packages via pip (no source build)

The Python pipeline — calibration, integration, indexing, refinement, grain processing — installs from PyPI and does not need the source build below:

pip install "midas-suite[all]"

midas-index and midas-fit-grain bundle C/OpenMP executables (midas_indexer, midas_fitgrain) and are published as sdist, so pip compiles them during install. CMake and ninja come along automatically; you need a C compiler and OpenMP (macOS: brew install libomp gcc; Linux: gcc; Windows: Visual Studio Build Tools). If either is missing the install still succeeds and silently leaves you on the much slower Python-only path — so check:

import midas_index.backend_c as b;      print(b.available())   # True = c-omp indexer present
import midas_fit_grain.backend_c as f;  print(f.available())   # True = c-omp refiner present

See packages/midas_suite/README.md for the full breakdown of what pip does and does not give you.

Prerequisites

Platform	Requirements
macOS	Homebrew, LLVM, libomp, GCC, CMake, jemalloc
Linux	GCC ≥ 9, CMake ≥ 3.16
Windows	WSL with Ubuntu

MIDAS automatically downloads and builds these C/C++ dependencies during compilation: NLOPT, LIBTIFF, FFTW, HDF5, BLOSC, BLOSC-2, ZLIB, LIBZIP.

Clone

git clone https://github.com/marinerhemant/MIDAS.git
cd MIDAS

macOS

1. Install Homebrew (if not already installed):

   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

   Without sudo access, install to your home directory:

   mkdir homebrew && curl -L https://github.com/Homebrew/brew/tarball/main | tar xz --strip-components 1 -C homebrew
   eval "$(homebrew/bin/brew shellenv)"
   brew update --force --quiet
   chmod -R go-w "$(brew --prefix)/share/zsh"

   Add Homebrew to your PATH:

   echo 'eval $(/opt/homebrew/bin/brew shellenv)' >> ~/.zshrc
   source ~/.zshrc

2. Install dependencies:

   brew install llvm libomp gcc cmake jemalloc

3. Configure environment variables:

   echo 'export PATH="/opt/homebrew/opt/llvm/bin:$PATH"' >> ~/.zshrc
   echo 'export LDFLAGS="-L/opt/homebrew/opt/llvm/lib $LDFLAGS"' >> ~/.zshrc
   echo 'export CPPFLAGS="-I/opt/homebrew/opt/llvm/include $CPPFLAGS"' >> ~/.zshrc
   echo 'export LDFLAGS="-L/opt/homebrew/opt/libomp/lib $LDFLAGS"' >> ~/.zshrc
   echo 'export CPPFLAGS="-I/opt/homebrew/opt/libomp/include $CPPFLAGS"' >> ~/.zshrc
   echo 'CC=/opt/homebrew/opt/gcc/bin/gcc-15' >> ~/.zshrc
   echo 'export CC' >> ~/.zshrc
   echo 'CXX=/opt/homebrew/opt/gcc/bin/g++-15' >> ~/.zshrc
   echo 'export CXX' >> ~/.zshrc
   source ~/.zshrc

4. Build:

   ./build.sh

Linux

./build.sh

Windows (WSL)

sudo ./build_wsl_windows.sh

Build Options

# Disable CUDA support
./build.sh --cuda OFF

# Specify a custom nvcc compiler path
./build.sh --nvcc /usr/local/cuda-12/bin/nvcc

# Or pass it directly to cmake
cd build && cmake .. -DCMAKE_CUDA_COMPILER=/path/to/cuda/bin/nvcc && cmake --build . -j

# Specify target CUDA architectures (default is "86;90")
cd build && cmake .. -DCMAKE_CUDA_ARCHITECTURES="75;80" && cmake --build . -j

# Build and run benchmarks
./build.sh --test all

# Build a single target
cd build
cmake --build . --target IntegratorFitPeaksGPUStream

Updating After git pull

Each time you pull the latest changes, rebuild the C binaries:

cd MIDAS/build
cmake --build .

Python Environment

conda env create -f environment.yml
conda activate midas_env

Or install the required packages manually:

pip install numpy scipy h5py parsl

---

Quick Start

1. Calibrate the detector geometry → FF_Calibration
2. Run FF-HEDM grain indexing and fitting → FF_Analysis
3. Visualize results interactively → FF_Interactive_Plotting
4. Match/stitch grains across load states or layers → FF_Match_Stack_Reconstructions
5. Reconstruct NF-HEDM orientation maps → NF_Analysis
6. Validate with forward simulation → Forward_Simulation

See the manuals README for the full step-by-step checklist.

Example: Run FF-HEDM on the included test dataset

conda activate midas_env
cd FF_HEDM/Example
python ../workflows/ff_MIDAS.py -paramFN ps_ff.txt

Example: Run FF-HEDM with GPU acceleration

python FF_HEDM/workflows/ff_MIDAS.py -paramFN ps_ff.txt -useGPU 1

Example: Resume a failed pipeline from its last checkpoint

# Auto-detect last completed stage and resume:
python FF_HEDM/workflows/ff_MIDAS.py -paramFN ps_ff.txt -resume /path/to/consolidated.h5

# Or restart from a specific stage:
python FF_HEDM/workflows/ff_MIDAS.py -paramFN ps_ff.txt -restartFrom refinement

Example: Reprocess old results (generate MergeMap.csv + consolidated HDF5)

python FF_HEDM/workflows/ff_MIDAS.py -reprocess 1 -resultFolder /path/to/results/

Example: Match grains across load states

python utils/match_grains.py match \
  --state1 unloaded/Grains.csv \
  --state2 loaded/Grains.csv \
  --space-group 225 --mode combined --weights 2.0 50.0

---

Benchmark Testing

MIDAS includes automated benchmark tests that validate every pipeline using synthetic data and built-in reference comparisons. Each test also runs pre-flight checks (binary existence, staleness, package availability) and prints an environment fingerprint to make failures easy to diagnose.

Quick Start

# Build and run all benchmarks
./build.sh --test all

# Or run benchmarks individually:
./build.sh --test ff    # FF-HEDM only
./build.sh --test nf    # NF-HEDM only

Running Benchmarks Separately

conda activate midas_env

# FF-HEDM: Forward simulation → indexing → grain recovery (6-stage HDF5 comparison)
python tests/test_ff_hedm.py -nCPUs 4

# NF-HEDM: Forward simulation → reconstruction → orientation comparison (>80% < 0.25°)
python tests/test_nf_hedm.py -nCPUs 8

# Calibration + Integration: CeO2 ring fitting + peak strain residual
python tests/test_calibration_integration.py -nCPUs 4

# Phase ID: CeO2 detected, Au absent, lattice constant within 500 ppm
python tests/test_phase_id.py -nCPUs 4

# Tomography: Shepp-Logan phantom → FBP reconstruction → Pearson correlation > 0.85
python tests/test_tomo.py -nCPUs 4

If a Test Fails

All tests support built-in diagnostic flags for troubleshooting:

# Generate a JSON diagnostic report with full environment and comparison details
python tests/test_ff_hedm.py -nCPUs 4 --diagnose

# Save the generated output file alongside the reference for manual comparison
python tests/test_ff_hedm.py -nCPUs 4 --save-on-fail

Send the generated diagnostic report to hsharma@anl.gov for assistance.

See tests/README.md for detailed documentation on what each test computes, how results are compared, and exact pass/fail criteria.

---

Citing MIDAS

If you use MIDAS in your research, please cite:

In review, citations coming soon.

FF-HEDM methodology:

H. Sharma, R. M. Huizenga & S. E. Offerman, "A fast methodology to determine the characteristics of thousands of grains using three-dimensional X-ray diffraction. I. Overlapping diffraction peaks and parameters of the experimental setup," J. Appl. Cryst. 45, 693–704 (2012). DOI: 10.1107/S0021889812025563

H. Sharma, R. M. Huizenga & S. E. Offerman, "A fast methodology to determine the characteristics of thousands of grains using three-dimensional X-ray diffraction. II. Volume, centre-of-mass position, crystallographic orientation and strain state of grains," J. Appl. Cryst. 45, 705–718 (2012). DOI: 10.1107/S0021889812025599

---

Acknowledgments

• SGInfo library for HKL calculations
• ODFPF package (Cornell) for misorientation functions

---

License

MIDAS is released under the UChicago Argonne open-source license.

Copyright © 2012, UChicago Argonne, LLC. All rights reserved.

This product includes software produced by UChicago Argonne, LLC under Contract No. DE-AC02-06CH11357 with the Department of Energy.