use cached safety length to skip navigation for electrons and positrons with no magnetic field

[ggalgoczi]

Summary

AdePT uses cached safety distance to speed up navigation for electrons and positrons during tracking in magnetic fields. (This is achievable because MSC calculates the safety distance anyway so it does not need to be calculated again.) This PR proposes to do the same even when no magnetic field is present.

The underlying optimization is that if the proposed step length is smaller than safety length, then there is no point in doing the geometry lookup of the next volume since we know the particle will stay in the current volume.

Running using Example1 with no magnetic field with 1000 electrons with 10 GeV in full LHCb I see 11.09% event loop time speed up and 2.94% wall time when including physics and geometry init. However the results are not byte identical. From 16 events 10 have identical energy deposition in the whole detector but 6 have different energy deposition. In the worst case the difference was 0.021%. I traced this to cases where VecGeom calculates a too large safety (ComputeSafety returns more than the distance to the next boundary that ComputeStepAndNextVolume finds from the same position and navigation state), so the electron can actually skip surfaces.

Was this observed before? I have minimal experience with VecGeom. I will likely file a separate issue with a reproducer.

Because of this I made it a runtime option that is off by default. This does not help every case, eg. when gammas dominate the simulation (on cms2018 -- with magnetic field off -- the wall time is unchanged within noise).

I understand that this is not useful for LHC experiments, since they do have magnetic field almost everywhere but it will be very useful for DUNE far detector, etc. where there is no magnetic field.

Configuration

The kernel option is:

enableApproximateSafetyNavigationSkip

It defaults to false and is exposed through:

/adept/enableApproximateSafetyNavigationSkip true

The value is copied through AdePTConfiguration, AdePTTransportConfig, and TransportKernelOptions

Environment:

• AdePT base d5b15f6
• Stock G4HepEm
• Geant4 11.4.1
• RTX 4090 (sm_89)
• CUDA 13.0
• VecGeom 2.0.0-rc9 (49d44f1f6), VECGEOM_NAV=tuple
• Zero magnetic field
• 10 GeV electrons
• 16 events
• 8 Geant4 worker threads
• 1000 primaries per event
• In repository fullLHCb and cms2018 GDML files

Reproduce

example1 -m fullLHCb_skip.mac

fullLHCb_skip.mac — remove the /adept/enableApproximateSafetyNavigationSkip line for the baseline

/run/numberOfThreads 8
/control/verbose 0
/run/verbose 0
/process/verbose 0
/tracking/verbose 0
/event/verbose 0
/detector/filename examples/data/LHCb_Upgrade_fullLHCb.gdml
/adept/setVerbosity 0
/adept/setMillionsOfTrackSlots 8
/adept/setMillionsOfHitSlots 8
/adept/setCUDAStackLimit 8192
/adept/setTrackInAllRegions true
/adept/enableApproximateSafetyNavigationSkip true
/detector/setField 0 0 0 tesla
/run/setCut 0.7 mm
/run/initialize
/eventAction/verbose 1
/gun/setDefault
/gun/particle e-
/gun/energy 10 GeV
/gun/number 1000
/gun/position 0 0 0
/gun/print false
/gun/randomizeGun true
/gun/addParticle e- weight 1 energy 10 GeV
/gun/minPhi 0 deg
/gun/maxPhi 360 deg
/gun/minTheta 10 deg
/gun/maxTheta 170 deg
/run/beamOn 16

Related AdePT work

Similar features that make use of the same principle:

• #687, inherit safety from charged tracks
• #658, use cached safety and update it in B-field propagation
• #681, improve initialization of tracks

[agheata]

I traced this to cases where VecGeom calculates a too large safety (ComputeSafety returns more than the distance to the next boundary that ComputeStepAndNextVolume finds from the same position and navigation state), so the electron can actually skip surfaces.

How did you trace this? Do you have a reproducer (point/direction giving bad safety value)? Of course this should never happen, but so far we have mainly shape regressions, not a global one. I should add a regression in VecGeom doing exactly this: generating arbitrary steps, validating if they are within safety, or using the distance to boundary if not.

[agheata]

I have added a regression in VecGeom to fix this: https://gitlab.cern.ch/VecGeom/VecGeom/-/merge_requests/1476
After running it thoroughly on the LHC geometries, I had to fix a couple of minor inconsistencies for particles on surface for cone and torus, all at 1e-7-1e-8 so unlikely to produce divergence in AdePT. I would check the correctness of how you cache and use the safety, or else please give me some reproducer

[ggalgoczi]

I have added a regression in VecGeom to fix this: https://gitlab.cern.ch/VecGeom/VecGeom/-/merge_requests/1476 After running it thoroughly on the LHC geometries, I had to fix a couple of minor inconsistencies for particles on surface for cone and torus, all at 1e-7-1e-8 so unlikely to produce divergence in AdePT. I would check the correctness of how you cache and use the safety, or else please give me some reproducer

Thank you @agheata . This PR did not introduce new caching. I see what the bug is likely to be, I will write it up in few days.

[SeverinDiederichs]

Hi @ggalgoczi,
thank you for the finding, that is great! Indeed we usually don't consider the no magnetic field case, that's why it is not optimized at all. If there is some actual interest, there are some other low hanging fruits (e.g. having a run time option with or without a B field, which would save a lot of registers and could significantly improve performance).

For the current improvement, I agree that something seems to be off if the safety is not safe. Therefore, it would be best to fix that problem, and then make the change directly, without making it just an option.

I am still away this week, but happy to discuss next week.
On a side note: the same change should also be done in the electrons_split.cuh for the split kernels, which are typically faster for settings which heavily use the GPU (and slower for those that don't).