CoDICE: L3_lo-direct-events: normalization bug - Card 3472

[mstarkey2158]

@jtniehof @pleasant-menlo @mmshaw

Lynn Kistler discovered that the normalizaion and normalization_per_event variables contained fill values or 0's for all DCR events (i.e., events without a valid APD_ID). These in fact are valid events and should be included in the normalization computations (an oversight on my end).

The current processing code applies a mask to events without a valid apd_id which is the cause of this issue. I have looked through the github processing code and identified this issue in the following codes:

rebin_direct_events_by_energy_and_spin_sector.py at line 81-82 (see screenshot attached)

• We need to remove the apd_id_mask. Only keep the spin_sector_mask and energy_indices_mask

I think the remaining lookup function for the normalization_per_event variable is ok since it uses the above function.
I am happy to join a call if needed.

I do think this should be corrected for data release 2 if possible.

[jtniehof]

#127 specifically added this filter. The main discussion was in email (subject "CoDICE Lo direct-events apd_id = 0", first email from Nicholas at Menlo on June 17). Summary is that the 3d VDF was failing if apd_id==0, and so they were excluded from both the 3D VDF calculation and the normalization.

If we include these events in the normalization but not the 3D VDF, are we going to have the 3D VDF normalized properly? I presume they might just go into different priority bins?

[mstarkey2158]

#127 specifically added this filter. The main discussion was in email (subject "CoDICE Lo direct-events apd_id = 0", first email from Nicholas at Menlo on June 17). Summary is that the 3d VDF was failing if apd_id==0, and so they were excluded from both the 3D VDF calculation and the normalization.

If we include these events in the normalization but not the 3D VDF, are we going to have the 3D VDF normalized properly? I presume they might just go into different priority bins?

According to Lynn, the 3D VDFs would be valid after updating the normalization factor to include the DCRs. The normalization gives us the 'true' count level at any point that the instrument would measure if it could detect all events perfectly. The 3D VDFs should still exclude APD_Id = 0 when filling the count matrix. So, the normalization and normalization_per_event would be the only things to change.

[jtniehof]

Ah, right, because it's norm-per-event, and it's a question of where these events are included or not.

Looking at d6b88af , we do not want the apd_id == 0 filtering in the direct event code: calculate_normalization_factor -> rebin_direct_events_for_normalization -> rebin_direct_events_by_energy_and_spin_sector. However, we do want it in the 3D VDF code,rebin_to_counts_by_species_elevation_and_spin_sector.

I believe we do still want to filter the normalization to exclude apd_id > CODICE_LO_NUM_AZIMUTH_BINS, so a lot of d6b88af changes to that codepath remain in place (it's just removing the comparison to 0).

@mstarkey2158 , is this accurate?

[mstarkey2158]

Ah, right, because it's norm-per-event, and it's a question of where these events are included or not.

Looking at d6b88af , we do not want the apd_id == 0 filtering in the direct event code: calculate_normalization_factor -> rebin_direct_events_for_normalization -> rebin_direct_events_by_energy_and_spin_sector. However, we do want it in the 3D VDF code,rebin_to_counts_by_species_elevation_and_spin_sector.

I believe we do still want to filter the normalization to exclude apd_id > CODICE_LO_NUM_AZIMUTH_BINS, so a lot of d6b88af changes to that codepath remain in place (it's just removing the comparison to 0).

@mstarkey2158 , is this accurate?

Yes, good point. For the events with APD ID > 24, I am not sure if we want to keep those in the normalization calculation or not. Those events correspond to: 30 = ASIC Reported Invalid Channel, and 31 = Energy Timeout. So I guess these could still be valid DCR events that simply had something go wrong in the final APD energy measurement. I will ask Lynn Kistler about this and get back to you.

[jtniehof]

From email:

Positions 28/31 are included in the SW priority rates
Ok, so based on that information we should not limit the normalization computation based on APD ID at all. Also, we should make sure that we are not applying a filter for position as well (which I don’t think we do).

So we should do no filtering on the normalization, but before creating the 3D VDF we should toss any events that do not match 0 < APD_ID <= 24.

I'm going to mark this as ready for implementation.

[mstarkey2158]

From email:

Positions 28/31 are included in the SW priority rates
Ok, so based on that information we should not limit the normalization computation based on APD ID at all. Also, we should make sure that we are not applying a filter for position as well (which I don’t think we do).

So we should do no filtering on the normalization, but before creating the 3D VDF we should toss any events that do not match 0 < APD_ID <= 24.

I'm going to mark this as ready for implementation.

Correct. When creating the 3D VDFs, we should only include events with a valid APD ID since we bin the data based on apd_id into the 3D count array.