Skip to content

Add variables to Hi L1B DE product#2749

Merged
subagonsouth merged 3 commits intoIMAP-Science-Operations-Center:devfrom
subagonsouth:2746-hi-l1b---add-spin_set_met-and-packet_full-variables
Feb 23, 2026
Merged

Add variables to Hi L1B DE product#2749
subagonsouth merged 3 commits intoIMAP-Science-Operations-Center:devfrom
subagonsouth:2746-hi-l1b---add-spin_set_met-and-packet_full-variables

Conversation

@subagonsouth
Copy link
Contributor

Change Summary

Overview

Adds two new variables to the Hi L1B Direct Event product

New Variables

  1. esa_step_met (float64, epoch dimension)
    - MET time when the ESA was stepped, computed from esa_step_seconds + esa_step_milliseconds / 1000
    - Provides a convenient floating-point representation of the ESA step timestamp
  2. ccsds_qf (uint8, epoch dimension)
    - Quality flag bitmask for each CCSDS packet
    - Bit 0 (PACKET_FULL): Set when packet contains 664 events (maximum capacity)

Implementation Details

  • Added de_esa_step_met() function to compute ESA step MET
  • Added de_ccsds_qf() function to compute CCSDS quality flags
  • Added ImapHiL1bDeFlags flag class to quality_flags.py following existing patterns
  • Added variable attribute definitions to imap_hi_variable_attrs.yaml
  • L1B DE output now contains 17 data variables (previously 15)

Files Changed

  • imap_processing/hi/hi_l1b.py - Added two new processing functions
  • imap_processing/quality_flags.py - Added ImapHiL1bDeFlags class
  • imap_processing/cdf/config/imap_hi_variable_attrs.yaml - Added variable definitions
  • imap_processing/tests/hi/test_hi_l1b.py - Added tests for new functions

Closes: #2746

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the IMAP-Hi L1B Direct Event (DE) product generation to include two new epoch-dimension variables that carry (1) a floating-point ESA step MET timestamp and (2) a CCSDS packet-level quality flag bitmask.

Changes:

  • Add esa_step_met derived from esa_step_seconds + esa_step_milliseconds.
  • Add ccsds_qf bitmask with PACKET_FULL (bit 0) set for packets containing 664 events.
  • Extend variable attribute configuration and add unit tests for the new helpers.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
imap_processing/hi/hi_l1b.py Adds de_esa_step_met() and de_ccsds_qf() and injects their outputs into the L1B DE dataset.
imap_processing/quality_flags.py Introduces ImapHiL1bDeFlags IntFlag for DE packet-level quality bits.
imap_processing/cdf/config/imap_hi_variable_attrs.yaml Adds attribute definitions for hi_de_esa_step_met and hi_de_ccsds_qf.
imap_processing/tests/hi/test_hi_l1b.py Updates dataset variable count expectations and adds unit tests for the new functions.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

subagonsouth and others added 2 commits February 20, 2026 16:05
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
# Filter out fill/out-of-range indices (e.g., uint16 FILLVAL 65535)
valid_mask = (ccsds_indices >= 0) & (ccsds_indices < n_packets)

# If there are no valid events, all packets keep default quality flag 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would think that if there were no valid events you would want everything flagged? Not with ImapHiL1bDeFlags.PACKET_FULL but another one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could add another flag to indicate an empty packet but currently, the only flag I have defined indicates that the CCSDS packet contained the maximum number of events. You are correct that a packet having no packets probably indicates a problem but that may be caught elsewhere.

@subagonsouth subagonsouth self-assigned this Feb 23, 2026
@subagonsouth subagonsouth merged commit d4ad77d into IMAP-Science-Operations-Center:dev Feb 23, 2026
14 checks passed
@subagonsouth subagonsouth deleted the 2746-hi-l1b---add-spin_set_met-and-packet_full-variables branch February 23, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Hi L1B - add spin_set_met and packet_full variables

3 participants