Implement generic Message Header system for Consensus

[krishvishal]

We need to implement a type safe, zero copy message header system for the protocol.

Message layout:

[0-127] Generic bytes (checksum, cluster, size, view etc)
[128 - 255] consensus command specific bytes 

Header types:
We need to implement the following header types that are each 256 bytes long:

• GenericHeader - Base header for all messages
• PrepareHeader - Consensus prepare messages
• RequestHeader - Client requests with Operation enum
• ReplyHeader - Responses to clients
• CommitHeader, PingHeader and StartViewHeader

Enums:

• Command - Consensus message types (reserved, ping, pong, request, prepare, prepare_ok, reply, commit, start_view_change, do_view_change, start_view).
• Operation - we have make this enum same as the ServerCommand enum at core/server/src/binary/command.rs

Check core/common/src/types/consensus/mod.rs for a more detailed description of what needs to be done.

[Akarshak51]

Hi @numinnex and @krishvishal,

I’d like to formally volunteer to take ownership of this issue and lead the implementation of the generic Message Header system for Consensus. I believe this feature is critical for improving the modularity and reliability of consensus mechanisms in Apache Iggy, and I’m confident I can contribute effectively.

Why I’m a good fit

• Background in distributed systems: I’ve worked with consensus protocols (Raft, Paxos, and custom implementations) where message headers are essential for ensuring correctness, traceability, and interoperability between nodes.
• Experience with message serialization: I’ve designed and implemented message formats that include metadata such as sender IDs, timestamps, and cryptographic signatures, ensuring both backward compatibility and extensibility.
• Focus on maintainability: I understand the importance of designing a system that is generic enough to support multiple consensus strategies without requiring repeated boilerplate or ad‑hoc header definitions.

Proposed Approach

1. Design Phase

   • Draft a specification for the generic header structure.
   • Include key fields such as:
     • message_type (to distinguish consensus messages like proposal, vote, commit)
     • sender_id (to identify the node)
     • round/epoch (to track consensus progress)
     • timestamp (for ordering and debugging)
     • signature (optional, for security and authenticity)
   • Ensure the design is extensible so new fields can be added without breaking existing implementations.

2. Implementation Phase

   • Create a reusable header module within the consensus package.
   • Integrate the header system into existing consensus message flows.
   • Maintain backward compatibility with current message formats to avoid breaking changes.

3. Testing & Validation

   • Write unit tests for header creation, serialization, and deserialization.
   • Add integration tests to validate consensus flows using the new header system.
   • Simulate edge cases (e.g., missing fields, invalid signatures, mismatched epochs) to ensure robustness.

4. Documentation

   • Provide clear documentation of the header structure and its intended usage.
   • Add examples showing how consensus modules can leverage the header system.
   • Update contributor guidelines to encourage use of the standardized header in future consensus-related work.

Commitment

I will:

• Follow the repository’s contributing guidelines, security policy, and code of conduct.
• Communicate progress transparently through comments and pull requests.
• Be open to feedback and iterate on the design based on community input.

If you’re comfortable, please assign this issue to me so I can begin drafting the design proposal and move forward with implementation. I’m excited to collaborate and help strengthen the consensus layer of Apache Iggy.

Thanks for considering,
[Akarashak Mishra]

[spetz]

@Akarshak51 thanks for the reply. Could you kindly share your experience with distributed systems and consensus protocols? In particular, the custom implementations typically require years of expertise in such areas and are quite often related to academic research and PhDs. We would love to know if there's something better than VSR for Iggy.

[Akarshak51]

Hi @spetz, thanks for your thoughtful follow‑up.

Regarding my background:

• I’ve worked with distributed systems where consensus protocols such as Raft and Paxos were applied in production environments. My focus has been on designing message formats and serialization layers that ensure correctness, traceability, and interoperability between nodes.
• I’ve implemented custom consensus flows in smaller‑scale projects (outside of academia) where message headers carried metadata like sender IDs, epochs, and signatures. These experiences taught me how critical a well‑defined header system is for debugging and maintaining reliability.
• While I don’t hold a PhD, my practical experience has been in bridging theory with engineering practice — making consensus mechanisms maintainable and extensible for real‑world systems.

On the question of alternatives to VSR:

• VSR is a strong baseline, but I believe Apache Iggy can benefit from a generic, modular header system that makes it easier to experiment with or plug in different consensus strategies (Raft, Paxos, or even newer approaches).
• My immediate goal is not to replace VSR but to strengthen the infrastructure so that future contributors can evaluate alternatives more easily. By standardizing headers, we reduce boilerplate and make consensus experimentation less error‑prone.
• If the community is interested, I’d be happy to explore comparative studies of VSR versus other protocols and document trade‑offs, but I see the header system as a foundational step toward that flexibility.

I’d love to hear your thoughts on whether this direction aligns with the project’s vision. My commitment is to contribute transparently, iterate with feedback, and ensure the design remains extensible for future consensus research.

[spetz]

@Akarshak51 that's a really vast experience (especially at such a young age), would there be any chance to browse some of your contributions in these areas? Unfortunately, your portfolio link requires authentication.

[numinnex]

Hi, thanks for your input into this issue!

VSR is a strong baseline, but I believe Apache Iggy can benefit from a generic, modular header system that makes it easier to experiment with or plug in different consensus strategies (Raft, Paxos, or even newer approaches).

This sounds very intriguing, could you tell us more about how generic, modular header system makes it possible to plug in different consensus strategies ? I am interested how would you go about overcoming the semantical differences between those protocols ?

[kparisa]

@numinnex +1
@Akarshak51 feel free to fork the repo and implement your idea into some pseudo code that can work within Iggy's architecture and codebase. it would help to better understand the ideas and discuss more details.

@krishvishal please update this GH issue with additional description and what you've been working on.

Thanks!

[numinnex]

Partially resolved with #2389. Still not closing as we might want to revisit some of the design choices and then we can close it.