DAOS-18017 build: update mercury to 2.4.1 #16943

soumagne · 2025-10-03T15:48:27Z

remove libfabric dependency on DAOS rpm

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2025-10-03T15:48:43Z

Ticket title is 'Update mercury version to 2.4.1'
Status is 'Awaiting Verification'
Labels: 'scrubbed_2.6.5'
Job should run at elevated priority (1)
https://daosio.atlassian.net/browse/DAOS-18017

daosbuild3 · 2025-11-06T22:02:23Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16943/13/execution/node/959/log

daosbuild3 · 2025-11-07T09:01:34Z

Test stage NLT on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16943/16/testReport/

daosbuild3 · 2025-11-07T11:18:30Z

Test stage Test RPMs on EL 8.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16943/16/execution/node/974/log

soumagne · 2025-11-10T16:49:58Z

Jenkinsfile

 // To use a test branch (i.e. PR) until it lands to master
 // I.e. for testing library changes
-//@Library(value='pipeline-lib@your_branch') _
+@Library(value='pipeline-lib@soumagne/mercury_libfabric') _


@phender this is the best and easiest solution I think for now (using the PR daos-stack/pipeline-lib#480), this way there is only minimal changes required to the Jenkinsfiles.

Reminder to ourselves that we need to land the pipeline-lib change first

to be frank, I'm not completely sure of the order that should go in, if I look at the comment it says:

// The @library line needs to be edited in a PR for adding a test. // Adding a test requires two PRs. One to add the test. // That PR should be landed with out deleting the PR branch. // Then a second PR submitted to comment out the @Library line, and when it // is landed, both PR branches can be deleted.

so in theory I think we want to land that PR first so that it uses the branch when it lands, then land the pipeline-lib PR without deleting the branch, and finally merge a new PR that comments out that line again so it uses the default branch. That's how I see it but maybe I'm missing something?

Usually we land the pipeline-lib PR (and delete the branch). Then update this daos PR back to the default / uncommented pipeline-lib. Then land this PR if testing is okay

well imo if you do that then you are going to break all PRs / CIs until this PR has landed :)

Ok, I misunderstood what this PR was doing. I did some testing locally and for example trying to install this new mercury with 2.6.4 works as expected

sudo dnf install mercury-2.4.1~rc5-1.el8 daos-2.6.4 ... Installing: daos 2.6.4-7.3224.g8a20f44f.el8 mercury 2.4.1~rc5-1.el8

So, PRs that do NOT have this change will be okay because their version of daos still depends on mercury and on libfabric.
Then, when pipeline-lib says to install mercury-libfabric it doesn't actually matter for those PRs because mercury-libfabric really just installs mercury and libfabric.

So I see two options. Either what you mentioned about leaving the pipeline-lib reference in the landing of this PR, OR something like this:

Remove the pipeline-lib reference in this PR.

Keep the daos -> libfabric dependency for now for compatibility

Land this PR, since with (1) and (2) it should still work.

Land the pipeline-lib PR

New daos PR to remove the daos -> libfabric dependency

IMO I prefer this approach so we don't have to leave the pipeline-lib reference in place

PRs that do NOT have this change will be okay

you are allowed to install mercury without mercury-libfabric or mercury-ucx but all tests will fail because the providers / plugins won't be found. What this PR does is separate the libfabric mercury plugin that allows mercury to interface with libfabric. Without it, mercury cannot use libfabric.

Keep the daos -> libfabric dependency for now for compatibility

No, this does not help in any way and no with (1) and (2) it won't work because you do need mercury-libfabric. This is not a simple alias of some sort, to expand on what I just mentioned above, it installs a separate library that is similar to mercury-ucx (you can look at that as an example). This library is then dynamically loaded by mercury.

Then I don't know how to land any of this without breaking CI

Well that's what I've just been trying to explain :) I'm glad we're reaching the same conclusion. The real issue is that rpms are globally visible once merged, so because of that, any change of that sort will face the same issues.

daltonbohning · 2025-11-10T21:17:37Z

utils/rpms/daos.sh

 EXTRA_OPTS+=("--rpm-attr" "0755,root,root:${sysconfdir}/daos/certs")

-DEPENDS=( "mercury >= ${mercury_full}" "${libfabric_lib} >= ${libfabric_full}" )
+DEPENDS=( "mercury >= ${mercury_version}" )


DAOS no longer depends on libfabric directly? So mercury is the only access to libfabric?

that's right, only the mercury plugin does. The idea is that we need to be able to install DAOS on systems that only have UCX so we don't want to always a pull a libfabric dependency (DAOS itself does not need it anymore anyway).

utils/rpms/daos.spec

utils/rpms/mercury.sh

daltonbohning · 2025-11-10T21:23:59Z

Jenkinsfile

 // To use a test branch (i.e. PR) until it lands to master
 // I.e. for testing library changes
-//@Library(value='pipeline-lib@your_branch') _
+@Library(value='pipeline-lib@soumagne/mercury_libfabric') _


Reminder to ourselves that we need to land the pipeline-lib change first

frostedcmos

Changes LGTM, would prefer someone more knowledgeable than me with CI process and possible alternatives to approve

soumagne · 2025-11-13T20:28:25Z

@phender @daltonbohning please review this PR. I'm also happy to discuss offline to walk you through it and think about better solutions if there are any. It is passing the CI and I will need to get that merged as there are a number of fixes we need to pick for customers.
For background you can also read more about the need for separating mercury-libfabric in that ticket: https://daosio.atlassian.net/browse/DAOS-13244

soumagne · 2025-11-13T20:56:14Z

From discussion with @mchaarawi since this PR will affect 2.6 also, should get 2.6 merge approval before moving further.

Remove libfabric dependency on DAOS rpm Keep debug log enabled Add patch for version info Signed-off-by: Jerome Soumagne <[email protected]>

Signed-off-by: Jerome Soumagne <[email protected]>

soumagne force-pushed the soumagne/mercury_241rc4 branch 5 times, most recently from c90845f to 595eaa3 Compare October 6, 2025 16:35

soumagne force-pushed the soumagne/mercury_241rc4 branch 3 times, most recently from d01bbb0 to ec4dcea Compare October 28, 2025 16:43

daos-stack deleted a comment from daosbuild3 Oct 28, 2025

soumagne changed the title ~~DAOS-18017 build: update mercury to 2.4.1rc4~~ DAOS-18017 build: update mercury to 2.4.1rc5 Oct 28, 2025

soumagne force-pushed the soumagne/mercury_241rc4 branch from ec4dcea to 513ed05 Compare November 6, 2025 00:25

soumagne mentioned this pull request Nov 6, 2025

DAOS-16501 build: Support of mercury with ASan #16917

Open

6 tasks

soumagne force-pushed the soumagne/mercury_241rc4 branch 2 times, most recently from fd2be22 to c409827 Compare November 6, 2025 18:56

daos-stack deleted a comment from daosbuild3 Nov 6, 2025

soumagne marked this pull request as ready for review November 10, 2025 16:32

soumagne requested review from a team as code owners November 10, 2025 16:32

soumagne commented Nov 10, 2025

View reviewed changes

soumagne requested review from frostedcmos, jgmoore-or and phender November 10, 2025 16:54

daltonbohning reviewed Nov 10, 2025

View reviewed changes

soumagne force-pushed the soumagne/mercury_241rc4 branch from ca3a47d to 52baeb4 Compare November 12, 2025 18:27

soumagne requested a review from daltonbohning November 13, 2025 20:17

frostedcmos reviewed Nov 13, 2025

View reviewed changes

soumagne requested a review from mchaarawi November 13, 2025 20:28

soumagne force-pushed the soumagne/mercury_241rc4 branch from 779d38b to cfea0f3 Compare November 24, 2025 23:02

soumagne requested review from a team as code owners November 24, 2025 23:02

github-actions bot added the priority Ticket has high priority (automatically managed) label Nov 24, 2025

soumagne changed the title ~~DAOS-18017 build: update mercury to 2.4.1rc5~~ DAOS-18017 build: update mercury to 2.4.1 Nov 25, 2025

soumagne force-pushed the soumagne/mercury_241rc4 branch from cfea0f3 to 1e3c1fb Compare November 26, 2025 17:57

soumagne added 2 commits December 2, 2025 11:45

DAOS-18017 build: update mercury to 2.4.1

d90326e

Remove libfabric dependency on DAOS rpm Keep debug log enabled Add patch for version info Signed-off-by: Jerome Soumagne <[email protected]>

DAOS-18017 build: test new mercury-libfabric

55cda7d

Signed-off-by: Jerome Soumagne <[email protected]>

soumagne force-pushed the soumagne/mercury_241rc4 branch from 1e3c1fb to 55cda7d Compare December 2, 2025 17:46

jolivier23 approved these changes Dec 4, 2025

View reviewed changes

frostedcmos approved these changes Dec 9, 2025

View reviewed changes

DAOS-18017 build: update mercury to 2.4.1 #16943

Are you sure you want to change the base?

DAOS-18017 build: update mercury to 2.4.1 #16943

Uh oh!

Conversation

soumagne commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Oct 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Nov 6, 2025

Uh oh!

daosbuild3 commented Nov 7, 2025

Uh oh!

daosbuild3 commented Nov 7, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frostedcmos left a comment

Choose a reason for hiding this comment

Uh oh!

soumagne commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

soumagne commented Nov 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

6 participants

soumagne commented Oct 3, 2025 •

edited

Loading

github-actions bot commented Oct 3, 2025 •

edited

Loading

soumagne commented Nov 13, 2025 •

edited

Loading