DAOS-17893 vos: check objects before freeing #17293

janekmi · 2025-12-18T16:35:02Z

DAOS-17893 is a ticket reporting a crash in DAVv2 happened while freeing an allocation which happened because the provided offset was not a beginning of a memory block but the memory block at hand was not a run. The allocator itself could check for this kind of discrepancies and report before the process will be terminated by a SIGFPE signal. But the higher the issue would be caught the more information we could potentially recover from the crash.

One such place, notorious to be involved in this kind of incidents is the VOS garbage collector (e.g. DAOS-18049). Possibly not because it is more buggy than any other piece of DAOS rather it is a place where we enumerate large chunks of the VOS metadata in order to free the requested objects and all their descendants.

Hence, this PR introduces a few asserts into the GC code so whenever it is possible to validate the offset GC is about to free whether it actually points to an object we expect to live there we assert it actually is as expected and we dump its contents if not for further investigation.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

daosbuild3 · 2025-12-18T16:39:32Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/1/execution/node/279/log

daosbuild3 · 2025-12-18T16:40:12Z

Test stage Build on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/1/execution/node/287/log

daosbuild3 · 2025-12-18T16:41:32Z

Test stage Build on EL 9.6 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/1/execution/node/371/log

github-actions · 2025-12-18T16:45:11Z

Errors are Unable to load ticket data
https://daosio.atlassian.net/browse/DAOS-17893

daosbuild3 · 2025-12-18T16:47:56Z

Test stage Build on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/1/execution/node/405/log

daosbuild3 · 2025-12-18T18:29:41Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/2/execution/node/301/log

daosbuild3 · 2025-12-18T18:30:13Z

Test stage Build on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/2/execution/node/317/log

daosbuild3 · 2025-12-18T18:38:15Z

Test stage Build on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/2/execution/node/411/log

daosbuild3 · 2025-12-18T20:11:55Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/3/execution/node/301/log

daosbuild3 · 2025-12-18T20:12:18Z

Test stage Build on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/3/execution/node/317/log

Signed-off-by: Jan Michalski <[email protected]>

assert + memory dump for debug. Signed-off-by: Jan Michalski <[email protected]>

daosbuild3 · 2025-12-18T20:22:56Z

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/4/execution/node/302/log

daosbuild3 · 2025-12-18T20:23:23Z

Test stage Build on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/4/execution/node/318/log

daosbuild3 · 2025-12-18T20:31:05Z

Test stage Build on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/4/execution/node/412/log

Signed-off-by: Jan Michalski <[email protected]>

daosbuild3 · 2025-12-19T15:15:55Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17293/5/testReport/

daosbuild3 · 2025-12-19T15:18:42Z

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/5/execution/node/668/log

daosbuild3 · 2025-12-19T15:47:01Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17293/5/testReport/

- add d_log_memory_ut to utest.yaml - change D_EMIT to D_FATAL so it is easily visible - move d_log_memory() call before d_alt_assert() call Signed-off-by: Jan Michalski <[email protected]>

- fix DTX_ACT_BLOB_MAGIC assert - remove the ILOG assert (left a comment to spare making the mistake again) Signed-off-by: Jan Michalski <[email protected]>

daosbuild3 · 2025-12-22T15:35:07Z

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-17293/6/execution/node/668/log

daosbuild3 · 2025-12-22T15:35:50Z

Test stage Unit Test bdev on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17293/6/testReport/

daosbuild3 · 2025-12-22T15:38:32Z

Test stage Unit Test bdev with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17293/6/testReport/

daosbuild3 · 2025-12-22T16:05:40Z

Test stage Unit Test with memcheck on EL 8.8 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-17293/6/testReport/

- fix the magic assert for committed DBDs Signed-off-by: Jan Michalski <[email protected]>

NiuYawei · 2025-12-25T02:17:56Z

src/vos/vos_gc.c

+	 * Since the key's structure does not have a magic value and the ilog root (which has
+	 * a magic value) is already destroyed at this stage there is no way to verify the pointer
+	 * actually points to a valid data.
+	 */


Why the assert is removed? It was incorrect?

Yes. It was incorrect. As I have written in the comment, at this stage the key's ILOG is already destroyed, so we cannot use it. A pity.

Nasf-Fan

Hence, this PR introduces a few asserts into the GC code so whenever it is possible to validate the offset GC is about to free whether it actually points to an object we expect to live there we assert it actually is as expected and we dump its contents if not for further investigation.

The commit message is out of date. Please update when you have other chance to refresh the patch. Thanks.

Nasf-Fan · 2025-12-31T02:16:03Z

src/include/gurt/debug.h

+		D_FATAL("Assertion '%s' failed: " fmt, #cond, ##__VA_ARGS__);                      \
+		d_log_memory((uint8_t *)ptr, size);                                                \
+		if (d_alt_assert != NULL)                                                          \
+			d_alt_assert(0, #cond, __FILE__, __LINE__);                                \


"else assert(0);" ?

janekmi requested review from a team as code owners December 18, 2025 16:35

janekmi requested review from NiuYawei and sherintg December 18, 2025 16:36

janekmi force-pushed the janekmi/DAOS-17893-GC-asserts branch from 6bbd198 to 0824861 Compare December 18, 2025 18:24

janekmi force-pushed the janekmi/DAOS-17893-GC-asserts branch from 0824861 to 1508a5e Compare December 18, 2025 20:05

janekmi added 2 commits December 18, 2025 20:17

DAOS-17893 gurt: introduce d_log_memory()

db123e2

Signed-off-by: Jan Michalski <[email protected]>

DAOS-17893 vos: check objects before freeing

3ad502c

assert + memory dump for debug. Signed-off-by: Jan Michalski <[email protected]>

janekmi force-pushed the janekmi/DAOS-17893-GC-asserts branch from 1508a5e to 3ad502c Compare December 18, 2025 20:17

NiuYawei previously approved these changes Dec 19, 2025

View reviewed changes

DAOS-17893 gurt: add missing pthread (fix)

bb1c783

Signed-off-by: Jan Michalski <[email protected]>

janekmi dismissed NiuYawei’s stale review via bb1c783 December 19, 2025 14:42

janekmi added 2 commits December 22, 2025 14:55

DAOS-17893 gurt: introduce d_log_memory() (fix)

2dcebbf

- add d_log_memory_ut to utest.yaml - change D_EMIT to D_FATAL so it is easily visible - move d_log_memory() call before d_alt_assert() call Signed-off-by: Jan Michalski <[email protected]>

DAOS-17893 vos: check objects before freeing (fix)

ddce4f5

- fix DTX_ACT_BLOB_MAGIC assert - remove the ILOG assert (left a comment to spare making the mistake again) Signed-off-by: Jan Michalski <[email protected]>

DAOS-17893 vos: check objects before freeing (fix)

34dc3b7

- fix the magic assert for committed DBDs Signed-off-by: Jan Michalski <[email protected]>

janekmi requested review from Nasf-Fan and NiuYawei December 24, 2025 12:18

NiuYawei reviewed Dec 25, 2025

View reviewed changes

janekmi requested a review from NiuYawei December 29, 2025 13:22

NiuYawei approved these changes Dec 30, 2025

View reviewed changes

Nasf-Fan approved these changes Dec 31, 2025

View reviewed changes

janekmi requested a review from a team December 31, 2025 14:42

DAOS-17893 vos: check objects before freeing #17293

Are you sure you want to change the base?

DAOS-17893 vos: check objects before freeing #17293

Uh oh!

Conversation

janekmi commented Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

github-actions bot commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 18, 2025

Uh oh!

daosbuild3 commented Dec 19, 2025

Uh oh!

daosbuild3 commented Dec 19, 2025

Uh oh!

daosbuild3 commented Dec 19, 2025

Uh oh!

daosbuild3 commented Dec 22, 2025

Uh oh!

daosbuild3 commented Dec 22, 2025

Uh oh!

daosbuild3 commented Dec 22, 2025

Uh oh!

daosbuild3 commented Dec 22, 2025

Uh oh!

NiuYawei Dec 25, 2025

Choose a reason for hiding this comment

Uh oh!

janekmi Dec 29, 2025

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan left a comment

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Dec 31, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

janekmi commented Dec 18, 2025 •

edited

Loading