fs/inode: bug fix about fdlist_extend, fdlist_install and eventfd/timerfd_blocking_io

[Donny9]

Summary

This PR contains four bug fixes and improvements to the NuttX file system layer, specifically focusing on file descriptor management and event handling:

1. Fix fd allocation exceeding OPEN_MAX (fs/inode/fs_files.c)

The OPEN_MAX boundary check in fdlist_extend() was incorrectly using orig_rows instead of (orig_rows + 1) when validating whether the new allocation would exceed limits. This allowed file descriptor allocation to succeed even when it would exceed OPEN_MAX, potentially causing memory corruption or exceeding system resource limits.

Example: With OPEN_MAX=256 and CONFIG_NFILE_DESCRIPTORS_PER_BLOCK=32, when orig_rows=8 (already at 256 fds), the old code would check 32 * 8 > 256 (false, allows extension to 288), while the fixed code correctly checks 32 * 9 > 256 (true, blocks extension).

2. Fix fd tag loss during parent-to-child inheritance (fs/inode/fs_files.c)

When child processes inherited file descriptors from parent via fdlist_copy(), the fd tags (fd_tag_fdsan and fd_tag_fdcheck) were not being preserved. This caused assertion failures when child processes closed inherited fds, as the fdcheck/fdsan subsystems expected matching tags.

The fix adds a copy parameter to fdlist_install() to distinguish between new fd allocation (initialize fresh tags) and fd inheritance (preserve parent tags). This is critical for any application using fork/clone with inherited file descriptors when CONFIG_FDSAN or CONFIG_FDCHECK are enabled.

3. Fix busy loop in eventfd/timerfd (fs/vfs/fs_eventfd.c, fs/vfs/fs_timerfd.c)

Updated semaphore handling to properly advance to the next waiting task, preventing busy loops in event and timer file descriptor operations. Without this fix, certain polling scenarios could cause the system to spin unnecessarily.

Impact

Users:

• Applications using fork/clone with CONFIG_FDSAN or CONFIG_FDCHECK will no longer crash when closing inherited file descriptors
• Systems can no longer exceed OPEN_MAX limit, preventing potential memory corruption
• Debug builds will clearly identify file descriptor leaks instead of obscure failures
• Event and timer fd operations will be more efficient without busy loops

Build Process:

• No build process changes
• All changes are runtime behavior fixes

Hardware:

• No hardware-specific changes
• Fixes apply to all architectures

Documentation:

• No documentation changes required

Security:

• Prevents potential buffer overflow/memory corruption from exceeding OPEN_MAX
• Improves system stability and resource management

Compatibility:

• All changes are backward compatible
• Fixes broken functionality without changing APIs
• May expose existing bugs in applications that were exceeding OPEN_MAX undetected

Testing

Test 1: OPEN_MAX Boundary Check

Test Procedure:

// Test program that attempts to open more than OPEN_MAX files
int fds[300];
int count = 0;
for (int i = 0; i < 300; i++) {
    fds[i] = open("/dev/null", O_RDONLY);
    if (fds[i] < 0) {
        printf("Failed at count %d, errno=%d\n", count, errno);
        break;
    }
    count++;
}

Before Fix:

Could allocate up to 288 fds (exceeded OPEN_MAX=256 by 32)
Potential memory corruption
After Fix:

Correctly stops at or before OPEN_MAX (253-256 depending on stdio fds)
Returns -EMFILE when limit reached
No memory corruption

Test 2: Fork with Inherited FDs

Test Procedure:

// Parent opens files
int fd1 = open("/tmp/test.txt", O_RDWR);
int fd2 = socket(AF_INET, SOCK_STREAM, 0);

// Fork child process
pid_t pid = fork();
if (pid == 0) {
    // Child closes inherited fds
    close(fd1);
    close(fd2);
    exit(0);
}

After Fix:

Child successfully closes inherited fds
fd_tag_fdsan and fd_tag_fdcheck properly preserved
No crashes, all tests pass
Test Log:
Ran complete test suite with uv_run_tests_main including:

run_helper_tcp4_echo_server
Multiple fork/close cycles
File descriptor inheritance scenarios
All tests PASSED without assertions

OSTest Verification:

Ran full OSTest suite with all patches applied:

nsh> ostest
...

All tests PASSED