add check for existance of output dir for memtest #5574
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
PrometheusPi
added
component: tools
ikbuibui
previously approved these changes
Dec 9, 2025
ikbuibui
force-pushed
the
add_checkFilesystem_memtest
branch
from
dce8646 to
af1c9f3
Compare
Contributor
|
Force pushed to re-trigger the CI because the gitlab runner was stuck |
Member
|
@PrometheusPi you need to force push again, you should not assign someone to the pull requests else the CI bot can not handle the PR. |
PrometheusPi
force-pushed
the
add_checkFilesystem_memtest
branch
from
af1c9f3 to
018fd29
Compare
Contributor
|
Why don't you also print the output in the failing case? It might still be valuable information in there. |
Member
Author
|
@chillenzer good point - I will add that |
psychocoderHPC
requested changes
Dec 12, 2025
| echo "Error: $0 did not find directory: $old_path (on host: $host_name with rank: $host_rank)" >&2 | ||
| echo "error message of memtest is:" >&2 | ||
| echo -e "$output" >&2 | ||
| exit 2 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I encountered an issue on OLCF Frontier where we ran through the memtest loop, saw a defective node/GPU, but could not write out the error file. @psychocoderHPC and I suspect that the file system was not available or up-to-date on the defective node, thus preventing any useful error log.
To prevent this kind of bug in the future, this PR adds a check that writes to stderr if the directory to which to write the error log is not available.