Ticket 3132

There is mail program, "smail", that comes with Slurm in the "contribs/seff" directory and in the "slurm-seff" RPM that will probably do what you want. It waits for the job to complete and then includes accounting information in the email.

Moe,
thanks for the suggestion.
However the user wants to have control from within the cluster since he has a pretty complicated set of scripts to manage submission.

DV

Tim,
this issue is still open for us.
As I said in my previous reply the email route is not feasible. Is there a specific reason of this inconsistency? Is it a database issue?

Davide

Slurm does not have a mechanism for printing post-job statistics as described. I'm aware that other resource managers do have such a feature, and I've seen several similar requests on slurm-dev before. If desired I can open an enhancement request for it, but cannot say when/if this would be addressed.

There is a delay sending job statistics to the database, and the final collection time happens as the job completes. Running sstat during the epilog delays this final update to the database, so you're seeing slightly-old data.

The data for the individual steps themselves (launched through srun) will have been finalized, and I'd expect that to be usually consistent. But there is still a race condition between delivering the final result set to slurmdbd and the sstat query retrieving these values. There are a number of scalability and reliability issues that lead to this asynchronous behavior, and this asynchronous behavior will always mean that retrieving the accounting data while the job has not completed (it is not complete until the epilog finishes) may lead to inaccuracies such as what you're seeing here.

One potential alternative may be to, within the job, run the profiled portion as a separate step, and then use sstat within the batch script after that step has completed to retrieve the accounting information for that single step. You'd want to check that the step has been marked as complete, and retry until it has, to avoid similar issues. (Or you could just add a 'sleep 30' in between the srun and sstat calls and assume that's close enough.)

Tim,
this issue is still open for us.
As I said in my previous reply the email route is not feasible. Is there a specific reason of this inconsistency? Is it a database issue?

Davide

Tim,
I apologize for the noise. The message got sent twice.

Anyway, I will certainly join the club of the requestors of having the post-completion statistics option added to Slurm. That would certainly be helpful to many of our users and it would also be simpler to use.

Thanks.

Davide

One correction to my prior message - you'd want to use sacct, not sstat. sstat only works while the step is still executing.

I've opened 3207 to track that enhancement request, although there are some serious architectural hurdles to overcome to provide such a feature, and I can't say when/if we'd address that.

Perfect. Thanks Tim.

Davide

Was there anything else I can answer on this, or can I mark this one back to resolved? 

Bug 3207 will cover any potential enhancement to provide this internally; we've been discussing it internally and have a rough idea of how to implement that, but won't necessarily get to it.

If Vanderbilt would be interested in sponsoring that work that's a quick way to get it moved to the top of the list, we can take that discussion direct to email if so.

Marking closed - bug 3207 will cover the potential enhancement.