Is it possible to find a job's StdOut and StdErr paths after it has completed and been purged from the controller (in sacct, for example)? Is the data even saved in accounting?
Hi Matt, This information is not currently stored in the database. The structure that is used to write this information to the database, or to a flat file, is a job_record (src/slurmctld/slurmctld.h) which has no support for the std_err, std_out or std_in. -Jason
Matt, After looking at this again and talking internally we think you should be able to use the jobcomp plugin, specifically elasticsearch to save this information. https://slurm.schedmd.com/jobcompplugins.html -Jason
Thanks. That would be useful for the system administrators, but is it advisable to give end users access to ElasticSearch?
Matt, >That would be useful for the system administrators, but is it advisable to give end users access to ElasticSearch? That question is best answered by their documentation. https://www.elastic.co/blog/found-elasticsearch-security It does look like elasticsearch has some role/profiles support so that might be an option however, if you want users to access data then you would do this through some other dashboard or reporting tool. https://www.elastic.co/guide/en/elastic-stack-overview/current/built-in-roles.html https://www.elastic.co/guide/en/elastic-stack-overview/current/setting-up-authentication.html
Matt, One other note about elasticsearch and the std_err and std_out. If you do not specify the -e or -o then just the work_dir is shown. > "queue_wait" : 1, > "work_dir" : "/home/jason/slurm/17.11", > "std_in" : "/dev/null", > "ntasks" : 0, Slurm will use the work_dir as the location to put the std_err and std_out if not specified so you can image an implied location here. > WorkDir=/home/jason/slurm/17.11 > StdErr=/home/jason/slurm/17.11/slurm-42.out > StdOut=/home/jason/slurm/17.11/slurm-42.out However, if you do specify a -e/-o > "queue_wait" : 0, > "work_dir" : "/home/jason/slurm/17.11", > "std_err" : "/tmp/some.error", > "std_in" : "/dev/null", > "std_out" : "/tmp/some.out", > "ntasks" : 0, I am going to close out this issue as info given.
Matt - I'm looking at getting this included in slurmdbd in 20.02 based on further prompting through email. One alternative I can suggest for now is to use a job_submit plugin to store this data in the AdminComment field in some fashion. That field is explicitly designed to give you - the admin - a text column in the database to store arbitrary data in. Some sites have taken to packing additional accounting detail in json format, in advance of us adding/extending the data we capture ourselves. - Tim
(In reply to Tim Wickberg from comment #8) > One alternative I can suggest for now is to use a job_submit plugin to store > this data in the AdminComment field in some fashion. That field is > explicitly designed to give you - the admin - a text column in the database > to store arbitrary data in. Some sites have taken to packing additional > accounting detail in json format, in advance of us adding/extending the data > we capture ourselves. Understand the suggestion. Unfortunately these fields can contain replacement values (%j, %a, %A), which is not trivial to implement.
(In reply to Matt Ezell from comment #9) > (In reply to Tim Wickberg from comment #8) > > Understand the suggestion. Unfortunately these fields can contain > replacement values (%j, %a, %A), which is not trivial to implement. Hi Tim, Any chance this is getting some attention? The last time we spoke of it, Jess/you mentioned possibly being able to get this into the 20.11 release. Thanks, Tony.
Tim/Jess, Would you provide an update on the state of this issue? Thanks, Tony.
Tim/Jess, Do you know if this is slated to be worked into an upcoming update? If not, would you provide me a Statement of Work at your earliest opportunity? Thanks, Tony
> Do you know if this is slated to be worked into an upcoming update? If not, > would you provide me a Statement of Work at your earliest opportunity? We'll have an SoW over to you next week. - Tim
SOW details sent to Tony and Matt 👍
