Ticket 6886

Hi Matt,

 This information is not currently stored in the database. The structure that is used to write this information to the database, or to a flat file, is a job_record (src/slurmctld/slurmctld.h) which has no support for the std_err, std_out or std_in. 

-Jason

Matt,

 After looking at this again and talking internally we think you should be able to use the jobcomp plugin, specifically elasticsearch to save this information.

https://slurm.schedmd.com/jobcompplugins.html

-Jason

Thanks.  That would be useful for the system administrators, but is it advisable to give end users access to ElasticSearch?

Matt,

>That would be useful for the system administrators, but is it advisable to give end users access to ElasticSearch?

 That question is best answered by their documentation.

https://www.elastic.co/blog/found-elasticsearch-security

 It does look like elasticsearch has some role/profiles support so that might be an option however, if you want users to access data then you would do this through some other dashboard or reporting tool. 

https://www.elastic.co/guide/en/elastic-stack-overview/current/built-in-roles.html

https://www.elastic.co/guide/en/elastic-stack-overview/current/setting-up-authentication.html

Matt, 

One other note about elasticsearch and the std_err and std_out.

If you do not specify the -e or -o then just the work_dir is shown. 

>          "queue_wait" : 1,
>          "work_dir" : "/home/jason/slurm/17.11",
>          "std_in" : "/dev/null",
>          "ntasks" : 0,

Slurm will use the work_dir as the location to put the std_err and std_out if not specified so you can image an implied location here.



>   WorkDir=/home/jason/slurm/17.11
>   StdErr=/home/jason/slurm/17.11/slurm-42.out
>   StdOut=/home/jason/slurm/17.11/slurm-42.out

However, if you do specify a -e/-o


>          "queue_wait" : 0,
>          "work_dir" : "/home/jason/slurm/17.11",
>          "std_err" : "/tmp/some.error",
>          "std_in" : "/dev/null",
>          "std_out" : "/tmp/some.out",
>          "ntasks" : 0,


I am going to close out this issue as info given.

Matt -

I'm looking at getting this included in slurmdbd in 20.02 based on further prompting through email.

One alternative I can suggest for now is to use a job_submit plugin to store this data in the AdminComment field in some fashion. That field is explicitly designed to give you - the admin - a text column in the database to store arbitrary data in. Some sites have taken to packing additional accounting detail in json format, in advance of us adding/extending the data we capture ourselves.

- Tim

(In reply to Tim Wickberg from comment #8)
> One alternative I can suggest for now is to use a job_submit plugin to store
> this data in the AdminComment field in some fashion. That field is
> explicitly designed to give you - the admin - a text column in the database
> to store arbitrary data in. Some sites have taken to packing additional
> accounting detail in json format, in advance of us adding/extending the data
> we capture ourselves.

Understand the suggestion.  Unfortunately these fields can contain replacement values (%j, %a, %A), which is not trivial to implement.

(In reply to Matt Ezell from comment #9)
> (In reply to Tim Wickberg from comment #8)

> 
> Understand the suggestion.  Unfortunately these fields can contain
> replacement values (%j, %a, %A), which is not trivial to implement.

Hi Tim,

Any chance this is getting some attention?  The last time we spoke of it, Jess/you mentioned possibly being able to get this into the 20.11 release.

Thanks,

Tony.

Tim/Jess,

Would you provide an update on the state of this issue?

Thanks,

Tony.

Tim/Jess,

Do you know if this is slated to be worked into an upcoming update?  If not, would you provide me a Statement of Work at your earliest opportunity?  

Thanks,

Tony

> Do you know if this is slated to be worked into an upcoming update?  If not,
> would you provide me a Statement of Work at your earliest opportunity?  

We'll have an SoW over to you next week.

- Tim

SOW details sent to Tony and Matt  👍