We have a request from a user group to enable "TrackWCKey" in our cluster. Their use case is quite different from what is documented in the WCKey manual page [1], since a user would like to perform sub-accounting for each of his multiple projects rather than what is anticipated in [1]: 1. This particular group of users wants to run jobs under their multiple distinct projects, and associate some WCKey with each of their jobs using "sbatch --wckey=xxx" [2]. Here "xxx" is any user-selected key which won't require the use of "sacctmgr" for defining it. All other users of the cluster are not expected to use WCKeys at this point in time. 2. On a daily basis the users would like to use "sacct" to extract his own job records including WCKeys, and then attribute cluster usage to the various project WCKeys. We have some questions regarding the intended use of WCKeys and any necessary cluster reconfiguration: 3. Is the WCKeys usage in items 1. and 2. feasible? 4. We would NOT use AccountingStorageEnforce=wckeys in slurm.conf because this should be voluntary and there is no need for any enforcement. Currently our slurm.conf has: AccountingStorageEnforce=associations,limits,qos,safe TrackWCKey=no 5. I presume that we must set "TrackWCKey=yes" in slurm.conf as well as slurmdbd.conf and reconfigure/restart the daemons, right? 6. Are there any caveats or points of caution associated with enabling "TrackWCKey" on a running cluster? Thanks a lot, Ole [1] https://slurm.schedmd.com/wckey.html [2] https://slurm.schedmd.com/sbatch.html#OPT_wckey
Hi, Your understanding is correct. Feasibility (1–3): Yes, this works. With TrackWCKey=yes and without AccountingStorageEnforce=wckeys, users can submit sbatch --wckey=xxx with any self-chosen string. The key does not need to be pre-defined with sacctmgr. Users can then extract their own records with sacct: sacct -u <user> ... --format=JobID,WCKey,Elapsed,... or sacct -u <user> --wckeys=projA,projB --format=JobID,WCKey,Elapsed,... Enforcement (4): Right, keep AccountingStorageEnforce=associations,limits,qos,safe. Enforcement is not required for this use case. Configuration (5): Yes. Set TrackWCKey=yes in both slurm.conf and slurmdbd.conf. - slurm.conf makes WCKeys appear in sacct, so the day-to-day per-project extraction works. - slurmdbd.conf additionally populates sacctmgr list wckey and the sreport roll-up tables. After editing the config, SIGHUP or restart slurmdbd and run scontrol reconfigure. Caveats (6): - Tracking applies only to jobs that start after the change. - Jobs submitted without --wckey are recorded against the default key, shown with a leading *. - With enforcement disabled, keys are created on first use. Typos create separate keys, but case does not matter. You can audit the set of keys for typos with sacctmgr list wckey. Let me know if you have any other questions. Best, Nathan Bulloch
(In reply to Nathan Bulloch from comment #1) > Your understanding is correct. Thank you so much for confirming out interpretation of the WCKey manual page [1]! We'll implement this change when we're back from the ISC HPC conference next week. > Let me know if you have any other questions. Actually, the WCKey manual page [1] is quite difficult to interpret, since it just explains the concept as "A WCKey is an orthogonal way to do accounting against possibly unrelated accounts. This can be useful where users from different accounts are all working on the same project.". IMHO, it would seem relevant to write an expanded documentation of the wckeys concept to make it more understandable. It seems to me that a wckey is simply a tag which users may or may not attach to their jobs so they will appear in the Slurm database for accounting purposes. The manual sections on AccountingStorageEnforce and sacctmgr seem to be optional and should be moved to the bottom of the manual. The use case for AccountingStorageEnforce is not obvious to me and may need to be elaborated a bit to show its usefulness. Thanks a lot, Ole [1] https://slurm.schedmd.com/wckey.html
I'm out of the office, back on June 26. Jeg er ikke på kontoret, tilbage igen 26. juni. Best regards / Venlig hilsen, Ole Holm Nielsen
Happy to help! I'll look into clarifying the wckey documentation (tracked in 25433). Let me know if anything comes up once the changes are in place. Mvh, Nathan Bulloch
