We'd like to be able to change the account on jobs that have completed. Use cases are: (1) the user chose the wrong account when running the job; (2) something went wrong in the allocations process and the user was associated with an account they shouldn't have been associated with, and they ran a job with that account; (3) change in account naming scheme, e.g., we replace some test account(s) used early on with a set of production accounts, and we want jobs ran early on some test account to now be attached to one of the production accounts If this is already possible and I'm missing something, please point me in the right direction. If it's not possible, is this functionality that could be enabled?
BTW, it's fine and, in fact, desirable if this capability is restricted to Admin or Admin+Operator roles.
Hello Jake, I can ask around the team on your behalf in regards to this question, as the use case seems reasonable enough. Hopefully, there is something that already exists that allows for the changing of account for completed jobs that is sanctioned as a valid process. Best Regards, Tyler Connel
Hi Jake, I did have a discussion with the team about this and there is not presently a mechanism for changing account for jobs post-facto. There was, previously, a ticket involving changing WCKey post-facto, but doing the same for account would produce additional complications. In particular, changing account for previous jobs would not immediately affect fairshare or limit calculations. From what I'm told, sreport and sacct would be affected by changes to account for post-facto jobs. I've looped in @wickberg now to see what next steps should be regarding this ticket. Best, Tyler Connel
Hi Jake, There was a fairly detailed discussion on this ticket between the team and there were some remaining questions about what would be desired here. I'll do my best to provide a synopsis of the discussion: There are many associations throughout the system with respect to an account value. Changing the account value associated for a job system-wide (with respect to the sytemctld daemon, etc.) is something that would be quite difficult. Otherwise, changing an account value in the accounting database and having that change reflect in subsequent calls to client applications (e.g. sreport) would be possible. In the former case, re-calculations of fairshare and/or limit algorithms would be required, but would be next to impossible having lapsed the original context in which the job was run. In the latter, only historical information would be affected and the system would be able to continue running otherwise normally. Would you be able to determine whether your use case fits into the first or the second scenario? Best Regards, Tyler Connel
It seems that our use cases generally fit into the second scenario because oftentimes (esp. for any significant number of jobs that we'd like to update) they are identified quite a bit after the jobs have completed, at which point historical usage calculations on associations have been allowed to decay somewhat or a lot. So I think we can live w/ the understanding that a procedure to change accounts on completed jobs would only affect entries in the DB and would NOT affect things in slurmctld's domain. (It would matter to us to be able to adjust historical usage on QOS, because we use that method for applying limits related to funding deposits. E.g., if a job was originally charged to account_a and we adjust the DB so that it's not charged to account_b, we'd like to be able to refund its charge on a QOS associated with account_a and add the charge onto a QOS associated with account_b. But that would be covered by our request to allows setting RawUsage on a QOS to a non-zero value in issue 17549.) Thanks!
@Jake, I have confidence based on my prior conversations with the team that we could support changing account for historical data. I'd have to get a final confirmation now that we've nailed down the particular use case. As for the nature of the change, what sort of interface would you like for changing account association for particular jobs? I imagine additional options to sacctmgr might make the most sense. Best Regards, Tyler Connel
Sorry for the delayed response. sacctmgr does sound like a reasonable way to make the change. I don't think the particular syntax matters to us, just the functionality. At a minimum we'd need to be able to change the account on a job by its job ID, e.g.: sacctmgr modify job where jobid=11111 set account=new_account It'd be nice to able to bulk change all jobs submitted with some old account to the new, e.g.: sacctmgr modify job where account=old_account set account=new_account (looking at the suggested functionality in 17549 as an example) Also nice (but increasingly niche) to be able to target jobs for a given user, e.g.: sacctmgr modify job where user=dude set account=new_account Or both user and account: sacctmgr modify job where user=dude account=old_account set account=new_account Thanks!
@Jake, Thank you for the detail in your last response. I spoke to my team and this is something that we would be able to support, but it would be considered as an enhancement. You should receive a SOW (statement of work) from a representative shortly. Feel free to reach out if you have any additional questions thereafter. Best Regards, Tyler Connel
