Ticket 12619

Summary:	sacctmgr archive dump/load across major version changes?
Product:	Slurm	Reporter:	Troy Baer <troy>
Component:	Documentation	Assignee:	Ben Roberts <ben>
Status:	RESOLVED FIXED	QA Contact:	Tim Wickberg <tim>
Severity:	4 - Minor Issue
Priority:	---	CC:	tdockendorf
Version:	20.11.8
Hardware:	Linux
OS:	Linux
Site:	Ohio State OSC	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:	21.08.5, 22.05.pre1	Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description Troy Baer 2021-10-06 11:14:53 MDT

Does sacctmgr archive dump/load work across major version changes?

For example, let's say that I were to run "sacctmgr archive dump jobs  PurgeJobAfter=6months" against a slurmdbd instance running 20.11.8 to reduce the database size, then updated that slurmdbd instance from 20.11.8 to 21.08.2.  Would a subsequent "sacctmgr archive load" using a dump file from before the update be expected to work?  The sacctmgr man page doesn't say (that I can find, anyway).

Comment 1 Nate Rini 2021-10-06 13:02:31 MDT

(In reply to Troy Baer from comment #0)
> Does sacctmgr archive dump/load work across major version changes?
> 
> For example, let's say that I were to run "sacctmgr archive dump jobs 
> PurgeJobAfter=6months" against a slurmdbd instance running 20.11.8 to reduce
> the database size, then updated that slurmdbd instance from 20.11.8 to
> 21.08.2.  Would a subsequent "sacctmgr archive load" using a dump file from
> before the update be expected to work?  The sacctmgr man page doesn't say
> (that I can find, anyway).

Yes, it should be able to load. Is it not working?

Comment 2 Troy Baer 2021-10-06 13:27:38 MDT

(In reply to Nate Rini from comment #1)
> Yes, it should be able to load. Is it not working?

We hadn't tried it yet, as we've just started testing and planning for 21.08.  We were looking at it as a hedge in case the DB schema changes involved in the 21.08 slurmdbd update was going to be on the order of hours with our prod database, but we've since found that the schema changes take on the order of ~7-10 minutes rather than hours.

Nonetheless, it would be helpful if the documentation made a statement that sacctmgr archive dump/load is expected to work across major versions.

Comment 3 Nate Rini 2021-10-06 13:32:53 MDT

(In reply to Troy Baer from comment #2)
> (In reply to Nate Rini from comment #1)
> > Yes, it should be able to load. Is it not working?
> 
> We hadn't tried it yet, as we've just started testing and planning for
> 21.08.  We were looking at it as a hedge in case the DB schema changes
> involved in the 21.08 slurmdbd update was going to be on the order of hours
> with our prod database, but we've since found that the schema changes take
> on the order of ~7-10 minutes rather than hours.

When in doubt, take a MySQL dump.
 
> Nonetheless, it would be helpful if the documentation made a statement that
> sacctmgr archive dump/load is expected to work across major versions.
Sure, please verify that it works.

Comment 4 Trey Dockendorf 2021-10-06 14:46:02 MDT

I did a dump on 20.11.8:

sacctmgr archive dump directory=/var/lib/slurmdbd.archive Jobs PurgeJobAfter=6 Steps PurgeStepAfter=6 Events PurgeEventAfter=6

I then upgraded to 21.08.2 and did the archive load:

find /var/lib/slurmdbd.archive -type f -exec sacctmgr -i archive load file={} \;

As best I can tell this worked without issue.

Comment 5 Nate Rini 2021-10-06 15:21:08 MDT

(In reply to Trey Dockendorf from comment #4)
> As best I can tell this worked without issue.

Great, I'll swap this to a doc bug and will update when the doc change is upstream.

Comment 11 Ben Roberts 2021-12-06 15:28:31 MST

Hi Troy,

We have checked in an update to the docs, clarifying that you can import archive files created with previous versions of Slurm.

https://github.com/SchedMD/slurm/commit/739934579d580f04d6db99e9f63cc1017044eb88

This change will be visible in the online docs with the release of 21.08.5.

Thanks,
Ben