Ticket 6084

Summary:	Migrating databases
Product:	Slurm	Reporter:	David Rhey <drhey>
Component:	Database	Assignee:	Nate Rini <nate>
Status:	RESOLVED INFOGIVEN	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	nate
Version:	18.08.0
Hardware:	Linux
OS:	Linux
Site:	University of Michigan	Alineos Sites:	---
Atos/Eviden Sites:	---	Confidential Site:	---
Coreweave sites:	---	Cray Sites:	---
DS9 clusters:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Linux Distro:	---
Machine Name:		CLE Version:
Version Fixed:		Target Release:	---
DevPrio:	---	Emory-Cloud Sites:	---

Description David Rhey 2018-11-26 07:53:05 MST

Hello,

We have a situation where we might need to move the DBD entirely from one host to another, and we don't want to lose any historic data. I've found bugs 2826 and 5891, which seem to indicate that changing the StateSaveLocation shouldn't be an issue.

We are thinking that we'd shutdown the DBD, do the dump, restore the dump to the new host, update the slurm.conf to account for the new storage host location and restart the daemons.

Does this seem acceptable? Are there any caveats or things we're missing/overlooking?

Thanks, as always!

David

Comment 1 Nate Rini 2018-11-26 08:54:04 MST

David

Will the install of Mysql/Maria stay the same version?

--Nate

Comment 2 David Rhey 2018-11-26 09:01:15 MST

Nate,

Yes, the version of MariaDB/MySQL will remain the same. Although you bring up an interesting point that I hadn't thought of viz. upgrading databases at the package level (i.e. an RPM update of MariaDB).

Thanks!

David

Comment 3 Nate Rini 2018-11-26 09:24:15 MST

(In reply to David Rhey from comment #2)
> Yes, the version of MariaDB/MySQL will remain the same. Although you bring
> up an interesting point that I hadn't thought of viz. upgrading databases at
> the package level (i.e. an RPM update of MariaDB).
Care needs to be taken when upgrading SQL during a migration along the lines you have planned.

(In reply to David Rhey from comment #0)
> We are thinking that we'd shutdown the DBD, do the dump, restore the dump to
> the new host, update the slurm.conf to account for the new storage host
> location and restart the daemons.
> 
> Does this seem acceptable? Are there any caveats or things we're
> missing/overlooking?
Please make sure to leave the old slurmdbd and Maria server down, but unchanged, until you are sure there are no issues. You will also want to check the slurmctld and slurmdbd logs for any warnings or errors during startup. 

Beyond that, I see no issues with your plan.

--Nate

Comment 4 David Rhey 2018-11-26 09:28:22 MST

Thanks, Nate!

I have no plans to actually do an immediate upgrade after a migration, but it did beg the question on whether or not it could be done at some point.

I think we can call this one closed.

David

Comment 5 Nate Rini 2018-11-26 09:31:44 MST

(In reply to David Rhey from comment #4)
> it did beg the question on whether or not it could be done at some point.
There is no reason a Mysql upgrade won't work. Just make sure to take backups in case there are any issues with the upgrade itself and avoid doing multiple upgrades/changes at once.

--Nate