Ticket 6084 - Migrating databases
Summary: Migrating databases
Status: RESOLVED INFOGIVEN
Alias: None
Product: Slurm
Classification: Unclassified
Component: Database (show other tickets)
Version: 18.08.0
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Nate Rini
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2018-11-26 07:53 MST by David Rhey
Modified: 2018-11-26 09:31 MST (History)
1 user (show)

See Also:
Site: University of Michigan
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description David Rhey 2018-11-26 07:53:05 MST 
Hello,

We have a situation where we might need to move the DBD entirely from one host to another, and we don't want to lose any historic data. I've found bugs 2826 and 5891, which seem to indicate that changing the StateSaveLocation shouldn't be an issue.

We are thinking that we'd shutdown the DBD, do the dump, restore the dump to the new host, update the slurm.conf to account for the new storage host location and restart the daemons.

Does this seem acceptable? Are there any caveats or things we're missing/overlooking?

Thanks, as always!

David
Comment 1 Nate Rini 2018-11-26 08:54:04 MST 
David

Will the install of Mysql/Maria stay the same version?

--Nate
Comment 2 David Rhey 2018-11-26 09:01:15 MST 
Nate,

Yes, the version of MariaDB/MySQL will remain the same. Although you bring up an interesting point that I hadn't thought of viz. upgrading databases at the package level (i.e. an RPM update of MariaDB).

Thanks!

David
Comment 3 Nate Rini 2018-11-26 09:24:15 MST 
(In reply to David Rhey from comment #2)
> Yes, the version of MariaDB/MySQL will remain the same. Although you bring
> up an interesting point that I hadn't thought of viz. upgrading databases at
> the package level (i.e. an RPM update of MariaDB).
Care needs to be taken when upgrading SQL during a migration along the lines you have planned.

(In reply to David Rhey from comment #0)
> We are thinking that we'd shutdown the DBD, do the dump, restore the dump to
> the new host, update the slurm.conf to account for the new storage host
> location and restart the daemons.
> 
> Does this seem acceptable? Are there any caveats or things we're
> missing/overlooking?
Please make sure to leave the old slurmdbd and Maria server down, but unchanged, until you are sure there are no issues. You will also want to check the slurmctld and slurmdbd logs for any warnings or errors during startup. 

Beyond that, I see no issues with your plan.

--Nate
Comment 4 David Rhey 2018-11-26 09:28:22 MST 
Thanks, Nate!

I have no plans to actually do an immediate upgrade after a migration, but it did beg the question on whether or not it could be done at some point.

I think we can call this one closed.

David
Comment 5 Nate Rini 2018-11-26 09:31:44 MST 
(In reply to David Rhey from comment #4)
> it did beg the question on whether or not it could be done at some point.
There is no reason a Mysql upgrade won't work. Just make sure to take backups in case there are any issues with the upgrade itself and avoid doing multiple upgrades/changes at once.

--Nate