Ticket 8029

Summary:	Job data not in database after upgrade of slurm controller to 19.05.3
Product:	Slurm	Reporter:	Tony Racho <antonio-ii.racho>
Component:	Database	Assignee:	Marcin Stolarek <mstolarek>
Status:	RESOLVED FIXED	QA Contact:
Severity:	4 - Minor Issue
Priority:	---	CC:	agilmoreno, antonio-ii.racho, mstolarek
Version:	19.05.3
Hardware:	Linux
OS:	Linux
Site:	CRAY	Slinky Site:	---
Alineos Sites:	---	Atos/Eviden Sites:	---
Confidential Site:	---	Coreweave sites:	---
Cray Sites:	NIWA/WELLINGTON	DS9 clusters:	---
Google sites:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:	19.05.4
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description Tony Racho 2019-10-30 20:50:33 MDT

We have upgraded our DB to 19.05.3. (two days ago) from 18.08.7.

We upgraded our primary controller to 19.05.3 today from 18.08.7

Jobs are running fine, only that completed jobs or job data is NOT in the database when queried via sacct while the jobs that were started before the upgrade their data is in the database.

What could be the issue here? Any settings?

Comment 1 Tony Racho 2019-10-30 20:55:50 MDT

Our compute nodes are still on 18.08.7 and will be upgraded later on.

Comment 2 Tony Racho 2019-10-30 21:26:33 MDT

We've found out that this is due to the sacct binary difference in version 18 and 19.

Any workaround for that?

Comment 3 Albert Gil 2019-10-31 07:56:40 MDT

Hi Tony,

> We've found out that this is due to the sacct binary difference in version
> 18 and 19.

Do you mean that the job doesn't appear when you use sacct version 19.05, but it appears when you use sacct version 18.08, with slurmdbd always on 19.05.
Is that right?

Can you provide a job example of that?
Can you post the specific sacct command that you run and its ouputs?
Once you have a JobID(Raw) showing that behavior, can you post the its information in the DB with a command like this:

MariaDB [slurm_acct_db_1905]> select * from <CLUSTER_NAME>_job_table where id_job=<JOB_ID_RAW>;

Thanks,
Albert

Comment 8 Marcin Stolarek 2019-10-31 10:43:37 MDT

Comment on attachment 12189 [details]
initialize cond_job->db_falgs to DBFLAG_NOSET (v2)

Tony,

I think that I was able to reproduce the issue. Are you able to apply the attached patch, rebuild slurm, restart slurmdbd and check if it works as expected?

I've tested the patch, however, I have to add that it didn't pass full SchedMD review process and is not yet scheduled for release.

cheers,
Marcin

Comment 9 Tony Racho 2019-10-31 14:17:23 MDT

(In reply to Albert Gil from comment #3)
> Hi Tony,
> 
> > We've found out that this is due to the sacct binary difference in version
> > 18 and 19.
> 
> Do you mean that the job doesn't appear when you use sacct version 19.05,
> but it appears when you use sacct version 18.08, with slurmdbd always on
> 19.05.
> Is that right?
> 
> Can you provide a job example of that?
> Can you post the specific sacct command that you run and its ouputs?
> Once you have a JobID(Raw) showing that behavior, can you post the its
> information in the DB with a command like this:
> 
> MariaDB [slurm_acct_db_1905]> select * from <CLUSTER_NAME>_job_table where
> id_job=<JOB_ID_RAW>;
> 
> Thanks,
> Albert

Hi Albert:

When using sacct 18, the jobs that were ran or completed after the slurm controller upgrade to 19.05.3 could not be seen but when using sacct 19 it can see the jobs or return output.

Cheers,
Tony

Comment 10 Marcin Stolarek 2019-10-31 15:20:48 MDT

Tony,

Thanks for replay, the information confirms that the patch from comment 8 should fix the issue for you.

Can you apply and verify it in your environment?

Cheers,
Marcin

Comment 11 Marcin Stolarek 2019-11-01 05:20:19 MDT

Tony,

I'm lowering the severity of this bug report to 4 because we have both easy workaround - use the new version of sacct and the patch that should fix the compatibility issue. Do you agree?

Please let me know if need any help with the patch application.

cheers,
Marcin

Comment 12 Tony Racho 2019-11-01 05:23:02 MDT

(In reply to Marcin Stolarek from comment #11)
> Tony,
> 
> I'm lowering the severity of this bug report to 4 because we have both easy
> workaround - use the new version of sacct and the patch that should fix the
> compatibility issue. Do you agree?
> 
> Please let me know if need any help with the patch application.
> 
> cheers,
> Marcin

Hi Marcin:

Thank you. 

We upgraded all our slurmd clients to 19.05.3 to resolve the issue. 

Thanks again for your help. 

This can be closed. 

Cheers,
Tony

Comment 14 Marcin Stolarek 2019-11-14 01:41:50 MST

Tony,

The patch for compatibility issue you've seen was merged to main repository[1] and will be released as part of Slurm 19.05.4.

cheers,
Marcin


[1] https://github.com/SchedMD/slurm/commit/6158e479e4e5ae5509c7d330c4879886cc49a342