Ticket 8029 - Job data not in database after upgrade of slurm controller to 19.05.3
Summary: Job data not in database after upgrade of slurm controller to 19.05.3
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: Database (show other tickets)
Version: 19.05.3
Hardware: Linux Linux
: 4 - Minor Issue
Assignee: Marcin Stolarek
QA Contact:
URL:
Depends on:
Blocks:
 
Reported: 2019-10-30 20:50 MDT by Tony Racho
Modified: 2019-11-14 01:41 MST (History)
3 users (show)

See Also:
Site: CRAY
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: NIWA/WELLINGTON
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 19.05.4
Target Release: ---
DevPrio: ---
Emory-Cloud Sites: ---

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this ticket.
Description Tony Racho 2019-10-30 20:50:33 MDT 
We have upgraded our DB to 19.05.3. (two days ago) from 18.08.7.

We upgraded our primary controller to 19.05.3 today from 18.08.7

Jobs are running fine, only that completed jobs or job data is NOT in the database when queried via sacct while the jobs that were started before the upgrade their data is in the database.

What could be the issue here? Any settings?
Comment 1 Tony Racho 2019-10-30 20:55:50 MDT 
Our compute nodes are still on 18.08.7 and will be upgraded later on.
Comment 2 Tony Racho 2019-10-30 21:26:33 MDT 
We've found out that this is due to the sacct binary difference in version 18 and 19.

Any workaround for that?
Comment 3 Albert Gil 2019-10-31 07:56:40 MDT 
Hi Tony,

> We've found out that this is due to the sacct binary difference in version
> 18 and 19.

Do you mean that the job doesn't appear when you use sacct version 19.05, but it appears when you use sacct version 18.08, with slurmdbd always on 19.05.
Is that right?

Can you provide a job example of that?
Can you post the specific sacct command that you run and its ouputs?
Once you have a JobID(Raw) showing that behavior, can you post the its information in the DB with a command like this:

MariaDB [slurm_acct_db_1905]> select * from <CLUSTER_NAME>_job_table where id_job=<JOB_ID_RAW>;

Thanks,
Albert
Comment 8 Marcin Stolarek 2019-10-31 10:43:37 MDT 
Comment on attachment 12189 [details]
initialize cond_job->db_falgs to DBFLAG_NOSET (v2)

Tony,

I think that I was able to reproduce the issue. Are you able to apply the attached patch, rebuild slurm, restart slurmdbd and check if it works as expected?

I've tested the patch, however, I have to add that it didn't pass full SchedMD review process and is not yet scheduled for release.

cheers,
Marcin
Comment 9 Tony Racho 2019-10-31 14:17:23 MDT 
(In reply to Albert Gil from comment #3)
> Hi Tony,
> 
> > We've found out that this is due to the sacct binary difference in version
> > 18 and 19.
> 
> Do you mean that the job doesn't appear when you use sacct version 19.05,
> but it appears when you use sacct version 18.08, with slurmdbd always on
> 19.05.
> Is that right?
> 
> Can you provide a job example of that?
> Can you post the specific sacct command that you run and its ouputs?
> Once you have a JobID(Raw) showing that behavior, can you post the its
> information in the DB with a command like this:
> 
> MariaDB [slurm_acct_db_1905]> select * from <CLUSTER_NAME>_job_table where
> id_job=<JOB_ID_RAW>;
> 
> Thanks,
> Albert

Hi Albert:

When using sacct 18, the jobs that were ran or completed after the slurm controller upgrade to 19.05.3 could not be seen but when using sacct 19 it can see the jobs or return output.

Cheers,
Tony
Comment 10 Marcin Stolarek 2019-10-31 15:20:48 MDT 
Tony,

Thanks for replay, the information confirms that the patch from comment 8 should fix the issue for you.

Can you apply and verify it in your environment?

Cheers,
Marcin
Comment 11 Marcin Stolarek 2019-11-01 05:20:19 MDT 
Tony,

I'm lowering the severity of this bug report to 4 because we have both easy workaround - use the new version of sacct and the patch that should fix the compatibility issue. Do you agree?

Please let me know if need any help with the patch application.

cheers,
Marcin
Comment 12 Tony Racho 2019-11-01 05:23:02 MDT 
(In reply to Marcin Stolarek from comment #11)
> Tony,
> 
> I'm lowering the severity of this bug report to 4 because we have both easy
> workaround - use the new version of sacct and the patch that should fix the
> compatibility issue. Do you agree?
> 
> Please let me know if need any help with the patch application.
> 
> cheers,
> Marcin

Hi Marcin:

Thank you. 

We upgraded all our slurmd clients to 19.05.3 to resolve the issue. 

Thanks again for your help. 

This can be closed. 

Cheers,
Tony
Comment 14 Marcin Stolarek 2019-11-14 01:41:50 MST 
Tony,

The patch for compatibility issue you've seen was merged to main repository[1] and will be released as part of Slurm 19.05.4.

cheers,
Marcin


[1] https://github.com/SchedMD/slurm/commit/6158e479e4e5ae5509c7d330c4879886cc49a342