Hi Ole,

The job script table is targeted by the 'PurgeJobAfter' field (same for the job env table if enabled). I will note that when enabled, the script table is expected to become particularly large, so you may want to set that purge interval shorter than you otherwise would to keep the database size manageable.

Let me know if you have any further questions.

Best regards,
Stephen

Hi Stephen,

(In reply to Stephen Kendall from comment #1)
> The job script table is targeted by the 'PurgeJobAfter' field (same for the
> job env table if enabled). I will note that when enabled, the script table
> is expected to become particularly large, so you may want to set that purge
> interval shorter than you otherwise would to keep the database size
> manageable.

To clarify my request, I believe that I'm asking for a different type purge functionality.  Our policy is to *never* purge any of the job information, so that our database still contains accounting information of all jobs back to the "beginning of time" a number of years ago.

What I'm asking for is, if we were to configure job_script archival, how to purge *only* the <clustername>_job_script_table in the database.  For example, 7 days worth of scripts for the purpose of user support, while never purging any jobs at all from the database for the purpose of accounting.

Do you think this will become feasible at some point along the roadmap?

Thanks,
Ole

I see, thanks for the clarification. I'm not aware of any prior discussion of this idea, and it would need to be done in a way that doesn't cause issues for other sites who are already using the existing behavior. I will check internally on this.

Best regards,
Stephen

(In reply to Stephen Kendall from comment #3)
> I see, thanks for the clarification. I'm not aware of any prior discussion
> of this idea, and it would need to be done in a way that doesn't cause
> issues for other sites who are already using the existing behavior. I will
> check internally on this.

Thanks for checking.  If purging this table is not going to be implemented, would my second questionin comment 0 be feasible?

If such purging parameters are not included in the current roadmaps, is there a safe way to clear the <clustername>_job_script_table in the database at regular intervals?

Thanks,
Ole

(In reply to Stephen Kendall from comment #3)
> I see, thanks for the clarification. I'm not aware of any prior discussion
> of this idea, and it would need to be done in a way that doesn't cause
> issues for other sites who are already using the existing behavior. I will
> check internally on this.

Actually, IMHO, introducing a new "PurgeJobScript" parameter in slurmdbd.conf shouldn't break anyone's current setup because it would be an optional parameter.

Thanks,
Ole

So far we are looking further into this for inclusion on a future version, but we don't have it on the roadmap for a specific release. One of the complications is that currently, the job script purges don't directly look at the time, but instead look at what jobs have already been purged and remove any scripts that were associated with those jobs. Thus adding script-only purges would require the addition of a separate purge path.

(In reply to Ole.H.Nielsen@fysik.dtu.dk from comment #4)
> If such purging parameters are not included in the current roadmaps, is
> there a safe way to clear the <clustername>_job_script_table in the database
> at regular intervals?

In general, we do not recommend any manual interventions in the accounting tables as it is easy to mistakenly delete the wrong rows or make other changes to the table that would cause errors or inconsistent results from 'slurmdbd'. If such interventions are taken, they should always be done with 'slurmdbd' stopped and after a full backup of the database.

I was also able to find an alternate method of saving scripts that might be more practical, especially since this enhancement would be at least several months out from release. Using a 'job_submit' script, it would be possible to dump each submitted job script to a file on the controller. At that point you can easily prune old ones as you see fit. However, this does have the disadvantage of occurring before a job ID is assigned to the job, so it makes it more of a challenge to match up with jobs visible on the controller or in the accounting database.
https://slurm.schedmd.com/job_submit_plugins.html

In any case, I would suggest enabling 'AccountingStoreFlags=job_script' temporarily to determine how quickly the table fills up. You can determine the on disk size of the database files in the configured datadir ('/var/lib/mysql/' on most installations). You may find that the scripts generated by your environment are practical to retain for a long time as the storage demands are highly workload-dependent based on the number of jobs and length of the scripts. With close monitoring, you will be able to estimate how quickly the table will grow in size well before it will slow down upgrades or other tasks. And if you find that the size looks manageable, you can leave it enabled long-term.

Best regards,
Stephen

Hi Stephen,

Thanks a lot for the explanation of the complexity of purging job scripts based on job completion time.  IMHO, this would be the preferred functionality, and I hope the developers will consider adding a new "PurgeJobScript" parameter in slurmdbd.conf in the not too distant future.  

Probably most sites don't want the job scripts in the databases after a few weeks, when no more user support requests about specific jobids could be anticipated.  On the other hand, job accounting requires jobs to be stored for long times (probably years; we store job accounting records back to day zero).

I appreciate the warnings about database manipulation, and I will avoid doing that.

Probably we will refrain from enabling 'AccountingStoreFlags=job_script' at all, because those scripts would remain in the database forever.  I realize that it's a very useful functionality, but retaining a lean and efficient database, also across major updates, will most likely have priority at our site.

Best regards,
Ole

Hi Ole,

We have checked in enhancements to add specific purge options for script and env. In 'slurmdbd.conf', these take the form of 'PurgeJobScriptAfter' and 'PurgeJobEnvAfter' and the corresponding Archive boolean fields. This was split into several different commits, here's the one containing that added the parsing for the config options:
https://github.com/SchedMD/slurm/commit/204d1f112549ebc1a37ff8465788fbffff6ad4e3

We are still fine tuning the related documentation but all the config parsing and handling in the code is checked in ahead of 26.05. Let us know if you have any further questions.

Best regards,
Stephen

Hi Stephen,

(In reply to Stephen Kendall from comment #10)
> We have checked in enhancements to add specific purge options for script and
> env. In 'slurmdbd.conf', these take the form of 'PurgeJobScriptAfter' and
> 'PurgeJobEnvAfter' and the corresponding Archive boolean fields. This was
> split into several different commits, here's the one containing that added
> the parsing for the config options:
> https://github.com/SchedMD/slurm/commit/
> 204d1f112549ebc1a37ff8465788fbffff6ad4e3
> 
> We are still fine tuning the related documentation but all the config
> parsing and handling in the code is checked in ahead of 26.05. Let us know
> if you have any further questions.

Thanks very much for making these enhancements!  We appreciate very much the upcoming  'PurgeJobScriptAfter' and
'PurgeJobEnvAfter' parameters for slurmdbd.conf, since this will enable us to safely use the AccountingStoreFlags=job_script flag 
without our database growing indefinitely!  I trust that many other sites are going to find this feature very useful!

A bit of warning for those sites enabling the new purge parameters is to introduce them *very* gradually as described in my Wiki page [1],
for example:

PurgeJobScriptAfter=2000days
PurgeJobEnvAfter=2000days

and lower the values little by little over time.

Thanks,
Ole

[1] https://wiki.fysik.dtu.dk/Niflheim_system/Slurm_database/#setting-database-purge-parameters