Hello, We are about to stand up some reporting information for our group based on output from sreport, specifically "cluster AccountUtilizationByUser". The intent is to have a process that runs in cron and updates info that other groups will consume. My initial thought would be that we would run an sreport for the previous day and log that information. However, the thought occurred to me that we have some partitions with pretty long limits - 10 days. I'm concerned that a report for the previous day would not be accurate if there are still long running jobs in the queue that started more than a day ago. I see this description in the sreport man page. sreport is used to generate reports of job usage and cluster utilization for Slurm jobs saved to the Slurm Database, slurmdbd. Report data comes from hourly, daily, and monthly rollups of accounting data that occur automatically in the background. Data will be pulled from the rollup table with the longest interval that can satisfy the requested report period. For example, a report on the time range "Start=01/01 End=03/01" will be able to pull from the monthly rollup table, while a report with "Start=01/01-03:00 End=03/01" will need to use the hourly rollup table. So this is encouraging and makes it sound like long running jobs are accounted for as they progress and not after they are finished. Can you please confirm that? In other words, will an "sreport cluster AccountUtilizationByUser" with times for the previous day be accurate? You can assume we won't run the previous day right at midnight but a couple hours later.
In 'sreport', your guess is correct - jobs that are running when a rollup is performed will have usage so far included in the rollup period. For example, this job was submitted shortly before the top of the hour and shows some usage in that hourly report: $ sreport cluster AccountUtilizationByUser start=14:00 end=15:00; sacct -XS now-2hour -o jobid,jobname,start,end,alloccpus -------------------------------------------------------------------------------- Cluster/Account/User Utilization 2025-03-27T14:00:00 - 2025-03-27T14:59:59 (3600 secs) Usage reported in CPU Minutes -------------------------------------------------------------------------------- Cluster Account Login Proper Name Used Energy --------- --------------- --------- --------------- -------- -------- minesofm+ root 3 0 minesofm+ main 3 0 minesofm+ main stephen ,,, 3 0 JobID JobName Start End AllocCPUS ------------ ---------- ------------------- ------------------- ---------- 399 wrap 2025-03-27T14:56:37 2025-03-27T15:05:48 1 Let me know if you have any further questions. Best regards, Stephen
Very good, thanks for the info. How frequently is do these rollups get updated? I'm wondering how long after midnight we should wait to run the report for the previous day.
The rollups start at the beginning of their respective time period. In this case, the daily rollups start at midnight, then will take some time to process the data. Although the amount of time it takes to complete may vary based on the amount of data and specs of the database server, I would expect it to go quite quickly (a few minutes) in most cases. Best regards, Stephen
