SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | Question about sreport and job timing | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Darby Vicker <darby.vicker-1> |
| Component: | Accounting | Assignee: | Stephen Kendall <stephen> |
| Status: | OPEN --- | QA Contact: | |
| Severity: | 4 - Minor Issue | ||
| Priority: | --- | ||
| Version: | 24.11.0 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | Johnson Space Center | Alineos Sites: | --- |
| Atos/Eviden Sites: | --- | Confidential Site: | --- |
| Coreweave sites: | --- | Cray Sites: | --- |
| DS9 clusters: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | ||
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
In 'sreport', your guess is correct - jobs that are running when a rollup is performed will have usage so far included in the rollup period. For example, this job was submitted shortly before the top of the hour and shows some usage in that hourly report: $ sreport cluster AccountUtilizationByUser start=14:00 end=15:00; sacct -XS now-2hour -o jobid,jobname,start,end,alloccpus -------------------------------------------------------------------------------- Cluster/Account/User Utilization 2025-03-27T14:00:00 - 2025-03-27T14:59:59 (3600 secs) Usage reported in CPU Minutes -------------------------------------------------------------------------------- Cluster Account Login Proper Name Used Energy --------- --------------- --------- --------------- -------- -------- minesofm+ root 3 0 minesofm+ main 3 0 minesofm+ main stephen ,,, 3 0 JobID JobName Start End AllocCPUS ------------ ---------- ------------------- ------------------- ---------- 399 wrap 2025-03-27T14:56:37 2025-03-27T15:05:48 1 Let me know if you have any further questions. Best regards, Stephen Very good, thanks for the info. How frequently is do these rollups get updated? I'm wondering how long after midnight we should wait to run the report for the previous day. The rollups start at the beginning of their respective time period. In this case, the daily rollups start at midnight, then will take some time to process the data. Although the amount of time it takes to complete may vary based on the amount of data and specs of the database server, I would expect it to go quite quickly (a few minutes) in most cases. Best regards, Stephen |
Hello, We are about to stand up some reporting information for our group based on output from sreport, specifically "cluster AccountUtilizationByUser". The intent is to have a process that runs in cron and updates info that other groups will consume. My initial thought would be that we would run an sreport for the previous day and log that information. However, the thought occurred to me that we have some partitions with pretty long limits - 10 days. I'm concerned that a report for the previous day would not be accurate if there are still long running jobs in the queue that started more than a day ago. I see this description in the sreport man page. sreport is used to generate reports of job usage and cluster utilization for Slurm jobs saved to the Slurm Database, slurmdbd. Report data comes from hourly, daily, and monthly rollups of accounting data that occur automatically in the background. Data will be pulled from the rollup table with the longest interval that can satisfy the requested report period. For example, a report on the time range "Start=01/01 End=03/01" will be able to pull from the monthly rollup table, while a report with "Start=01/01-03:00 End=03/01" will need to use the hourly rollup table. So this is encouraging and makes it sound like long running jobs are accounted for as they progress and not after they are finished. Can you please confirm that? In other words, will an "sreport cluster AccountUtilizationByUser" with times for the previous day be accurate? You can assume we won't run the previous day right at midnight but a couple hours later.