SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | sreport historical pending jobs | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Amit Kumar <ahkumar> |
| Component: | Accounting | Assignee: | Albert Gil <albert.gil> |
| Status: | RESOLVED INFOGIVEN | QA Contact: | |
| Severity: | 4 - Minor Issue | ||
| Priority: | --- | CC: | albert.gil, felip.moll |
| Version: | 17.11.7 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | SMU | Alineos Sites: | --- |
| Atos/Eviden Sites: | --- | Confidential Site: | --- |
| Coreweave sites: | --- | Cray Sites: | --- |
| DS9 clusters: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Linux Distro: | CentOS |
| Machine Name: | M2 | CLE Version: | |
| Version Fixed: | Target Release: | --- | |
| DevPrio: | --- | Emory-Cloud Sites: | --- |
| Ticket Depends on: | 6188 | ||
| Ticket Blocks: | |||
|
Description
Amit Kumar
2018-11-19 14:47:58 MST
Hi Amit,
> I would like to be able to report pending jobs historically.
We are not sure about what do you exactly want.
What is the metric / value that you want to obtain or compute?
Could you provide us with a simple example of a workload submission and execution and the report that you would like to get from it?
Thanks,
Albert
Hi Albert, I can use sacct to get historical information with lot of details including submit time, start time and elapsed time etc. But no direct way to find out how many jobs were pending/Running in a partition during a particular time window in the past? I could us the above information from sacct to parse and compute that but no easy way to get an historical snapshopt of running/pending jobs. Is this something we can obtain from sreport? And from my digging it felt like we couldn't. Thank you, Amit Hi Albert: Wondering what is bug 6188 ?? What does it mean by it depends on #6188? Can you please give more insight? Hi Amit, (In reply to Amit Kumar from comment #4) > Hi Albert: Wondering what is bug 6188 ?? What does it mean by it depends on > #6188? Can you please give more insight? It is just an internal discussion at SchedMD about how to improve sacct documentation and features, for your case but also for others. You can just ignore it. Going back to your request: Do you know the "Reserved" and "Allocated" values of the sreport? Isn't it what you are looking for? You asked for the number of jobs in pending or running, but please note that in a time window a job could be in different states, therefore you will have duplicates. By the other side sreport tracks these different states in the time window computing and aggregating the actual amount of time that the workload (set of jobs) was waiting for resources (Reserved) and running (Allocated) in a time window. Isn't it what you want? For example, given a cluster of only 4 nodes and 2 CPUs each, if we run this workload: $ srun -N 4 -c2 sleep 240 & # Fills up the cluster for 4 minutes $ srun -N 4 -c1 sleep 60 & Then, once sreport is ready (usually after 1 hour) it should report something like: $ sreport -T CPU -t Min Cluster Utilization Format=Allocated,Reserved End=now -------------------------------------------------------------------------------- Cluster Utilization 2018-12-06T00:00:00 - 2018-12-07T15:59:59 Usage reported in TRES Minutes -------------------------------------------------------------------------------- Allocate Reserved -------- -------- 36 16 So, you've run 36 CPU-Minutes (32 the 1st and 4 the 2nd) and waited 16 CPU-minutes in pending (the 2nd job requested 4 CPUs and waited 4 minutes). Is that what you are looking for? Albert Hi Amit! Just following up here. Did the proposed idea of using the sreport output based on TRES Allocated and Reserved work for you? Regards, Albert Hi Albert, Sorry for no response. Allocated and Reserved does not tell the story I would like to tell. I would like to say that "at a certain time" X number of jobs were pending/waiting in a particular Queue and Y jobs were Running. May be I am trying to look at the historical output of squeue at a given time, which tells me exactly how many are waiting and running etc. Is there a way I can pull this out? Thank you, Amit (In reply to Amit Kumar from comment #7) > I would like to say that "at a certain time" X number of jobs were > pending/waiting in a particular Queue and Y jobs were Running. > > May be I am trying to look at the historical output of squeue at a given > time, which tells me exactly how many are waiting and running etc. Let's try to do it by example. Let's have a small and empty cluster with 4 node and 2 CPUs each, and I run this small workload: $ srun --begin=now+30 -c 2 sleep 60 & # Eligible after 30 sec $ srun --begin=now+30 -c 2 sleep 60 & $ srun --begin=now+30 -c 2 sleep 60 & $ srun --begin=now+30 -c 2 sleep 60 & $ srun --begin=now+60 -c 2 sleep 60 & # Eligible after 60 sec $ srun --begin=now+60 -c 2 sleep 60 & The timeline of the workload will consists in 6 main phases: - Phase 1: All jobs will be NonEligible because they have a begin time in the future - Phase 2: After 30 seconds the first 4 jobs will be Eligible for the scheduler becoming Pending - Phase 3: After some instants the scheduler will do its job and the 4 Eligible jobs will Start Running - Phase 4: After 30 seconds the last 2 jobs will be also Eligible but will be Pending because the cluster is full - Phase 5: After 30 seconds the first 4 jobs will End and the last 2 jobs will Start Running - Phase 6: After 60 seconds everything will be done We can see the time of these these phases quering the whole time window with sacct (-X to avoid steps info): $ sacct -X --format JobID,Submit,Eligible,Start,End JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- 57 15:57:38 15:58:08 15:58:16 15:59:16 58 15:57:40 15:58:10 15:58:16 15:59:16 59 15:57:41 15:58:11 15:58:16 15:59:16 60 15:57:43 15:58:13 15:58:16 15:59:16 61 15:57:50 15:58:50 15:59:16 16:00:16 62 15:57:51 15:58:51 15:59:16 16:00:16 Phase 1: time < 15:58:08 Phase 2: 15:58:08 < time < 15:58:16 Phase 3: 15:58:16 < time < 15:58:50 Phase 4: 15:58:50 < time < 15:59:16 Phase 5: 15:59:17 < time < 16:00:16 Phase 6: 15:58:08 < time < 15:58:16 The first thing to note is that the --starttime and --endtime of sacct will allow ass to query the state of the cluster on that time window. So, we will use the --starttime and --endtime options query the queue in each phase (or in your words: "to look historical output of squeue at a given time"). Phase 1: We get nothing because NonEligible jobs are not in any valid State. $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=15:57:38 --endtime=15:58:07 JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- Phase 2: In this phase we get 4 Pending jobs and none Running $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=15:57:38 --endtime=15:58:15 --state Pending JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- 57 15:57:38 15:58:08 15:58:16 15:59:16 58 15:57:40 15:58:10 15:58:16 15:59:16 59 15:57:41 15:58:11 15:58:16 15:59:16 60 15:57:43 15:58:13 15:58:16 15:59:16 $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=15:57:38 --endtime=15:58:15 --state Running JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- Phase 3: We have 4 Running jobs and none Pending $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=15:58:17 --endtime=15:58:48 --state Pending JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=15:58:17 --endtime=15:58:48 --state Running JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- 57 15:57:38 15:58:08 15:58:16 15:59:16 58 15:57:40 15:58:10 15:58:16 15:59:16 59 15:57:41 15:58:11 15:58:16 15:59:16 60 15:57:43 15:58:13 15:58:16 15:59:16 Phase 4: We have still the 4 Running jobs, but as the last 2 became Eligible, they appear now as Pending $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=15:58:48 --endtime=15:59:15 --state Pending JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- 61 15:57:50 15:58:50 15:59:16 16:00:16 62 15:57:51 15:58:51 15:59:16 16:00:16 $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=15:58:48 --endtime=15:59:15 --state Running JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- 57 15:57:38 15:58:08 15:58:16 15:59:16 58 15:57:40 15:58:10 15:58:16 15:59:16 59 15:57:41 15:58:11 15:58:16 15:59:16 60 15:57:43 15:58:13 15:58:16 15:59:16 Phase 5: The first 4 jobs are End, so we will only see 2 jobs Running $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=15:59:17 --endtime=16:00:16 --state Pending JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=15:59:17 --endtime=16:00:16 --state Running JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- 61 15:57:50 15:58:50 15:59:16 16:00:16 62 15:57:51 15:58:51 15:59:16 16:00:16 Phase 6: Empty queue again because all the work is done $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=16:00:17 --state Pending JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- $ sacct -X --format JobID,Submit,Eligible,Start,End --starttime=16:00:17 --state Running JobID Submit Eligible Start End ------------ ------------------- ------------------- ------------------- ------------------- Please note that the State column is not show in the example because it will be always "COMPLETED" as we are running the queries once the workload is finished. The State column shows always the current state of the job. So, if I understood correctly your question, you have to: - Use sacct, not sreport - Use the --startime and --entime options to define the (small) time window where you want to "look the historical output of squeue" - Do one query per each State that you want to list using the --state option And please not that if your time window is "big", the same job could be in two different states and will appear in different queries of sacct. Is this what you want? Albert Ops!
Sorry, the copy&paste is not always a good idea..
> Phase 1: time < 15:58:08
> Phase 2: 15:58:08 < time < 15:58:16
> Phase 3: 15:58:16 < time < 15:58:50
> Phase 4: 15:58:50 < time < 15:59:16
> Phase 5: 15:59:17 < time < 16:00:16
> Phase 6: 15:58:08 < time < 15:58:16
Phase 6 is actually: 16:00:17 > time
Hi Albert, Thank you for the details in trying to answer my question. Please allow me sometime to walk through this and get back to you. Also wanted to mention that given I will be out of town beginning tomorrow if I cannot get back to you by the end of today can you please leave this ticket open until I can get back sometime late next week? Thank you, Amit Hi Amit, > Thank you for the details in trying to answer my question. Please allow me > sometime to walk through this and get back to you. You're welcome! And sorry for long posting.. The final summary is probably just enough: >> If I understood correctly your question, you have to: >> - Use sacct, not sreport >> - Use the --startime and --entime options to define the (small) time window where you want to "look the historical output of squeue" >> - Do one query per each State that you want to list using the --state option > Also wanted to mention > that given I will be out of town beginning tomorrow if I cannot get back to > you by the end of today can you please leave this ticket open until I can > get back sometime late next week? Sure! Have a nice days! Albert Hi Albert, Before the break I tried your suggestion, but I found inconsistent results and could not get the same results as you. I am guessing this is because you have a very clean system and I have lot of data and completed jobs. I will summarize what I find in a day or two and then troubleshoot this further. Thank you, Amit Hi Amit, > Before the break I tried your suggestion, but I found inconsistent results > and could not get the same results as you. I am guessing this is because you > have a very clean system and I have lot of data and completed jobs. I will > summarize what I find in a day or two and then troubleshoot this further. We are working on some bugs regarding wrong/inconsistent values reported by sacct (eg bug 5717). If it is Ok for you, I'm closing this bug as infogiven because using sacct as proposed in comment 8 should accomplish your goal, right? We can handle these inconsistent values of sacct that you are facing in a new bug whenever you need. Is that fine? Thanks, Albert |