Hello,

For a running job we could see what exactly CPUs/GPUs/etc is allocated. E.g.
[root@DCALPH000 ~]# scontrol show -dd jobid=1360317 | grep IDs
     Nodes=dcalph134 CPU_IDs=24 Mem=0 GRES_IDX=gpu:p100(IDX:0)
[root@DCALPH000 ~]# 


We believe that  access to the same information will be useful for already completed jobs.

1) That will allow to reconstruct what really happened on cluster at any given time (time interval).
1.1) For example. If we'd like to research preemptor / preemptee relation. Let me explain: if we see overlap of the start,end pair for some job (having different PriorityTier)  and see that these jobs did share some nodes - there are still no sufficient evidences to conclude about preemption as it is not clear if there was any overlap in terms of e.g. CPU cores allocated on each of the nodes in question.
1.2) If we trying to calculate core seconds usage for each user per interval (say for each 10 minutes) we need to know exactly whether each jobs was running or suspended (preempted) inside that interval. And if job was suspended - then for how long it was preempted within this particular interval

2) For GPU we have already setup server level DCGM/prometheus/graphana dashboard. And we see now historical metrics for each GPU of any server. But job centric presentation as still impossible n as Slurm does not save allocated GPU IDs.

Of course we could still look into our own custom completion plugin(s). But we'd like to avoid any potential duplication of efforts. So even if SchedMD is not planning to look into this in a foreseeable future it would be at least interesting to understand where: slurm accounting db?/elastic search completion plugin?/somewhere else it seems logical to SchedMD to store information about exact CPU/GPU assignments.