Ticket 13518

Summary: slurmrestd GrpSubmit and MaxJobsAccrue
Product: Slurm Reporter: Matt Ezell <ezellma>
Component: slurmrestdAssignee: Nate Rini <nate>
Status: RESOLVED INFOGIVEN QA Contact:
Severity: 4 - Minor Issue    
Priority: --- CC: barlowat
Version: 21.08.5   
Hardware: Linux   
OS: Linux   
See Also: https://bugs.schedmd.com/show_bug.cgi?id=13727
Site: ORNL-OLCF Alineos Sites: ---
Atos/Eviden Sites: --- Confidential Site: ---
Coreweave sites: --- Cray Sites: ---
DS9 clusters: --- HPCnow Sites: ---
HPE Sites: --- IBM Sites: ---
NOAA SIte: --- OCF Sites: ---
Recursion Pharma Sites: --- SFW Sites: ---
SNIC sites: --- Linux Distro: ---
Machine Name: CLE Version:
Version Fixed: Target Release: ---
DevPrio: --- Emory-Cloud Sites: ---
Ticket Depends on: 12897    
Ticket Blocks:    

Description Matt Ezell 2022-02-25 13:31:42 MST 
Hi, 

This is a follow-up from https://bugs.schedmd.com/show_bug.cgi?id=13254.

Our development team encountered an issue setting “GrpSubmit” and “MaxJobsAccrue” with the QOS API endpoint with OpenAPI DB version 0.0.38 on Slurm 22.05.0-0pre1 through the master branch around 3pm EST, February 24th. Previously, we encountered the same issues in the Slurm versions 20.11.8, 21-8-4-1, and 21-8-5-1 on OpenAPI DB version 0.0.36 and 0.0.37.

I think the problem is there is no way to set the QOS fields “GrpSubmit” and “MaxJobsAccrue” through the API. Perhaps you could add the following:
{
  "QOS": [
    {
      "grp": {
          "submit": NUMBER
       },
       "max": {
           "jobs: {
              "accruing": NUMBER
           }
       }
   ]
}


Bug Reproducer

---

API Request
POST request to /slurmdb/v0.0.38/qos
Body:
{
  "QOS": [
    {
      "limits": {
        "max": {
          "active_jobs": {
            "accruing": 5,
            "count": 10
          },
          "wall_clock": {
            "per": {
              "qos": 15,
              "job": 20
            }
          },
          "jobs": {
            "per": {
              "account": 25,
              "user": 30
            }
          },
          "accruing": {
            "per": {
              "account": 35,
              "user": 40
            }
          }
        }
      },
      "name": "test1",
      "priority": 45
    }
  ]
}


Response
{
  "meta": {
    "plugin": {
      "type": "openapi/dbv0.0.38",
      "name": "Slurm OpenAPI DB v0.0.38"
    },
    "Slurm": {
      "version": {
        "major": 22,
        "micro": 0,
        "minor": 5
      },
      "release": "22.05.0-0pre1"
    }
  },
  "errors": []
}


Actual Saved Fields
[vagrant@rats-slurm ~]$ sacctmgr list qos format=GrpSubmit,MaxJobsAccrue
GrpSubmit MaxJobsAccrue
--------- -------------
                      0
        0            0
[vagrant@rats-slurm ~]$


Relevant Slurmrestd Log
-- Logs begin at Thu 2022-02-24 17:53:42 UTC. --
Feb 24 20:14:22 rats-slurm slurmrestd[16036]: debug2: _on_body: [[localhost]:45318] received 589 bytes for HTTP body length 590/589 bytes
Feb 24 20:14:22 rats-slurm slurmrestd[16036]: operations_router: [[localhost]:45318] POST /slurmdb/v0.0.38/qos
Feb 24 20:14:22 rats-slurm slurmrestd[16036]: debug:  rest_auth/local: slurm_rest_auth_p_authenticate: slurm_rest_auth_p_authenticate: [[localhost]:45318] socket authentication only supported on UNIX sockets
Feb 24 20:14:22 rats-slurm slurmrestd[16036]: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:45318] attempting user_name vagrant token authentication pass through
Feb 24 20:14:22 rats-slurm slurmrestd[16036]: debug4: _resolve_mime: [[localhost]:45318] accepts */* with q=1.000000
Feb 24 20:14:22 rats-slurm slurmrestd[16036]: debug4: _resolve_mime: [[localhost]:45318] found accepts */*=application/json with q=1.000000
Feb 24 20:14:22 rats-slurm slurmrestd[16036]: debug3: _resolve_mime: [[localhost]:45318] mime read: application/json write: application/json
Feb 24 20:14:22 rats-slurm slurmrestd[16036]: serializer/json: _try_parse: _try_parse: WARNING: Extra 1 characters after JSON string detected
Feb 24 20:14:22 rats-slurm slurmrestd[16036]: debug:  accounting_storage/slurmdbd: _connect_dbd_conn: Sent PersistInit msg
Feb 24 20:14:23 rats-slurm slurmrestd[16036]: debug4: parse_http: [[localhost]:45318] parsed 971/971 bytes
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug:  parse_http: [[localhost]:45318] Accepted HTTP connection
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug:  _on_url: [[localhost]:45318] url path: /slurmdb/v0.0.38/qos query: (null)
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: slurmrestd: operations_router: [[localhost]:45318] POST /slurmdb/v0.0.38/qos
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: slurmrestd: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:45318] attempting user_name vagrant token authentication pass through
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: slurmrestd: serializer/json: _try_parse: _try_parse: WARNING: Extra 1 characters after JSON string detected
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug2: _on_header_value: [[localhost]:45318] Header: Host Value: localhost:6250
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug2: _on_header_value: [[localhost]:45318] Header: User-Agent Value: insomnia/2021.7.2
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug2: _on_header_value: [[localhost]:45318] Header: Cookie Value: __profilin=p%3Dt
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug2: _on_header_value: [[localhost]:45318] Header: X-SLURM-USER-NAME Value: vagrant
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug2: _on_header_value: [[localhost]:45318] Header: X-SLURM-USER-TOKEN Value: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2NTA5MTE5MTksImlhdCI6MTY0NTcyNzkxOSwic3VuIjoidmFncmFudCJ9.NC0QrX0Ugic1pL-x-wYCfplvJOQTCDCVtQbUOxR0lyI
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug2: _on_header_value: [[localhost]:45318] Header: Content-Type Value: application/json
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug2: _on_header_value: [[localhost]:45318] Header: Accept Value: */*
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug2: _on_header_value: [[localhost]:45318] Header: Content-Length Value: 589
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug3: _on_headers_complete: [[localhost]:45318] HTTP/1.1 connection
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug2: _on_body: [[localhost]:45318] received 589 bytes for HTTP body length 590/589 bytes
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: operations_router: [[localhost]:45318] POST /slurmdb/v0.0.38/qos
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug:  rest_auth/local: slurm_rest_auth_p_authenticate: slurm_rest_auth_p_authenticate: [[localhost]:45318] socket authentication only supported on UNIX sockets
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: rest_auth/jwt: slurm_rest_auth_p_authenticate: [[localhost]:45318] attempting user_name vagrant token authentication pass through
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug4: _resolve_mime: [[localhost]:45318] accepts */* with q=1.000000
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug4: _resolve_mime: [[localhost]:45318] found accepts */*=application/json with q=1.000000
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug3: _resolve_mime: [[localhost]:45318] mime read: application/json write: application/json
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: serializer/json: _try_parse: _try_parse: WARNING: Extra 1 characters after JSON string detected
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug:  accounting_storage/slurmdbd: _connect_dbd_conn: Sent PersistInit msg
Feb 24 20:14:32 rats-slurm slurmrestd[16036]: debug4: parse_http: [[localhost]:45318] parsed 971/971 bytes
Comment 1 Nate Rini 2022-03-01 16:30:10 MST 
(In reply to Matt Ezell from comment #0)
> Our development team encountered an issue setting “GrpSubmit” and
> “MaxJobsAccrue” with the QOS API endpoint with OpenAPI DB version 0.0.38 on
> Slurm 22.05.0-0pre1 through the master branch around 3pm EST, February 24th.

Please feel free to just give the hash of the commit as a reference.
Comment 2 Nate Rini 2022-03-01 16:37:41 MST 
(In reply to Matt Ezell from comment #0)
> I think the problem is there is no way to set the QOS fields “GrpSubmit” and
> “MaxJobsAccrue” through the API. Perhaps you could add the following:

Please note that in bug#13195 we are going to sync all the names in dbv38 plugin to the names used in sacctmgr/sacct to make naming easier for everyone.
Comment 6 Nate Rini 2022-03-02 13:22:18 MST 
(In reply to Matt Ezell from comment #0)
> “MaxJobsAccrue” through the API

Please verify MaxJobsAccrue is the correct field desired as that is not a field of QOS but of associations currently under "max/jobs/accruing".
Comment 7 Nate Rini 2022-03-29 08:39:06 MDT 
(In reply to Nate Rini from comment #6)
> (In reply to Matt Ezell from comment #0)
> > “MaxJobsAccrue” through the API
> 
> Please verify MaxJobsAccrue is the correct field desired as that is not a
> field of QOS but of associations currently under "max/jobs/accruing".

Reducing severity while waiting for feedback.
Comment 8 Aaron Barlow 2022-03-29 11:25:03 MDT 
The hash of the commit I used on February 25th: c600cc9.

I see that MaxJobsAccrue exists for QOS since I can run "sacctmgr list qos format=MaxJobsAccrue" and get a non-null result. However, maybe that will be changing soon? I see under https://slurm.schedmd.com/resource_limits.html#qos, that MaxJobsAccruePerAccount and MaxJobsAccruePerUser are now options. I would be fine if I could set either MaxJobsAccrue or both MaxJobsAccruePer[User|Account].
Comment 11 Nate Rini 2022-03-29 11:47:36 MDT 
(In reply to Aaron Barlow from comment #8)
> The hash of the commit I used on February 25th: c600cc9.
> 
> I see that MaxJobsAccrue exists for QOS since I can run "sacctmgr list qos
> format=MaxJobsAccrue" and get a non-null result. However, maybe that will be
> changing soon? I see under
> https://slurm.schedmd.com/resource_limits.html#qos, that
> MaxJobsAccruePerAccount and MaxJobsAccruePerUser are now options. I would be
> fine if I could set either MaxJobsAccrue or both
> MaxJobsAccruePer[User|Account].

Looking at the code. When dumping MaxJobsAccrue for a QOS, that is an alias for MaxJobsAccruePerUser.  When dumping MaxJobsAccrue for an association, that is a field upon itself. 'MaxJobsAccrue' is not documented under 'SPECIFICATIONS FOR QOS' but only under 'GENERAL SPECIFICATIONS FOR ASSOCIATION BASED ENTITIES' in the sacctmgr man pages. It looks like an undocumented backward compatibility parameter was being used here when dumping 'MaxJobsAccrue' with QOS.
Comment 12 Aaron Barlow 2022-03-30 10:03:20 MDT 
For the MaxJobsAccrue part of this bug report then, adding the ability to set MaxJobsAccruePerUser and MaxJobsAccruePerAccount through the API would satisfy my need for MaxJobsAccrue to be set.
Comment 23 Nate Rini 2022-05-04 18:33:33 MDT 
(In reply to Aaron Barlow from comment #12)
> For the MaxJobsAccrue part of this bug report then, adding the ability to
> set MaxJobsAccruePerUser and MaxJobsAccruePerAccount through the API would
> satisfy my need for MaxJobsAccrue to be set.

Please note that both of these fields are added by bug#12796 to dbv0.0.38 plugin in Slurm-22.05.
> https://github.com/SchedMD/slurm/commit/de6bcf65025eb83b2f819a368bf6c07978601a53

Please give a test to make sure it meets your needs. If not, please respond to this ticket and we can continue debugging.