Hello, We are testing integration with an external dkube mlaas application with our HPC cluster using slurm's rest api. Our developers have the following observation/question: After installing slumrestd and enabling auth/jwt. https://slurm.schedmd.com/jwt.html 1) The way JWT is working is different from what we originally thought. Slumrestd is able to verify only the JWTs which are generated by slurm. scontrol token username=$USER The above command returns the token, and using that in curl APIs works. But, passing DKube generated JWT does not work as slurm does not recognize it. Is there a way for us to generate the token remotely or would each user have to generate their own token and pass that back to dkube to use. This would limit us in automating our workflow and I am sure there is a better way to do this and I might just be missing something. Any guidance you can provide would be helpful. Regards, -Simran
(In reply to Simran from comment #0) > 1) The way JWT is working is different from what we originally thought. > Slumrestd is able to verify only the JWTs which are generated by slurm. > scontrol token username=$USER > The above command returns the token, and using that in curl APIs works. But, > passing DKube generated JWT does not work as slurm does not recognize it. Any JWT token passed to the slurmrestd (which is sent to slurmctld) must have the correct claims and use "HS256" signature signed with the same key as provided to slurmctld. > sun -> username > exp -> unix timestamp of expiration date > iat -> unix timestamp of creation date (optional) An easy way to verify the payload has the correct claims is put them into https://jwt.io/. Please make sure to only test tokens that are from test systems that have been revoked/invalidated.
Thanks Nate for the quick reply. (In reply to Nate Rini from comment #1) > (In reply to Simran from comment #0) > > 1) The way JWT is working is different from what we originally thought. > > Slumrestd is able to verify only the JWTs which are generated by slurm. > > scontrol token username=$USER > > The above command returns the token, and using that in curl APIs works. But, > > passing DKube generated JWT does not work as slurm does not recognize it. > > Any JWT token passed to the slurmrestd (which is sent to slurmctld) must > have the correct claims and use "HS256" signature signed with the same key > as provided to slurmctld. When you say signed with the same key as provided to slurmctld, where/how are we providing the key to slurmctld. Is this via some slurm config or via the api call somehow? > > > sun -> username > > exp -> unix timestamp of expiration date > > iat -> unix timestamp of creation date (optional) > > An easy way to verify the payload has the correct claims is put them into > https://jwt.io/. Please make sure to only test tokens that are from test > systems that have been revoked/invalidated.
(In reply to Simran from comment #2) > (In reply to Nate Rini from comment #1) > > (In reply to Simran from comment #0) > > Any JWT token passed to the slurmrestd (which is sent to slurmctld) must > > have the correct claims and use "HS256" signature signed with the same key > > as provided to slurmctld. > > When you say signed with the same key as provided to slurmctld, where/how > are we providing the key to slurmctld. Is this via some slurm config or via > the api call somehow? The key is contents of jwt_hs256.key file in your StateSaveLocation. See this for better example on setup: > https://slurm.schedmd.com/jwt.html In your case, I would expect you to copy the key from dkube instead of randomly generating one using openssl command.
Hi Nate, Due to security reasons dkube is planning on regenerating the signing key on a regular basis. Would a workflow like this be supported by slurm? How could we automate this process on slurm and ensure that the key gets regenerated. Do other customers using slurmrestd keep the same key once created or do they also use some automated process to regenerate the keys. Assuming slurmctld would need to be restarted every time a new key is generated as well. Thanks, -Simran
(In reply to Simran from comment #4) > Due to security reasons dkube is planning on regenerating the signing key on > a regular basis. Does dkube provide a method to verify a JWT key and determine which user it belongs to? > Would a workflow like this be supported by slurm? How long is a "regular basis"? > How could we automate this process on slurm and ensure that the key gets > regenerated. > Do other customers using slurmrestd keep the same key once > created or do they also use some automated process to regenerate the keys. This looks like a better use case for an authentication proxy. > Assuming slurmctld would need to be restarted every time a new key is > generated as well. Correct, but only slurmctld (and slurmdbd in 20.11 if any accounting methods are used).
Thanks Nate. They have agreed to not change the signing key so we should not have a need to update this key often. They were thinking on a quaterly basis, I think. Is there a way to get a systemd script to start slurmrestd. During our installation we can find systemd scripts for slurmctld and slurmdbd but not slurmrestd. Regards, -Simran (In reply to Nate Rini from comment #5) > (In reply to Simran from comment #4) > > Due to security reasons dkube is planning on regenerating the signing key on > > a regular basis. > Does dkube provide a method to verify a JWT key and determine which user it > belongs to? > > > Would a workflow like this be supported by slurm? > How long is a "regular basis"? > > > How could we automate this process on slurm and ensure that the key gets > > regenerated. > > Do other customers using slurmrestd keep the same key once > > created or do they also use some automated process to regenerate the keys. > > This looks like a better use case for an authentication proxy. > > > Assuming slurmctld would need to be restarted every time a new key is > > generated as well. > Correct, but only slurmctld (and slurmdbd in 20.11 if any accounting methods > are used).
(In reply to Simran from comment #6) > They have agreed to not change the signing key so we should > not have a need to update this key often. They were thinking on a quarterly > basis, I think. Understood. There is the option to setup an authentication proxy that will use pam (or the auth system provided by dkube) to avoid the need for Slurm to use the same key as dkube. But if that works for your site, then it works for us. > Is there a way to get a systemd script to start slurmrestd. An example is here: > https://github.com/naterini/slurm-gcp/commit/468d1f609dce934068b48e17945234453ff2e7f6#diff-e09e083af9035d60cde196b1095a7a0a4d54d94df6601a9c1fde3c43a3964315R646-R663 I'll have to look at getting one upstream.
(In reply to Nate Rini from comment #7) > I'll have to look at getting one upstream. Slurm 20.11 will now generate unit file: > https://github.com/SchedMD/slurm/commit/e4b0cbe3dd2deef2da70fe91638208bd684c35f4 Do you have any more questions?
(In reply to Nate Rini from comment #8) > (In reply to Nate Rini from comment #7) > > I'll have to look at getting one upstream. > Slurm 20.11 will now generate unit file: > > https://github.com/SchedMD/slurm/commit/e4b0cbe3dd2deef2da70fe91638208bd684c35f4 > > Do you have any more questions? Not at the moment Nate. Feel free to close this request and I can open a new one if something comes up. Thanks, -Simran
Closing per your response. Thanks, --Nate
