SchedMD - Slurm Support – Full Text Ticket Listing
- Home
- | New
- | Browse
- | Search
- | [?]
- | Reports
- | Help
- | New Account
- | Log In
- | Forgot Password
| Summary: | ntasks and ntasks-per-node are ignored when launching job using aprun -B | ||
|---|---|---|---|
| Product: | Slurm | Reporter: | Jason Coverston <jason.coverston> |
| Component: | Cray ALPS | Assignee: | Danny Auble <dauble> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | 2 - High Impact | ||
| Priority: | --- | CC: | dauble |
| Version: | 14.03.7 | ||
| Hardware: | Linux | ||
| OS: | Linux | ||
| Site: | CSCS - Swiss National Supercomputing Centre | Slinky Site: | --- |
| Alineos Sites: | --- | Atos/Eviden Sites: | --- |
| Confidential Site: | --- | Coreweave sites: | --- |
| Cray Sites: | --- | DS9 clusters: | --- |
| Google sites: | --- | HPCnow Sites: | --- |
| HPE Sites: | --- | IBM Sites: | --- |
| NOAA SIte: | --- | NoveTech Sites: | --- |
| Nvidia HWinf-CS Sites: | --- | OCF Sites: | --- |
| Recursion Pharma Sites: | --- | SFW Sites: | --- |
| SNIC sites: | --- | Tzag Elita Sites: | --- |
| Linux Distro: | --- | Machine Name: | |
| CLE Version: | Version Fixed: | 14.03.9 14.11.0rc2 | |
| Target Release: | --- | DevPrio: | --- |
| Emory-Cloud Sites: | --- | ||
| Attachments: |
attachment-11172-0.html
Logs from santis and brisi attachment-13914-0.html basil_interface.c attachment-7624-0.html attachment-17490-0.html attachment-25734-0.html |
||
Here is an example of the behavior on the XC30 system running CLE 5.2.25 with SLURM 2.5.4
"ntasks=1" is ignored unless combined with " --ntasks-per-node ". This is also true on XC30 running CLE 5.1.29 with SLURM2.5.4
#!/bin/bash -l
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=00:05:00
###export OMP_NUM_THREADS=1
echo "The current job ID is $SLURM_JOB_ID"
echo "Running on $SLURM_JOB_NUM_NODES nodes"
echo "Using $SLURM_TASKS_PER_NODE tasks per node"
echo "A total of $SLURM_NTASKS tasks is used"
aprun -B hostname
sleep 120
20:52:53 nina@santis01:$ sbatch job
Submitted batch job 1894
20:52:59 nina@santis01:$ apstat -r
ResId ApId From Arch PEs N d Memory State
37842 136996 batch:1894 XT 1 1 1 2048 NID list,conf,claim
A 37842 136997 batch:1894 XT 1 - - 2048 conf,claim
20:53:03 nina@santis01: $ apstat -rvvv
ResId ApId From Arch PEs N d Memory State
37842 136996 batch:1894 XT 1 1 1 2048 NID list,conf
Reservation detail
Res[0]: apid 136996, pagg 0x200016d02, resId 37842, user nina, NID list,
gid 30544, account 20778, time 0, normal
Batch System ID = 1894
Reservation flags = 0x140000
Created at Wed Sep 17 20:52:59 2014
Number of commands 1, control network fanout 32
Cmd[0]: BASIL -n 1 -N 1 -j 1, 2048MB, XT, nodes 1, exclusive
Reservation list entries: 1
PE 0, cmd 0, nid 14, CPU 0x1, map 0x1, accels 0
Customer heavily uses aprun -B. I have logs that I can pass if needed. Thanks! Jason Jason, send logs and slurm.conf if you can. I know cscs had a heavily modified 2.5.4. I am wondering if they didn't send us a patch. Were the results you sent done with vanilla slurm? Did you run the tests or just report what they reported? On September 22, 2014 12:22:19 AM PDT, bugs@schedmd.com wrote: >http://bugs.schedmd.com/show_bug.cgi?id=1115 > >--- Comment #2 from Jason Coverston <jcovers@cray.com> --- >Customer heavily uses aprun -B. > >I have logs that I can pass if needed. > >Thanks! > >Jason > >-- >You are receiving this mail because: >You are on the CC list for the bug. Hi Danny, Below is the conf file for both systems. brisi is the name of the new tds machine running 14.03.7. This issue was discovered by the onsite Cray person. Logs will follow soon. Cheers, Jason 09:55:18 nina@santis01:~ $ scontrol show config Configuration data as of 09:55:49 AccountingStorageBackupHost = fojorina02 AccountingStorageEnforce = associations,limits AccountingStorageHost = fojorina01 AccountingStorageLoc = N/A AccountingStoragePort = 6819 AccountingStorageType = accounting_storage/slurmdbd AccountingStorageUser = N/A AccountingStoreJobComment = YES AcctGatherEnergyType = acct_gather_energy/none AcctGatherNodeFreq = 0 sec AuthType = auth/munge BackupAddr = santis01 BackupController = santis01 BatchStartTimeout = 10 sec BOOT_TIME = 8 Sep 14:41 CacheGroups = 0 CheckpointType = checkpoint/none ClusterName = santis CompleteWait = 0 sec ControlAddr = sdb ControlMachine = sdb CryptoType = crypto/munge DebugFlags = Priority DefMemPerNode = UNLIMITED DisableRootJobs = NO EnforcePartLimits = NO Epilog = /opt/cray/ccm/default/etc/slurm.epilog.frontend EpilogMsgTime = 2000 usec EpilogSlurmctld = /opt/slurm/default/etc/epilogslurmctld.sh FastSchedule = 2 FirstJobId = 1 GetEnvTimeout = 2 sec GresTypes = gpu,gpu_mem GroupUpdateForce = 0 GroupUpdateTime = 600 sec HASH_VAL = Match HealthCheckInterval = 0 sec HealthCheckProgram = (null) InactiveLimit = 1800 sec JobAcctGatherFrequency = 30 sec JobAcctGatherType = jobacct_gather/none JobCheckpointDir = /var/slurm/checkpoint JobCompHost = localhost JobCompLoc = /var/log/slurm_jobcomp.log JobCompPort = 0 JobCompType = jobcomp/none JobCompUser = root JobCredentialPrivateKey = (null) JobCredentialPublicCertificate = (null) JobFileAppend = 0 JobRequeue = 1 JobSubmitPlugins = def_partition,get_group_prio,lua KillOnBadExit = 0 KillWait = 30 sec LaunchType = launch/aprun Licenses = (null) LicensesUsed = (null) MailProg = /usr/bin/mail MaxJobCount = 20000 MaxJobId = 4294901760 MaxMemPerNode = 32768 MaxStepCount = 40000 MaxTasksPerNode = 128 MessageTimeout = 60 sec MinJobAge = 10 sec MpiDefault = none MpiParams = (null) NEXT_JOB_ID = 1895 OverTimeLimit = 0 min PluginDir = /opt/slurm/2.5.4/lib/slurm PlugStackConfig = /opt/slurm/default/etc/plugstack.conf PreemptMode = OFF PreemptType = preempt/none PriorityDecayHalfLife = 7-00:00:00 PriorityCalcPeriod = 00:05:00 PriorityFavorSmall = 0 PriorityFlags = 0 PriorityMaxAge = 14-00:00:00 PriorityUsageResetPeriod = MONTHLY PriorityType = priority/multifactor PriorityWeightAge = 2000 PriorityWeightFairShare = 0 PriorityWeightJobSize = 10000 PriorityWeightPartition = 30000 PriorityWeightQOS = 0 PrivateData = none ProctrackType = proctrack/sgi_job Prolog = /opt/cray/ccm/default/etc/slurm.prolog.frontend PrologSlurmctld = /opt/slurm/default/etc/prologslurmctld.sh PrologFlags = Alloc PropagatePrioProcess = 0 PropagateResourceLimits = NONE PropagateResourceLimitsExcept = (null) RebootProgram = (null) ReconfigFlags = (null) ResumeProgram = (null) ResumeRate = 300 nodes/min ResumeTimeout = 60 sec ResvOverRun = 0 min ReturnToService = 1 SallocDefaultCommand = (null) SchedulerParameters = (null) SchedulerPort = 7321 SchedulerRootFilter = 1 SchedulerTimeSlice = 30 sec SchedulerType = sched/backfill SelectType = select/cray SelectTypeParameters = CR_MEMORY,CR_ONE_TASK_PER_CORE SlurmUser = root(0) SlurmctldDebug = info SlurmctldLogFile = /ufs/slurm/log/slurmctld.log SlurmSchedLogFile = (null) SlurmctldPort = 6817 SlurmctldTimeout = 120 sec SlurmdDebug = info SlurmdLogFile = /ufs/slurm/log/%h.log SlurmdPidFile = /var/run/slurmd.pid SlurmdPort = 6818 SlurmdSpoolDir = /ufs/slurm/spool/%n SlurmdTimeout = 300 sec SlurmdUser = root(0) SlurmSchedLogLevel = 0 SlurmctldPidFile = /var/run/slurmctld.pid SLURM_CONF = /opt/slurm/default/etc/slurm.conf SLURM_VERSION = 2.5.4 SrunEpilog = (null) SrunProlog = (null) StateSaveLocation = /ufs/slurm/spool/state SuspendExcNodes = (null) SuspendExcParts = (null) SuspendProgram = (null) SuspendRate = 60 nodes/min SuspendTime = NONE SuspendTimeout = 30 sec SwitchType = switch/none TaskEpilog = (null) TaskPlugin = task/none TaskPluginParam = (null type) TaskProlog = /opt/slurm/default/etc/TaskProlog.sh TmpFS = /tmp TopologyPlugin = topology/none TrackWCKey = 0 TreeWidth = 50 UsePam = 0 UnkillableStepProgram = (null) UnkillableStepTimeout = 60 sec VSizeFactor = 0 percent WaitTime = 0 sec Slurmctld(primary/backup) at sdb/santis01 are UP/UP 09:55:49 nina@santis01:~ $ 10:12:45 nina@brisi01:~ $ scontrol show config Configuration data as of 10:12:58 AccountingStorageBackupHost = (null) AccountingStorageEnforce = none AccountingStorageHost = localhost AccountingStorageLoc = /var/log/slurm_jobacct.log AccountingStoragePort = 0 AccountingStorageType = accounting_storage/none AccountingStorageUser = root AccountingStoreJobComment = YES AcctGatherEnergyType = acct_gather_energy/none AcctGatherFilesystemType = acct_gather_filesystem/none AcctGatherInfinibandType = acct_gather_infiniband/none AcctGatherNodeFreq = 0 sec AcctGatherProfileType = acct_gather_profile/none AuthInfo = (null) AuthType = auth/munge BackupAddr = brisi01 BackupController = brisi01 BatchStartTimeout = 10 sec BOOT_TIME = 26 Aug 15:52 CacheGroups = 0 CheckpointType = checkpoint/none ClusterName = brisi CompleteWait = 0 sec ControlAddr = sdb-p2 ControlMachine = sdb-p2 CoreSpecPlugin = core_spec/none CryptoType = crypto/munge DebugFlags = Priority DefMemPerNode = 65536 DisableRootJobs = NO DynAllocPort = 0 EnforcePartLimits = NO Epilog = /opt/cray/ccm/default/etc/slurm.epilog.frontend EpilogMsgTime = 2000 usec EpilogSlurmctld = /opt/slurm/default/etc/epilogslurmctld.sh ExtSensorsType = ext_sensors/none ExtSensorsFreq = 0 sec FairShareDampeningFactor = 1 FastSchedule = 2 FirstJobId = 1 GetEnvTimeout = 2 sec GresTypes = (null) GroupUpdateForce = 0 GroupUpdateTime = 600 sec HASH_VAL = Match HealthCheckInterval = 0 sec HealthCheckNodeState = ANY HealthCheckProgram = (null) InactiveLimit = 1800 sec JobAcctGatherFrequency = 30 JobAcctGatherType = jobacct_gather/none JobAcctGatherParams = (null) JobCheckpointDir = /var/slurm/checkpoint JobCompHost = localhost JobCompLoc = /var/log/slurm_jobcomp.log JobCompPort = 0 JobCompType = jobcomp/none JobCompUser = root JobContainerPlugin = job_container/none JobCredentialPrivateKey = (null) JobCredentialPublicCertificate = (null) JobFileAppend = 0 JobRequeue = 1 JobSubmitPlugins = (null) KeepAliveTime = SYSTEM_DEFAULT KillOnBadExit = 0 KillWait = 30 sec LaunchType = launch/aprun Licenses = (null) LicensesUsed = (null) MailProg = /usr/bin/mail MaxArraySize = 1001 MaxJobCount = 20000 MaxJobId = 4294901760 MaxMemPerNode = 65536 MaxStepCount = 40000 MaxTasksPerNode = 128 MessageTimeout = 60 sec MinJobAge = 10 sec MpiDefault = none MpiParams = (null) NEXT_JOB_ID = 997 OverTimeLimit = 0 min PluginDir = /opt/slurm/14.03.7/lib/slurm PlugStackConfig = /opt/slurm/14.03.7/etc/plugstack.conf PreemptMode = OFF PreemptType = preempt/none PriorityDecayHalfLife = 7-00:00:00 PriorityCalcPeriod = 00:05:00 PriorityFavorSmall = 0 PriorityFlags = 0 PriorityMaxAge = 14-00:00:00 PriorityUsageResetPeriod = MONTHLY PriorityType = priority/multifactor PriorityWeightAge = 2000 PriorityWeightFairShare = 0 PriorityWeightJobSize = 10000 PriorityWeightPartition = 30000 PriorityWeightQOS = 0 PrivateData = none ProctrackType = proctrack/sgi_job Prolog = /opt/cray/ccm/default/etc/slurm.prolog.frontend PrologSlurmctld = /opt/slurm/default/etc/prologslurmctld.sh PrologFlags = Alloc PropagatePrioProcess = 0 PropagateResourceLimits = NONE PropagateResourceLimitsExcept = (null) RebootProgram = (null) ReconfigFlags = (null) ResumeProgram = (null) ResumeRate = 300 nodes/min ResumeTimeout = 60 sec ResvEpilog = (null) ResvOverRun = 0 min ResvProlog = (null) ReturnToService = 1 SallocDefaultCommand = (null) SchedulerParameters = (null) SchedulerPort = 7321 SchedulerRootFilter = 1 SchedulerTimeSlice = 30 sec SchedulerType = sched/backfill SelectType = select/alps SelectTypeParameters = CR_CORE_MEMORY,CR_ONE_TASK_PER_CORE SlurmUser = root(0) SlurmctldDebug = debug SlurmctldLogFile = /ufs/slurm/log/slurmctld.log SlurmSchedLogFile = (null) SlurmctldPort = 6817 SlurmctldTimeout = 120 sec SlurmdDebug = debug SlurmdLogFile = /ufs/slurm/log/%h.log SlurmdPidFile = /var/run/slurmd.pid SlurmdPlugstack = (null) SlurmdPort = 6818 SlurmdSpoolDir = /ufs/slurm/spool/%n SlurmdTimeout = 300 sec SlurmdUser = root(0) SlurmSchedLogLevel = 0 SlurmctldPidFile = /var/run/slurmctld.pid SlurmctldPlugstack = (null) SLURM_CONF = /opt/slurm/14.03.7/etc/slurm.conf SLURM_VERSION = 14.03.7 SrunEpilog = (null) SrunProlog = (null) StateSaveLocation = /ufs/slurm/spool/state SuspendExcNodes = (null) SuspendExcParts = (null) SuspendProgram = (null) SuspendRate = 60 nodes/min SuspendTime = NONE SuspendTimeout = 30 sec SwitchType = switch/none TaskEpilog = (null) TaskPlugin = task/none TaskPluginParam = (null type) TaskProlog = /opt/slurm/default/etc/TaskProlog.sh TmpFS = /tmp TopologyPlugin = topology/none TrackWCKey = 0 TreeWidth = 50 UsePam = 0 UnkillableStepProgram = (null) UnkillableStepTimeout = 60 sec VSizeFactor = 0 percent WaitTime = 0 sec Slurmctld(primary/backup) at sdb-p2/brisi01 are UP/UP I can confirm the allocation is different from 2.5 to 14.03. The difference appears to originate in 2.6 (commit d7060246b363e6fa27607f040eb888bbfd896f58). I will point out it appears if the user used srun instead of aprun, or specifies the PE count on aprun things would work correctly. It appears the change was needed since the change allocates the entire node instead of just the 1 PE. This would make sense since the whole node is being charged to the user. In my testing it appears in 2.5 I got only 1 PE with or without --ntasks-per-node=1. My guess was the change made it possible for the user to rightfully gain access to all the PEs in the node since the user was being charged for them. You will notice if you do an scontrol show job the job in 2.5 has been allocated all the PEs on the node displayed as NumCPUs in the scontrol output. But in this case the user can at most only run on 1 PE unless -n is stated in the allocation line. It appears in 2.6 this was "fixed" to give access to all the PEs on the node to correctly match what was allocated. This appears to be a change in behavior, but I am not sure this is a bug as the user can still get the previous behavior by requesting it or using srun. A question is would it be possible to use srun instead of aprun? Or explicitly put -N1 -n1 on your aprun line? If srun is ran with the --launch-cmd option it will give the correct aprun line to use. If there is reasonable justification why what was changed in 2.6 is incorrect we can revert the change, but from my point of view 2.5 contained a bug that was fixed. They appear to only be benefiting from what I feel is the incorrect behavior. Let me know what you think. Created attachment 1252 [details]
Logs from santis and brisi
ping Sorry Danny, Am back in the office now. I'll check to see if I have an answer from the site yet. Cheers, Jason (In reply to Danny Auble from comment #7) > ping Hi Danny, The customer is beginning to use the new version more and more. Prepping for the upgrade. They use aprun -B heavily. Just got this from the customer: URGENT request from site ( customer): I think that we would like to have the same functionality we had before, so aprun -B must match as much as possible the slurm parameter provided with the sbatch script. ...they are used to use aprun -B in ordere to reflect slurm's parameter on the allocation and not rewrite everytime in a different notation (aprun) the desired layout. Jason, I don't think going forward we will revert this change, as we feel the previous behavior represents a bug instead of the other way around. If the customer is unable to use normal Slurm submission methods (srun) they can locally carry a revert of the commit from comment 5. This obviously leads to an undesirable scenario as support becomes harder to do since the release they run will no longer be standard. And the user would need to carry a separate patch around and patch when they install. This will also lead to inconsistency when/if they begin to use srun to submit jobs with potential unknown/inconsistant results. I'll look at this more and see if there is another option, but what the code does now is correct as it allocates the user the amount of resources they are being charged. This is the way other plugins in Slurm work that allocate whole nodes. The previous behavior only allocated a fraction of what the user in reality was being charged. We feel this is incorrect with regards to accounting and to what the user indeed was allocated. Hi Danny, I'm waiting for comment from the customer. In the meantime: - Regarding the current behavior. We have so many sites that use the machines differently. Some force ntasks per node on every user to equal the amount of each compute node. A user can then shape their app with aprun as they like. I tend to favor this approach. - With the new "resFullNode" feature in ALPS the user is allocated the entire node even if they request less than available. This is really for systems with hyperthreading. But it seems to match the current Slurm behavior. I'm asking if the customer has this set. - Some of the other batch systems will allocate less than available if the user requests it. Not saying you need to be like the others at all. Just an fyi. In the meantime a side conversation. Isn't the way a user is charged up to the customer? As we know on Cray systems the entire node is "allocated" even though a user may only request 1 task. However, couldn't it be up to the customer to choose to only charge them for the tasks they request versus the resources that are wasted? (In reply to Jason Coverston from comment #11) > Hi Danny, > > I'm waiting for comment from the customer. My hope is they will start to transition away from aprun and use srun. This is the future and they gain step accounting with srun. I know in 2.5 srun was a wrapper that they didn't like, it is not that way anymore. Perhaps they are unaware of that. > > In the meantime: > > - Regarding the current behavior. We have so many sites that use the > machines differently. Some force ntasks per node on every user to equal the > amount of each compute node. A user can then shape their app with aprun as > they like. I tend to favor this approach. I agree, this is how they do it on other non-cray linear systems, grab the node and run on part of it. If they really want the old method I am thinking of putting in a SelectTypeParam that will allow it, but not sure how things will work with srun or such since we viewed the old method as a bug. > > - With the new "resFullNode" feature in ALPS the user is allocated the > entire node even if they request less than available. This is really for > systems with hyperthreading. But it seems to match the current Slurm > behavior. I'm asking if the customer has this set. I doubt this would matter to Slurm since unless you are running native mode you can only run 1 job per node. Since the select/linear plugin is what is used underneath the select/alps this is what is configured, it allocates the whole node. With Native mode they can get what they are looking for as they can layer on top of select/cons_res instead of linear. > > - Some of the other batch systems will allocate less than available if the > user requests it. Not saying you need to be like the others at all. Just an > fyi. We could do this obviously (since 2.5 was done this way) :). We just didn't see it as "correct". > > In the meantime a side conversation. > > Isn't the way a user is charged up to the customer? As we know on Cray > systems the entire node is "allocated" even though a user may only request 1 > task. However, couldn't it be up to the customer to choose to only charge > them for the tasks they request versus the resources that are wasted? Slurm accounting is always for the resources a user has been allocated whether they use them or not. Anyone can allocate a few nodes with salloc and just sit on them. Slurm doesn't care if they actually run on the resources or not. I know some people track how much power is being used during the job and tie that to total cost of a job as well. Hi Danny, Waiting for the customer to digest the info. Thanks! Jason Hi Danny, I spoke with the customer today. This is their top priority BUG. We discussed moving their user community to either aprun with full options or srun but they feel this would be a very difficult change. Unfortunately we did not catch this early enough for them to give their users sufficient notification and to prepare examples on how they could migrate to the new more full option set. All their current user community documents contain aprun -B examples. The customer is requesting the change be reverted to the older behavior which will allow a user to request a subset of the total node's resources. My thoughts would be to make this configurable to not break customers that are now used to the current behavior. But first, I need some clarification: How would using srun make this work? The ALPS reservation would still be rounded up correct? They cannot just replace aprun -B with srun? Some more thoughts from the customer: Agreed that the scheduler allocate the entire node resource to user and that' s what they' re being charged for. If user choose to run his/her job on few cores ( via -ntask and ntasks-per-node ) then we should honor the request as well. Often in time MPI jobs is prone to be killed due to Out Of Memory ( OOM ) if all cores are allocated. Thanks for the info Jason. In the examples you showed srun could be a drop in replacement for aprun -B. I am guessing in almost all there cases where aprun only has the -B option you could replace it directly with srun and it would work as they would want. I understand this is asking the user to do something which isn't always the easiest thing, but it seems a pretty small change. I am sure not all the aprun in their scripts only contain the -B option so it might not be that easy to switch. How things work is this, srun knows what the environment is and talks with the slurmctld to do the right thing. Aprun directly doesn't do this, it's not Slurm so why would it :)? If it did read in env variables and or talk to the slurmctld aprun could be "fixed" to do the correct Slurm thing as well, but I doubt that is an attractive option as well ;). Since aprun is not Slurm it is relying on whatever was setup for it and then goes in runs in the exact allocation. When running srun it creates an aprun that does the correct thing. Without srun you would need the user to do this magic. What we tried to do with srun was to make things easier so you only have to learn one thing when going between systems. When using srun then the comments from the user are valid, since -ntask and ntasks-per-node are Slurm ideas aprun doesn't know anything about them, but srun does and does the correct thing. I can see what I can do to add a SelectTypeParameter to allow this to work the way they want, but we would not treat this as normal since we don't feel running aprun directly is normal just like on any other system we don't feel running mpirun or mpiexec would be considered normal. The patch will be against 14.11 but will probably be able to be backported to 14.03. If the customer isn't able to change scripts would this work for the customer? Jason could you request the customers 2.5 src/plugins/cray/basil_interface.c file? Just so I get what it looks like there as things have changed most likely from the vanilla version of the code? Hi Danny, Unfortunately the customer is still requesting a patch for this issue. We have tried using srun and observed some undesirable behavior when using srun with options. They have wrapped aprun which isn't playing nice. And the wrapper cannot be removed. The matching Cray BUG's severity now has been raised to critical. It has been flagged as impacting the acceptance. I'm afraid I need to ask for a patch for this issue. Jason I would be interested in the undesirable behaviour is. If I can get their 2.5 file I requested I can get a patch for them. On October 8, 2014 12:49:51 PM PDT, bugs@schedmd.com wrote: >http://bugs.schedmd.com/show_bug.cgi?id=1115 > >--- Comment #17 from Jason Coverston <jcovers@cray.com> --- >Hi Danny, > >Unfortunately the customer is still requesting a patch for this issue. > >We have tried using srun and observed some undesirable behavior when >using srun >with options. They have wrapped aprun which isn't playing nice. And the >wrapper >cannot be removed. > >The matching Cray BUG's severity now has been raised to critical. It >has been >flagged as impacting the acceptance. > >I'm afraid I need to ask for a patch for this issue. > >Jason > >-- >You are receiving this mail because: >You are on the CC list for the bug. >You are the assignee for the bug. Created attachment 1301 [details]
attachment-13914-0.html
(In reply to Danny Auble from comment #18) > I would be interested in the undesirable behaviour is. > > If I can get their 2.5 file I requested I can get a patch for them. Thanks Danny. Sorry I haven't gotten it already. I will try to get it shortly. There was only one example they gave me about an srun requesting 0 memory (this was a test for an unrelated ALPS bug) which failed. I'll try to get more examples for undesirable behavior. Thanks, jason Created attachment 1304 [details]
basil_interface.c
Attached is the basil_interface.c
I was able to get this into 14.03 with a new entry to the cray.conf file SubAllocate=yes After putting this into the cray.conf and restarting the slurmctld they will get the previous behavior. The commit is https://github.com/SchedMD/slurm/commit/a9dc50d4183427947301023b92104bd250a66bbc Please reopen if they see any abnormalities. It should be stressed to them to at least start planning to switch to srun, since that is what we support. I am not sure what they mean by "the wrapper can not be removed" in Comment 17. Do you happen to know? Any more info on what they don't like about the srun solution would be greatly appreciated. Perhaps open new bugs since we would be happy to fix srun if a problem indeed does exist. Thanks Danny! Verified this works on snake-p1. Still waiting on info regarding the wrapper. Will pass along the suggestion of beginning to plan for the eventual move to srun. Thanks again!! Jason Danny, Can you provide an explanation as to why "salloc -N 1" will allocate 1 task with the patch, versus the entire node? There seems to be a little confusion at the customer site with this particular salloc command using the -N nodes option only. Apparently some people were used to running this and getting the entire node. I was beginning to explain that it's because you have not specified any other options, like ntasks or ntasks-per-node. And that Slurm is then making the assumption in this case that -N essentially means tasks. But I wanted to check with you first as to the reason why, and perhaps you can explain it better than me! Thanks, Jason Your explanation is correct. In Slurm if you request nodes you have to at least have 1 task per node. Since they don't mention any thing like task count in the allocation Slurm will default to 1 task per node. If you do >salloc -N3 -n2 salloc: Warning: can't run 2 processes on 3 nodes, setting nnodes to 2 salloc: Granted job allocation 65543 it makes it a little clearer on this point. With the salloc -N1 the entire node will get allocated, but it defaults to only one task per node. With the new option it will only tell ALPS to reserve 1 pe per node. In all cases normal/ALPS Slurm land using select/linear the user will be allocated the entire node, only with this patch they will only be able to run on what they originally requested. With select/linear --ntasks-per-node on the allocation just uses that to allocate the correct number of nodes and set the default for sruns running that don't specify otherwise. In normal Slurm, and without this new option, the user can easily override this by using --ntasks-per-node with any srun inside the allocation to do whatever they want as long as it works in the allocation given. With this new option they are stuck with at most with what they originally asked for even though the whole node is allocated to them in Slurm. In example, on a normal select/linear system (running on a node with 8 cores) this is how it works... >salloc -N1 >srun hostname snowflake >srun -n2 hostname snowflake snowflake >srun -n8 hostname snowflake snowflake snowflake snowflake snowflake snowflake snowflake snowflake Without the new option the Cray/ALPS system will work the same way, but with the option you will get this... >salloc -N1 >srun hostname snowflake >srun -n2 hostname srun: error: Unable to create job step: Requested node configuration is not available >srun -n8 hostname srun: error: Unable to create job step: Requested node configuration is not available In both cases you can only run on the most allocated, but with the new option ALPS is told to only reserve a sub portion of the allocated node(s) so at most you will only be able to use that instead of what is being charged to the user. This only comes as an issue if the person ever wants to do different types of steps than was originally requested. With this customer it doesn't appear this ever happens. If they ever worked on a different cluster running select/linear I could see how they were used to the normal behavior that is described above. Thanks Danny. They want Slurm to assume the highest ntasks per node if not specified. To me this would work on a system which has the same nodes per system. But on a system like snake-p1 with different sized nodes this would be difficult. I would suggest they write a job submit plugin that does that. We try not to assume what the user wants in most cases. Like you said, on a heterogeneous system or could get very confusing. The user could always request this on the command line so they always hey what they want. Is this the same customer as before? On October 11, 2014 8:55:44 AM PDT, bugs@schedmd.com wrote: >http://bugs.schedmd.com/show_bug.cgi?id=1115 > >--- Comment #26 from Jason Coverston <jcovers@cray.com> --- >Thanks Danny. > >They want Slurm to assume the highest ntasks per node if not specified. >To me >this would work on a system which has the same nodes per system. But on >a >system like snake-p1 with different sized nodes this would be >difficult. > >-- >You are receiving this mail because: >You are on the CC list for the bug. >You are the assignee for the bug. Created attachment 1318 [details]
attachment-7624-0.html
(In reply to Danny Auble from comment #27) > I would suggest they write a job submit plugin that does that. We try not > to assume what the user wants in most cases. Like you said, on a > heterogeneous system or could get very confusing. The user could always > request this on the command line so they always hey what they want. Is this > the same customer as before? yes, same customer. They just had me open http://bugs.schedmd.com/show_bug.cgi?id=1159 That bug seems to desire the method this bug didn't like. On October 11, 2014 10:02:57 AM PDT, bugs@schedmd.com wrote: >http://bugs.schedmd.com/show_bug.cgi?id=1115 > >--- Comment #29 from Jason Coverston <jcovers@cray.com> --- >(In reply to Danny Auble from comment #27) >> I would suggest they write a job submit plugin that does that. We >try not >> to assume what the user wants in most cases. Like you said, on a >> heterogeneous system or could get very confusing. The user could >always >> request this on the command line so they always hey what they want. >Is this >> the same customer as before? > >yes, same customer. They just had me open >http://bugs.schedmd.com/show_bug.cgi?id=1159 > >-- >You are receiving this mail because: >You are on the CC list for the bug. >You are the assignee for the bug. Created attachment 1320 [details]
attachment-17490-0.html
(In reply to Danny Auble from comment #30) > That bug seems to desire the method this bug didn't like. I am in agreement with you. And getting more confused every time I go back and forth. I will send a mail and ask them to clearly explain what they want. I agree. You know they could get both behaviours without the option and with srun :-). On October 11, 2014 10:12:44 AM PDT, bugs@schedmd.com wrote: >http://bugs.schedmd.com/show_bug.cgi?id=1115 > >--- Comment #32 from Jason Coverston <jcovers@cray.com> --- >(In reply to Danny Auble from comment #30) >> That bug seems to desire the method this bug didn't like. > >I am in agreement with you. And getting more confused every time I go >back and >forth. > >I will send a mail and ask them to clearly explain what they want. > >-- >You are receiving this mail because: >You are on the CC list for the bug. >You are the assignee for the bug. Created attachment 1321 [details]
attachment-25734-0.html
|
Created attachment 1251 [details] attachment-11172-0.html It' s a common practice that users launch their job using aprun -B. On our Haswell TDS running CLE5.2.25 and SLURM 14.03.7 it just discovered that "ntasks=1" has never been enforced even if it' s combined with " ntasks-per-node=1". 1.) CLE 5.2.25 with SLURM 14.03.7 Jobscript: #!/bin/bash -l #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --ntasks-per-node=1 aprun -B hostname Output: 20:35:50 nina@brisi01: $ sbatch job Submitted batch job 972 20:36:01 nina@brisi01:$ apstat -r ResId ApId From Arch PEs N d Memory State 38020 134843 batch:972 XT 24 24 1 2730 NID list,conf,claim A 38020 134844 batch:972 XT 24 - - 2730 conf,claim 20:36:05 nina@brisi01:$ apstat -rvvvv ResId ApId From Arch PEs N d Memory State 38020 134843 batch:972 XT 24 24 1 2730 NID list,conf Reservation detail Res[0]: apid 134843, pagg 0x1d00018adf, resId 38020, user nina, NID list, gid 30544, account 20778, time 0, normal Batch System ID = 972 Reservation flags = 0x940000 Created at Wed Sep 17 20:36:02 2014 Number of commands 1, control network fanout 32 Cmd[0]: BASIL -n 24 -N 24 -j 1, 2730MB, XT, nodes 1, exclusive Reservation list entries: 24 PE 0, cmd 0, nid 58, CPU 0x1, map 0x3, accels 0 PE 1, cmd 0, nid 58, CPU 0x2, map 0x3, accels 0 PE 2, cmd 0, nid 58, CPU 0x4, map 0x3, accels 0 PE 3, cmd 0, nid 58, CPU 0x8, map 0x3, accels 0 PE 4, cmd 0, nid 58, CPU 0x10, map 0x3, accels 0 PE 5, cmd 0, nid 58, CPU 0x20, map 0x3, accels 0 PE 6, cmd 0, nid 58, CPU 0x40, map 0x3, accels 0 PE 7, cmd 0, nid 58, CPU 0x80, map 0x3, accels 0 PE 8, cmd 0, nid 58, CPU 0x100, map 0x3, accels 0 PE 9, cmd 0, nid 58, CPU 0x200, map 0x3, accels 0 PE 10, cmd 0, nid 58, CPU 0x400, map 0x3, accels 0 PE 11, cmd 0, nid 58, CPU 0x800, map 0x3, accels 0 PE 12, cmd 0, nid 58, CPU 0x1000, map 0x3, accels 0 PE 13, cmd 0, nid 58, CPU 0x2000, map 0x3, accels 0 PE 14, cmd 0, nid 58, CPU 0x4000, map 0x3, accels 0 PE 15, cmd 0, nid 58, CPU 0x8000, map 0x3, accels 0 PE 16, cmd 0, nid 58, CPU 0x10000, map 0x3, accels 0 PE 17, cmd 0, nid 58, CPU 0x20000, map 0x3, accels 0 PE 18, cmd 0, nid 58, CPU 0x40000, map 0x3, accels 0 PE 19, cmd 0, nid 58, CPU 0x80000, map 0x3, accels 0 PE 20, cmd 0, nid 58, CPU 0x100000, map 0x3, accels 0 PE 21, cmd 0, nid 58, CPU 0x200000, map 0x3, accels 0 PE 22, cmd 0, nid 58, CPU 0x400000, map 0x3, accels 0 PE 23, cmd 0, nid 58, CPU 0x800000, map 0x3, accels 0