Ticket 510

Summary:	Cores/CPUs
Product:	Slurm	Reporter:	Ulf Markwardt <Ulf.markwardt>
Component:	Other	Assignee:	Moe Jette <jette>
Status:	RESOLVED FIXED	QA Contact:
Severity:	5 - Enhancement
Priority:	---	CC:	da
Version:	2.6.x
Hardware:	Linux
OS:	Linux
Site:	Universitat Dresden (Germany)	Slinky Site:	---
Alineos Sites:	---	Atos/Eviden Sites:	---
Confidential Site:	---	Coreweave sites:	---
Cray Sites:	---	DS9 clusters:	---
Google sites:	---	HPCnow Sites:	---
HPE Sites:	---	IBM Sites:	---
NOAA SIte:	---	NoveTech Sites:	---
Nvidia HWinf-CS Sites:	---	OCF Sites:	---
Recursion Pharma Sites:	---	SFW Sites:	---
SNIC sites:	---	Tzag Elita Sites:	---
Linux Distro:	---	Machine Name:
CLE Version:		Version Fixed:	2.6.6
Target Release:	---	DevPrio:	---
Emory-Cloud Sites:	---

Description Ulf Markwardt 2013-11-13 17:44:06 MST

Dear developers,

normally Slurm means by "CPU" a compute core (like in "--cpus-per-task"). Now I see the entity "core" in the option "--ntasks-per-core". 

The wording is a little confusing. Maybe replace "CPU" by "core" whenever possible.

Thank you
Ulf

Comment 1 Ulf Markwardt 2013-11-13 17:52:18 MST

http://slurm.schedmd.com/srun.html

Now I am comletely confused:

"For example srun -c2 --threads-per-core=1 prog may allocate two cores for the job, but if each of those cores contains two threads, the job allocation will include four CPUs. The job step allocation will then launch two threads per CPU for a total of two tasks. "

As default I think ntasks=1.

-c 2 : 2 CPUs(=cores) per task
If each core creates 2 threads I end up with 4 threads. Not with 2 tasks, 2 threads each.

Right or wrong?

Thanks
Ulf

Comment 2 Moe Jette 2013-11-14 04:35:41 MST

Normally, a "CPU" means a hyperthread rather than a core. I tried to clarify this in a new FAQ here:

https://github.com/SchedMD/slurm/commit/97f8a5016c57443ec917a16a5353421096364151

With respect to your example,
--threads-per-core=1 indicates that you want to use one thread per core
-c (or --cpus-per-task) = 2 indicates that you want 2 CPUs
Combined the two options and you get 2 threads on different cores.

If your resource allocation unit is cores, then your job would be allocated 2 full cores. If your resource allocation unit is CPUs, the job would be allocated 2 threads on different cores.

Comment 3 Moe Jette 2013-11-14 04:53:57 MST

I wanted to add that if your SelectTypeParameters=CR_Core_Memory, then while a CPU is considered a thread, resources are allocated to jobs at the level of cores. CR_CPU_Memory could be used to allocate resources at the level of threads, but then two different jobs could share a core, which is normally not desirable.

Comment 4 Ulf Markwardt 2013-11-14 05:41:53 MST

Dear Moe,

> With respect to your example,
> --threads-per-core=1 indicates that you want to use one thread per core
> -c (or --cpus-per-task) = 2 indicates that you want 2 CPUs
> Combined the two options and you get 2 threads on different cores.
> 
> If your resource allocation unit is cores, then your job would be allocated
> 2 full cores. If your resource allocation unit is CPUs, the job would be
> allocated 2 threads on different cores.

Ok, this is how I understand it. So please change the description (citation) in srun.html accordingly -> I do not get 2 TASKS.

Thank you
Ulf

Comment 5 Moe Jette 2013-11-14 05:55:44 MST

So in your case,

srun -c2 --ntasks-per-core=1 hostname

prints the hostname 4 times (i.e. 4 tasks)?

I only see 2 tasks.

Comment 6 Moe Jette 2013-11-14 06:26:46 MST

Perhaps you could attach your slurm.conf file. Perhaps your system is configured to allocate sockets rather than cores?

Comment 7 Moe Jette 2014-03-26 10:01:09 MDT

If this is still a problem, please attach your slurm.conf file and a log demonstrating the problem.

Comment 8 Ulf Markwardt 2014-03-27 01:42:18 MDT

Since Tuesday, we run Slurm 2.6.6. I see no more problems anymore. 
Thank you
Ulf

Comment 9 Moe Jette 2014-03-27 02:21:17 MDT

fixed in 2.6.6