Created attachment 667 [details] attachment-17777-0.html Currently vsize_max and similar values are tracked with a 32bit uint (in kilobytes): slurm-2.6.6-2/slurm/slurmdb.h: uint32_t vsize_max; which makes the max value that it can track to be 4TBs. That's too small for our UV (it has 12TBs) and it is easy to see that it will be a limiting factor for many more machines in future. Can you please consider changing vsize_max and other such values to uint64_t? Thanks
vsize_max is in KB (rss is as well), so I think 12TB should fit into a uint32_t. The more concerning issue would be how we read that info from proc. Could you send me a line from /proc/<pid>/stat of a process that is using all 12TB of memory (or more than 4TB of memory anyway). Danny On 03/03/14 08:39, bugs@schedmd.com wrote: > Site -Other- > Bug ID 619 <http://bugs.schedmd.com/show_bug.cgi?id=619> > Summary Increase vsize_max to uint64_6 > Product SLURM > Version 2.6.6 > Hardware Linux > OS Linux > Status UNCONFIRMED > Severity 5 - Enhancement > Priority --- > Component Accounting > Assignee david@schedmd.com > Reporter plazonic@princeton.edu > CC da@schedmd.com, david@schedmd.com, jette@schedmd.com > > Currently vsize_max and similar values are tracked with a 32bit uint (in > kilobytes): > > slurm-2.6.6-2/slurm/slurmdb.h: uint32_t vsize_max; > > which makes the max value that it can track to be 4TBs. That's too small for > our UV (it has 12TBs) and it is easy to see that it will be a limiting factor > for many more machines in future. Can you please consider changing vsize_max > and other such values to uint64_t? > > Thanks > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You are on the CC list for the bug. >
Since we are reading these in as unsigned longs (which in most systems is a uint64_t) it does make sense to change them to uint64_t. On 03/03/14 09:36, Danny Auble wrote: > vsize_max is in KB (rss is as well), so I think 12TB should fit into a > uint32_t. > > The more concerning issue would be how we read that info from proc. > > Could you send me a line from /proc/<pid>/stat of a process that is > using all 12TB of memory (or more than 4TB of memory anyway). > > Danny > > On 03/03/14 08:39, bugs@schedmd.com wrote: >> Site -Other- >> Bug ID 619 <http://bugs.schedmd.com/show_bug.cgi?id=619> >> Summary Increase vsize_max to uint64_6 >> Product SLURM >> Version 2.6.6 >> Hardware Linux >> OS Linux >> Status UNCONFIRMED >> Severity 5 - Enhancement >> Priority --- >> Component Accounting >> Assignee david@schedmd.com >> Reporter plazonic@princeton.edu >> CC da@schedmd.com, david@schedmd.com, jette@schedmd.com >> >> Currently vsize_max and similar values are tracked with a 32bit uint (in >> kilobytes): >> >> slurm-2.6.6-2/slurm/slurmdb.h: uint32_t vsize_max; >> >> which makes the max value that it can track to be 4TBs. That's too small for >> our UV (it has 12TBs) and it is easy to see that it will be a limiting factor >> for many more machines in future. Can you please consider changing vsize_max >> and other such values to uint64_t? >> >> Thanks >> ------------------------------------------------------------------------ >> You are receiving this mail because: >> >> * You are on the CC list for the bug. >> >
Created attachment 668 [details] attachment-28899-0.html
I am afraid that a 32bit in KBs is 4TBs: 2^32/1024(to convert to MB)/1024(to convert to GB)/1024(to convert to TB)=4 After all that's why we had the 3.2TB value go negative on us when it was a signed int. I cannot right now do more then look at the current state of /proc/[0-9]*/stat - machine is fully utilized. These are the entries with biggest value on spot 23: 552514 (turb) R 551177 552514 551174 0 -1 50340096 19242 421 0 0 5407400 129 0 0 20 0 1 0 120783143 2140473208832 17042 18446744073709551615 4194304 5883284 140737488290544 140737488285136 46912499178413 0 0 0 25183486 0 0 0 17 126 0 0 2 0 0 607187 (turb) R 605813 607187 605810 0 -1 50340096 19242 421 0 0 14744133 124 0 0 20 0 1 0 111430483 2140473208832 17042 18446744073709551615 4194304 5883284 140737488290544 140737488285136 4910943 0 0 0 25183486 0 0 0 17 234 0 0 1 0 0 660981 (turb) R 659606 660981 659603 0 -1 50340096 19243 421 0 0 14449968 172 0 1 20 0 1 0 111724367 2140473208832 17043 18446744073709551615 4194304 5883284 140737488290544 140737488285664 5008168 0 0 0 25183486 0 0 0 17 498 0 0 1 0 0 827422 (turb) R 826111 827422 826108 0 -1 50340096 19241 421 0 0 8762451 140 0 1 20 0 1 0 117421123 2140473208832 17041 18446744073709551615 4194304 5883284 140737488290544 140737488285136 4305539 0 0 0 25183486 0 0 0 17 954 0 0 2 0 0 852531 (turb) R 851308 852531 851306 0 -1 50340096 19243 421 0 0 8192721 109 0 0 20 0 1 0 117993240 2140473208832 17043 18446744073709551615 4194304 5883284 140737488290544 140737488285136 5011147 0 0 0 25183486 0 0 0 17 90 0 0 1 0 0
Looks like my math was bad. (Darn you google). I'll see how big of a deal this would be. When a change happens it will not be in 2.6 and might not be in 14.03. I'll let you know. On 03/03/14 10:00, bugs@schedmd.com wrote: > > *Comment # 4 <http://bugs.schedmd.com/show_bug.cgi?id=619#c4> on bug > 619 <http://bugs.schedmd.com/show_bug.cgi?id=619> from Josko Plazonic > <mailto:plazonic@princeton.edu> * > I am afraid that a 32bit in KBs is 4TBs: > > 2^32/1024(to convert to MB)/1024(to convert to GB)/1024(to convert to TB)=4 > > After all that's why we had the 3.2TB value go negative on us when it was a > signed int. > > I cannot right now do more then look at the current state of /proc/[0-9]*/stat > - machine is fully utilized. These are the entries with biggest value on spot > 23: > > 552514 (turb) R 551177 552514 551174 0 -1 50340096 19242 421 0 0 5407400 129 0 > 0 20 0 1 0 120783143 2140473208832 17042 18446744073709551615 4194304 5883284 > 140737488290544 140737488285136 46912499178413 0 0 0 25183486 0 0 0 17 126 0 0 > 2 0 0 > 607187 (turb) R 605813 607187 605810 0 -1 50340096 19242 421 0 0 14744133 124 0 > 0 20 0 1 0 111430483 2140473208832 17042 18446744073709551615 4194304 5883284 > 140737488290544 140737488285136 4910943 0 0 0 25183486 0 0 0 17 234 0 0 1 0 0 > 660981 (turb) R 659606 660981 659603 0 -1 50340096 19243 421 0 0 14449968 172 0 > 1 20 0 1 0 111724367 2140473208832 17043 18446744073709551615 4194304 5883284 > 140737488290544 140737488285664 5008168 0 0 0 25183486 0 0 0 17 498 0 0 1 0 0 > 827422 (turb) R 826111 827422 826108 0 -1 50340096 19241 421 0 0 8762451 140 0 > 1 20 0 1 0 117421123 2140473208832 17041 18446744073709551615 4194304 5883284 > 140737488290544 140737488285136 4305539 0 0 0 25183486 0 0 0 17 954 0 0 2 0 0 > 852531 (turb) R 851308 852531 851306 0 -1 50340096 19243 421 0 0 8192721 109 0 > 0 20 0 1 0 117993240 2140473208832 17043 18446744073709551615 4194304 5883284 > 140737488290544 140737488285136 5011147 0 0 0 25183486 0 0 0 17 90 0 0 1 0 0 > ------------------------------------------------------------------------ > You are receiving this mail because: > > * You are on the CC list for the bug. >
Created attachment 669 [details] attachment-25054-0.html
Easy to get lots with numbers this big :) but that's the future - new machines with Intel E7 v2 can go at least up to 6TBs and that's without having a UV kind of monster. Anyway, I assumed it was more intrusive, which is why I made it a different bugzilla entry - thanks for looking into it!
OK, I was able to get this info 14.03. The commits are here... cde5181e7fc2ba8d33a94831658e667a623c5176 f12dedb393e98a2d927b88dde8af708210173068 3e5ddedc0a4e159b93c517c55cc3605877f0d1e3 00bfa484577ae2427dc412e7711966eaca4e4eac The patch will not work with 2.6, but the good news is 14.03 will be available by the end of the month so you will not have to wait long to get it. Let me know if you see any problems with these patches.
