Ticket 5212

Hi Alessandro,

I'm looking over your stuff and am preparing an answer for you.

--Isaac

(In reply to Isaac Hartung from comment #2)
> Hi Alessandro,
> 
> I'm looking over your stuff and am preparing an answer for you.
> 
> --Isaac

Thank you Isaac,
please don't hesitate to ask any additional information that you think may prove useful.

Alessandro

Could you attach your slurm.conf?

Created attachment 6975 [details]
slurm.conf

(In reply to Isaac Hartung from comment #4)
> Could you attach your slurm.conf?

Sure, attached in the previous comment.

Seeing as your trees are all of the same depth, I can't imagine the depth_oblivious options would help as it is intended to "improve fairness between accounts in deep and/or irregular hierarchies".


setting the fairshare field on your child accounts to "parent" might interest you:

https://slurm.schedmd.com/resource_limits.html

Hi Alessandro,

Have you been able to try those changes?  Do you have any other concerns regarding fair-share?

Regards

Dear Isaac,
sorry for not having replied before.

Unfortunately, setting the child account fairshare to parent won't help us. In my first message i forgot to mention that we already use parent: this is how the regular user fairshare is defined. In our user environment, we have that more users can belong to a single project (account). Such project has its own share, and all users have to consume it with the same (local) priority, that goes down for all the users of the same project if one of them submit some jobs accounted on it. On the other hand, the single project is never related to any of the others, even if they have the same father. So the effect of the siblings in the fairshare calculation should be null.
So, we want the child account share to be the first considered, and therefore the users share is parent. This cannot be replicated for the "father account-son account" relationship, because if we do that only the father shares count, and this is not what we want (a project with many cpu hours to spend may see its jobs waiting for too long because a smaller project has consumed some of the fathers' shares).

Our idea is that SLURM fair-share system simply wasn't thought for the account priority model that we are trying to implement, so we have to force a bit our hands to find a way to make it behave as we want. In this moment we are trying to implement the second strategy i showed you in my first message, that is to assign to the fathers the sum of the shares of their sons instead of a symbolic "1", and to keep DEPTH_OBLIVIOUS so that such shares are redistributed to all accounts keeping the proportion with the effective dimension of the project itself. We are finding difficulties with writing the script that actually implements them, and our system administrators are also assigned to many other tasks so they don't have much time for this. Therefore this activity is currently slowed down and this is why I hadn't replied to you yet.

Finally, I thought of another way to rephrase our question. In your manual,  you explain the simplified Fair-Share formula which is quite individual, i.e. it takes into account only the fair-share factor, the normalized shares and usage, adn the dampening factor related to the single account. Then you explain the actual formula that adds to the mix all the parameters related fo father-son and sibling relationship, and I assume this is actually the formula used when computing the final fair-share value. So our question can be probably become: is there a way to substitute the complete fair-share formula with the simplified fair-share formula, so that parent relationships are not taken into consideration?

I will update you when we are able to implement our changes and i will report about its effects.
Thank you for reading and best regards,
Alessandro

>So our question can be probably become: is there a way to substitute the >complete fair-share formula with the simplified fair-share formula, so that >parent relationships are not taken into consideration?

No this is not possible.  Fair tree as outlined in the docs is the only fair share algorithm available.

On another note, if you are using DEPTH_OBLIVIOUS, you are already not engaging the FAIR_TREE calculations:

DEPTH_OBLIVIOUS
If set, priority will be calculated based similar to the normal multifactor calculation, but depth of the associations in the tree do not adversely effect their priority. [[[---This option precludes the use of FAIR_TREE.---]]] (emphasis added).

Hi Alessandro,

Seeing as you want to use DEPTH_OBLIVIOUS and it precludes FAIR_TREE, it seems we have addressed the issue raised by this ticket.  I am going to close this ticket, but should you have any further, related questions, please post them here and/or reopen this ticket.

Regards,

Isaac

Dear Isaac,

Sorry that I'm back to you after so much time. I too agree that this ticket can be closed, but before that I would like to show you the results of our last update, in the hope it may be interesting for you as well.

There are also some things I would like to clarify about our original question.  First of all, we know that DEPTH_OBLIVIOUS is excluding FAIR_TREE effects. If you recall this excerpt from my first correspondance:
"In our first attempt, we set up PriorityFlags=FAIR_TREE as you usually recommend. However, this resulted in an unwanted behaviour, since when an account consumed some cpu hours, we saw a drop of the fair-share value even for accounts of the same family that haven't consumed anything yet. We solved this by recurring at PriorityFlags=DEPTH_OBLIVIOUS."

After that, for me the idea of further discussing FAIR_TREE effects was completely out of question, and in fact i have never mentioned it again once. The problem for us is that, after having eradicated some of effect of the account relationship by moving to DEPTH OBLIVIOUS, there were still some unwanted effects that have to do with relationship and that we didn't want either. I refer here to the problem of the fathers with the same amount of resources that are distributed among an uneven number of sons, so that fathers with a lot of children scatter the Norm_shares quota more sparsely than father with a few children. This was happening when DEPTH_OBLIVIOUS was already activated, and FAIR_TREE was off.

So our question has never been "Is it a good idea to activate DEPTH_OBLIVIOUS instead of FAIR_TREE?", but rather "given that we want to use DEPTH_OBLIVIOUS, how can we overcome this persisting unwanted behaviour?". I admit that the question of the simplified formula was kind of a long shot, but let's say that if it existed a third flag "NO_RELATIONS" we would take it.

So I am still of the idea that I expressed in my last mail:
"Our idea is that SLURM fair-share system simply wasn't thought for the account priority model that we are trying to implement, so we have to force a bit our hands to find a way to make it behave as we want."

That's what we did. If you remember, I told you that I would have come back to you after we were able to apply some changes in the calculation of our norm_shares. I attach two text files, that are the result of the command "sshare -l --format=account,rawshares,normshares". The file oldshares.txt has been produced before the change: every father account has a share of "1", so they all have the same amount of norm_shares because they all refer directly to root. Such amount is splitten among the sons, that vary in number from father to father: so you can see that fathers with few sons assign to them much bigger values of norm_shares than fathers of many sons. Each son has also its pre-assigned amount of raw_shares, and there is at least a respect of the proportions between the siblings on the assignment of norm_shares, i.e. accounts with many raw shares will get more norm shares than accounts of the same father and with less raw shares. But that is not enough: you can take as an example the "pra16" family, that are the biggest projects in our cluster and they have more raw shares than anyone else. The biggest son, pra16_4235, has norm_shares=0.001191, while a small account but with few siblings like sigu_tarta17_0 has norm_shares=0.008891, way more than the supposedly bigger one. You also have to imagine an undisplayed third layer of hierarchy that are the users assigned to each account, all set to fairshare=parent.

The second file, newshares.txt, has been produced after the change. What we did is to implement a script that sums the raw shares of each account belonging to the same family, and assigns that number to the father, instead of the default "1". In this way there is a proportion between fathers as well as between sons, so the global proportion is respected. For example, the father pra16 has now way more shares than the father sigu, and their sons behave accordingly: now pra16_4235 has norm_shares=0.008037 and sigu_tarta17_0 has norm_shares=0.000027. So mission (supposedly) accomplished.

There is still some work to do for us on tuning the fair-share (we need to find a Dampening Factor that reduces the consume of shares for each job and makes our model resemble more like a monthly budget linearization), but at least we have solved this issue. I wanted to share this with you, because it may be interesting for you to confront with users that have this kind of necessities when dealing with fair share, and you may think in the upcoming versions of SLURM to implement a solution for this kind of situation, handled by the scheduler itself.

Thank you for all the attention and the support.
Regards,
Alessandro

Created attachment 7417 [details]
First file - old configuration

Created attachment 7418 [details]
Second file - new configuration