Ticket 3103 - Further SLURM/pmix communication improvements
Summary: Further SLURM/pmix communication improvements
Status: RESOLVED DUPLICATE of ticket 3909
Alias: None
Product: Slurm
Classification: Unclassified
Component: Contributions (show other tickets)
Version: 16.05.4
Hardware: Linux Linux
: 5 - Enhancement
Assignee: Danny Auble
QA Contact:
URL:
: 3625 (view as ticket list)
Depends on:
Blocks:
 
Reported: 2016-09-20 10:15 MDT by Artem Polyakov
Modified: 2017-08-04 10:56 MDT (History)
0 users

See Also:
Site: -Other-
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed:
Target Release: 17.11
DevPrio: ---
Emory-Cloud Sites: ---


Attachments
POC code (1.26 KB, patch)
2016-09-20 10:15 MDT, Artem Polyakov
Details | Diff
Set of patches improving communication performance (37.54 KB, application/x-bzip)
2017-03-27 13:20 MDT, Artem Polyakov
Details
One more patch to update copyrights of all affected files (20.26 KB, patch)
2017-04-22 13:50 MDT, Artem Polyakov
Details | Diff
One more patch to update copyrights of all affected files (v2) (20.23 KB, patch)
2017-04-26 15:10 MDT, Artem Polyakov
Details | Diff
Fix EIO shutdown (2.42 KB, patch)
2017-04-27 13:58 MDT, Artem Polyakov
Details | Diff
Cleanup debug bugs (1.24 KB, patch)
2017-04-27 13:59 MDT, Artem Polyakov
Details | Diff
Fix EIO shutdown (2.66 KB, patch)
2017-04-27 14:36 MDT, Artem Polyakov
Details | Diff
Cleanup debug stuff (1.24 KB, patch)
2017-04-27 14:36 MDT, Artem Polyakov
Details | Diff
The full set of patches in one tar (39.95 KB, application/x-bzip)
2017-04-27 14:53 MDT, Artem Polyakov
Details
Full set of patches v3 (39.94 KB, application/x-bzip)
2017-04-27 14:56 MDT, Artem Polyakov
Details

Note You need to log in before you can comment on or make changes to this ticket.
Description Artem Polyakov 2016-09-20 10:15:32 MDT
Created attachment 3521 [details]
POC code

Hello,
I'm opening this bug report to discuss the problem that I'm seeing with SLURM/pmix performance.
When I'm comparing it to OMPI performance I see that SLURM/pmix has serious overhead compared to OMPI implementation.

The root cause is that SLURM is creating threads and in case of loaded node those threads are not necessarily will be immediately scheduled. This causes high latencies.

One of the steps to fix that is the bug https://bugs.schedmd.com/show_bug.cgi?id=3102 but it covers only the sender.

slurmd is the receiver that will forward data to some slurmstepd. With the attached hacky patch I can get comparable performance (to OMPI implementation).
 
What I'm about to suggest is to enable another thread in slurmd that will handle connection requests immediately. It will sit on the another well-known port and we will have some API to acces this one.

All the current activity will continue to flow through the common path but there will be an option to reduce the latency.
Comment 1 Artem Polyakov 2016-09-20 23:22:34 MDT
We are OK with doing implementation, but I want to sync on the architecture.

I'll provide the draft soon.
Comment 5 Danny Auble 2016-12-14 10:17:52 MST
Artem,

I'm interested in your idea to see if it has merit.  But it isn't clear to me if it will give much.  I'm interested to see what you come up with though.  Perhaps you should look at the persistent connection code in 17.02.  That might reduce the amount of info needing processing.  If you are going to make a separate thread to handle these it makes sense to make it persistent as that would help with performance.
Comment 6 Artem Polyakov 2016-12-14 12:05:28 MST
Hi, Danny.

Here is some update on this:

A. I'm actually now working on the new incarnation of this idea. I want to use slurm's communication primitives to initiate connections: sender port # will be in the slurm proto message when it's a first time you communicate with the peer. Once the message is received - a direct persistent tcp connection will be established between 2 stepd's and response and all other communication will continue through it.

B. We can still look for the option of a separate thread that processes messages immediately for this approach. It can be used to forward short messages that can be read quickly and then there will be no forward delay.

So my plan is the following:
1. Implement the adea of persistent stepd-s connection and see how it will work.
2. If I will still get a performance hit for the first initiation message sent through slurm proto - I will implement a proof of a concept to see if (B) will help.
Comment 7 Danny Auble 2016-12-14 12:19:07 MST
This seems like a good path forward, just make sure you use the already existing persistent connection code already in Slurm if possible.  Not sure how well this will work in practice though, but I'm interested in what you come up with.
Comment 8 Artem Polyakov 2016-12-14 13:52:18 MST
Can you point me on the API?.
Comment 9 Danny Auble 2016-12-14 14:24:17 MST
Look at the way the slurmdbd or the new fed_mgr works on the slurmctld.
Comment 10 Artem Polyakov 2016-12-14 14:25:35 MST
ok, thank you.
Comment 11 Artem Polyakov 2016-12-16 22:09:37 MST
Danny,

I quick checked persistent connection implementation and it doesn't seem suitable for this task.
It looks like a client-server model while with pmix direct modex it is more peer-to-peer model. If I understand correctly with persistent sockets the server creates one thread per each connection which is what I highly want to avoid for several reasons:
1. In theory there might be a connection from each stepd in the job which will hurt scalability. In practice amount of connections is expected to be log(num_nodex) but still we don't want to create this much of threads.
2. The problem that I'm currently trying to solve is related to context switching delays in case of highly loaded nodes (we are running side to side with CPU-aggressive MPI application).

So I'm more inclined to use lower level slurm API to manage sockets and use EIO for managing a set of fd's.
Comment 12 Danny Auble 2016-12-22 11:38:12 MST
Hum, perhaps I misunderstood what you were looking to pass on the connection.

But if you can come up with a better solution by all means give it a stab.

I'm interested with what you come up with.
Comment 13 Danny Auble 2017-01-19 14:53:35 MST
Artem, just a reminder we are code freezing tomorrow for 17.02.  I am guessing anything you will be doing will make it into 17.11.
Comment 14 Artem Polyakov 2017-01-19 14:59:09 MST
Thank you, Danny.

I'm still working on the feature. Do you think it is possible to introduce it in 17.01.1 given that it only affects internal intercommunication of PMIx plugin?
Comment 15 Danny Auble 2017-01-19 15:00:33 MST
It will be questionable to get it into any 17.02 release.  When you are done we can look at it and decide then.
Comment 16 Artem Polyakov 2017-01-19 15:14:43 MST
Understood. I almost have it finished. But was constantly interrupted by some urgent tasks.
Comment 17 Artem Polyakov 2017-03-27 13:20:10 MDT
Set of patches

This set of patches introduce two important features:
1. Direct TCP connection between stepd's
2. Non-blocking send/receives.

Those two features are self-contained and improve the performance of the PMIX working in the direct-modex mode.
They also are strategic steps for subsequent near future improvements.

We have a week+ of successful MTT runs for our internal test suite containing 1500+ MPI applications, so we have some level of confidence that it no regression was introduced. We will be running this on daily basis.

Application of this patchset:
* Please apply with git am to preserve commit messages. Some of them include related performance data that I'd like to store for the records.
* Note that after patch #6 (0006-mpi-pmix-Rework-communication-framework-step-1.patch) autogen.sh should be invoked because this patch introduces changes to the build system (because of new *.c/*.h files).
Comment 18 Artem Polyakov 2017-03-27 13:20:58 MDT
Created attachment 4253 [details]
Set of patches improving communication performance
Comment 19 Artem Polyakov 2017-03-27 13:23:01 MDT
*** Ticket 3625 has been marked as a duplicate of this ticket. ***
Comment 20 Artem Polyakov 2017-04-10 09:07:14 MDT
Any news on this?
Comment 21 Artem Polyakov 2017-04-21 11:36:36 MDT
We are now contributing our testing results through MTT to OMPI community:
https://mtt.open-mpi.org/index.php?do_redir=2419

This is the set of the tests that successfully executing on for my github branch with this set of patches. We running it 1 month already and it shows stable and good results: we have ~126000 successful test runs since we contributing to OMPI MTT and 2x more since we started the testing.

Are there any news on your end? Can I somehow help to speedup the review process? We soon will have some changes on top of this one and we want all of them in the next SLURM release.
Comment 22 Artem Polyakov 2017-04-22 13:50:46 MDT
Created attachment 4403 [details]
One more patch to update copyrights of all affected files

I just realized that I forgot to update copyrights.
This patch fixes that.
Comment 23 Danny Auble 2017-04-26 12:02:53 MDT
Artem, I am just starting to get around to this.  Your copyright patch appears to have a few cut-paste issues.  Almost every new file has 

pmix_common.h - PMIx common declarations and includes

in it.  I am fairly confident that is incorrect ;).  If you could redo that patch it would be great.

Thanks!
Comment 24 Artem Polyakov 2017-04-26 15:10:05 MDT
Created attachment 4424 [details]
One more patch to update copyrights of all affected files (v2)

Danny, thanks for catching that!
Comment 25 Artem Polyakov 2017-04-27 13:58:27 MDT
Created attachment 4435 [details]
Fix EIO shutdown
Comment 26 Artem Polyakov 2017-04-27 13:59:34 MDT
Created attachment 4436 [details]
Cleanup debug bugs
Comment 27 Artem Polyakov 2017-04-27 14:00:05 MDT
Danny, please note, that I've attached 2 more cleanup patches.
Comment 28 Artem Polyakov 2017-04-27 14:36:20 MDT
Created attachment 4437 [details]
Fix EIO shutdown
Comment 29 Artem Polyakov 2017-04-27 14:36:56 MDT
Created attachment 4438 [details]
Cleanup debug stuff
Comment 30 Artem Polyakov 2017-04-27 14:53:41 MDT
Created attachment 4439 [details]
The full set of patches in one tar

Danny, I also fixed some commit messages (for some reasons git format-patch concatenated multiple lines in some of the commit messages)

Please, use this patch set.
Comment 31 Danny Auble 2017-04-27 14:56:27 MDT
Artem, if it makes sense could you make less patches?  I haven't merged it yet.  I am totally fine if it doesn't make sense, but if these are patches on top of patches merging them together would be better for me.  If they are different sets of functionality then please keep them separate.
Comment 32 Artem Polyakov 2017-04-27 14:56:48 MDT
Created attachment 4440 [details]
Full set of patches v3

In v3 I added "Signed-off" lines
Comment 33 Artem Polyakov 2017-04-27 14:58:30 MDT
Danny,
I kept this separation intentionally.
If you have problems with merging - I can rebase my branch against the latest slurm/master and any other branch that you need and provide rebased patches.
Comment 34 Artem Polyakov 2017-05-22 08:22:58 MDT
Hello, any updates?
Comment 35 Danny Auble 2017-05-22 08:43:22 MDT
Not yet
Comment 36 Artem Polyakov 2017-06-19 18:23:43 MDT
Please, note that I've submitted implementation of the next step of PMIx plugin development roadmap: UCX library support.
https://bugs.schedmd.com/show_bug.cgi?id=3909

This implementation is based on this changes. Also #3909 fixes coding style, so please do not apply code formatting after this will be merged as it will distort #3909 patchset.
Comment 37 Danny Auble 2017-08-04 10:56:49 MDT
Closing, moving all discussion to Bug 3909.

*** This ticket has been marked as a duplicate of ticket 3909 ***