Created attachment 3521 [details] POC code Hello, I'm opening this bug report to discuss the problem that I'm seeing with SLURM/pmix performance. When I'm comparing it to OMPI performance I see that SLURM/pmix has serious overhead compared to OMPI implementation. The root cause is that SLURM is creating threads and in case of loaded node those threads are not necessarily will be immediately scheduled. This causes high latencies. One of the steps to fix that is the bug https://bugs.schedmd.com/show_bug.cgi?id=3102 but it covers only the sender. slurmd is the receiver that will forward data to some slurmstepd. With the attached hacky patch I can get comparable performance (to OMPI implementation). What I'm about to suggest is to enable another thread in slurmd that will handle connection requests immediately. It will sit on the another well-known port and we will have some API to acces this one. All the current activity will continue to flow through the common path but there will be an option to reduce the latency.
We are OK with doing implementation, but I want to sync on the architecture. I'll provide the draft soon.
Artem, I'm interested in your idea to see if it has merit. But it isn't clear to me if it will give much. I'm interested to see what you come up with though. Perhaps you should look at the persistent connection code in 17.02. That might reduce the amount of info needing processing. If you are going to make a separate thread to handle these it makes sense to make it persistent as that would help with performance.
Hi, Danny. Here is some update on this: A. I'm actually now working on the new incarnation of this idea. I want to use slurm's communication primitives to initiate connections: sender port # will be in the slurm proto message when it's a first time you communicate with the peer. Once the message is received - a direct persistent tcp connection will be established between 2 stepd's and response and all other communication will continue through it. B. We can still look for the option of a separate thread that processes messages immediately for this approach. It can be used to forward short messages that can be read quickly and then there will be no forward delay. So my plan is the following: 1. Implement the adea of persistent stepd-s connection and see how it will work. 2. If I will still get a performance hit for the first initiation message sent through slurm proto - I will implement a proof of a concept to see if (B) will help.
This seems like a good path forward, just make sure you use the already existing persistent connection code already in Slurm if possible. Not sure how well this will work in practice though, but I'm interested in what you come up with.
Can you point me on the API?.
Look at the way the slurmdbd or the new fed_mgr works on the slurmctld.
ok, thank you.
Danny, I quick checked persistent connection implementation and it doesn't seem suitable for this task. It looks like a client-server model while with pmix direct modex it is more peer-to-peer model. If I understand correctly with persistent sockets the server creates one thread per each connection which is what I highly want to avoid for several reasons: 1. In theory there might be a connection from each stepd in the job which will hurt scalability. In practice amount of connections is expected to be log(num_nodex) but still we don't want to create this much of threads. 2. The problem that I'm currently trying to solve is related to context switching delays in case of highly loaded nodes (we are running side to side with CPU-aggressive MPI application). So I'm more inclined to use lower level slurm API to manage sockets and use EIO for managing a set of fd's.
Hum, perhaps I misunderstood what you were looking to pass on the connection. But if you can come up with a better solution by all means give it a stab. I'm interested with what you come up with.
Artem, just a reminder we are code freezing tomorrow for 17.02. I am guessing anything you will be doing will make it into 17.11.
Thank you, Danny. I'm still working on the feature. Do you think it is possible to introduce it in 17.01.1 given that it only affects internal intercommunication of PMIx plugin?
It will be questionable to get it into any 17.02 release. When you are done we can look at it and decide then.
Understood. I almost have it finished. But was constantly interrupted by some urgent tasks.
Set of patches This set of patches introduce two important features: 1. Direct TCP connection between stepd's 2. Non-blocking send/receives. Those two features are self-contained and improve the performance of the PMIX working in the direct-modex mode. They also are strategic steps for subsequent near future improvements. We have a week+ of successful MTT runs for our internal test suite containing 1500+ MPI applications, so we have some level of confidence that it no regression was introduced. We will be running this on daily basis. Application of this patchset: * Please apply with git am to preserve commit messages. Some of them include related performance data that I'd like to store for the records. * Note that after patch #6 (0006-mpi-pmix-Rework-communication-framework-step-1.patch) autogen.sh should be invoked because this patch introduces changes to the build system (because of new *.c/*.h files).
Created attachment 4253 [details] Set of patches improving communication performance
*** Ticket 3625 has been marked as a duplicate of this ticket. ***
Any news on this?
We are now contributing our testing results through MTT to OMPI community: https://mtt.open-mpi.org/index.php?do_redir=2419 This is the set of the tests that successfully executing on for my github branch with this set of patches. We running it 1 month already and it shows stable and good results: we have ~126000 successful test runs since we contributing to OMPI MTT and 2x more since we started the testing. Are there any news on your end? Can I somehow help to speedup the review process? We soon will have some changes on top of this one and we want all of them in the next SLURM release.
Created attachment 4403 [details] One more patch to update copyrights of all affected files I just realized that I forgot to update copyrights. This patch fixes that.
Artem, I am just starting to get around to this. Your copyright patch appears to have a few cut-paste issues. Almost every new file has pmix_common.h - PMIx common declarations and includes in it. I am fairly confident that is incorrect ;). If you could redo that patch it would be great. Thanks!
Created attachment 4424 [details] One more patch to update copyrights of all affected files (v2) Danny, thanks for catching that!
Created attachment 4435 [details] Fix EIO shutdown
Created attachment 4436 [details] Cleanup debug bugs
Danny, please note, that I've attached 2 more cleanup patches.
Created attachment 4437 [details] Fix EIO shutdown
Created attachment 4438 [details] Cleanup debug stuff
Created attachment 4439 [details] The full set of patches in one tar Danny, I also fixed some commit messages (for some reasons git format-patch concatenated multiple lines in some of the commit messages) Please, use this patch set.
Artem, if it makes sense could you make less patches? I haven't merged it yet. I am totally fine if it doesn't make sense, but if these are patches on top of patches merging them together would be better for me. If they are different sets of functionality then please keep them separate.
Created attachment 4440 [details] Full set of patches v3 In v3 I added "Signed-off" lines
Danny, I kept this separation intentionally. If you have problems with merging - I can rebase my branch against the latest slurm/master and any other branch that you need and provide rebased patches.
Hello, any updates?
Not yet
Please, note that I've submitted implementation of the next step of PMIx plugin development roadmap: UCX library support. https://bugs.schedmd.com/show_bug.cgi?id=3909 This implementation is based on this changes. Also #3909 fixes coding style, so please do not apply code formatting after this will be merged as it will distort #3909 patchset.
Closing, moving all discussion to Bug 3909. *** This ticket has been marked as a duplicate of ticket 3909 ***