Created attachment 4795 [details] Set of patches introducing UCX support Hello, This patchset introduces support for UCX library (http://www.openucx.org/) that provides an abstraction to multiple high-performance fabrics. This patchset depends on bug report #3103 (https://bugs.schedmd.com/show_bug.cgi?id=3103). Note: * After applying this patchset autogen.sh has to be executed to update Makefile.in's * This patchset contains coding style fix that afects #3103 as well. Please don't do formatting after #3103 is applied.
Some more info: * All of the old functionality is available and selectable through the environment variables. * We thoroughly tested this with our MPI test suite containing more than 1K tests. We are planning to do some additional work to further improve the jobstart performance and support new PMIx versions.
Created attachment 4798 [details] v1 patchset + 2 fixes
Thanks Artem - I'm tagging as an enhancement request to look into merging these ahead of the 17.11 release. Such an extension will obviously take us a bit of effort to review and validate. Are you able to expand upon the testing process that these patches have been run through so far? thanks, - Tim
Hello, Tim. We are using Open MPI's MTT to run a set of MPI applications. The following are the results for https://bugs.schedmd.com/show_bug.cgi?id=3103 patchset: https://mtt.open-mpi.org/index.php?do_redir=2453 We ran the same set of tests with this patchset, all tests are successful. MTT vary PMIx modes (full/direct modex), communication part: * SLURM API that was originally used in pmix plugin * TCP direct connection (introduced in https://mtt.open-mpi.org/index.php?do_redir=2453) * UCX direct connection (introduced in this patchset)
Also note, that until SLURM is explicitly compiled "--with-ucx" UCX code will not be active.
Here is the link to my GitHub branch in case it'll be useful: https://github.com/artpol84/slurm/commits/pmix/ucx_submission
Actually after I told you about our tests I decided to go and double-check beyond the "SUCCESS" statuses and I found the bug related to collectives seq number calculation. The fix is under testing now and I will ensure that all outputs are clean. I will attach the patch separately as Danny asked me, it will be small.
Created attachment 4814 [details] Fix-collectives-sequence-number-calculation
Created attachment 4815 [details] Fix-collectives-sequence-number-calculation v2
Now MTT results look clean.
Created attachment 4843 [details] 0026-mpi-pmix-fix-UCX-cleanup.patch
Created attachment 4844 [details] 0027-mpi-pmix-Cleanup-UCX-progress.patch
Artem, you can tar these up, I just didn't want more added to a tarball I had already processed.
Created attachment 4845 [details] New set of fixes
Created attachment 4874 [details] Fix the typo causing a perf bug This bug was introduced in patch #4 (0004-mpi-pmix-Infrastructural-changes-for-future-UCX-supp.patch). Maybe it makes sense to apply it right after this one to exclude the error instead of fixing it in patch #28. Problem description: mpi/pmix: Fix a typo in I/O engine code Replace "bool" with "int" as the return type of `pmixp_io_fd()`. This was causing an interesting hidden bug affecting the performance. Since this function was returning boolean value it was always returning "1" instead of actual fd number. fd=1 is set to /dev/null for slurmstepd, this is a char device that is always read/write ready from the poll() perspective. So poll was continuously interrupting and progress was fine, but CPU usage was ~100%.
Artem, could you please prepare a patch that changes empty parameter functions from -pmixp_dconn_conn_type_t pmixp_dconn_connect_type() +pmixp_dconn_conn_type_t pmixp_dconn_connect_type(void) on the new code? I am about halfway through the entire code patch only looking at formatting, it looks like only minor issues so far. Note, all new code dealing with debug output should be one line no matter how long, parameters on new lines i.e. - PMIXP_ERROR("Can't find address for host " - "%s, check slurm.conf", nodename); + PMIXP_ERROR("Can't find address for host %s, check slurm.conf", + nodename);
Hit enter too fast on that last one. Continuing... And function declarations should be -pmixp_conn_t * -pmixp_conn_new_temp(pmixp_conn_proto_t proto, int fd, - pmixp_conn_new_msg_cb_t nmsg_cb) +pmixp_conn_t *pmixp_conn_new_temp(pmixp_conn_proto_t proto, int fd, + pmixp_conn_new_msg_cb_t nmsg_cb) Most of the time you got it right, but not always. New multiline comments should be /* * comment * comment */ Also Slurm is 'Slurm' not 'SLURM' ;).
At the moment I am only looking for a patch to fill in the 'void' parameter, I am taking care of all the other format issues.
Ok, I'll prepare the patch asap
Created attachment 4966 [details] Fix zero-argument functions formatting I think I addressed them all.
Created attachment 4967 [details] Fix zero-argument functions formatting v2 Looks like I missed some files in the prev commit.
Created attachment 5019 [details] 0030-mpi-pmix-fix-singleton-progress-fan-in.patch This patch fixes the single-node case.
Created attachment 5020 [details] 0030-mpi-pmix-fix-singleton-progress-fan-in_v2.patch Fix the enum
*** Ticket 3103 has been marked as a duplicate of this ticket. ***
Created attachment 5027 [details] 0031-mpi-pmix-Fix-the-case-when-only-part-of-allocation-i.patch Danny, I noticed that you commited all of the patches to the Slurm/master. When testing today I found one more bug in the implementation. Attaching it here.
Created attachment 5028 [details] 0032-mpi-pmix-Fix-UCX-tags.patch This bug was discovered as part of a next step involving collectives optimizations. It was observed for the 32K+ allgatherv message sizes.
Thanks Artem, all of these patches have been committed. Please reopen if anything else needs to be added/fixed.
Created attachment 5034 [details] 0033-mpi-pmix-Fix-the-case-where-UCX-fails-to-connect.patch During the debug of the UCX connection issues with Danny we noticed a segfault: srun --mpi=pmix -n2 -N2 ~/pmix/1/build/test/pmix_client -n 2 --job-fence slurmstepd: error: smd1 [0] /pmixp_dconn_ucx.c:666 [_ucx_connect] mpi/pmix: ERROR: ucp_ep_create failed: Destination is unreachable slurmstepd: error: smd1 [0] /pmixp_dconn.h:244 [pmixp_dconn_connect] mpi/pmix: ERROR: Cannot establish direct connection to smd2 (1) slurmstepd: error: smd1 [0] /pmixp_server.c:653 [_process_extended_hdr] mpi/pmix: ERROR: Unable to connect to 1 slurmstepd: error: smd2 [1] /pmixp_dconn_ucx.c:666 [_ucx_connect] mpi/pmix: ERROR: ucp_ep_create failed: Destination is unreachable slurmstepd: error: smd2 [1] /pmixp_dconn.h:244 [pmixp_dconn_connect] mpi/pmix: ERROR: Cannot establish direct connection to smd1 (0) slurmstepd: error: smd2 [1] /pmixp_server.c:653 [_process_extended_hdr] mpi/pmix: ERROR: Unable to connect to 0 slurmstepd: error: smd1 [0] /pmixp_dconn_ucx.c:666 [_ucx_connect] mpi/pmix: ERROR: ucp_ep_create failed: Destination is unreachable [smd1:3391 :0] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x18) It can be seen from it that output that UCX connection establishment attempt is performed twice. This was triggering a bug with double finalization of the peer pending messages list. This patch fixes it. gdb trace: Thread 8 (Thread 0x7f5c44133700 (LWP 3720)): #0 0x00007f5c4bea1ded in pause () at ../sysdeps/unix/syscall-template.S:84 #1 0x00007f5c451882c5 in ucs_debug_freeze () at ../../../../ucx/src/ucs/debug/debug.c:728 ---Type <return> to continue, or q <return> to quit--- #2 0x00007f5c45188a6d in ucs_error_freeze (message=0x7f5c44131a20 "Caught signal 11 (Segmentation fault: address not mapped to object at address 0x18)", error_type=0x7f5c45191fe8 "address not mapped to object") at ../../../../ucx/src/ucs/debug/debug.c:847 #3 ucs_handle_error (error_type=error_type@entry=0x7f5c45191fe8 "address not mapped to object", message=message@entry=0x7f5c45191e05 "Caught signal %d (%s: %s%s)") at ../../../../ucx/src/ucs/debug/debug.c:1011 #4 0x00007f5c45188450 in ucs_debug_handle_error_signal (signo=signo@entry=11, cause=0x7f5c45191fe8 "address not mapped to object", fmt=fmt@entry=0x7f5c451920b4 " at address %p") at ../../../../ucx/src/ucs/debug/debug.c:951 #5 0x00007f5c451885f2 in ucs_error_signal_handler (signo=11, info=0x7f5c441322b0, context=<optimized out>) at ../../../../ucx/src/ucs/debug/debug.c:974 #6 <signal handler called> #7 0x00007f5c464cb271 in pmixp_list_enq (l=0x7f5c466daef0 <_free_list>, elem=0x0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_utils.h:181 #8 0x00007f5c464cb54b in pmixp_rlist_fini (l=0x55592d7c45b8) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_utils.h:358 #9 0x00007f5c464cdc3b in _ucx_connect (_priv=0x55592d7c4550, ep_data=0x7f5c3000a810, ep_len=226, init_msg=0x0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_dconn_ucx.c:704 #10 0x00007f5c464ca423 in pmixp_dconn_connect_do (dconn=0x55592d7c3308, ep_data=0x7f5c3000a810, ep_len=226, init_msg=0x0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_dconn.c:115 #11 0x00007f5c464bc1d1 in pmixp_dconn_connect (dconn=0x55592d7c3308, ep_data=0x7f5c3000a810, ep_len=226, init_msg=0x0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_dconn.h:227 #12 0x00007f5c464be777 in _process_extended_hdr (hdr=0x7f5c30000e04, buf=0x7f5c30000e40) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_server.c:651 #13 0x00007f5c464c014c in _slurm_new_msg (conn=0x7f5c30000ec0, _hdr=0x7f5c30000e00, msg=0x7f5c3000a5b0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_server.c:1128 #14 0x00007f5c464bba47 in pmixp_conn_progress_rcv (conn=0x7f5c30000ec0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_conn.h:109 #15 0x00007f5c464c02b8 in pmixp_server_slurm_conn (fd=42) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_server.c:1153 #16 0x00007f5c464b14eb in _server_conn_read (obj=0x7f5c30000940, objs=0x55592d720338) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_agent.c:155 #17 0x000055592b9cd6b0 in _poll_handle_event (revents=1, obj=0x7f5c30000940, objList=0x55592d720338) at /home/da/slurm/17.11/slurm/src/common/eio.c:493 #18 0x000055592b9cd474 in _poll_dispatch (pfds=0x7f5c30000e70, nfds=4, map=0x7f5c30000c00, objList=0x55592d720338) at /home/da/slurm/17.11/slurm/src/common/eio.c:445 #19 0x000055592b9ccfc0 in eio_handle_mainloop (eio=0x7f5c300008d0) at /home/da/slurm/17.11/slurm/src/common/eio.c:333 ---Type <return> to continue, or q <return> to quit--- #20 0x00007f5c464b1c35 in _agent_thread (unused=0x0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_agent.c:282 #21 0x00007f5c4be986da in start_thread (arg=0x7f5c44133700) at pthread_create.c:456 #22 0x00007f5c4bbd2d7f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
Created attachment 5035 [details] 0034-mpi-pmix-Enforce-parameter-checking-in-pmix_-r-list.patch Enforce some parameter checking for the pmixp_[r]list with respect to a problems observed with UCX connect.
Created attachment 5038 [details] 0035-mpi-pmix-Fix-UCX-connection-error-case-handling.patch Abort if we can't communicate through direct connect while it was requested.
Created attachment 5043 [details] 0036-mpi-pmix-rework-tree-based-collective.patch Rework collectives to benefit from the direct connect.
Created attachment 5044 [details] 0036-mpi-pmix-rework-tree-based-collective_v2.patch
Created attachment 5053 [details] 0037-mpi-pmix-Add-collectives-evaluation-tool.patch Add the internal tool to measure the performance of collectives.
Created attachment 5054 [details] 0038-mpi-pmix-Addittional-UCX-settings-to-ensure-safe-for.patch Some fixes to UCX to ensure that fork() can be done safely.
Danny, to keep track: patches #0033-0038 are pending for the review.
Created attachment 5055 [details] 0039-mpi-pmix-Disable-collectives-debug-output-by-default.patch Trivial - disable debug by default
Created attachment 5058 [details] 0040-mpi-pmix-Fixed-ucx-configure.patch
Created attachment 5072 [details] 0040-mpi-pmix-Fixed-ucx-configure_v2.patch
Thanks Artem, these have all been checked in. Please reopen if more is needed. Note, I do not have hardware to test this on so we are relying on your testing.
Thank you, Danny. We continuing our testing and will add bugfixes if found. We also will have one more feature - early connection initiation. It should be fairly small change, but we need to do some perf tuning before submission.
Created attachment 5253 [details] Early wireup slurm daemons if requested. This is a pretty simple feature that will be off by default. But I'd like to have it in 17.11 for performance reasons.
Artem it appears this patch only works if the patch from bug 4131 is applied.
Thanks Boris, this has been committed 7938e8c6b6ce. Please reopen if anything else comes up.
Created attachment 5402 [details] Fix UCX initialization
Danny, there was some miscommunication with UCX developers and this unneeded parameter causes rare data corruptions and needs to be removed.
Thanks Artem c862002c33