Ticket 3909 - SLURM/PMIx add support for UCX point-to-point library
Summary: SLURM/PMIx add support for UCX point-to-point library
Status: RESOLVED FIXED
Alias: None
Product: Slurm
Classification: Unclassified
Component: Contributions (show other tickets)
Version: 17.11.x
Hardware: Linux Linux
: 5 - Enhancement
Assignee: Danny Auble
QA Contact:
URL:
: 3103 (view as ticket list)
Depends on: 4131
Blocks:
  Show dependency tree / graph
 
Reported: 2017-06-19 18:20 MDT by Artem Polyakov
Modified: 2017-10-19 06:46 MDT (History)
0 users

See Also:
Site: -Other-
Slinky Site: ---
Alineos Sites: ---
Atos/Eviden Sites: ---
Confidential Site: ---
Coreweave sites: ---
Cray Sites: ---
DS9 clusters: ---
Google sites: ---
HPCnow Sites: ---
HPE Sites: ---
IBM Sites: ---
NOAA SIte: ---
NoveTech Sites: ---
Nvidia HWinf-CS Sites: ---
OCF Sites: ---
Recursion Pharma Sites: ---
SFW Sites: ---
SNIC sites: ---
Tzag Elita Sites: ---
Linux Distro: ---
Machine Name:
CLE Version:
Version Fixed: 17.11.0-pre3(rc1)
Target Release: 17.11
DevPrio: ---
Emory-Cloud Sites: ---


Attachments
Set of patches introducing UCX support (55.10 KB, application/x-bzip)
2017-06-19 18:20 MDT, Artem Polyakov
Details
v1 patchset + 2 fixes (55.83 KB, application/x-bzip)
2017-06-20 21:10 MDT, Artem Polyakov
Details
Fix-collectives-sequence-number-calculation (2.06 KB, patch)
2017-06-22 19:07 MDT, Artem Polyakov
Details | Diff
Fix-collectives-sequence-number-calculation v2 (1.84 KB, patch)
2017-06-22 19:13 MDT, Artem Polyakov
Details | Diff
0026-mpi-pmix-fix-UCX-cleanup.patch (823 bytes, patch)
2017-06-28 11:39 MDT, Artem Polyakov
Details | Diff
0027-mpi-pmix-Cleanup-UCX-progress.patch (2.49 KB, patch)
2017-06-28 11:39 MDT, Artem Polyakov
Details | Diff
New set of fixes (2.19 KB, application/x-bzip)
2017-06-28 12:01 MDT, Artem Polyakov
Details
Fix the typo causing a perf bug (1.15 KB, patch)
2017-07-05 13:45 MDT, Artem Polyakov
Details | Diff
Fix zero-argument functions formatting (4.53 KB, patch)
2017-07-26 18:38 MDT, Artem Polyakov
Details | Diff
Fix zero-argument functions formatting v2 (5.50 KB, patch)
2017-07-27 00:26 MDT, Artem Polyakov
Details | Diff
0030-mpi-pmix-fix-singleton-progress-fan-in.patch (802 bytes, patch)
2017-08-03 16:20 MDT, Artem Polyakov
Details | Diff
0030-mpi-pmix-fix-singleton-progress-fan-in_v2.patch (799 bytes, patch)
2017-08-03 16:44 MDT, Artem Polyakov
Details | Diff
0031-mpi-pmix-Fix-the-case-when-only-part-of-allocation-i.patch (1.13 KB, patch)
2017-08-05 15:04 MDT, Artem Polyakov
Details | Diff
0032-mpi-pmix-Fix-UCX-tags.patch (1.26 KB, patch)
2017-08-07 11:01 MDT, Artem Polyakov
Details | Diff
0033-mpi-pmix-Fix-the-case-where-UCX-fails-to-connect.patch (1.63 KB, patch)
2017-08-08 01:00 MDT, Artem Polyakov
Details | Diff
0034-mpi-pmix-Enforce-parameter-checking-in-pmix_-r-list.patch (3.22 KB, patch)
2017-08-08 01:02 MDT, Artem Polyakov
Details | Diff
0035-mpi-pmix-Fix-UCX-connection-error-case-handling.patch (1.43 KB, patch)
2017-08-08 11:38 MDT, Artem Polyakov
Details | Diff
0036-mpi-pmix-rework-tree-based-collective.patch (59.89 KB, patch)
2017-08-08 22:06 MDT, Artem Polyakov
Details | Diff
0036-mpi-pmix-rework-tree-based-collective_v2.patch (61.69 KB, patch)
2017-08-09 02:42 MDT, Artem Polyakov
Details | Diff
0037-mpi-pmix-Add-collectives-evaluation-tool.patch (9.96 KB, patch)
2017-08-09 12:01 MDT, Artem Polyakov
Details | Diff
0038-mpi-pmix-Addittional-UCX-settings-to-ensure-safe-for.patch (1007 bytes, patch)
2017-08-09 12:02 MDT, Artem Polyakov
Details | Diff
0039-mpi-pmix-Disable-collectives-debug-output-by-default.patch (688 bytes, patch)
2017-08-09 12:11 MDT, Artem Polyakov
Details | Diff
0040-mpi-pmix-Fixed-ucx-configure.patch (3.00 KB, patch)
2017-08-09 13:48 MDT, Artem Polyakov
Details | Diff
0040-mpi-pmix-Fixed-ucx-configure_v2.patch (3.49 KB, patch)
2017-08-10 16:28 MDT, Artem Polyakov
Details | Diff
Early wireup slurm daemons if requested. (4.18 KB, patch)
2017-09-19 21:42 MDT, Artem Polyakov
Details | Diff
Fix UCX initialization (497 bytes, patch)
2017-10-18 23:34 MDT, Artem Polyakov
Details | Diff

Note You need to log in before you can comment on or make changes to this ticket.
Description Artem Polyakov 2017-06-19 18:20:48 MDT
Created attachment 4795 [details]
Set of patches introducing UCX support

Hello,

This patchset introduces support for UCX library (http://www.openucx.org/) that provides an abstraction to multiple high-performance fabrics.

This patchset depends on bug report #3103 (https://bugs.schedmd.com/show_bug.cgi?id=3103).

Note:
* After applying this patchset autogen.sh has to be executed to update Makefile.in's
* This patchset contains coding style fix that afects #3103 as well. Please don't do formatting after #3103 is applied.
Comment 1 Artem Polyakov 2017-06-19 18:48:54 MDT
Some more info:

* All of the old functionality is available and selectable through the environment variables.
* We thoroughly tested this with our MPI test suite containing more than 1K tests.


We are planning to do some additional work to further improve the jobstart performance and support new PMIx versions.
Comment 2 Artem Polyakov 2017-06-20 21:10:39 MDT
Created attachment 4798 [details]
v1 patchset + 2 fixes
Comment 3 Tim Wickberg 2017-06-21 14:46:50 MDT
Thanks Artem -

I'm tagging as an enhancement request to look into merging these ahead of the 17.11 release. Such an extension will obviously take us a bit of effort to review and validate.

Are you able to expand upon the testing process that these patches have been run through so far?

thanks,
- Tim
Comment 4 Artem Polyakov 2017-06-21 15:46:00 MDT
Hello, Tim.

We are using Open MPI's MTT to run a set of MPI applications. The following are the results for https://bugs.schedmd.com/show_bug.cgi?id=3103 patchset:
https://mtt.open-mpi.org/index.php?do_redir=2453

We ran the same set of tests with this patchset, all tests are successful.
MTT vary PMIx modes (full/direct modex), communication part:
* SLURM API that was originally used in pmix plugin
* TCP direct connection (introduced in https://mtt.open-mpi.org/index.php?do_redir=2453)
* UCX direct connection (introduced in this patchset)
Comment 5 Artem Polyakov 2017-06-21 16:03:24 MDT
Also note, that until SLURM is explicitly compiled "--with-ucx" UCX code will not be active.
Comment 6 Artem Polyakov 2017-06-21 16:44:47 MDT
Here is the link to my GitHub branch in case it'll be useful:
https://github.com/artpol84/slurm/commits/pmix/ucx_submission
Comment 7 Artem Polyakov 2017-06-22 13:56:16 MDT
Actually after I told you about our tests I decided to go and double-check beyond the "SUCCESS" statuses and I found the bug related to collectives seq number calculation.
The fix is under testing now and I will ensure that all outputs are clean.
I will attach the patch separately as Danny asked me, it will be small.
Comment 8 Artem Polyakov 2017-06-22 19:07:22 MDT
Created attachment 4814 [details]
Fix-collectives-sequence-number-calculation
Comment 9 Artem Polyakov 2017-06-22 19:13:11 MDT
Created attachment 4815 [details]
Fix-collectives-sequence-number-calculation v2
Comment 10 Artem Polyakov 2017-06-22 19:34:30 MDT
Now MTT results look clean.
Comment 11 Artem Polyakov 2017-06-28 11:39:21 MDT
Created attachment 4843 [details]
0026-mpi-pmix-fix-UCX-cleanup.patch
Comment 12 Artem Polyakov 2017-06-28 11:39:58 MDT
Created attachment 4844 [details]
0027-mpi-pmix-Cleanup-UCX-progress.patch
Comment 13 Danny Auble 2017-06-28 11:55:42 MDT
Artem, you can tar these up, I just didn't want more added to a tarball I had already processed.
Comment 14 Artem Polyakov 2017-06-28 12:01:20 MDT
Created attachment 4845 [details]
New set of fixes
Comment 15 Artem Polyakov 2017-07-05 13:45:14 MDT
Created attachment 4874 [details]
Fix the typo causing a perf bug

This bug was introduced in patch #4 (0004-mpi-pmix-Infrastructural-changes-for-future-UCX-supp.patch). Maybe it makes sense to apply it right after this one to exclude the error instead of fixing it in patch #28.

Problem description:
mpi/pmix: Fix a typo in I/O engine code

Replace "bool" with "int" as the return type of `pmixp_io_fd()`. This
was causing an interesting hidden bug affecting the performance.
Since this function was returning boolean value it was always
returning "1" instead of actual fd number. fd=1 is set to /dev/null
for slurmstepd, this is a char device that is always read/write ready
from the poll() perspective. So poll was continuously interrupting and
progress was fine, but CPU usage was ~100%.
Comment 16 Danny Auble 2017-07-26 15:35:29 MDT
Artem, could you please prepare a patch that changes empty parameter functions from

-pmixp_dconn_conn_type_t pmixp_dconn_connect_type()
+pmixp_dconn_conn_type_t pmixp_dconn_connect_type(void)

on the new code?

I am about halfway through the entire code patch only looking at formatting, it looks like only minor issues so far.

Note, all new code dealing with debug output should be one line no matter how long, parameters on new lines

i.e.

-		PMIXP_ERROR("Can't find address for host "
-			    "%s, check slurm.conf", nodename);
+		PMIXP_ERROR("Can't find address for host %s, check slurm.conf",
+ nodename);
Comment 17 Danny Auble 2017-07-26 15:38:29 MDT
Hit enter too fast on that last one.

Continuing...

And function declarations should be

-pmixp_conn_t *
-pmixp_conn_new_temp(pmixp_conn_proto_t proto, int fd,
-		    pmixp_conn_new_msg_cb_t nmsg_cb)
+pmixp_conn_t *pmixp_conn_new_temp(pmixp_conn_proto_t proto, int fd,
+				  pmixp_conn_new_msg_cb_t nmsg_cb)

Most of the time you got it right, but not always.

New multiline comments should be 

/*
 * comment
 * comment
 */

Also Slurm is 'Slurm' not 'SLURM' ;).
Comment 18 Danny Auble 2017-07-26 15:40:55 MDT
At the moment I am only looking for a patch to fill in the 'void' parameter, I am taking care of all the other format issues.
Comment 19 Artem Polyakov 2017-07-26 18:00:37 MDT
Ok, I'll prepare the patch asap
Comment 20 Artem Polyakov 2017-07-26 18:38:54 MDT
Created attachment 4966 [details]
Fix zero-argument functions formatting

I think I addressed them all.
Comment 21 Artem Polyakov 2017-07-27 00:26:08 MDT
Created attachment 4967 [details]
Fix zero-argument functions formatting v2

Looks like I missed some files in the prev commit.
Comment 22 Artem Polyakov 2017-08-03 16:20:17 MDT
Created attachment 5019 [details]
0030-mpi-pmix-fix-singleton-progress-fan-in.patch

This patch fixes the single-node case.
Comment 23 Artem Polyakov 2017-08-03 16:44:12 MDT
Created attachment 5020 [details]
0030-mpi-pmix-fix-singleton-progress-fan-in_v2.patch

Fix the enum
Comment 24 Danny Auble 2017-08-04 10:56:49 MDT
*** Ticket 3103 has been marked as a duplicate of this ticket. ***
Comment 25 Artem Polyakov 2017-08-05 15:04:29 MDT
Created attachment 5027 [details]
0031-mpi-pmix-Fix-the-case-when-only-part-of-allocation-i.patch

Danny, I noticed that you commited all of the patches to the Slurm/master.
When testing today I found one more bug in the implementation. Attaching it here.
Comment 26 Artem Polyakov 2017-08-07 11:01:23 MDT
Created attachment 5028 [details]
0032-mpi-pmix-Fix-UCX-tags.patch

This bug was discovered as part of a next step involving collectives optimizations. It was observed for the 32K+ allgatherv message sizes.
Comment 27 Danny Auble 2017-08-07 11:50:41 MDT
Thanks Artem, all of these patches have been committed.  Please reopen if anything else needs to be added/fixed.
Comment 28 Artem Polyakov 2017-08-08 01:00:46 MDT
Created attachment 5034 [details]
0033-mpi-pmix-Fix-the-case-where-UCX-fails-to-connect.patch

During the debug of the UCX connection issues with Danny we noticed a segfault:

srun --mpi=pmix -n2 -N2 ~/pmix/1/build/test/pmix_client -n 2 --job-fence
slurmstepd: error: smd1 [0] /pmixp_dconn_ucx.c:666 [_ucx_connect] mpi/pmix: ERROR: ucp_ep_create failed: Destination is unreachable
slurmstepd: error: smd1 [0] /pmixp_dconn.h:244 [pmixp_dconn_connect] mpi/pmix: ERROR: Cannot establish direct connection to smd2 (1)
slurmstepd: error: smd1 [0] /pmixp_server.c:653 [_process_extended_hdr] mpi/pmix: ERROR: Unable to connect to 1
slurmstepd: error: smd2 [1] /pmixp_dconn_ucx.c:666 [_ucx_connect] mpi/pmix: ERROR: ucp_ep_create failed: Destination is unreachable
slurmstepd: error: smd2 [1] /pmixp_dconn.h:244 [pmixp_dconn_connect] mpi/pmix: ERROR: Cannot establish direct connection to smd1 (0)
slurmstepd: error: smd2 [1] /pmixp_server.c:653 [_process_extended_hdr] mpi/pmix: ERROR: Unable to connect to 0
slurmstepd: error: smd1 [0] /pmixp_dconn_ucx.c:666 [_ucx_connect] mpi/pmix: ERROR: ucp_ep_create failed: Destination is unreachable
[smd1:3391 :0] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x18)

It can be seen from it that output that UCX connection establishment attempt is performed twice. This was triggering a bug with double finalization of the peer pending messages list. This patch fixes it.

gdb trace:
Thread 8 (Thread 0x7f5c44133700 (LWP 3720)):
#0 0x00007f5c4bea1ded in pause () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f5c451882c5 in ucs_debug_freeze () at ../../../../ucx/src/ucs/debug/debug.c:728
---Type <return> to continue, or q <return> to quit--- 
#2 0x00007f5c45188a6d in ucs_error_freeze (message=0x7f5c44131a20 "Caught signal 11 (Segmentation fault: address not mapped to object at address 0x18)", 
error_type=0x7f5c45191fe8 "address not mapped to object") at ../../../../ucx/src/ucs/debug/debug.c:847
#3 ucs_handle_error (error_type=error_type@entry=0x7f5c45191fe8 "address not mapped to object", message=message@entry=0x7f5c45191e05 "Caught signal %d (%s: %s%s)")
at ../../../../ucx/src/ucs/debug/debug.c:1011
#4 0x00007f5c45188450 in ucs_debug_handle_error_signal (signo=signo@entry=11, cause=0x7f5c45191fe8 "address not mapped to object", fmt=fmt@entry=0x7f5c451920b4 " at address %p")
at ../../../../ucx/src/ucs/debug/debug.c:951
#5 0x00007f5c451885f2 in ucs_error_signal_handler (signo=11, info=0x7f5c441322b0, context=<optimized out>) at ../../../../ucx/src/ucs/debug/debug.c:974
#6 <signal handler called>
#7 0x00007f5c464cb271 in pmixp_list_enq (l=0x7f5c466daef0 <_free_list>, elem=0x0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_utils.h:181
#8 0x00007f5c464cb54b in pmixp_rlist_fini (l=0x55592d7c45b8) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_utils.h:358
#9 0x00007f5c464cdc3b in _ucx_connect (_priv=0x55592d7c4550, ep_data=0x7f5c3000a810, ep_len=226, init_msg=0x0)
at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_dconn_ucx.c:704
#10 0x00007f5c464ca423 in pmixp_dconn_connect_do (dconn=0x55592d7c3308, ep_data=0x7f5c3000a810, ep_len=226, init_msg=0x0)
at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_dconn.c:115
#11 0x00007f5c464bc1d1 in pmixp_dconn_connect (dconn=0x55592d7c3308, ep_data=0x7f5c3000a810, ep_len=226, init_msg=0x0)
at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_dconn.h:227
#12 0x00007f5c464be777 in _process_extended_hdr (hdr=0x7f5c30000e04, buf=0x7f5c30000e40) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_server.c:651
#13 0x00007f5c464c014c in _slurm_new_msg (conn=0x7f5c30000ec0, _hdr=0x7f5c30000e00, msg=0x7f5c3000a5b0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_server.c:1128
#14 0x00007f5c464bba47 in pmixp_conn_progress_rcv (conn=0x7f5c30000ec0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_conn.h:109
#15 0x00007f5c464c02b8 in pmixp_server_slurm_conn (fd=42) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_server.c:1153
#16 0x00007f5c464b14eb in _server_conn_read (obj=0x7f5c30000940, objs=0x55592d720338) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_agent.c:155
#17 0x000055592b9cd6b0 in _poll_handle_event (revents=1, obj=0x7f5c30000940, objList=0x55592d720338) at /home/da/slurm/17.11/slurm/src/common/eio.c:493
#18 0x000055592b9cd474 in _poll_dispatch (pfds=0x7f5c30000e70, nfds=4, map=0x7f5c30000c00, objList=0x55592d720338) at /home/da/slurm/17.11/slurm/src/common/eio.c:445
#19 0x000055592b9ccfc0 in eio_handle_mainloop (eio=0x7f5c300008d0) at /home/da/slurm/17.11/slurm/src/common/eio.c:333
---Type <return> to continue, or q <return> to quit---
#20 0x00007f5c464b1c35 in _agent_thread (unused=0x0) at /home/da/slurm/17.11/slurm/src/plugins/mpi/pmix/pmixp_agent.c:282
#21 0x00007f5c4be986da in start_thread (arg=0x7f5c44133700) at pthread_create.c:456
#22 0x00007f5c4bbd2d7f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:105
Comment 29 Artem Polyakov 2017-08-08 01:02:45 MDT
Created attachment 5035 [details]
0034-mpi-pmix-Enforce-parameter-checking-in-pmix_-r-list.patch

Enforce some parameter checking for the pmixp_[r]list with respect to a problems observed with UCX connect.
Comment 30 Artem Polyakov 2017-08-08 11:38:38 MDT
Created attachment 5038 [details]
0035-mpi-pmix-Fix-UCX-connection-error-case-handling.patch

Abort if we can't communicate through direct connect while it was requested.
Comment 31 Artem Polyakov 2017-08-08 22:06:22 MDT
Created attachment 5043 [details]
0036-mpi-pmix-rework-tree-based-collective.patch

Rework collectives to benefit from the direct connect.
Comment 32 Artem Polyakov 2017-08-09 02:42:52 MDT
Created attachment 5044 [details]
0036-mpi-pmix-rework-tree-based-collective_v2.patch
Comment 33 Artem Polyakov 2017-08-09 12:01:21 MDT
Created attachment 5053 [details]
0037-mpi-pmix-Add-collectives-evaluation-tool.patch

Add the internal tool to measure the performance of collectives.
Comment 34 Artem Polyakov 2017-08-09 12:02:22 MDT
Created attachment 5054 [details]
0038-mpi-pmix-Addittional-UCX-settings-to-ensure-safe-for.patch

Some fixes to UCX to ensure that fork() can be done safely.
Comment 35 Artem Polyakov 2017-08-09 12:04:37 MDT
Danny, to keep track: patches #0033-0038 are pending for the review.
Comment 36 Artem Polyakov 2017-08-09 12:11:56 MDT
Created attachment 5055 [details]
0039-mpi-pmix-Disable-collectives-debug-output-by-default.patch

Trivial - disable debug by default
Comment 37 Artem Polyakov 2017-08-09 13:48:41 MDT
Created attachment 5058 [details]
0040-mpi-pmix-Fixed-ucx-configure.patch
Comment 38 Artem Polyakov 2017-08-10 16:28:40 MDT
Created attachment 5072 [details]
0040-mpi-pmix-Fixed-ucx-configure_v2.patch
Comment 39 Danny Auble 2017-08-30 16:11:56 MDT
Thanks Artem, these have all been checked in.  Please reopen if more is needed.  Note, I do not have hardware to test this on so we are relying on your testing.
Comment 40 Artem Polyakov 2017-08-30 16:21:32 MDT
Thank you, Danny.

We continuing our testing and will add bugfixes if found.
We also will have one more feature - early connection initiation. It should be fairly small change, but we need to do some perf tuning before submission.
Comment 41 Artem Polyakov 2017-09-19 21:42:00 MDT
Created attachment 5253 [details]
Early wireup slurm daemons if requested.

This is a pretty simple feature that will be off by default. But I'd like to have it in 17.11 for performance reasons.
Comment 42 Danny Auble 2017-10-10 08:13:04 MDT
Artem it appears this patch only works if the patch from bug 4131 is applied.
Comment 43 Danny Auble 2017-10-10 14:42:29 MDT
Thanks Boris, this has been committed 7938e8c6b6ce.

Please reopen if anything else comes up.
Comment 44 Artem Polyakov 2017-10-18 23:34:52 MDT
Created attachment 5402 [details]
Fix UCX initialization
Comment 45 Artem Polyakov 2017-10-19 00:02:54 MDT
Danny,
there was some miscommunication with UCX developers and this unneeded parameter causes rare data corruptions and needs to be removed.
Comment 46 Danny Auble 2017-10-19 06:46:40 MDT
Thanks Artem

c862002c33