Add appendix with use case descriptions

[SteVwonder]

Introduces the \copySignature and \pasteSignature macros for easy copying and pasting of function signatures across the document, along with a convert-to-copysig.py script that can convert old-style signatures to the new \copySignature format (idempotently).

The new appendix contains use case documents for:

• Business Card Exchange (Business Card Exchange for Process-to-Process Wire-up #191)
• Debugging/Tools (Debugging #216)
• Hybrid Programming Models (Hybrid Programming Models #232)

Co-authored by @spophale and @jjhursey

PDF copy with the changes (updated Jan 19, 2021)

[SteVwonder]

Looks like I borked the rebase that split the commits up into the three sections. I'll figure out where I went wrong and re-push.

[SteVwonder]

Please use emoji reactions ON THIS COMMENT to indicate your position on this proposal.

• You do not need to vote on every proposal
• If you have no opinion, don't vote - that is also useful data
• If you've already commented on this issue, please still vote so
  we know your current thoughts
• Not all proposals solve exactly the same problem, so we may end
  up accepting proposals that appear to have some overlap
  This is not a binding majority-rule vote, but it will be a very
  significant input into the corresponding ASC decision.

Here are the meanings for the emojis:

• Hooray or Rocket: I support this so strongly that I
  want to be an advocate for it
• Heart: I think this is an ideal solution
• Thumbs up: I'd be happy with this solution
• Confused: I'd rather we not do this, but I can tolerate it
• Thumbs down: I'd be actively unhappy, and may even consider
  other technologies instead
  If you want to explain in more detail, feel free to add another
  comment, but please also vote on this comment.

[jjhursey]

@SteVwonder for the CI failure here is a patch:

diff --git a/bin/check-multi-declare.py b/bin/check-multi-declare.py
index 584a3f97..85150c0b 100755
--- a/bin/check-multi-declare.py
+++ b/bin/check-multi-declare.py
@@ -39,8 +39,8 @@ if __name__ == "__main__":
             print("-"*50)
 
         # subsection.A is Appendix A: Python Bindings
-        # subsection.B is Appendix B: Revision History
-        p = subprocess.Popen("grep \"newlabel{"+ref_str+"\" pmix-standard.aux | grep -v \"subsection.A\|subsection.B\"",
+        # subsection.C is Appendix C: Revision History
+        p = subprocess.Popen("grep \"newlabel{"+ref_str+"\" pmix-standard.aux | grep -v \"subsection.A\|subsection.C\"",
                              stdout=subprocess.PIPE, stderr=subprocess.STDOUT, shell=True, close_fds=True)
         sout = p.communicate()[0].decode("utf-8").splitlines()
         if p.returncode != 0:

Since the revision chapter is now Appendix C, we just needed to update this part of the CI check.

The other fix for PMIX_DEBUG_JOB we talked about on the call.

[jjhursey]

@SteVwonder I posted a fix with the debugger diagram fixes, and fixes for the CI and PMIX_DEBUG_JOB items noted above. They are in SteVwonder#1
Let me know what you think.

[SteVwonder]

@jjhursey and @rhc54: I'm a little stuck debugging the hybrid programming model example. Pretty everything is working as expected, but I cannot get one of the event callbacks to fire. The code example (sources/hybrid-prog-model.c added in this PR) essentially sets up two callbacks for the PMIX_OPENMP_PARALLEL_ENTERED event. The first callback (parallel_region_OMP_cb) is registered to be first. The second callback (parallel_region_MPI_cb) is registered to come after the first. As far as I can tell, only the second callback fires not the first. Any ideas or suggestions?

Here is the output of the program when run as a single rank under prrte (the relevant bit is the last two lines):

Registered MPI programming model
Registering event handler for model declaration
Entered model_declared_cb
Registered event handler for model declaration
Entered openmp_handler
Registered OpenMP event handler for OpenMP parallel region entered
Entered mpi_handler
Registered MPI event handler for OpenMP parallel region entered
Notifying OpenMP parallel region about to be entered
Entered parallel_region_MPI_cb

[rhc54]

Let me take a look - could be an error in the code or a bug in the event notification system.

[rhc54]

Okay, combination of errors on both sides. I have fixed the library side (see openpmix/openpmix#2046). The problem on the example side was that you weren't waiting for notification to occur before exiting. Remember, PMIx_Notify_event is asynchronous - the function call returns only to let you know that it successfully understood your request. It hasn't actually done the notification yet. You have to add flags to the event handlers to track the notification progress.

Here is the corrected example code:

#include <stdio.h>

#include <pmix.h>

static volatile bool OMP_region_entered = false;
static volatile bool MPI_region_entered = false;

//<EG BEGIN ID="declare_model_cb">
static void model_declared_cb(size_t evhdlr_registration_id,
                              pmix_status_t status, const pmix_proc_t *source,
                              pmix_info_t info[], size_t ninfo,
                              pmix_info_t results[], size_t nresults,
                              pmix_event_notification_cbfunc_fn_t cbfunc,
                              void *cbdata) {
  printf("Entered %s\n", __FUNCTION__);
  int n;
  for (n = 0; n < ninfo; n++) {
    if (PMIX_CHECK_KEY(&info[n], PMIX_PROGRAMMING_MODEL) &&
        strcmp(info[n].value.data.string, "MPI") == 0) {
      /* ignore our own declaration */
      break;
    } else {
      /* actions to perform when another model registers */
    }
  }
  if (NULL != cbfunc) {
    /* tell the event handler that we are only a partial step */
    cbfunc(PMIX_EVENT_PARTIAL_ACTION_TAKEN, NULL, 0, NULL, NULL, cbdata);
  }
}
//<EG END ID="declare_model_cb">

//<EG BEGIN ID="omp_thread">
static void parallel_region_OMP_cb(size_t evhdlr_registration_id,
                                   pmix_status_t status,
                                   const pmix_proc_t *source,
                                   pmix_info_t info[], size_t ninfo,
                                   pmix_info_t results[], size_t nresults,
                                   pmix_event_notification_cbfunc_fn_t cbfunc,
                                   void *cbdata) {
  printf("Entered %s\n", __FUNCTION__);
  /* do what we need OpenMP to do on entering a parallel region */
  if (NULL != cbfunc) {
    /* tell the event handler that we are only a partial step */
    cbfunc(PMIX_EVENT_PARTIAL_ACTION_TAKEN, NULL, 0, NULL, NULL, cbdata);
  }
    OMP_region_entered = true;
}
//<EG END ID="omp_thread">

//<EG BEGIN ID="mpi_thread">
static void parallel_region_MPI_cb(size_t evhdlr_registration_id,
                                   pmix_status_t status,
                                   const pmix_proc_t *source,
                                   pmix_info_t info[], size_t ninfo,
                                   pmix_info_t results[], size_t nresults,
                                   pmix_event_notification_cbfunc_fn_t cbfunc,
                                   void *cbdata) {
  printf("Entered %s\n", __FUNCTION__);
  /* do what we need MPI to do on entering a parallel region */
  if (NULL != cbfunc) {
    /* do what we need MPI to do on entering a parallel region */
    cbfunc(PMIX_EVENT_ACTION_COMPLETE, NULL, 0, NULL, NULL, cbdata);
  }
    MPI_region_entered = true;
}
//<EG END ID="mpi_thread">

static int openmp_handler() {
  pmix_info_t *info;
  int rc;

  printf("Entered %s\n", __FUNCTION__);

  //<EG BEGIN ID="omp_thread">
  bool is_true = true;
  pmix_status_t code = PMIX_OPENMP_PARALLEL_ENTERED;
  PMIX_INFO_CREATE(info, 2);
  PMIX_INFO_LOAD(&info[0], PMIX_EVENT_HDLR_NAME, "OpenMP-Master", PMIX_STRING);
  PMIX_INFO_LOAD(&info[1], PMIX_EVENT_HDLR_FIRST, &is_true, PMIX_BOOL);
  rc = PMIx_Register_event_handler(&code, 1, info, 2, parallel_region_OMP_cb, NULL, NULL);
  if (rc < 0)
    fprintf(stderr, "%s: Failed to register event handler for OpenMP region entrance\n", __FUNCTION__);
  PMIX_INFO_FREE(info, 2);
  //<EG END ID="omp_thread">
  printf("Registered OpenMP event handler for OpenMP parallel region entered\n");

  return rc;
}

static int mpi_handler() {
  pmix_info_t *info;
  int rc;

  printf("Entered %s\n", __FUNCTION__);

  //<EG BEGIN ID="mpi_thread">
  pmix_status_t code = PMIX_OPENMP_PARALLEL_ENTERED;
  PMIX_INFO_CREATE(info, 2);
  PMIX_INFO_LOAD(&info[0], PMIX_EVENT_HDLR_NAME, "MPI-Thread", PMIX_STRING);
  PMIX_INFO_LOAD(&info[1], PMIX_EVENT_HDLR_AFTER, "OpenMP-Master", PMIX_STRING);
  rc = PMIx_Register_event_handler(&code, 1, info, 2, parallel_region_MPI_cb, NULL, NULL);
  if (rc < 0)
    fprintf(stderr, "%s: Failed to register event handler for OpenMP region entrance\n", __FUNCTION__);
  PMIX_INFO_FREE(info, 2);
  //<EG END ID="mpi_thread">
  printf("Registered MPI event handler for OpenMP parallel region entered\n");

  return rc;
}

int main() {
  //<EG BEGIN ID="declare_model">
  pmix_proc_t myproc;
  pmix_info_t *info;

  PMIX_INFO_CREATE(info, 4);
  PMIX_INFO_LOAD(&info[0], PMIX_PROGRAMMING_MODEL, "MPI", PMIX_STRING);
  PMIX_INFO_LOAD(&info[1], PMIX_MODEL_LIBRARY_NAME, "FooMPI", PMIX_STRING);
  PMIX_INFO_LOAD(&info[2], PMIX_MODEL_LIBRARY_VERSION, "1.0.0", PMIX_STRING);
  PMIX_INFO_LOAD(&info[3], PMIX_THREADING_MODEL, "pthread", PMIX_STRING);
  pmix_status_t rc = PMIx_Init(&myproc, info, 4);
  PMIX_INFO_FREE(info, 4);
  //<EG END ID="declare_model">
  printf("Registered MPI programming model\n");

  printf("Registering event handler for model declaration\n");
  //<EG BEGIN ID="declare_model_cb">
  pmix_status_t code = PMIX_MODEL_DECLARED;
  rc = PMIx_Register_event_handler(&code, 1, NULL, 0, model_declared_cb, NULL, NULL);
  //<EG END ID="declare_model_cb">
    if (rc < 0) {
        fprintf(stderr, "Failed to register event handler for model declaration\n");
        goto fin;
    }
  printf("Registered event handler for model declaration\n");

  openmp_handler();
  mpi_handler();

  printf("Notifying OpenMP parallel region about to be entered\n");
  //<EG BEGIN ID="notify_event">
    PMIX_INFO_CREATE(info, 1);
    PMIX_INFO_LOAD(&info[0], PMIX_EVENT_NON_DEFAULT, NULL, PMIX_BOOL);
  rc = PMIx_Notify_event(PMIX_OPENMP_PARALLEL_ENTERED, &myproc, PMIX_RANGE_PROC_LOCAL, info, 1, NULL, NULL);
    if (rc < 0) {
        fprintf(stderr, "Failed to notify OpenMP region entered\n");
        goto fin;
    }

    while (!OMP_region_entered || !MPI_region_entered) {
        sleep(1);
    }
    fprintf(stderr, "Test completed\n");

fin:
    PMIx_Finalize(NULL, 0);
  //<EG END ID="notify_event">
}

Ideally, you would use pthread conditioned waits instead of that sleep function call, but this should hopefully be good enough for now.

[rhc54]

Oh...duh...instead of using those volatiles to know when you can exit, you could just pass a callback function to PMIx_Notify_event and the lib will execute that function when all notifications are complete. You would then only have to monitor for when that one function call is done.

[rhc54]

And here is that updated version of the code:

#include <stdio.h>

#include <pmix.h>

//<EG BEGIN ID="declare_model_cb">
static void model_declared_cb(size_t evhdlr_registration_id,
                              pmix_status_t status, const pmix_proc_t *source,
                              pmix_info_t info[], size_t ninfo,
                              pmix_info_t results[], size_t nresults,
                              pmix_event_notification_cbfunc_fn_t cbfunc,
                              void *cbdata) {
  printf("Entered %s\n", __FUNCTION__);
  int n;
  for (n = 0; n < ninfo; n++) {
    if (PMIX_CHECK_KEY(&info[n], PMIX_PROGRAMMING_MODEL) &&
        strcmp(info[n].value.data.string, "MPI") == 0) {
      /* ignore our own declaration */
      break;
    } else {
      /* actions to perform when another model registers */
    }
  }
  if (NULL != cbfunc) {
    /* tell the event handler that we are only a partial step */
    cbfunc(PMIX_EVENT_PARTIAL_ACTION_TAKEN, NULL, 0, NULL, NULL, cbdata);
  }
}
//<EG END ID="declare_model_cb">

//<EG BEGIN ID="omp_thread">
static void parallel_region_OMP_cb(size_t evhdlr_registration_id,
                                   pmix_status_t status,
                                   const pmix_proc_t *source,
                                   pmix_info_t info[], size_t ninfo,
                                   pmix_info_t results[], size_t nresults,
                                   pmix_event_notification_cbfunc_fn_t cbfunc,
                                   void *cbdata) {
  printf("Entered %s\n", __FUNCTION__);
  /* do what we need OpenMP to do on entering a parallel region */
  if (NULL != cbfunc) {
    /* tell the event handler that we are only a partial step */
    cbfunc(PMIX_EVENT_PARTIAL_ACTION_TAKEN, NULL, 0, NULL, NULL, cbdata);
  }
}
//<EG END ID="omp_thread">

//<EG BEGIN ID="mpi_thread">
static void parallel_region_MPI_cb(size_t evhdlr_registration_id,
                                   pmix_status_t status,
                                   const pmix_proc_t *source,
                                   pmix_info_t info[], size_t ninfo,
                                   pmix_info_t results[], size_t nresults,
                                   pmix_event_notification_cbfunc_fn_t cbfunc,
                                   void *cbdata) {
  printf("Entered %s\n", __FUNCTION__);
  /* do what we need MPI to do on entering a parallel region */
  if (NULL != cbfunc) {
    /* do what we need MPI to do on entering a parallel region */
    cbfunc(PMIX_EVENT_ACTION_COMPLETE, NULL, 0, NULL, NULL, cbdata);
  }
}
//<EG END ID="mpi_thread">

static int openmp_handler() {
  pmix_info_t *info;
  int rc;

  printf("Entered %s\n", __FUNCTION__);

  //<EG BEGIN ID="omp_thread">
  bool is_true = true;
  pmix_status_t code = PMIX_OPENMP_PARALLEL_ENTERED;
  PMIX_INFO_CREATE(info, 2);
  PMIX_INFO_LOAD(&info[0], PMIX_EVENT_HDLR_NAME, "OpenMP-Master", PMIX_STRING);
  PMIX_INFO_LOAD(&info[1], PMIX_EVENT_HDLR_FIRST, &is_true, PMIX_BOOL);
  rc = PMIx_Register_event_handler(&code, 1, info, 2, parallel_region_OMP_cb, NULL, NULL);
  if (rc < 0)
    fprintf(stderr, "%s: Failed to register event handler for OpenMP region entrance\n", __FUNCTION__);
  PMIX_INFO_FREE(info, 2);
  //<EG END ID="omp_thread">
  printf("Registered OpenMP event handler for OpenMP parallel region entered\n");

  return rc;
}

static int mpi_handler() {
  pmix_info_t *info;
  int rc;

  printf("Entered %s\n", __FUNCTION__);

  //<EG BEGIN ID="mpi_thread">
  pmix_status_t code = PMIX_OPENMP_PARALLEL_ENTERED;
  PMIX_INFO_CREATE(info, 2);
  PMIX_INFO_LOAD(&info[0], PMIX_EVENT_HDLR_NAME, "MPI-Thread", PMIX_STRING);
  PMIX_INFO_LOAD(&info[1], PMIX_EVENT_HDLR_AFTER, "OpenMP-Master", PMIX_STRING);
  rc = PMIx_Register_event_handler(&code, 1, info, 2, parallel_region_MPI_cb, NULL, NULL);
  if (rc < 0)
    fprintf(stderr, "%s: Failed to register event handler for OpenMP region entrance\n", __FUNCTION__);
  PMIX_INFO_FREE(info, 2);
  //<EG END ID="mpi_thread">
  printf("Registered MPI event handler for OpenMP parallel region entered\n");

  return rc;
}

static void notify_complete(pmix_status_t status, void *cbdata)
{
    volatile bool *flag = (volatile bool*)cbdata;
    *flag = true;
}

int main() {
  //<EG BEGIN ID="declare_model">
  pmix_proc_t myproc;
  pmix_info_t *info;
    volatile bool wearedone = false;

  PMIX_INFO_CREATE(info, 4);
  PMIX_INFO_LOAD(&info[0], PMIX_PROGRAMMING_MODEL, "MPI", PMIX_STRING);
  PMIX_INFO_LOAD(&info[1], PMIX_MODEL_LIBRARY_NAME, "FooMPI", PMIX_STRING);
  PMIX_INFO_LOAD(&info[2], PMIX_MODEL_LIBRARY_VERSION, "1.0.0", PMIX_STRING);
  PMIX_INFO_LOAD(&info[3], PMIX_THREADING_MODEL, "pthread", PMIX_STRING);
  pmix_status_t rc = PMIx_Init(&myproc, info, 4);
  PMIX_INFO_FREE(info, 4);
  //<EG END ID="declare_model">
  printf("Registered MPI programming model\n");

  printf("Registering event handler for model declaration\n");
  //<EG BEGIN ID="declare_model_cb">
  pmix_status_t code = PMIX_MODEL_DECLARED;
  rc = PMIx_Register_event_handler(&code, 1, NULL, 0, model_declared_cb, NULL, NULL);
  //<EG END ID="declare_model_cb">
    if (rc < 0) {
        fprintf(stderr, "Failed to register event handler for model declaration\n");
        goto fin;
    }
  printf("Registered event handler for model declaration\n");

  openmp_handler();
  mpi_handler();

  printf("Notifying OpenMP parallel region about to be entered\n");
  //<EG BEGIN ID="notify_event">
    PMIX_INFO_CREATE(info, 1);
    PMIX_INFO_LOAD(&info[0], PMIX_EVENT_NON_DEFAULT, NULL, PMIX_BOOL);
  rc = PMIx_Notify_event(PMIX_OPENMP_PARALLEL_ENTERED, &myproc, PMIX_RANGE_PROC_LOCAL, info, 1, notify_complete, (void*)&wearedone);
    if (rc < 0) {
        fprintf(stderr, "Failed to notify OpenMP region entered\n");
        goto fin;
    }

    while (!wearedone) {
        struct timespec ts;
        ts.tv_sec = 0;
        ts.tv_nsec = 100000;
        nanosleep(&ts, NULL);
    }
    fprintf(stderr, "Test completed\n");

fin:
    PMIx_Finalize(NULL, 0);
  //<EG END ID="notify_event">
}

[rhc54]

I also realized I short-changed you on a description of the errors in this example. Here's a more complete list:

• in your event handler registrations, you defined two pmix_info_t attributes, but you provided a parameter value of ninfo=1 instead of ninfo=2 to the call to PMIx_Register_event_handler. Thus, the second attribute was ignored. Unfortunately, those carried the directives about ordering of the events, which is why only the second was executed - it came first and it returned event_complete.
• the range of your notification about entering the parallel region was PMIX_RANGE_GLOBAL. This instructed PMIx to propagate the event everywhere, which in turn requires that the process be connected to a server. Since we are running this as a singleton instead of under a PMIx server, the library returned an error. I'll have to address this a bit later as I suppose we could argue that "global" means "as far as you can send it", but for now I'm assuming you really only want to notify other threads within the same process. PMIX_RANGE_PROC_LOCAL is the correct range for that.
• while you didn't register a default event handler, it is always a good idea to tell PMIx_Notify_event not to pass the event to a default handler when you have setup specific ones to handle the event being generated.
• as I noted before, you needed to wait for notification to complete before exiting
• the example code failed to call "finalize" before terminating - not a huge deal, but we do want to encourage complete and proper code

[SteVwonder]

Thanks @rhc54! Just pushed a commit with your changes. I confirmed that they worked for me when compiled against the latest master of OpenPMIx and PRRTE.

@jjhursey: I noticed after merging your PR that I'm now getting compilation errors about PMIX_LAUNCH_DIRECTIVE which was re-added in #330. Since this PR is now in the "can only add commits, can't rebase" phase, should we just deal with the rebase/error after the reading/vote has occurred?

[jjhursey]

Yeah we can deal with it after the reading presentation at the 1Q 2021 meeting.

[jjhursey]

Attached is the PDF rendering of the current state of this PR - for the 1Q 2021 reading
pmix-standard.pdf

[SteVwonder]

In preparation of the Q2 meeting, here is the latest copy of the standard after the rebase to fix the missing reference: pmix-standard.pdf

[kathrynmohror]

Congratulations @SteVwonder! Your PR passed the second vote and is accepted into the standard!

[SteVwonder]

Thanks @kathrynmohror!

And many thanks to @jjhursey and @spophale for their contributions!

[jjhursey]

PMIx ASC 2Q 2021: May 11 & May 13

• Passed second vote: 7 yes / 0 no / 1 abstain
• Accepted as stable for v5.0
• I will merge this in today