# -*- text -*-
#
# Copyright (c) 2021-2026 Nanook Consulting  All rights reserved.
# $COPYRIGHT$
#
# Additional copyrights may follow
#
# $HEADER$
#
# This is the US/English general help file for PRRTE's prun.
#
[version]

%s (%s) %s

%s
#
[usage]

%s (%s) %s

Usage: %s [OPTION]...

Submit job to the PMIx Reference RTE

The following list of command line options are available. Note that
more detailed help for any option can be obtained by adding that
option to the help request as "--help <option>".

+----------------------+-----------------------------------------------+
|                      | General Options                               |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "-h" | "--help"      | This help message                             |
+----------------------+-----------------------------------------------+
| "-h" | "--help       | Help for the specified option                 |
| <arg0>"              |                                               |
+----------------------+-----------------------------------------------+
| "-v" | "--verbose"   | Enable typical debug options                  |
+----------------------+-----------------------------------------------+
| "-V" | "--version"   | Print version and exit                        |
+----------------------+-----------------------------------------------+

+----------------------+-----------------------------------------------+
|                      | Debug Options                                 |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "--display <arg0>"   | Comma-delimited list of options for           |
|                      | displaying information about the allocation   |
|                      | and job (see "--help display")                |
+----------------------+-----------------------------------------------+
| "--timeout           | Timeout the job after the specified number of |
| <seconds>"           | seconds                                       |
+----------------------+-----------------------------------------------+
| "--spawn-timeout     | Timeout the job if spawn takes more than the  |
| <seconds>"           | specified number of seconds                   |
+----------------------+-----------------------------------------------+
| "--get-stack-traces" | Get stack traces of all application procs on  |
|                      | timeout                                       |
+----------------------+-----------------------------------------------+
| "--report-state-on-  | Report all job and process states upon        |
| timeout"             | timeout                                       |
+----------------------+-----------------------------------------------+
| "--stop-on-exec"     | If supported, stop each specified process at  |
|                      | start of execution                            |
+----------------------+-----------------------------------------------+
| "--stop-in-init"     | Direct the specified processes to stop in     |
|                      | "PMIx_Init"                                   |
+----------------------+-----------------------------------------------+
| "--stop-in-app"      | Direct the processes to stop at an            |
|      [=breakpoint]   | application-controlled location, optionally   |
|                      | naming the breakpoint at which to stop        |
+----------------------+-----------------------------------------------+
| "--no-aggregate-     | Do not aggregate help messages issued by      |
| help"                | multiple processes - report each one as it is |
|                      | received                                      |
+----------------------+-----------------------------------------------+

+----------------------+-----------------------------------------------+
|                      | Output Options                                |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "--output <args>"    | Comma-delimited list of options that control  |
|                      | how output is generated (see "--help output") |
+----------------------+-----------------------------------------------+
| "--xterm <ranks>"    | Create a new xterm window and display output  |
|                      | from the specified ranks there                |
+----------------------+-----------------------------------------------+

+----------------------+-----------------------------------------------+
|                      | Input Options                                 |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "--stdin <rank>"     | Specify the process to receive stdin ["all",  |
|                      | "none", or a single rank] (default: "0",      |
|                      | indicating rank 0)                            |
+----------------------+-----------------------------------------------+

+----------------------+-----------------------------------------------+
|                      | Placement Options                             |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "--mapby <type>" |   | Mapping Policy for job or for individual app  |
| "--map-by <type>"    |                                               |
+----------------------+-----------------------------------------------+
| "--rankby <type>" |  | Ranking Policy for job or for individual app  |
| "--rank-by <type>"   |                                               |
+----------------------+-----------------------------------------------+
| "--bindto <type>" |  | Binding policy for job or for individual app  |
| "--bind-to <type>"   |                                               |
+----------------------+-----------------------------------------------+

+----------------------+-----------------------------------------------+
|                      | Launch Options                                |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "--rtos <arg0>" |    | Comma-delimited list of runtime directives    |
| "--runtime-options   | for the job (e.g., do not abort if a process  |
| <arg0>"              | exits on non-zero status)                     |
+----------------------+-----------------------------------------------+
| "-c" | "--np         | Number of processes to run                    |
| <num_procs>"         |                                               |
+----------------------+-----------------------------------------------+
| "-n" | "--n          | Number of processes to run                    |
| <num_procs>"         |                                               |
+----------------------+-----------------------------------------------+
| "-N <num_procs>"     | Number of processes to run per node           |
+----------------------+-----------------------------------------------+
| "--app <filename>"   | Provide an appfile listing the application    |
|                      | contexts to launch                            |
+----------------------+-----------------------------------------------+
| "-H" | "--host       | List of hosts to invoke processes on          |
| <hosts>"             |                                               |
+----------------------+-----------------------------------------------+
| "--add-host <hosts>" | List of hosts to add to the DVM prior to      |
|                      | launching the given app                       |
+----------------------+-----------------------------------------------+
| "--activate <hosts>" | List of allocated nodes to bring into the DVM |
|                      | (start a daemon on) prior to launching the    |
|                      | given app                                     |
+----------------------+-----------------------------------------------+
| "--hostfile          | Provide a hostfile                            |
| <filename>"          |                                               |
+----------------------+-----------------------------------------------+
| "--machinefile       | Provide a hostfile (synonym for "--hostfile") |
| <filename>"          |                                               |
+----------------------+-----------------------------------------------+
| "--add-hostfile      | Provide a hostfile listing hosts to add to    |
| <filename>"          | the DVM prior to launching the given app      |
+----------------------+-----------------------------------------------+
| "--path <path>"      | PATH to be used to look for executables to    |
|                      | start processes                               |
+----------------------+-----------------------------------------------+
| "--prtemca <key>     | Pass context-specific PRRTE MCA parameters    |
| <value>"             | ("key" is the parameter name; "value" is the  |
|                      | parameter value)                              |
+----------------------+-----------------------------------------------+
| "--pmixmca <key>     | Pass context-specific PMIx MCA parameters;    |
| <value>"             | they are considered global if only one        |
|                      | context is specified ("key" is the parameter  |
|                      | name; "value" is the parameter value)         |
+----------------------+-----------------------------------------------+
| "--app-pmix-prefix <dir>" | Prefix to be used by app procs to look   |
|                      | for their PMIx installation on remote nodes.  |
|                      | This is the location of the top-level         |
|                      | directory for the installation. In the        |
|                      | absence of providing an application-specific  |
|                      | prefix, the PMIx prefix (if given) used by    |
|                      | PRRTE's own executables will be applied       |
|                      | unless the "--no-app-prefix" directive is     |
|                      | given.                                        |
+----------------------+-----------------------------------------------+
| "--no-app-prefix"    | Do not provide a prefix directive to this     |
|                      | application.                                  |
+----------------------+-----------------------------------------------+
| "--tune <filename>"  | File(s) containing MCA params for tuning      |
|                      | operations                                    |
+----------------------+-----------------------------------------------+
| "--preload-files     | Preload the comma separated list of files to  |
| <filenames>"         | the remote machines current working directory |
|                      | before starting the remote process.           |
+----------------------+-----------------------------------------------+
| "--pset <name>"      | User-specified name assigned to the processes |
|                      | in their given application                    |
+----------------------+-----------------------------------------------+
| "-s" | "--preload-   | Preload the binary on the remote machine      |
| binary"              | before starting the remote process.           |
+----------------------+-----------------------------------------------+
| "--set-cwd-to-       | Set the working directory of the started      |
| session-dir"         | processes to their session directory          |
+----------------------+-----------------------------------------------+
| "--wd <dir>"         | Synonym for "--wdir"                          |
+----------------------+-----------------------------------------------+
| "--wdir <dir>"       | Set the working directory of the started      |
|                      | processes                                     |
+----------------------+-----------------------------------------------+
| "-x <name>"          | Export an environment variable, optionally    |
|                      | specifying a value (e.g., "-x foo" exports    |
|                      | the environment variable foo and takes its    |
|                      | value from the current environment; "-x       |
|                      | foo=bar" exports the environment variable     |
|                      | name foo and sets its value to "bar" in the   |
|                      | started processes; "-x foo*" exports all      |
|                      | current environmental variables starting with |
|                      | "foo")                                        |
+----------------------+-----------------------------------------------+
| "--set-env           | Set the named environmental variable to the   |
|     <name>=<value>"  | specified value. This will overwrite the      |
|                      | existing value, if it exists. Equivalent to   |
|                      | the "-x foo=val" option                       |
+----------------------+-----------------------------------------------+
| "--unset-env <name>" | Unset the named environmental variable. Note  |
|                      | "--unset-env foo*" unsets all current         |
|                      | environmental variables starting with "foo"   |
+----------------------+-----------------------------------------------+
| "--append-env        | Append the named environment variable with    |
| <name[c]> <value>"   | given value. The "[c]" must be appended to    |
|                      | the name to specify the separator to be used  |
|                      | when appending the value.                     |
+----------------------+-----------------------------------------------+
| "--prepend-env       | Prepend the named environment variable with   |
| <name[c]> <value>"   | given value. The "[c]" must be appended to    |
|                      | the name to specify the separator to be used  |
|                      | when prepending the value.                    |
+----------------------+-----------------------------------------------+
| "--gpu-support <val>"| Direct application to either enable (true) or |
|                      | disable (false) its internal library's GPU    |
|                      | support                                       |
+----------------------+-----------------------------------------------+
| "--memory-alloc-     | Comma-delimited list of memory allocation     |
| kinds <kinds>"       | kinds the application requires (passed to the |
|                      | application's library via PMIx)               |
+----------------------+-----------------------------------------------+
| "--alloc-id <val>"   | ID assigned by the host environment to the    |
|                      | allocation to be used for mapping the job     |
+----------------------+-----------------------------------------------+
| "--alloc-refid <val>"| Reference ID assigned by the user when the    |
|                      | allocation to be used for mapping the job     |
|                      | was requested                                 |
+----------------------+-----------------------------------------------+
| "--session-id <val>" | ID of the session whose allocation is to      |
|                      | be used for mapping the job                   |
+----------------------+-----------------------------------------------+

+----------------------+-----------------------------------------------+
|                      | Fault Tolerance Options                       |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "--enable-recovery"  | Enable recovery from process failure          |
|                      | (default: disabled)                           |
+----------------------+-----------------------------------------------+
| "--max-restarts      | Maximum number of times to restart a failed   |
| <num>"               | process                                       |
+----------------------+-----------------------------------------------+
| "--continuous"       | Job is to run until explicitly terminated     |
+----------------------+-----------------------------------------------+

+----------------------+-----------------------------------------------+
|                      | Specific Options                              |
+----------------------+-----------------------------------------------+
| Option               | Description                                   |
|======================|===============================================|
| "--do-not-connect"   | Do not connect to a server                    |
+----------------------+-----------------------------------------------+
| "--dvm-uri <uri>"    | Specify the URI of the DVM master, or the     |
|                      | name of the file (specified as                |
|                      | "file:filename") that contains that info      |
+----------------------+-----------------------------------------------+
| "--namespace <name>" | Namespace of the daemon to which we should    |
|                      | connect                                       |
+----------------------+-----------------------------------------------+
| "--pid <pid>"        | PID of the daemon to which we should connect  |
|                      | (integer PID or "file:<filename>" for file    |
|                      | containing the PID)                           |
+----------------------+-----------------------------------------------+
| "--system-server-    | First look for a system server and connect to |
| first"               | it if found                                   |
+----------------------+-----------------------------------------------+
| "--system-server-    | Connect only to a system-level server         |
| only"                |                                               |
+----------------------+-----------------------------------------------+
| "--wait-to-connect   | Delay specified number of seconds before      |
| <seconds>"           | trying to connect                             |
+----------------------+-----------------------------------------------+
| "--num-connect-      | Max number of times to try to connect         |
| retries <num>"       |                                               |
+----------------------+-----------------------------------------------+
| "--personality       | Specify the personality to be used            |
| <name>"              |                                               |
+----------------------+-----------------------------------------------+
| "--report-pid        | Print out PID on stdout ("-"), stderr ("+"),  |
| <arg0>"              | or a file [anything else]                     |
+----------------------+-----------------------------------------------+
| "--report-uri        | Print out URI on stdout ("-"), stderr ("+"),  |
| <arg0>"              | or a file [anything else]                     |
+----------------------+-----------------------------------------------+
| "--allow-run-as-     | Allow execution as root (**STRONGLY           |
| root"                | DISCOURAGED**)                                |
+----------------------+-----------------------------------------------+
| "--forward-signals   | Comma-delimited list of the signals (names or |
| <signals>"           | integers) to be forwarded to application      |
|                      | processes, replacing the default set ["none"  |
|                      | => forward nothing]. The default set is       |
|                      | "SIGTSTP", "SIGUSR1", "SIGUSR2", "SIGABRT",   |
|                      | "SIGALRM", and "SIGCONT"                      |
+----------------------+-----------------------------------------------+

%s
#
[dvm-uri]

Specify the URI of the DVM controller to connect to, or the name of the
file (specified as "file:<filename>") that contains that info.
#
[num-connect-retries]

Maximum number of times to try to connect to the DVM controller.
#
[pid]

PID of the DVM controller to connect to, given either as an integer or
as "file:<filename>" naming a file that contains the PID.
#
[namespace]

Namespace of the DVM controller to connect to.
#
[system-server-first]

First look for a system-level DVM controller and connect to it if found;
otherwise, fall back to searching for any other DVM controller.
#
[system-server-only]

Connect only to a system-level DVM controller — abort if one is not
found.
#
[wait-to-connect]

Delay the specified number of seconds between attempts to connect to the
DVM controller.
#
[x]

Export an environment variable, optionally specifying a value. For
example:

* "-x foo" exports the environment variable "foo" and takes its value
  from the current environment.

* "-x foo=bar" exports the environment variable name "foo" and sets its
  value to "bar" in the started processes.

* "-x foo*" exports all current environmental variables starting with
  "foo".
#
[set-env]

Set the named environmental variable to the specified value. This will
overwrite the existing value, if it exists. Equivalent to the "-x
foo=val" option.

These directives are applied in the order they are given on the command
line. "--set-env" replaces a value outright while "--prepend-env" and
"--append-env" edit the value already there, so the order is the result:
"--set-env FOO=1 --prepend-env FOO[:] x" leaves "FOO=x:1", while
"--prepend-env FOO[:] x --set-env FOO=1" leaves "FOO=1".
#
[unset-env]

Unset the named environmental variable. Note "--unset-env foo*" unsets
all current environmental variables starting with "foo".

These directives are applied in the order they are given on the command
line. "--set-env" replaces a value outright while "--prepend-env" and
"--append-env" edit the value already there, so the order is the result:
"--set-env FOO=1 --prepend-env FOO[:] x" leaves "FOO=x:1", while
"--prepend-env FOO[:] x --set-env FOO=1" leaves "FOO=1".
#
[append-env]

Append the given value to the named environment variable. The "[c]" must
be appended to the name to specify the separator to be used when
appending the value. For example:

   --append-env LD_LIBRARY_PATH[:] foo/lib

will result in:

   LD_LIBRARY_PATH=$LD_LIBRARY_PATH:foo/lib

These directives are applied in the order they are given on the command
line. "--set-env" replaces a value outright while "--prepend-env" and
"--append-env" edit the value already there, so the order is the result:
"--set-env FOO=1 --prepend-env FOO[:] x" leaves "FOO=x:1", while
"--prepend-env FOO[:] x --set-env FOO=1" leaves "FOO=1".
#
[prepend-env]

Prepend the given value to the named environment variable. The "[c]"
must be appended to the name to specify the separator to be used when
prepending the value. For example:

   --prepend-env LD_LIBRARY_PATH[:] foo/lib

will result in:

   LD_LIBRARY_PATH=foo/lib:$LD_LIBRARY_PATH

These directives are applied in the order they are given on the command
line. "--set-env" replaces a value outright while "--prepend-env" and
"--append-env" edit the value already there, so the order is the result:
"--set-env FOO=1 --prepend-env FOO[:] x" leaves "FOO=x:1", while
"--prepend-env FOO[:] x --set-env FOO=1" leaves "FOO=1".
#
[gpu-support]

Direct the application to either enable ("true") or disable ("false")
its internal library's GPU support. The value is passed to the
application's library via PMIx; a value that is neither true nor false
is refused.
#
[alloc-id]

Direct that the application be executed using the resources of the
allocation identified by the given ID, where that ID is the one the host
environment (e.g., the scheduler) assigned to the allocation when it was
created. The DVM reports this value back to the requester when an
allocation request completes, and it is the value returned by a query of
the allocation.

The named allocation must be one the requester is entitled to use — a
job can only be spawned onto resources its requester has been granted.
Entitlement is by namespace and by user: the namespace that requested
the allocation holds it, as does every job spawned into it, and so does
any tool run by the user the allocation was granted to. That last part
is what lets a later command reach an allocation an earlier one created,
since a tool's namespace lasts only as long as the command that made it.

Naming an allocation the DVM does not know about is an unrecoverable
error and reports "PMIX_ERR_NOT_FOUND"; naming one the requester may not
use is likewise unrecoverable and reports "PMIX_ERR_NO_PERMISSIONS". In
either case the job is not launched.

If no allocation is named, the job is mapped onto the allocation of the
session that requested it (the DVM's default session, in the case of a
tool such as "prun").

Note:

  The same allocation can be named in three different ways — by the
  host-assigned ID given here, by the reference ID the user attached
  to the allocation request ("--alloc-refid"), or by the numerical ID
  of the session that holds it ("--session-id"). These are alternative
  spellings of one directive and therefore cannot be combined on a
  single command line.
#
[alloc-refid]

Direct that the application be executed using the resources of the
allocation identified by the given reference ID, where that ID is the
one *the user* attached to the allocation request at the time the
allocation was made. This is the convenient counterpart to "--alloc-id":
it lets a script name an allocation using a label it chose itself,
without having to capture the identifier the host environment later
assigned to it.

The named allocation must be one the requester is entitled to use — a
job can only be spawned onto resources its requester has been granted.
Entitlement is by namespace and by user: the namespace that requested
the allocation holds it, as does every job spawned into it, and so does
any tool run by the user the allocation was granted to. That last part
is what lets a later command reach an allocation an earlier one created,
since a tool's namespace lasts only as long as the command that made it.

Naming an allocation the DVM does not know about is an unrecoverable
error and reports "PMIX_ERR_NOT_FOUND"; naming one the requester may not
use is likewise unrecoverable and reports "PMIX_ERR_NO_PERMISSIONS". In
either case the job is not launched.

If no allocation is named, the job is mapped onto the allocation of the
session that requested it (the DVM's default session, in the case of a
tool such as "prun").

Note:

  The same allocation can be named in three different ways — by the
  reference ID given here, by the ID the host environment assigned to
  the allocation ("--alloc-id"), or by the numerical ID of the session
  that holds it ("--session-id"). These are alternative spellings of
  one directive and therefore cannot be combined on a single command
  line.
#
[session-id]

Direct that the application be executed using the resources of the
session identified by the given numerical ID. A *session* is the
container the DVM uses to hold an allocation together with the jobs
running against it, and every allocation the DVM grants is assigned one.
The session ID is reported back to the requester when an allocation
request completes.

The value must be an unsigned 32-bit integer; anything else is rejected
before the job is submitted.

The named session must be one the requester is entitled to use — a job
can only be spawned onto resources its requester has been granted.
Entitlement is by namespace and by user: the namespace that requested
the session's allocation holds it, as does every job spawned into it,
and so does any tool run by the user it was granted to. That last part
is what lets a later command reach a session an earlier one created,
since a tool's namespace lasts only as long as the command that made it.

Naming a session the DVM does not know about is an unrecoverable error
and reports "PMIX_ERR_NOT_FOUND"; naming one the requester may not use
is likewise unrecoverable and reports "PMIX_ERR_NO_PERMISSIONS". In
either case the job is not launched.

If no session is named, the job is mapped onto the session that
requested it (the DVM's default session, in the case of a tool such as
"prun").

Note:

  The same allocation can be named in three different ways — by the
  session ID given here, by the ID the host environment assigned to
  the allocation ("--alloc-id"), or by the reference ID the user
  attached to the allocation request ("--alloc-refid"). These are
  alternative spellings of one directive and therefore cannot be
  combined on a single command line.
#
[hostfile]

#include#help-hostfile.txt#hostfile
#
[machinefile]

#include#help-hostfile.txt#machinefile

#
[host]

#include#help-hostfile.txt#host

#
[add-hostfile]

#include#help-hostfile.txt#add-hostfile

#
[add-host]

#include#help-dash-host.txt#add-host

#
[activate]

#include#help-dash-host.txt#activate

#
[np]

Specify the number of application processes to be started for this
application context. "-n", "-c" and "--n" are synonyms for "--np".
#
[report-pid]

Print the PID of this process: on stdout if the argument is "-", on
stderr if it is "+", into the already-open file descriptor it names if
it is a non-negative integer, and otherwise into the file it names.
#
[report-uri]

Print the PMIx contact URI of this process: on stdout if the argument is
"-", on stderr if it is "+", and otherwise into the file it names.
#
[forward-signals]

Comma-delimited list of the signals (names or integers) to be forwarded
to application processes ("none" = forward nothing).

The list *replaces* the default set rather than adding to it, so it
names every signal that is to be forwarded. The default set, used when
this option is not given, is SIGTSTP, SIGUSR1, SIGUSR2, SIGABRT,
SIGALRM, and SIGCONT.
#
[map-by]

#include#help-mapby.txt#map-by

#
[mapby]
#include#help-mapby.txt#mapby

#
[rank-by]

#include#help-rankby.txt#rank-by

#
[rankby]

#include#help-rankby.txt#rankby

#
[bind-to]

#include#help-bindto.txt#bind-to

#
[bindto]

#include#help-bindto.txt#bindto

#
[runtime-options]

#include#help-schizo-rtos.txt#rtos

#
[rtos]

#include#help-schizo-rtos.txt#rtos

#
[notifications]

#include#help-schizo-rtos.txt#notifications

#
[display]

#include#help-schizo-display.txt#display

#
[output]

#include#help-schizo-output.txt#output

#
[placement]

#include#help-placement.txt#placement

#
[placement-all]

#include#help-placement.txt#placement-all

#
[placement-fundamentals]

#include#help-placement.txt#placement-fundamentals

#
[placement-limits]

#include#help-placement.txt#placement-limits
#
[placement-examples]

#include#help-placement.txt#placement-examples
#
[placement-diagnostics]

#include#help-placement.txt#placement-diagnostics
#
[placement-deprecated]

#include#help-placement.txt#placement-deprecated

#
[rankfile]

Name of file to specify explicit task mapping

NOTE: deprecated in favor of "--map-by rankfile:file=<filename>"

#include#help-placement.txt#placement-rankfiles

#
[personality]

Specify the personality to be used. This governs selection of the plugin
responsible for defining and parsing the command line, harvesting and
forwarding environmental variables, and providing library-dependent
support to the launched processes. Examples include "ompi" for an
application compiled with Open MPI, "mpich" for one built against the
MPICH library, or "oshmem" for an OpenSHMEM application compiled against
SUNY's reference library.
#
[pmixmca]

Pass a PMIx MCA parameter

Syntax: "--pmixmca <key> <value>", where "key" is the parameter name and
"value" is the parameter value.
#
[prtemca]

Pass a PRRTE MCA parameter.

Syntax: "--prtemca <key> <value>", where "key" is the parameter name and
"value" is the parameter value.
#
[memory-alloc-kinds]

Comma-delimited list of the memory allocation kinds the application
requires. The list is passed through to the application's library (for
example, an MPI implementation) via PMIx; PRRTE itself does not
interpret the individual kinds.
#
[no-aggregate-help]

Do not aggregate help messages issued by the application processes.

By default, identical help output generated by many processes is
combined into a single report naming the number of processes that hit
it. This option turns that off, so every process's message is reported
as it is received.
#
[enable-recovery]

Enable recovery of failed processes in this job (default: disabled). A
process that fails is restarted, up to the limit given by
"--max-restarts", instead of terminating the job.
#
[max-restarts]

Maximum number of times a failed process in this job is to be restarted.
Only meaningful in combination with "--enable-recovery".
#
[continuous]

The job is to run until explicitly terminated — the runtime is not to
consider it complete when its processes exit.
#
[prun:executable-not-specified]

No executable was specified on the %s command line.

Aborting.
#
[prun:call-failed]

%s encountered a %s call failure.  This should not happen, and usually
indicates an error within the operating system itself. Specifically,
the following error occurred:

   %s

The only other available information that may be helpful is the errno
that was returned: %d.
#
[prun:proc-ordered-abort]

%s has exited due to process rank %lu with PID %lu on node %s calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by %s (as reported here).
#
[prun:proc-killed-by-release]

%s has exited because process rank %lu on node %s was killed when that
node was released from the DVM. Releasing a node removes its daemon,
which kills the application processes still running under it.

The node was released on request - by a process, a tool, or the
scheduler. Nothing drains a node before releasing it.

Any remaining processes in this job were terminated by %s. A job that
can carry on without the lost processes can say so with
"--runtime-options recoverable" and will not be terminated.
#
[prun:proc-exit-no-sync]

%s has exited due to process rank %lu with PID %lu on node %s exiting
improperly. There are three reasons this could occur:

1. this process did not call "init" before exiting, but others in the
job did. This can cause a job to hang indefinitely while it waits for
all processes to call "init". By rule, if one process calls "init",
then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
By rule, all processes that call "init" MUST call "finalize" prior to
exiting or it will be considered an "abnormal termination"

3. this process called "MPI_Abort" or "prte_abort" and the mca
parameter prte_create_session_dirs is set to false. In this case, the
run-time cannot detect that the abort call was an abnormal
termination. Hence, the only error message you will receive is this
one.

This may have caused other processes in the application to be
terminated by signals sent by %s (as reported here).

You can avoid this message by specifying -quiet on the %s command
line.
#
[prun:proc-aborted]

%s noticed that process rank %lu with PID %lu on node %s exited on
signal %d.
#
[prun:proc-aborted-strsignal]

%s noticed that process rank %lu with PID %lu on node %s exited on
signal %d (%s).
#
[prun:sys-limit-pipe]

%s was unable to launch the specified application as it encountered an
error:

Error: system limit exceeded on number of pipes that can be open

   Node: %s

when attempting to start process rank %lu.

This can be resolved by setting the mca parameter
opal_set_max_sys_limits to 1, increasing your limit descriptor setting
(using limit or ulimit commands), asking the system administrator for
that node to increase the system limit, or by rearranging your
processes to place fewer of them on that node.
#
[prun:sys-limit-files]

%s was unable to launch the specified application as it encountered an
error:

Error: system limit exceeded on number of files that can be open

   Node: %s

when attempting to start process rank %lu.

This can be resolved by setting the mca parameter
opal_set_max_sys_limits to 1, increasing your limit descriptor setting
(using limit or ulimit commands), asking the system administrator for
that node to increase the system limit, or by rearranging your
processes to place fewer of them on that node.
#
[prun:pipe-setup-failure]

%s was unable to launch the specified application as it encountered an
error:

Error: pipe function call failed when setting up I/O forwarding
subsystem

   Node: %s

while attempting to start process rank %lu.
#
[prun:sys-limit-children]

%s was unable to launch the specified application as it encountered an
error:

Error: system limit exceeded on number of processes that can be
started

   Node: %s

when attempting to start process rank %lu.

This can be resolved by either asking the system administrator for
that node to increase the system limit, or by rearranging your
processes to place fewer of them on that node.
#
[prun:failed-term-attrs]

%s was unable to launch the specified application as it encountered an
error:

Error: reading tty attributes function call failed while setting up
I/O forwarding system

   Node: %s

while attempting to start process rank %lu.
#
[prun:preload-collision]

%s was unable to launch the specified application because a file it was
asked to preload could not be placed in the working directory: something
else of that name is already there, and its contents are not what was to
be staged.

   Working directory: %s
   Node: %s

while attempting to start process rank %lu.

PRRTE will not overwrite a file it did not put there. Remove or rename
the file that is in the way, give the job a different working directory
(see the "--wdir" option), or preload the file under a name that does
not collide, and try again.
#
[prun:wdir-not-found]

%s was unable to launch the specified application as it could not find
the specified working directory:

   Working directory: %s
   Node: %s

while attempting to start process rank %lu.
#
[prun:wdir-not-accessible]

%s was unable to launch the specified application as it could not
access the specified working directory:

   Working directory: %s
   Node: %s

while attempting to start process rank %lu.
#
[prun:exe-not-found]

%s was unable to find the specified executable file, and therefore did
not launch the job.  This error was first reported for process rank
%lu; it may have occurred for other processes as well.

NOTE: A common cause for this error is misspelling a %s command
   line parameter option (remember that %s interprets the first
   unrecognized command line token as the executable).

   Node:       %s
   Executable: %s
#
[prun:exe-not-accessible]

%s was unable to launch the specified application as it could not
access an executable:

   Executable: %s
   Node: %s

while attempting to start process rank %lu.
#
[prun:pipe-read-failure]

%s was unable to launch the specified application as it encountered an
error:

Error: reading from a pipe function call failed while spawning a local
process

   Node: %s

while attempting to start process rank %lu.
#
[prun:proc-failed-to-start]

%s was unable to start the specified application as it encountered an
error:

   Error code: %d
   Error name: %s
   Node: %s

when attempting to start process rank %lu.
#
[prun:proc-failed-to-start-no-status]

%s was unable to start the specified application as it encountered an
error on node %s. More information may be available above.
#
[prun:ompi-server-filename-bad]

%s was unable to parse the filename where contact info for the pubsub-
server was to be found. The option we were given was:

   --pubsub-server %s

This appears to be missing the required ':' following the keyword
"file". Please remember that the correct format for this command line
option is:

   --pubsub-server file:path-to-file

where path-to-file can be either relative to the cwd or absolute.
#
[prun:ompi-server-filename-missing]

%s was unable to parse the filename where contact info for the pubsub-
server was to be found. The option we were given was:

   --pubsub-server %s

This appears to be missing a filename following the ':'. Please
remember that the correct format for this command line option is:

   --pubsub-server file:path-to-file

where path-to-file can be either relative to the cwd or absolute.
#
[prun:ompi-server-filename-access]

%s was unable to access the filename where contact info for the pubsub-
server was to be found. The option we were given was:

   --pubsub-server %s

Please remember that the correct format for this command line option
is:

   --pubsub-server file:path-to-file

where path-to-file can be either relative to the cwd or absolute, and
that you must have read access permissions to that file.
#
[prun:ompi-server-file-bad]

%s was unable to read the pubsub-server's contact info from the given
filename. The filename we were given was:

FILE: %s

Please remember that the correct format for this command line option
is:

--pubsub-server file:path-to-file

where path-to-file can be either relative to the cwd or absolute, and
that the file must have a single line in it that contains the PRTE uri
for the pubsub-server. Note that this is *not* a standard uri, but a
special format used internally by PRTE for communications. It can best
be generated by simply directing the pubsub-server to put its uri in a
file, and then giving %s that filename.
#
[prun:multiple-paffinity-schemes]

Multiple processor affinity schemes were specified (can only specify
one):

   Slot list: %s
   prte_paffinity_alone:  true

Please specify only the one desired method.
#
[prun:invalid-node-rank]

An invalid node rank was obtained - this is probably something that
should be reported to the PRRTE developers.
#
[prun:invalid-local-rank]

An invalid local rank was obtained - this is probably something that
should be reported to the PRRTE developers.
#
[prun:invalid-phys-cpu]

An invalid physical processor id was returned when attempting to set
processor affinity - please check to ensure that your system supports
such functionality. If so, then this is probably something that should
be reported to the PRRTE developers.
#
[prun:topo-not-supported]

An attempt was made to bind a process to a specific hardware topology
mapping (e.g., binding to a package) but the operating system does not
support such topology-aware actions.  Talk to your local system
administrator to find out if your system can support topology-aware
functionality (e.g., Linux Kernels newer than v2.6.18).

Systems that do not support processor topology-aware functionality
cannot use "bind to package" and other related functionality.

   Local host:        %s
   Action attempted:  %s %s
   Application name:  %s
#
[prun:not-enough-resources]

Not enough %s were found on the local host to meet the requested
binding action:

   Local host:        %s
   Action requested:  %s
   Application name:  %s

Please revise the request and try again.
#
[prun:invalid-slot-list-range]

A slot list was provided that exceeds the boundaries on available
resources:

   Local host: %s
   Slot list:  %s

Please check your boundaries and try again.
#
[prun:proc-comm-failed]

A critical communication path was lost to:

   My name:      %s
   Process name: %s
   Node:         %s
#
[prun:proc-mem-exceeded]

A process exceeded memory limits:

   Process name: %s
   Node:         %s
#
[prun:proc-stalled]

One or more processes appear to have stalled - a monitored file failed
to show the required activity.
#
[prun:proc-sensor-exceeded]

One or more processes have exceeded a specified sensor limit, but no
further info is available.
#
[prun:proc-heartbeat-failed]

%s failed to receive scheduled heartbeat communications from a remote
process:

   Process name: %s
   Node:         %s
#
[prun:non-zero-exit]

%s detected that one or more processes exited with non-zero status,
thus causing the job to be terminated. The first process to do so was:

   Process name: %s
   Exit code:    %d
#
[prun:negative-nprocs]

%s has detected that one or more applications was given a negative
number of processes to run:

   Application:  %s
   Num procs:    %d

Please correct this value and try again.
#
[prun:stop-on-exec]

%s was unable to stop the executable at first instruction:

   Error:     %s
   Nodename:  %s
   Rank:      %lu
#
[prun:executable-incorrectly-given]

The %s command was given with an application specified. %s is only
used to start the persistent DVM - it cannot be used with an
application.
#
[bad-option-input]

%s was given an option that expected a string argument:

   option: %s
   argument: %s
   expected: %s

Please correct the option and try again.
#
[file-open-error]

%s was unable to open the specified file provided as an option:

   option: %s
   argument: %s
   file: %s

Please correct the option and try again.
#
[bad-file]

%s was unable to read the necessary info from the provided file:

   option: %s
   argument: %s
   file: %s

Please correct the option or the file and try again.
#
[bad-dvm-option]

The --dvm <arg> option was provided, but the argument is one not
recognized:

   arg: %s

Please see "%s --help dvm" for a list of recognized options and
examples
#
[appfile-failure]

The --app <arg> option was provided, but the specified file was not
found or could not be opened:

   file: %s

Please correct the option and try again.
#
[appfile-with-app]

The --app <arg> option was provided together with an application on
the command line:

   file:        %s
   application: %s

An appfile supplies every application context of the job, so the
command line may not name one as well - the file's first line would
otherwise be appended to the command-line application's own arguments.
Options may still be given alongside --app: they are applied as if
written in front of the file's first line, so a job-level option (for
example, "--mapby") describes the whole job, while an option the
file's first line also gives (for example, "-n") is refused as a
duplicate.

Please move the application into the appfile, or drop --app, and try
again.
#
[N]

Specify the number of application processes to be started on each node.
"-N <num>" is shorthand for the placement directive "--mapby
ppr:<num>:node", and is converted to it.
#
[stop-on-exec]

If supported by the platform, stop each application process immediately
upon exec'ing it, pending release by a debugger. The directive applies
to all processes in the job. This is the same as "--rtos stop-on-exec".
#
[stop-in-init]

Include the "PMIX_DEBUG_STOP_IN_INIT" attribute in the application's job
info directing that the processes stop in "PMIx_Init" pending release.
The directive applies to all processes in the job. This is the same as
"--rtos stop-in-init".
#
[stop-in-app]

Include the "PMIX_DEBUG_STOP_IN_APP" attribute in the application's job
info directing that the processes stop at an application-determined
point pending release. The directive applies to all processes in the
job. This is the same as "--rtos stop-in-app".

An optional argument names the one breakpoint at which they are to stop
— e.g., "--stop-in-app=mpi-init". PRRTE cannot know where any given
breakpoint lives; all it can do is pass the name to the application in
the "PMIX_BREAKPOINT" environment variable and then wait for the "ready
for debug" event the application generates when it gets there. It is
therefore up to the application to recognize the name and stop in the
corresponding place. Given without an argument, the processes stop at
whichever such place they reach first.

Since the argument is read as a boolean when it spells one, a breakpoint
cannot be named "true", "false", or any other spelling of a truth value.
#
[wdir]

Set the working directory of the started processes. A relative path is
taken relative to the current working directory of the command, and
converted to an absolute path. Without this option (or
"--set-cwd-to-session-dir"), the processes start in the command's
current working directory. "--wd" is a synonym for "--wdir".
#
[wd]

Synonym for "--wdir".
#
[set-cwd-to-session-dir]

Set the working directory of the started processes to their session
directory. This is ignored if "--wdir" is also given.
#
[path]

Directory in which to find the executable of this application context:
the executable named on the command line is taken relative to it. A
relative directory is taken relative to the current working directory of
the command, and converted to an absolute path.
#
[pset]

User-specified name assigned to the processes in their given application
context. The processes are told it as the name of the PMIx process set
("PMIX_PSET_NAME") they belong to.
#
[preload-files]

Syntax: "--preload-files <files>"

Preload the comma-separated list of files to the remote machines'
current working directory before starting the remote process.
#
[preload-binary]

Syntax: "-s" or "--preload-binary"

Preload the binary on the remote machine before starting the remote
process.
#
[stdin]

Specify which application process is to receive the command's stdin:
"all", "none", or the rank of a single process (default: "0", indicating
rank 0). Any other value is refused.
#
[spawn-timeout]

Timeout the job if spawn takes more than the specified number of
seconds.
#
[report-state-on-timeout]

Report all job and process states upon timeout.
#
[get-stack-traces]

Get stack traces of all application processes still executing upon
timeout.
#
[do-not-connect]

Do not connect to a DVM controller.
#
[app-pmix-prefix]

Prefix to be used by an app to look for its PMIx installation on remote
nodes. This is the location of the top-level directory for the
installation. If the installation has not been moved, it would be the
value given to "--prefix" when the installation was configured.

Note that PRRTE cannot determine the exact name of the library
subdirectory under this location. For example, some systems will call it
"lib" while others call it "lib64". Accordingly, PRRTE will use the
library subdirectory name of the PMIx installation used to build PRRTE.

In the absence of providing an application-specific prefix, the PMIx
prefix (if given) used by PRRTE's own executables will be applied unless
the "--no-app-prefix" directive is given.
#
[no-app-prefix]

Do not apply any prefix to this application. This is needed when a
default PMIx prefix has been given to PRRTE, but the application has
been built against a PMIx library that (a) is different from the one
used by PRRTE, and (b) was not moved. Otherwise, PRRTE will apply its
default prefix to the application.
#
[allow-run-as-root]

Allow execution as root **(STRONGLY DISCOURAGED)**.

Running as root exposes the user to potentially catastrophic file system
corruption and damage — e.g., if the user accidentally points the root
of the session directory to a system required point, this directory and
all underlying elements will be deleted upon job completion, thereby
rendering the system inoperable.

It is recognized that some environments (e.g., containers) may require
operation as root, and that the user accepts the risks in those
scenarios. Accordingly, one can override PRRTE's run-as-root protection
by providing one of the following:

* The "--allow-run-as-root" command line directive

* Adding **BOTH** of the following environmental parameters:

    * "PRTE_ALLOW_RUN_AS_ROOT=1"
    * "PRTE_ALLOW_RUN_AS_ROOT_CONFIRM=1"

Again, we recommend this only be done if absolutely necessary.
#
[timeout]

Timeout the job if execution is not complete after the specified number
of seconds. The value must be a non-negative integer; zero means no
timeout. If this option is not given, the value of the "MPIEXEC_TIMEOUT"
environment variable, if set, is used instead. See also
"--report-state-on-timeout" and "--get-stack-traces".
#
[app]

Provide an appfile describing the application contexts to be launched.
Each line of the file that is neither blank nor a comment (a line whose
first non-blank character is "#") describes one application context,
written as it would be on the command line: the options that apply to
that context, followed by the executable and its arguments. Lines are
split at spaces; quotes are not interpreted.

Options given on the command line alongside "--app" are combined with
the first application context in the file, so job-level options (such as
"--mapby") apply to the whole job as usual, while an option that is also
given on the file's first line is refused as a duplicate. The command
line may not also name an application — an executable, or a
":"-separated application context — as well.
#
[xterm]

Display the output of the specified application processes in their own
xterm window. Ranks are given as a comma-delimited list of ranks and
inclusive ranges (for example, "1,3-6,9"), or as "all". A trailing "!"
(for example, "1,3!") keeps each window open after its process exits.
The xterm is started by the daemon on the node where the process runs,
so it needs a usable "DISPLAY" there: forward it with "-x DISPLAY", or
start the DVM with an "ssh -X" launch agent.
#
[tune]

Comma-delimited list of one or more files containing MCA parameters for
tuning DVM and/or application operations. The option may be given more
than once. A file named by a relative path is looked for first in the
current directory and then among the parameter sets installed with
PRRTE.

Syntax in the file is:

   param = value

with one parameter per line. Empty lines and lines beginning with the
"#" character are ignored, as is any whitespace around the "="
character. Quotes around the value are removed.

Each parameter is a *generic* MCA parameter, so it is treated exactly
like "--mca param value": it is applied to PRRTE if it belongs to a
PRRTE framework, and otherwise to PMIx. A parameter that belongs to
neither is an error, as is a parameter given twice with different
values. A parameter given explicitly on the command line ("--prtemca",
"--pmixmca") overrides the same parameter in a tune file.