prex - control tracing and manipulate probe points in a pro-
     cess or the kernel


     prex  [-o trace_file_name]  [-l libraries]  [-s kbytes_size]
     cmd [cmd-args...]

     prex  [-o trace_file_name]  [-l libraries]  [-s kbytes_size]
     -p pid

     prex -k [-s kbytes_size]


     The prex command is the part of the Solaris  tracing  archi-
     tecture that controls probes in a process or the kernel. See
     tracing(3TNF) for an overview of this tracing  architecture,
     including example source code using it.

     prex is the application used for external control of probes.
     It  automatically  preloads  the  libtnfprobe  library. prex
     locates all the probes in a target executable or the  kernel
     and  provides  an interface for the user to manipulate them.
     It allows a probe to be turned on for tracing, debugging, or
     both. Tracing generates a TNF (Trace Normal Form) trace file
     that can be converted to ASCII by tnfdump(1)  and  used  for
     performance analysis. Debugging generates a line to standard
     error whenever the probe is hit at run time.

     prex does not work on static executables. It only  works  on
     dynamic executables.

  Invoking prex
     There are three ways to invoke prex:

     1. Use prex to start the target  application  cmd.  In  this
        case,  the  target  application  need not be built with a
        dependency on libtnfprobe. See TNF_PROBE(3TNF). prex sets
        the  environment  variable LD_PRELOAD to load libtnfprobe
        into the target process. See ld(1). prex  then  uses  the
        environment variable PATH to find the target application.

     2. Attach prex to a running application. In this  case,  the
        running   target   application  should  have  libtnfprobe
        already linked in. Alternatively, the user  may  manually
        set  LD_PRELOAD  to  include  prior  to
        invoking the target.

     3. Use prex with the -k option to set prex to  kernel  mode.
        prex  can  then  be used to control probes in the Solaris
        kernel. In kernel mode, additional commands are  defined,
        and  some  commands  that  are  valid  in other modes are
        invalid. See Kernel Mode below.

  Control File Format and Command Language
     In a future release of prex, the  command  language  may  be
     moved to a syntax that is supported by an existing scripting
     language like ksh(1). In the meantime, the interface to prex
     is uncommitted.

        o  Commands should be in ASCII.

        o  Each command is terminated with the NEWLINE character.

        o  A command can be continued onto the next line by  end-
           ing the previous line with the backslash ("\") charac-

        o  Tokens in a command must be  separated  by  whitespace
           (one or more spaces or tabs).

        o  The "#" character implies that the rest of the line is
           a comment.

  Basic prex Commands
     Command                       Result
     % prex a.out                  Attaches prex to your  pro-
                                   gram and starts prex.
    prex> enable $all             Enables all the probes.
    prex> quit resume             Quits prex and resumes exe-
                                  cution of program.

  Control File Search Path
     There are two different methods of communicating with prex:

        o  By specifications in a control file. During  start-up,
           prex  searches for a file named  .prexrc in the direc-
           tories specified below. prex  does  not  stop  at  the
           first  one  it finds. This way a user can override any
           defaults that are set up. The search order is:


        o  By typing commands at the prex prompt.

     The command language for both methods is  the  same  and  is
     specified in USAGE. The commands that return output will not
     make sense in a control file. The output will go to standard

     When using prex on a target process, the target will  be  in
     one  of two states, running or stopped. This can be detected
     by the presence or absence  of  the  prex>  prompt.  If  the
     prompt  is  absent, it means that the target process is run-
     ning. Typing Control-C will stop the  target  pr  ocess  and
     return  the  user  to the prompt. There is no guarantee that
     Control-C will return to  a  prex  prompt  immediately.  For
     example,  if  the target process is stopped on a job control
     stop (SIGSTOP), then Control-C in prex will wait  until  the
     target  has  been continued (SIGCONT). See Signals to Target
     Program below for more information on signals and the target


     The following options are supported:

     -k    kernel mode: prex is used to  control  probes  in  the
           Solaris  kernel.  In  kernel mode, additional commands
           are defined, and some commands valid  in  other  modes
           are invalid. See Kernel Mode  below.

     -l libraries
           The libraries mentioned are linked in  to  the  target
           application  using LD_PRELOAD (see ld(1)). This option
           cannot be used when attaching to  a  running  process.
           The  argument  to  the  -l  option  should be a space-
           separated string enclosed in double quotes. Each token
           in  the  string  is  a  library  name.  It follows the
           LD_PRELOAD rules on how libraries should be  specified
           and where they will be found.

     -o trace_file_name
           File to be used for the trace output.  trace_file_name
           is  assumed  to  be  relative  to  the current working
           directory of prex (that is,  the  directory  that  the
           user was in when prex was started).

           If prex attaches to a process that is already tracing,
           the  new  trace_file_name  (if  provided)  will not be
           used. If no trace_file_name is specified, the  default
           is  /$TMPDIR/trace-pid  where pid is the process id of
           the target program. If TMPDIR  is  not  set,  /tmp  is

     -s kbytes_size
           Maximum size of the output trace file in  Kbytes.  The
           default  size of the trace kbytes_size is 4096 (2**10)
           bytes or 4 Mbytes for normal usage,  and  384  or  384
           kbytes  in  kernel  mode. The minimum size that can be
           specified is 128 Kbytes. The trace file can be thought
           of  as a least recently used circular buffer. Once the
           file has been filled, newer events will overwrite  the
           older ones.


     This section describes the usage of the prex utility.

     Probes are specified by a list of space-separated selectors.
     Selectors are of the form:


     (See TNF_PROBE(3TNF)). The "attribute=" is optional.  If  it
     is not specified, it defaults to "keys=".

     The attribute or value (generically called  "spec")  can  be
     any of the following:

     IDENT Any sequence of letters, digits, _, \, ., % not begin-
           ning with a digit. IDENT implies an exact match.

           Usually used to escape reserved words (any commands in
           the  command  language).  QUOTED_STR  implies an exact
           match and has to be enclosed in single quotes (' ').

           An ed(1) regular expression pattern match. REGEXP  has
           to  be  enclosed in slashes (/ /), A / can be included
           in a REGEXP by escaping it with a backslash \.

     The following grammar explains the syntax.

     selector_list ::=   |                /* empty */
                         selector_list selector
     selector ::=        spec=spec |  /* whitespace around `=' opt */
     spec ::=            IDENT |
                         QUOTED_STR |

     The terminals in the above grammar are:

     IDENT =       [a-zA-Z_\.%]{[a-zA-Z0-9_\.%]}+
     QUOTED_STR =  '[^\n']*'   /* any string in single quotes */
     REGEXP =      /[^\n/]*/   /* regexp's have to be in / / */

     This is a list of the remaining grammar that  is  needed  to
     understand  the  syntax  of the command language (defined in
     next subsection):

     filename ::=     QUOTED_STR    /* QUOTED_STR defined above */
     spec_list ::=    /* empty */ |
                      spec_list spec  /* spec defined above */
     fcn_handle ::=   &IDENT        /* IDENT defined above */
     set_name ::=     $IDENT        /* IDENT defined above */

  Command Language
     1. Set Creation and Set Listing

        create $set_name selector_list
        list     sets          # list the defined sets

        create can be used to define a set which contains  probes
        that match the selector_list. The set $all is pre-defined
        as /.*/ and it matches all the probes.

     2. Function Listing

        list     fcns        # list the available fcn_handle

        The user can list the different  functions  that  can  be
        connected  to  probe  points.  Currently,  only the debug
        function called &debug is available.

     3. Commands to Connect and Disconnect Probe Functions

        connect &fcn_handle $set_name
        connect &fcn_handle selector_list
        clear $set_name
        clear selector_list

        The connect command is used to  connect  probe  functions
        (which must be prefixed by `&') to probes. The probes are
        specified either as a single set  (with  a  `$'),  or  by
        explicitly  listing  the  probe selectors in the command.
        The probe function has to be one that is  listed  by  the
        list  fcns  command.  This  command  does  not enable the

        The clear command is used  to  disconnect  all  connected
        probe functions from the specified probes.

     4. Commands to Toggle the Tracing Mode

        trace $set_name
        trace selector_list
        untrace $set_name
        untrace selector_list

        The trace and untrace commands are  used  to  toggle  the
        tracing action of a probe point (that is, whether a probe
        will emit a trace record or not if it is hit). This  com-
        mand  does  not  enable the probes specified. Probes have
        tracing on by default. The most efficient way to turn off
        tracing  is by using the disable command. untrace is use-
        ful if you want debug output but no tracing.  If so,  set
        the  state  of  the  probe  to enabled, untraced, and the
        debug function connected.

     5. Commands to Enable and Disable Probes

        enable $set_name
        enable selector_list
        disable $set_name
        disable selector_list

        The enable and  disable  commands  are  used  to  control
        whether the probes perform the action that they have been
        set up for. To trace a probe, it has to be  both  enabled
        and traced (using the trace command). Probes are disabled
        by default. The list history command is used to list  the
        probe  control  commands  issued:  connect, clear, trace,
        untrace, enable, and  disable.  These  are  the  commands
        that are executed whenever a new shared object is brought
        in to the target program by dlopen(3DL). See the  subsec-
        tion, dlopen'ed Libraries, below for more information.

        The following table shows the actions  that  result  from
        specific  combinations of tracing, enabling, and connect-

        Enabled or   Tracing State     Debug State        Results
        Disabled       (On/Off)     (Connected/Cleared)    In
        Enabled          On             Connected        Tracing and

        Enabled          On             Cleared          Tracing only

        Enabled          Off            Connected        Debugging only

        Enabled          Off            Cleared          Nothing

        Disabled         On             Connected        Nothing

        Disabled         On             Cleared          Nothing

        Disabled         Off            Connected        Nothing

        Disabled         Off            Cleared          Nothing

     6. List History

        list history                  # lists probe control command history

        The list history command displays a  list  of  the  probe
        control  commands  previously  issued in the tracing ses-
        sion, for example, connect, clear, trace,  disable.  Com-
        mands  in  the  history  list are executed wherever a new
        shared object is  brought  into  the  target  program  by

     7. Commands to List Probes, List Values, or List Trace  File

        list spec_list probes $set_name    # list probes $all
        list spec_list probes selector_list   # list name probes file=test.c
        list values spec_list            # list values keys given in spec_list
        list tracefile                # list tracefile

        The first two commands list the selected  attributes  and
        values of the specified probes. They can be used to check
        the state of a probe. The third command lists the various
        values  associated  with  the  selected  attributes.  The
        fourth command lists the current tracefile.

     8. Help Command

        help topic

        To get a list of the  help  topics  that  are  available,
        invoke  the  help  command  with no arguments. If a topic
        argument is specified, help is printed for that topic.

     9. Source a File

        source filename

        The source command can be used to source a file  of  prex
        commands.  source  can  be  nested  (that  is, a file can
        source another file). filename is a quoted string.

        Process Control

        continue           # resumes the target process
        quit kill          # quit prex, kill target
        quit resume        # quit prex, continue target
        quit suspend       # quit prex, leave target suspended
        quit               # quit prex (continue or kill target)

        The default quit will continue the target process if prex
        attached  to  it. Instead, if prex had started the target
        program, quit will kill the target process.

  dlopen'ed Libraries
     Probes in shared objects that are brought in by  dlopen(3DL)
     are automatically set up according to the command history of
     prex. When a shared object is  removed  by  a  dlclose(3DL),
     prex  again needs to refresh its understanding of the probes
     in the target program. This implies that there is more  work
     to  do  for  dlopen(3DL) and dlclose(3DL) -so they will take
     slightly longer. If a user is not interested in this feature
     and   doesn't   want   to  interfere  with  dlopen(3DL)  and
     dlclose(3DL), detach prex from the target  to  inhibit  this

  Signals to Target Program
     prex does not interfere  with  signals  that  are  delivered
     directly  to  the target program. However, prex receives all
     signals normally generated from the terminal,  for  example,
     Control-C  (SIGINT),  and  Control-Z (SIGSTOP), and does not
     forward them to the target program.  To  signal  the  target
     program, use the kill(1) command from a shell.

  Interactions with Other Applications
     Process managing applications like dbx, truss(1),  and  prex
     cannot  operate  on  the same target program simultaneously.
     prex will not be able to attach to a target which  is  being
     controlled  by  another  application.  A  user can trace and
     debug a program serially  by  the  following  method:  first
     attach prex to target (or start target through prex), set up
     the probes using the command language, and  then  type  quit
     suspend.  The user can then attach dbx to the suspended pro-
     cess and debug it. A user can also  suspend  the  target  by
     sending  it a SIGSTOP signal, and then by typing quit resume
     to prex. In this case, the user should also send  a  SIGCONT
     signal  after  invoking dbx on the stopped process (else dbx
     will be hung).

  Failure of Event Writing Operations
     There are a few failure points that are possible when  writ-
     ing  out  events  to  a trace file, for example, system call
     failures. These failures result in a failure code being  set
     in  the  target  process.  The target process continues nor-
     mally, but no trace records are  written.  Whenever  a  user
     enters  Control-C to prex to get to a prex prompt, prex will
     check the failure code in the target and inform the user  if
     there was a tracing failure.

  Target Executing a Fork or exec
     If the target program does a fork(2), any  probes  that  the
     child  encounters will cause events to be logged to the same
     trace file. Events are annotated with a process  id,  so  it
     will  be  possible  to  determine which process a particular
     event came from. In multi-threaded programs, there is a race
     condition with a thread doing a fork while the other threads
     are still running.  For the trace file not to get corrupted,
     the  user  should either use fork1(2), or make sure that all
     other threads are quiescent when doing a fork(2),

     If the target  program  itself  (not  any  children  it  may
     fork(2))  does an exec(2), prex detaches from the target and
     exits. The user can reconnect prex with prex -p pid.

     A vfork(2) is generally followed quickly by an   exec(2)  in
     the  child,  and  in  the  interim,  the  child  borrows the
     parent's process while the parent waits for the exec(2). Any
     events  logged  by  the  child  from the parent process will
     appear to have been logged by the parent.

  Kernel Mode
     Invoking prex with the -k flag causes prex to run in  kernel
     mode.  In  kernel  mode, prex controls probes in the Solaris
     kernel. See tnf_kernel_probes(4) for  a  list  of  available
     probes  in  the Solaris kernel. A few prex commands are una-
     vailable in kernel mode; many other commands  are  valid  in
     kernel mode only.

     The -l, -o, and -p command-line options  are  not  valid  in
     kernel  mode  (that is, they may not be combined with the -k

     The rest of this section describes the  differences  in  the
     prex command language when running prex in kernel mode.

     1. prex will not stop the kernel

        When prex attaches to a running user  program,  it  stops
        the  user  program.  Obviously,  it  cannot  do this when
        attaching  to  the  kernel.   Instead,  prex  provides  a
        ``tracing  master  switch'':   no  probes  will  have any
        effect unless the tracing  master  switch  is  on.   This
        allows  the  user to iteratively select probes to enable,
        then enable them all at once by  turning  on  the  master

        The command

        ktrace [ on | off ]

        is used to inspect  and  set  the  value  of  the  master
        switch.  Without  an  argument,  prex reports the current
        state of the master switch.

        Since prex will not stop or kill the kernel, the

        quit resume


        quit kill

        commands are not valid in kernel mode.

     2. No functions may be attached to probes in the kernel

        In particular, the debug function is unavailable in  ker-
        nel mode.

     3. Trace output is written to an in-core buffer

        In kernel mode, a trace  output  file  is  not  generated
        directly,  in order to allow probes to be placed in time-
        critical code. Instead, trace output  is  written  to  an
        in-core  buffer,  and  copied  out by a separate program,

        The in-core buffer is not automatically created. The fol-
        lowing  prex command controls buffer allocation and deal-

        buffer [  alloc [  size ] |  dealloc ]

        Without an argument, the buffer command reports the  size
        of  the currently allocated buffer, if any. With an argu-
        ment of alloc [size], prex  allocates  a  buffer  of  the
        given  size. size is in bytes, with an optional suffix of
        'k' or 'm' specifying a multiplier of  1024  or  1048576,
        respectively.  If no  size is specified, the  size speci-
        fied on the command line with the -s option is used as  a
        default.  If the -s command line option was not used, the
        ``default default'' is 384 kilobytes.

        With an argument of dealloc, prex deallocates  the  trace
        buffer in the kernel.

        prex will reject attempts  to  turn  the  tracing  master
        switch  on when no buffer is allocated, and to deallocate
        the buffer when the tracing master  switch  is  on.  prex
        will  refuse  to  allocate  a  buffer when one is already
        allocated; use buffer dealloc first.

        prex will not allocate a buffer larger than one-half of a
        machine's physical memory.

     4. prex supports per-process probe enabling in the kernel

        In kernel mode,  it  is  possible  to  select  a  set  of
        processes  for  which probes are enabled. No trace output
        will be written when other processes traverse these probe
        points. This is called "process filter mode". By default,
        process filter mode is off, and all processes  cause  the
        generation  of  trace  records  when  they hit an enabled

        Some kernel events such as interrupts cannot  be  associ-
        ated  with  a  particular  user  process.  By convention,
        these events are considered to be generated by process id

        prex provides commands to turn process filter mode on and
        off, to get the current status of the process filter mode
        switch, to add and delete processes (by process id)  from
        the  process  filter set, and to list the current process
        filter set.

        The process filter set is maintained  even  when  process
        filter  mode  is  off,  but  has no effect unless process
        filter mode is on.

        When a process in the process filter set exits, its  pro-
        cess  id is automatically deleted from the process filter

        The command:

        pfilter [ on | off | add pidlist | delete pidlist ]

        controls the process filter switch,  and  process  filter
        set  membership.  With  no  arguments, pfilter prints the
        current process filter set and the state of  the  process
        filter mode switch:

        on or off
              set the state of the process filter mode switch.

        add pidlist

        delete pidlist
              add or delete processes  from  the  process  filter
              set.  pidlist  is  a comma-separated list of one or
              more process ids.


     See tracing(3TNF) for complete examples showing, among other
     things, the use of prex to do simple probe control.

     When either the process or kernel is started, all probes are

     Example 1: Set creation and set listing

     create $out name=/out/     # $out = probes with "out" in
                                #   value of "name" attribute
     create $foo /page/ name=biodone   # $foo = union of
            # probes with "page" in value of keys attribute
            # probes with "biodone" as value of "name" attribute
     list sets                  # list the defined sets
     list fcns                  # list the defined probe fcns

     Example 2: Commands to trace and connect probe functions

     trace foobar='on'          # exact match on foobar attribute
     trace $all                 # trace all probes (predefined set $all)
     connect &debug $foo        # connect debug func to probes in $foo

     Example 3: Commands to enable and disable probes

     enable  $all               # enable all probes
     enable /vm/ name=alloc     # enable the specified probes
     disable $foo               # disable probes in set $foo
     list history               # list probe control commands issued

     Example 4: Process control

     continue                   # resumes the target process
     ^C                         # stop target; give control to prex
     quit resume                # exit prex, leave process running
                                     # and resume execution of program

     Example 5: Kernel mode

     buffer alloc 2m            # allocate a 2 Megabyte buffer
     enable $all                # enable all probes
     trace $all                 # trace all probes
     ktrace on                  # turn tracing on
     ktrace off                 # turn tracing back off
     pfilter on                 # turn process filter mode on
     pfilter add 1379           # add pid 1379 to process filter
     ktrace on                  # turn tracing on
                                # (only pid 1379 will be traced)


           local prex initialization file

           user's prex initialization file

           process files


     See attributes(5) for descriptions of the  following  attri-

    |       ATTRIBUTE TYPE        |       ATTRIBUTE VALUE       |
    | Availability                | SUNWtnfc (32-bit)           |
    |                             | SUNWtnfcx (64-bit)          |


     ed(1), kill(1),  ksh(1),  ld(1),  tnfdump(1),  tnfxtract(1),
     truss(1),     exec(2),    fork(2),    fork1(2),    vfork(2),
     TNF_DECLARE_RECORD(3TNF),   TNF_PROBE(3TNF),   dlclose(3DL),
     dlopen(3DL),         gethrtime(3C),         libtnfctl(3TNF),
     tnf_process_disable(3TNF),                    tracing(3TNF),
     tnf_kernel_probes(4), attributes(5)


     Currently, the only probe function that is available is  the
     &debug  function.  When this function is executed, it prints
     out the arguments sent in to the probe as well as the  value
     associated with the sunw%debug attribute in the detail field
     (if any) to stderr.

     For example, for the following probe point:

     TNF_PROBE_2(input_values, "testapp main",
                     "sunw%debug 'have read input values successfully'",
                     tnf_long, int_input, x,
                     tnf_string, string_input, input);

     If x was 100 and input was the string  "success",  then  the
     output of the debug probe function would be:

     probe input_values; sunw%debug "have read input values successfully";
     int_input=100; string_input="success";

     Some non-SPARC hardware lacks a true high-resolution  timer,
     causing  gethrtime() to return the same value multiple times
     in succession. This can lead to problems in how  some  tools
     interpret  the trace file. This situation can be improved by
     interposing a version of  gethrtime(),  which  causes  these
     successive  values  to  be  artificially  incremented by one

         static mutex_t lock;
         static hrtime_t (*real_gethrtime)(void) = NULL;
         static hrtime_t last_time = 0;

         hrtime_t this_time;

         if (real_gethrtime == NULL) {
             real_gethrtime =
                  (hrtime_t (*)(void)) dlsym(RTLD_NEXT, "gethrtime");
         this_time = real_gethrtime();

         if (this_time <= last_time)
             this_time = ++last_time;
             last_time = this_time;

         return (this_time);

     Of course, this does not  increase  the  resolution  of  the
     timer,  so  timestamps for individual events are still rela-
     tively inaccurate. But this technique maintains ordering, so
     that  if  event  A causes event B, B never appears to happen
     before or at the same time as A.

     dbx is available with the Sun Workshop Products.


     prex should issue a notification when a process id has  been
     automatically deleted from the filter set.

     There is a known bug in prex which can result in  this  mes-

     Tracing shut down in target program due to an internal
     error - Please restart prex and target

     When prex runs as root, and the target process is not  root,
     and  the  tracefile is placed in a directory where it cannot
     be removed and re-created (a directory with the  sticky  bit
     on,  like  /tmp),mm then the target process will not be able
     to open the tracefile when it  needs  to.  This  results  in
     tracing being disabled.

     Changing any of the circumstances listed  above  should  fix
     the  problem. Either don't run prex as root, or run the tar-
     get process as root, or specify the tracefile in a directory
     other than /tmp.

Man(1) output converted with man2html