Author: Mugabi Siro

Category: LinuxCNC

Summary:

This entry discusses NML configuration and runtime under the LinuxCNC framework. References to sources and binaries are with respect to the root of the LinuxCNC v2.8.0~pre1 package.

Tags: linux linuxcnc

Table Of Contents

NML Background

According to the glossary of terms in the LinuxCNC developer Manual (available online):

[NML] Neutral Message Langauge provides a mechanism for handling
multiple types of messages in the same buffer, as well as simplifying
the interface for encoding and decoding buffers in neutral format and
the configuration mechanism.

The NML framework can be viewed as a high-level Inter Process Commnication (IPC) protocol, providing an abstraction for communication channels for different types of information -- control commands, status info, and error messages. The interfaces exported by -lnml are the same regardless of the actual underlying IPC mechanism in use. Under LinuxCNC, abstracted IPC mechanisms include shared memory (for IPC on the same host computer), and TCP sockets for remotely connected processes.

Behaviour of the NML framework is controlled by user settings in an NML configuration file. NML configuraton files are given the .nml extension. A pristine LinuxCNC download will include a few NML configuration files under configs/common/. The LinuxCNC src/config.h file (generated during build) defines e.g.

#define EMC2_DEFAULT_NMLFILE "${LINUXCNC_INSTALL_PATH}/configs/common/linuxcnc.nml"

for the default NML configuration file.

In a running LinuxCNC instance, a number of modules parse NML config files in order to establish NML communication channels (or simply NML channels) between each other. These programs will be referred to here as NML modules. Generally, NML modules will check against the NML_FILE setting in an INI file so as to determine which NML config file to use -- and fall back to the default NML config file if none was specified. Alternatively, some NML modules accept a commandline option for non-default NML config file specification1.

For command, error and status information, an NML module allocates separate NML channels. In other words, an NML channel is a communication path for specific type of information such as control commands, status info, and error messages.

A major component of the NML framework is the Communication Management System (CMS)2. It plays a central role in the low-level IPC abstraction mechanism of the NML communication framework. However, from the LinuxCNC user's perspective, the intricate details between CMS and NML interaction are unimportant. Sending/receiving information between NML modules can be abstractly referred to as CMS/NML buffer operations.

Also scattered throughout the the code that implements lib/libnml.so, are references to functions and data prefixed with RCS_*/rcs_*. According to the LinuxCNC Developer Manual (available online), libnml "was derived from the [US] NIST rcslib, but without all the multi-platform support and non-LinuxCNC specific code."

An understanding of NML configuration files is required by the user in order to properly setup, for example, remote LinuxCNC control and monitoring over a TCP/IP network. NML configuration is discussed in the next section. For the developer of an NML module, e.g. a user interface, some familiarity with the CMS/NML C++ classes will be necessary. See A LinuxCNC MDI Streaming Interface for a case study.

NML Configuration

The .nml file consists of two tables, each with a different Line Format:

  • A CMS/NML Buffer Table whose fields are in the Buffer Line Format.

  • An NML Process Table whose fields are in the Process Line Format. The term "process" here implies an executing program instance of an NML module. Following the naming conventions in .ini files, NML modules include:

    • The DISPLAY program (i.e. one of the various user interfaces e.g. bin/axis, bin/keystick, etc),
    • The EMCSERVER program (e.g. bin/linuxcncsvr) and,
    • The TASK program (e.g. bin/milltask).

Essentially, the configuration in an NML file establishes CMS/NML buffer connections between the NML modules. The EMCSERVER program acts as an intermediary between DISPLAY and TASK. Typically, a running LinuxCNC instance will feature a DISPLAY program sending commands to, and receiving status info and error messages from, the TASK program via EMCSERVER. The EMCSERVER program is the master for all NML channels. The scripts/linuxcnc shell script starts EMCSERVER before any of its NML client modules, i.e. DISPLAYand TASK.

Recall that the NML library implements a communication abstraction such that different underlying IPC mechanisms can be used transparently by the NML modules. For example, DISPLAY will continue to work across a network connection (TCP/IP) with EMCSERVER (and, hence, the rest of the LinuxCNC instance), just as it did in a "localhost" setup (SHared MEMory, SHMEM). The settings in the NML configuration file determine which IPC mechanism gets used.

Remote connections are especially useful in setups where EMCSERVER and the core LinuxCNC modules (i.e. TASK, and the hard realtime motion control and I/O modules) execute on an embedded device, while a high-level and graphics intensive user interface (such as bin/axis) runs on a separate PC workstation. Also note that under LinuxCNC, some of the combinations of NML configuration options are invalid, while others imply certain constraints.

NML Configuration File Format

CMS/NML Buffer Table

Fields in this table are in the following format:

Buffers  Name  Type  Host  size  nuet?  RPC No. (obsolete)  ID max_procs key [Type Specific Configs]

where:

  • A value B for Buffers indicates that the line is in NML buffer format.

  • Name specifies the CMS buffer identifier.

  • Type describes the buffer type. SHMEM is the only buffer type really useful with LinuxCNC. Other buffer types include LOCMEM, FILEMEM and PHANTOM.

    NOTE: This buffer type value, SHMEM, is not to be confused with the class SHMEM CMS construct (include/shmem.hh). The latter is used by lib/libnml.so when instantiating an NML channel that actually uses the OS' shared memory for IPC. On the other hand, this SHMEM value for the Buffer Table Type field is also specified for NML modules that use, say, class TCPMEM to instantiate NML channels for remote connections between two physical machines across a (TCP/IP) network. Internally, -lnml relies on the value of the Process Table Type field (see below), i.e. instead of this Buffer Table Type field value (i.e. SHMEM), to dertermine the actual IPC mechanism to use (e.g. Shared memory vs TCP sockets)3.

  • Host is either an IP address or hostname for the NML server

  • size size of the buffer

  • neut A boolean value that indicates if the data in the buffer is encoded in a machine independent format, or raw. Only useful in obscure/rare multi-processor configurations where different (and incompatible) arch's are sharing a block of memory.

  • RPC Number This field is obsolete and not used in LinuxCNC. Maintained for backward compatibility.

  • ID The NML buffer number. It is a unique ID used if a NML server controls multiple buffers. Seemingly redundant wrt Name.

  • max_procs Maximum number of processes allowed to connect to this buffer. Not clear why this restriction exists.

  • key this numerical identifier for a shared memory buffer (FIXME: need to verify this).

  • Buffer Type Specific Configs

    The buffer type implies additional configuration options whilst the OS in question precludes certain combinations. Only SHMEM Buffer Table type will be considered.

    NOTE: Some of the following notes/points were copied blindly from the LinuxCNC Developer Manual. Thorough code inspection and tests still pending:

    • TCP=(port number) Specifies which network port to use. From code inspection of src/libnml/cms/cms_cfg.cc:cms_create(), it seems that only the TCP protocol is supported. UDP and STCP are not.
    • queue Enables queued message passing
    • ascii Encode messages in a plain text format
    • disp Encode messages in a format suitable for display.
    • xdr Encode messages in External Data Representation. Using TPC implies XDR encoding. See rpc/xdr.h for details.
    • diag Enables diagnostics stored in the buffer (timings and bytes counts).
    • mutex=os_sem Default mode for providing semaphore locking of the buffer memory. mutex=mao split splits the buffer into half (or more) and allows a process to access part of the buffer whilst a second process writing to another part. Most NML messages are relatively short and can be copied to/from the buffer with minimal delays, so split access is really not essential.

The following snippet presents a typical CMS/NML Buffer Table config:

# Top-level buffers to EMC
B    emcCommand   SHMEM    localhost    8192    0    0    1    16 1001 TCP=5005 xdr queue
B    emcStatus    SHMEM    localhost    16384   0    0    2    16 1002 TCP=5005 xdr
B    emcError     SHMEM    localhost    8192    0    0    3    16 1003 TCP=5005 xdr queue

NML Process Table

Fields in this table are in the following format:

Procees Name Buffer Type Host Ops Server Timeout Master c_num [Type Specific Configs]
  • A value of P for the Process field identifies this line as a Process configuration.

  • Name process identifier. The value to specify here will depend on a hardcoded string constant in the NML module implementation. This is discussed in section NML Channels and Process Names.

  • Buffer The CMS/NML buffer.

  • Type indicates whether this process local or remote relative to the NML server. The value specified in this field, i.e. REMOTE vs. LOCAL, is what -lnml uses to determine which type of underlying IPC mechanism to use. Under LinuxCNC, specifying REMOTE will result in NML channels created by way of class TCPMEM. These class instances will establish TCP sessions between NML modules. On the other hand, LOCAL will imply using instances of class SHMEM to create NML channels that rely on OS shared memory IPC4. Apparently, LinuxCNC will use UNIX System V shared memory over POSIX shared memory by default5.

  • Host Specifies the network node where this process is running

  • Ops Gives the process read-only, write-only, or read/write access to the CMS buffer.

  • Server Specifies whether this process is an NML server or client.

  • Timeout Sets the timeout characteristics for accesses to the CMS buffer . The exact behaviour when timeout is set to zero or a negative value is unclear from NIST docs. Only INF and positive values are mentioned. However, from the rcslib code:

    • timeout > 0 blocking access until the timeout interval is reached or access to the NML buffer is available.
    • timeout = 0 access to the NML buffer is only possible if no other process is reading or writing at the time.
    • timeout < 0 or timeout = INF blocking access until the NML buffer is available.
  • Master Indicates if this process is responsible for creating and destroying the buffer.

  • c_num an integer between zero and max_procs - 1. TODO: This doesn't seem to have any effect. Need to verify from code execution trace.

NML Channels and Process Names

The NML module implementations contain a hard-coded string constant that determines what value to specify for the Name field of the NML Process Table in the .nml config file. Generally, the C++ code for an NML module implementation that instantiates NML channels will resemble:

...
// NML channel for control and motion commands
RCS_CMD_CHANNEL *emcCommandChannel = 
        new RCS_CMD_CHANNEL(emcFormat, "emcCommand", PROC_NAME, EMC_NMLFILE);
...
// NML channel for status info
RCS_STAT_CHANNEL *emcStatusChannel = 
        new RCS_STAT_CHANNEL(emcFormat, "emcStatus", PROC_NAME, EMC_NMLFILE);
...
// NML channel for error messages
NML *emcErrorChannel = 
        new NML(nmlErrorFormat, "emcError", PROC_NAME, EMC_NMLFILE);
...
// NML channel for tool commands
RCS_CMD_CHANNEL *toolCommandChannel = 
        new RCS_CMD_CHANNEL(emcFormat, "toolCmd", PROC_NAME, EMC_NMLFILE);
...
// NML channel for tool status info
RCS_STAT_CHANNEL *toolStatusChannel =
        new RCS_STAT_CHANNEL(emcFormat, "toolSts", PROC_NAME, EMC_NMLFILE);
...

where:

  • PROC_NAME expands to a string constant. This is the value to specify for the Name field of the NML Process Table of the .nml file.

  • EMC_NMLFILE expands to the path-qualified filename of the .nml configuration file in use. The manner in which LinuxCNC passes the required .nml file to NML module will depend on the current configuration.

Specifying PROC_NAME

  • EMCSERVER Example

    The implementation for the default EMCSERVER program, bin/linuxcncsvr, requires "emcsvr":

      File      Line
    0 emcsvr.cc 131 new RCS_CMD_CHANNEL(emcFormat, "emcCommand", "emcsvr", emc_nmlfile);
    1 emcsvr.cc 137 new RCS_STAT_CHANNEL(emcFormat, "emcStatus", "emcsvr", emc_nmlfile);
    2 emcsvr.cc 142 new NML(nmlErrorFormat, "emcError", "emcsvr", emc_nmlfile);
    3 emcsvr.cc 147 new RCS_CMD_CHANNEL(emcFormat, "toolCmd", "emcsvr", emc_nmlfile);
    4 emcsvr.cc 152 new RCS_STAT_CHANNEL(emcFormat, "toolSts", "emcsvr", emc_nmlfile);
    
  • TASK Example

    The implementation for the standard TASK program, bin/milltask, expects "emc":

      File           Line
    0 emctaskmain.cc 2831 new RCS_CMD_CHANNEL(emcFormat, "emcCommand", "emc", emc_nmlfile);
    1 emctaskmain.cc 2865 new RCS_STAT_CHANNEL(emcFormat, "emcStatus", "emc", emc_nmlfile);
    2 emctaskmain.cc 2896 new NML(nmlErrorFormat, "emcError", "emc", emc_nmlfile);
    ...
    8 taskclass.cc     96 new RCS_CMD_CHANNEL(emcFormat, "toolCmd", "emc", emc_nmlfile);
    9 taskclass.cc    110 new RCS_STAT_CHANNEL(emcFormat, "toolSts", "emc", emc_nmlfile);
    
  • DISPLAY Examples

    Several DISPLAY implementations assume the value "xemc":

    • bin/axis:

        File         Line
      0 emcmodule.cc  263 new RCS_STAT_CHANNEL(emcFormat, "emcStatus", "xemc", file);
      1 emcmodule.cc  701 new RCS_CMD_CHANNEL(emcFormat, "emcCommand", "xemc", file);
      2 emcmodule.cc  707 new RCS_STAT_CHANNEL(emcFormat, "emcStatus", "xemc", file);
      3 emcmodule.cc 1343 NML *c = new NML(emcFormat, "emcError", "xemc", file);
      
    • bin/keystick:

      7 keystick.cc  1362 emcCommandBuffer = new RCS_CMD_CHANNEL(emcFormat, "emcCommand", "xemc", emc_nmlfile);
      8 keystick.cc  1375 emcStatusBuffer = new RCS_STAT_CHANNEL(emcFormat, "emcStatus", "xemc", emc_nmlfile);
      9 keystick.cc  1400 emcErrorBuffer = new NML(nmlErrorFormat, "emcError", "xemc", emc_nmlfile);
      
    • bin/xlinuxcnc:

      d xemc.cc        94 emcCommandBuffer = new RCS_CMD_CHANNEL(emcFormat, "emcCommand", "xemc", emc_nmlfile);
      e xemc.cc       106 emcStatusBuffer = new RCS_STAT_CHANNEL(emcFormat, "emcStatus", "xemc", emc_nmlfile);
      f xemc.cc       130 emcErrorBuffer = new NML(nmlErrorFormat, "emcError", "xemc", emc_nmlfile);
      

    So, to determine what value to specify for the Name field (in the NML Process Table of the .nml config file) when using some other NML module, simply check against this NML API in its source code.

Specifying EMC_NMLFILE

The emc_nmlfile pointer variable shown in the code snippets above is actually a LinuxCNC global char array defined in src/emc/nml_intf/emccfg.c:

char emc_nmlfile[LINELEN] = DEFAULT_EMC_NMLFILE;

This source file is part of the lib/liblinuxcnc.a library6 which, generally, gets linked with the NML modules. The src/emc/nml_intf/emccfg.h header file includes:

/* default name of EMC NML file */
#define DEFAULT_EMC_NMLFILE EMC2_DEFAULT_NMLFILE

where EMC2_DEFAULT_NMLFILE is defined in src/config.h (generated during build). Nevertheless, recall that a different NML file can be specified via:

  • the NML_FILE entry of the EMC section in an INI file

  • alternatively, via a command line option if supported by the NML module e.g. bin/keystick -nml <NML_FILENAME>.

NML Channels and Information Flow Summary

The following figure presents an example illustrating NML information flow between NML modules during runtime. Here, the DISPLAY instance can either be executing on the same machine as the rest of the LinuxCNC instance, or be remotely connected it.

NML Channel, Summary

NML Setups, Examples

A prestine LinuxCNC-git download includes a few NML configuration file samples under configs/common.

NML Configuration for SHMEM IPC

The configs/common/linuxcnc.nml file is the default and is meant for setups where all instances of NML modules execute on the same physical machine.

NML Configuration for TCP IPC

The configs/common/{server.nml, client.nml} files are sample configurations for setups where the DISPLAY program executes on one machine, while the rest of the LinuxCNC instance runs on a separate machine across a TCP/IP network.

The default configuration is such that the computer running EMCSERVER is assigned IP address 192.168.0.4, while the machine running the remotely connected UI is 192.168.0.14. Recall that since the EMCSERVER program (by default, bin/linuxcncsvr) is the master of all NML channels, the remote LinuxCNC instance must first be started before running the UI on the local machine.

For purposes of illustration, a setup for the PUMA 560 work-cell OpenGL simulator is assumed here. Change accordingly.

Server side

  • The value for the size field for the following status CMS/NML buffer entries in the Buffer Table of the NML config file may have to be adjusted. E.g. emcStatus: from 10240 to 16384, and toolSts: from 4096 to 8192:

    $ cat /home/user/emc2/linuxcnc/configs/common/server.nml
    ...
    B emcStatus             SHMEM   localhost       16384   0       0       2           16 1002 TCP=5005 xdr
    ...
    B toolSts               SHMEM   localhost       8192    0       0       5           16 1005 TCP=5005 xdr
    

    The other options can be left at the default settings.

  • Include an NML_FILE parameter, and change the value of DISPLAY in a copy of the default puma560.ini file, say, svr_puma560.ini, e.g:

    $ cat /home/user/emc2/linuxcnc/configs/sim/axis/vismach/puma/svr_puma560.ini
    ...
    [EMC]
    ...
    NML_FILE = /home/user/emc2/linuxcnc/configs/common/server.nml
    ...
    [DISPLAY]
    #+ Name of display program, e.g., xemc
    DISPLAY = dummy
    ...
    

    where the dummy value for DISPLAY is to instruct scripts/linuxcnc not to expect a foreground UI, i.e. since the UI is now being run remotely.

  • launch server side with, say:

    $ linuxcnc configs/sim/axis/vismach/puma/svr_puma560.ini
    

At this point, only the tk window for the PUMA 560 work-cell simulation should appear:

LinuxCNC Remote Server Side

As shown, the EMCSERVER program now waits in the background for incoming TCP connections from a remote UI. For instance:

$ sudo netstat -ap | grep 5005
tcp        0      0 *:5005                  *:*                     LISTEN      2376/linuxcncsvr

Pressing RETURN on this foreground instance of scripts/linuxcnc will result in termination of this side of LinuxCNC.

Client side

FIXME: There seems to be a bug with this version of LinuxCNC preventing NML channel data transfers over TCP/IP connections. The TCPMEM::TCPMEM constructor gets invoked (see libnml/cms/cms_cfg.cc) and NML channels (TCP connections) are indeed established between server and client, e.g:

  • Launching a UI with, say,

    $ ${LINUXCNCDIR}/bin/keystick -ini ${LINUXCNCDIR}/configs/sim/axis/vismach/puma/puma560.ini \
         -nml ${LINUXCNCDIR}/configs/common/client.nml
    
    $ LD_LIBRARY_PATH=${LINUXCNCDIR}/linuxcnc/lib ./ne_linuxcnc_ui_puma \
         -ini ${LINUXCNCDIR}/linuxcnc/configs/sim/axis/vismach/puma/puma560.ini \
         -nml ${LINUXCNCDIR}/linuxcnc/configs/common/client.nml
    
    (etc)
    
  • then running the following command on the server side:

    $ sudo netstat -ap | grep 5005
    tcp        0      0 *:5005                  *:*                     LISTEN      2376/linuxcncsvr
    tcp        0      0 vm.local:5005           192.168.0.14:47710      ESTABLISHED 2376/linuxcncsvr
    tcp        0      0 vm.local:5005           192.168.0.14:55100      ESTABLISHED 2376/linuxcncsvr
    tcp        0      0 vm.local:5005           192.168.0.14:52493      ESTABLISHED 2376/linuxcncsvr
    

yet the CMS::write() virtual function still does not get overloaded/overriden by TCPMEM::write(). For example, an attempt by the UI to send a command via the NML command channel will result in errors such as:

libnml/cms/cms.cc 888: CMS::main_access called by xemc for emcCommand.
libnml/cms/cms.cc 890: This should never happen.
libnml/cms/cms.cc 891: Derived classes should either override main_access() or
libnml/cms/cms.cc 893: the functions that call it.(read(), write(), etc.)
libnml/cms/cms.cc 894: _local = 0x70c1c0

and from the backtrace:

emcCommandSend() --> NML::write() --> CMS::write() --> CMS::main_access()

I'm probably missing something -- but from code inspection I think (and to cite just a few):

// src/libnml/buffer/tcpmem.hh 
class TCPMEM:public CMS {
  ...
  CMS_STATUS write(void *data);
  CMS_STATUS write_if_read(void *data);

// src/libnml/buffer/tcpmem.cc 
CMS_STATUS TCPMEM::write(void *user_data)
CMS_STATUS TCPMEM::write_if_read(void *user_data)

should be:

// src/libnml/buffer/tcpmem.hh 
class TCPMEM:public CMS {
  ...
  CMS_STATUS write(void *user_data, int *); 
  CMS_STATUS write_if_read(void *user_data, int *);

// src/libnml/buffer/tcpmem.cc
CMS_STATUS_TCPMEM::write(void *user_data, int *dummy = NULL)
CMS_STATUS TCPMEM::write_if_read(void *user_data, int *dummy = NULL)

If this problem is a truly a bug, then it will also manifest upon UI launch in your case. In that case, consult with the experts on the LinuxCNC mailing list. The following steps may be performed to test...

  • Also change value of the size field for the emcStatus entry in the Buffer Table of the NML config file from 10240 to, say 16384, e.g:

    $ cat home/user/emc2/linuxcnc/configs/common/client.nml
    
    # Buffers
    # Name                  Type    Host             size    neut?   (old)   buffer# MP ---
    
    # Top-level buffers to EMC
    B emcCommand            SHMEM   localhost       8192    0       0       1       16 1001 TCP=5005 xdr queue
    B emcStatus             SHMEM   localhost       16384   0       0       2       16 1002 TCP=5005 xdr
    B emcError              SHMEM   localhost       8192    0       0       3       16 1003 TCP=5005 xdr queue
    
    # Processes
    # Name          Buffer          Type    Host              Ops     server? timeout master? cnum
    
    P xemc          emcCommand      REMOTE   192.168.0.4       W       0       10.0    0       10
    P xemc          emcStatus       REMOTE   192.168.0.4       R       0       10.0    0       10
    P xemc          emcError        REMOTE   192.168.0.4       R       0       10.0    0       10
    P xemc          toolCmd         REMOTE   192.168.0.4       W       0       10.0    0       10
    P xemc          toolSts         REMOTE   192.168.0.4       R       0       10.0    0       10
    
  • Launch the UI with this NML file, e.g. as illustrated above with bin/keystick. For other UIs, edit the INI file to include a NML_FILE parameter that points to client.nml -- assuming that the UI in question processes this INI file parameter.

Resources and Further Reading

  • LinuxCNC Developer Manual (available online).

  • LinuxCNC Integrator Manual (available online).

  • LinuxCNC User Manual (available online).

Footnotes

1. For example, the bin/keystick and ne_linuxcnc_ui_puma.cc UIs support the -nml NMLFILE option. The different types of NML modules are discussed in section NML Configuration. [go back]

2. See src/libnml/cms/cms.hh [go back].

3. See CMS::CMS() in src/libnml/cms/cms.cc [go back]

4. See cms_create() in src/libnml/cms/cms_cfg.cc [go back]

5. See rcs_shm_open() in src/libnml/os_intf/_shm.c [go back]

6. See src/emc/nml_intf/Submakefile [go back]