Author: Mugabi Siro

Category: General Linux

Summary:

A brief contrast of UNIX/Linux SHM IPC facilities: POSIX SHM vs. Shared File/Anonymous Mappings.

Tags: linux

UNIX Shared Memory IPC

In contrast to other Interprocess Communication (IPC) mechanisms such as FIFO's, message queues, and sockets, which involve system calls or data transfer between user and kernel memory, Shared Memory (SHM) allows fast data exchange via pointer references to data in a common region of memory shared among different processes. The kernel initializes the page-tables of the respective processes to point to the same pages of RAM, and a successful shared memory mapping forms a new Virtual Memory Area (VMA) in the process' address space. Nevertheless, since data in the shared memory region is visible to all processes involved, programs need to implement/adhere to strict synchronisation policies for shared data access. Synchronisation mechanisms include semaphores, file locks, mutexes and condition variables.

On most UNIX-like systems, and indeed Linux, there exist a variety of facilities for shared memory IPC:

  • System V SHM

  • Shared File/Anonymous mappings

  • POSIX SHM Objects

The interfaces exported by System V IPC facilities, including its flavor of SHM, are quite alien to the UNIX I/O paradigm; a historical artifact of their UNIX independent design. For example, unlike POSIX SHM and shared file/anonymous mappings which use a pathname and file descriptor to identify and handle, respectively, objects in programs, the System V SHM facility instead uses a System V IPC key (to identify objects) and identifier (to handle objects). The POSIX IPC facilities (message queues, semaphores and SHM) were designed to address these "unUNIX-like" peculiarities by their older System V counterparts. System V SHM will not be discussed any further.

Central to both POSIX SHM and shared file/anonymous mappings is the mmap(2) interface:

#include <sys/mman.h>

void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset);

int munmap(void *addr, size_t length);

A call to mmap(2) creates a new mapping in the virtual address space of the calling process. A common factor with both SHM IPC mechanisms, with respect to mmap(2) usage, is that the flags argument takes the MAP_SHARED value.

Shared File/Anonymous Mappings

The term memory mappings encompasses the facilities provided by two types of mappings: file mappings and anonymous mappings. Each type of mapping supports both private and shared memory mapping schemes. With respect to IPC, both types of memory mappings employ the shared mapping scheme, hence shared file mappings and shared anonymous mappings.

Non-IPC related uses of memory mappings include private file mappings1 and private anonymous mappings2. Private mapping schemes essentially translate to the MAP_PRIVATE setting for mmap(2)'s flag argument. Modifications to the region provided by these mappings are not visible to other processes.

Memory mappings disappear upon exec(2) but are inherited by a fork(2)'d child. Inherited private mappings are naturally copy-on-write (COW)3.

Shared File Mappings

These allow unrelated processes to share regions of memory in the interest of fast IPC. All processes mapping the same region of a file share the same physical pages of memory, which are initialized from a file region. The kernel performs on-demand loading of the pages of the mapping from the file. Modifications to the contents of the region are carried through to the underlying mapped file4. Note that unlike any other UNIX shared memory IPC facility, file-based mappings have the unique property of file-system persistence, i.e. data is retainable across system boots.

Calling msync(2) prior to munmap(2) ensures that the contents of the shared mapping get flushed to the disk file.

Shared Anonymous Mappings

In this case, SHM IPC is only possible between related processes e.g. a fork(2)'d child and its parent. Unlike typical mappings between parent and child which are private, and hence COW, shared anonymous mappings allow these related processes to share (and modify) the same pages in RAM. Note that anonymous mappings are not file-backed and their persistence is the lifetime of the processes involved. mmap(2) effects shared anonymous mappings via either inclusion of the MAP_ANONYMOUS flag, or by using a file descriptor obtained from an open(2) against /dev/zero.

POSIX Shared Memory Objects

Like shared file mappings, POSIX SHM enables unrelated processes to share a region of common memory. However, unlike the file-based memory-mappings, it does so without involving a disk file. Linux uses the RAM-based tmpfs as the "backing store" for POSIX SHM objects. For instance, on an Ubuntu AMD64 system:

$ mount | grep shm
none on /run/shm type tmpfs (rw,nosuid,nodev)

$ ls /dev/shm -l
lrwxrwxrwx 1 root root /dev/shm -> /run/shm

$ df -h | egrep '(^Filesystem|shm)'
Filesystem        Size  Used Avail Use% Mounted on
none              2.0G  1.4M  2.0G   1% /run/shm

This implies that persistence of a POSIX SHM object does not survive across system reboots - unlike shared file mappings. The duration of its lifetime lasts until it is explicitly deleted or the system is shut down. Nevertheless, this RAM-based approach avoids the overhead and unbounded latencies associated with disk-based file mappings e.g. relatively slow on-demand load operations of file pages from disk, the occassional sync with the disk file backing-store, etc. This makes POSIX SHM more suitable for use in (POSIX) realtime applications. In fact, building an application that employs the POSIX SHM interfaces requires specification of the -lrt static linker option.

Since POSIX SHM objects are not disk-based files, a special set of POSIX C API library functions, i.e. shm_open(3) and shm_unlink(3), exist to create and remove, respectively, these RAM-based objects:

#include <sys/mman.h>
#include <sys/stat.h>        /* For mode constants */
#include <fcntl.h>           /* For O_* constants */

int shm_open(const char *name, int oflag, mode_t mode);

int shm_unlink(const char *name);

Link with -lrt.

The file descriptor obtained from shm_open(3) is passed to mmap(2) to enable mapping of the POSIX SHM object into the caller's address space. File-based mappings, on the other hand, use conventional open(2) and close(2) with their file descriptors.

As with other uses of mmap(2), once the calling process successfully obtains a mapping, then the file descriptor can be closed without affecting the mapping. Nevertheless, a program may maintain it for future use with, say, fstat(2) and ftruncate(2). In particular, with POSIX SHM, ftruncate(2) must be used to set the size of the tmpfs-based object, since it initially has a length of zero.

The shm_unlink(3) call removes the specified SHM object. Note:

  • For existing mappings by multiple processes, if a program instance invokes shm_unlink(3), the rest of the mappings remain unaffected. Upon the termination of, or an munmap(2) by, the last process using the SHM object, the object is removed and its contents get lost.

  • After shm_unlink(3), any future shm_open(3) calls against the object (by the current program instance, or from any other process) will fail.

Further Reading

  • The Linux Programming Interface: A Linux and UNIX System Programming Handbook, Michael Kerrisk, No Starch Press, Inc. 2010. A most comprehensive and exhaustive coverage of Linux System Programming. 'Nuff sed!

Footnotes

1. A typical use case is when initializing a process' VMA's that correspond to its program's code and initialized data segments. For example, check out Sections & Segments and Linux VMA Mappings. [go back]

2. Commonly used for VMA allocation for .bss, i.e uninitialized data, of a process' data segment (for example, see Sections & Segments and Linux VMA Mappings), and for large dynamic memory allocation within a process' private address space (for example, see Private anonymous mappings in QEMU). [go back]

3. Recall that private anonymous mappings are zero filled, and so the fork(2)'d child won't see the contents of its parent's private anonymous mappings [go back]

4. Shared file mappings also serve the purpose of memory mapped I/O. For example, see mmaping MMIO and DMA regions. [go back]