Author: Mugabi Siro
Category: General Linux
A brief contrast of UNIX/Linux SHM IPC facilities: POSIX SHM vs. Shared File/Anonymous Mappings.Tags: linux
In contrast to other Interprocess Communication (IPC) mechanisms such as FIFO's, message queues, and sockets, which involve system calls or data transfer between user and kernel memory, Shared Memory (SHM) allows fast data exchange via pointer references to data in a common region of memory shared among different processes. The kernel initializes the page-tables of the respective processes to point to the same pages of RAM, and a successful shared memory mapping forms a new Virtual Memory Area (VMA) in the process' address space. Nevertheless, since data in the shared memory region is visible to all processes involved, programs need to implement/adhere to strict synchronisation policies for shared data access. Synchronisation mechanisms include semaphores, file locks, mutexes and condition variables.
On most UNIX-like systems, and indeed Linux, there exist a variety of facilities for shared memory IPC:
System V SHM
Shared File/Anonymous mappings
POSIX SHM Objects
The interfaces exported by System V IPC facilities, including its flavor of SHM, are quite alien to the UNIX I/O paradigm; a historical artifact of their UNIX independent design. For example, unlike POSIX SHM and shared file/anonymous mappings which use a pathname and file descriptor to identify and handle, respectively, objects in programs, the System V SHM facility instead uses a System V IPC key (to identify objects) and identifier (to handle objects). The POSIX IPC facilities (message queues, semaphores and SHM) were designed to address these "unUNIX-like" peculiarities by their older System V counterparts. System V SHM will not be discussed any further.
Central to both POSIX SHM and shared file/anonymous mappings is the
#include <sys/mman.h> void *mmap(void *addr, size_t length, int prot, int flags, int fd, off_t offset); int munmap(void *addr, size_t length);
A call to
mmap(2) creates a new mapping in the virtual address space of
the calling process. A common factor with both SHM IPC mechanisms, with respect to
mmap(2) usage, is that the
flags argument takes the
The term memory mappings encompasses the facilities provided by two types of mappings: file mappings and anonymous mappings. Each type of mapping supports both private and shared memory mapping schemes. With respect to IPC, both types of memory mappings employ the shared mapping scheme, hence shared file mappings and shared anonymous mappings.
Non-IPC related uses of memory mappings include private file mappings1 and private anonymous mappings2. Private mapping schemes essentially translate to the
MAP_PRIVATE setting for
flag argument. Modifications to the region provided by these mappings are not visible to other processes.
Memory mappings disappear upon
exec(2) but are inherited by a
child. Inherited private mappings are naturally copy-on-write (COW)3.
These allow unrelated processes to share regions of memory in the interest of fast IPC. All processes mapping the same region of a file share the same physical pages of memory, which are initialized from a file region. The kernel performs on-demand loading of the pages of the mapping from the file. Modifications to the contents of the region are carried through to the underlying mapped file4. Note that unlike any other UNIX shared memory IPC facility, file-based mappings have the unique property of file-system persistence, i.e. data is retainable across system boots.
msync(2) prior to
munmap(2) ensures that the contents of the shared mapping get flushed to the disk file.
In this case, SHM IPC is only possible between related processes e.g. a
fork(2)'d child and its parent. Unlike typical mappings between parent and child which are private, and hence COW, shared anonymous mappings allow these related processes to share (and modify) the same pages in RAM. Note that anonymous mappings are not file-backed and their persistence is the lifetime of the processes involved.
mmap(2) effects shared anonymous mappings via either inclusion of the
MAP_ANONYMOUS flag, or by using a file descriptor obtained from an
Like shared file mappings, POSIX SHM enables unrelated processes to share a region of common memory. However, unlike the file-based memory-mappings, it does so without involving a disk file. Linux uses the RAM-based tmpfs as the "backing store" for POSIX SHM objects. For instance, on an Ubuntu AMD64 system:
$ mount | grep shm none on /run/shm type tmpfs (rw,nosuid,nodev) $ ls /dev/shm -l lrwxrwxrwx 1 root root /dev/shm -> /run/shm $ df -h | egrep '(^Filesystem|shm)' Filesystem Size Used Avail Use% Mounted on none 2.0G 1.4M 2.0G 1% /run/shm
This implies that persistence of a POSIX SHM object does not survive across system reboots - unlike shared file mappings. The duration of its lifetime lasts until it is explicitly deleted or the system is shut down. Nevertheless, this RAM-based approach avoids the overhead and unbounded latencies associated with disk-based file mappings e.g. relatively slow on-demand load operations
of file pages from disk, the occassional sync with the disk file backing-store, etc.
This makes POSIX SHM more suitable for use in (POSIX) realtime applications. In fact, building an application that employs the POSIX SHM interfaces requires specification of the
-lrt static linker option.
Since POSIX SHM objects are not disk-based files, a special set of POSIX C API library functions, i.e.
shm_unlink(3), exist to create and remove, respectively, these RAM-based objects:
#include <sys/mman.h> #include <sys/stat.h> /* For mode constants */ #include <fcntl.h> /* For O_* constants */ int shm_open(const char *name, int oflag, mode_t mode); int shm_unlink(const char *name); Link with -lrt.
The file descriptor obtained from
shm_open(3) is passed to
mmap(2) to enable mapping of the POSIX SHM object into the caller's address space. File-based mappings, on the other hand, use conventional
close(2) with their file descriptors.
As with other uses of
mmap(2), once the calling process successfully obtains a mapping, then the file descriptor can be closed without affecting the mapping. Nevertheless, a program may maintain it for future use with, say,
ftruncate(2). In particular, with POSIX SHM,
ftruncate(2) must be used to set the size of the tmpfs-based object, since it initially has a length of zero.
shm_unlink(3) call removes the specified SHM object. Note:
For existing mappings by multiple processes, if a program instance
shm_unlink(3), the rest of the mappings remain unaffected.
Upon the termination of, or an
munmap(2) by, the last process using the SHM
object, the object is removed and its contents get lost.
shm_unlink(3), any future
shm_open(3) calls against the object (by the current program instance, or from any other process) will fail.
1. A typical use case is when initializing a process' VMA's that correspond to its program's code and initialized data segments. For example, check out Sections & Segments and Linux VMA Mappings. [go back]
2. Commonly used for VMA allocation for
.bss, i.e uninitialized data, of a process' data segment (for example, see Sections & Segments and Linux VMA Mappings), and for large dynamic memory allocation within a process' private address space (for example, see Private anonymous mappings in QEMU). [go back]
3. Recall that private anonymous mappings are zero filled, and so the
fork(2)'d child won't see the contents of its parent's private anonymous mappings [go back]