Moving your data closer to the core using it can make your program slower. Not because the move itself is slow. Because of what has to happen for the move to be safe. That is the NUMA problem, and it goes deeper than memory latency tables suggest.
What NUMA is at the Hardware Level
Non-uniform memory access (NUMA) means that not all memory is equally close to all CPUs. A CPU accessing memory on its own local node pays the lowest latency. The same CPU accessing memory on a remote node pays more, sometimes 2 to 3 times higher, because the request has to cross an interconnect to reach another memory controller.
Multi-socket servers are the classic NUMA case: two physical CPUs, each with its own DDR channels, connected by a high-speed link. QPI on Intel, Infinity Fabric on AMD. Each socket is a NUMA node.
The more interesting case for modern systems is NUMA within a single package. AMD EPYC and Threadripper processors use a chiplet design: multiple Core Complex Dies (CCDs), each with its own set of cores and L3 cache, connected via Infinity Fabric to an I/O die that handles memory controllers and PCIe. Depending on which CCD a thread runs on and which memory controller its data is attached to, it may be doing local or remote NUMA access without ever leaving the physical package. Consumer Ryzen chips with two CCDs (like the 5900X and 5950X) have the same topology. A thread on CCD0 accessing memory whose physical address maps to a controller attached to CCD1 pays a cross-fabric latency penalty.
numactl --hardware is the fastest way to see what your system looks like. If you have more than one NUMA node listed, this applies to you. If you have one node, the hardware either is a single-die design or the firmware has collapsed the topology into a single node for compatibility. Either way, the concepts here govern every system once it scales.
Detecting NUMA on Your System
Before reaching for libnuma, confirm what topology the kernel sees:
# Show number of NUMA nodes and memory per node
numactl --hardware
# If numactl is not installed
ls /sys/devices/system/node/ # one directory per node: node0, node1, ...
lscpu | grep -i "NUMA node" # shows node count and which CPUs map to each
# Show distances between nodes (lower is faster)
numactl --hardware | grep -A 5 "distances"
# Check if kernel automatic NUMA balancing is on
cat /proc/sys/kernel/numa_balancing # 1 = on, 0 = off
# Show per-node memory stats
cat /sys/devices/system/node/node0/meminfo
# Count TLB shootdowns happening on your system right now
cat /proc/interrupts | grep TLB
The distance table from numactl --hardware encodes relative latency between nodes. Distance 10 is local, 20 is the most common remote penalty on two-socket systems, but chiplet designs can have intermediate values.
The Unit of Movement: Pages, Not Bytes
Cache lines are the unit of movement between cores and caches. For NUMA, the unit is pages. The default page size on x86-64 is 4KB. When the kernel migrates memory to a different NUMA node, it moves whole pages, 4KB at minimum. A single stale byte in a 4KB page means the whole page gets moved.
This distinction matters for two reasons. First, it sets the granularity at which false sharing can happen across NUMA boundaries: if two threads on different nodes are accessing different data that happens to share a page, both pay the migration cost when either triggers a move. Second, it determines the cost of hugepages as a mitigation, which the later section covers.
The kernel’s automatic NUMA balancing (controlled by kernel.numa_balancing) works entirely at page granularity. It tracks which node’s threads are accessing each page and migrates pages toward the accessing threads over time. The intent is correct but the mechanism has a cost: every migration triggers a TLB shootdown.
Page Alignment Across NUMA Nodes
The cache line alignment lesson transfers directly to NUMA. For cache lines, the rule is: if two threads on different cores share a cache line they do not logically share, both pay coherence cost. Separate them onto different cache lines. The same logic applies to NUMA at page granularity.
If a struct has fields accessed by threads on node 0 and fields accessed by threads on node 1, and both sets of fields fit on the same 4KB page, any migration of that page triggers cross-node overhead for both. The fix is the same: align the per-node sections to a page boundary so the kernel can migrate them independently.
// Align NUMA-local sections to page boundaries
alignas(4096) struct NodeLocalData {
// only accessed by threads on node 0
std::atomic<uint64_t> counter;
char pad[4096 - sizeof(std::atomic<uint64_t>)];
};
alignas(4096) struct NodeLocalData node0_data;
alignas(4096) struct NodeLocalData node1_data;
The principle is identical to cache line alignment. The unit is different, 4096 bytes instead of 64, and the cost being avoided is page migration rather than cache invalidation.
TLBs: The Missing Hardware Protocol
Every virtual memory access goes through address translation. The CPU needs to convert a virtual address (what the program uses) to a physical address (where the data actually lives in DRAM). This mapping lives in the page table, maintained by the kernel. Looking it up on every memory access would be catastrophically slow, so each CPU core has a translation lookaside buffer (TLB), a small hardware cache of recent virtual-to-physical mappings.
The critical difference between TLBs and data caches: data caches have a hardware coherence protocol. The MESI protocol keeps every core’s L1/L2/L3 view of a cache line consistent automatically. TLBs have no such protocol. Each core’s TLB is a completely private cache with no hardware awareness of what other cores have cached. There is nothing in the silicon that keeps TLBs in sync.
When a virtual-to-physical mapping changes, the kernel is the sole source of ground truth (the master page table). Every CPU’s TLB may be holding a stale copy of the old mapping. The hardware will not detect this. If a CPU executes a load with a stale TLB entry, it will silently use the wrong physical address. The kernel has to handle this entirely in software.
TLB Shootdowns: What Actually Happens
When the kernel changes a page mapping, it sends an inter-processor interrupt (IPI) to every CPU that might have a stale TLB entry for that page. The receiving CPU:
- Pauses whatever it was executing and switches to kernel mode
- Runs the interrupt handler
- Executes
INVLPG(or a full TLB flush in some cases) to invalidate the specific entry - Returns to the interrupted code
This is a TLB shootdown. The name is accurate: the kernel shoots down stale entries across the system. The cost shows up as system time in profiling output, not your program’s user time. It is not idle time either. It is time the CPU spent in kernel mode on your behalf, processing an interrupt your program caused by having its memory move.
The shootdown cannot be disabled. It is not a kernel policy, it is a hardware requirement. The kernel cannot allow stale TLB entries to persist because the result would be silent memory corruption. What you can control is whether the migration happens at all.
A NUMA page migration shoots down TLBs on every CPU that has mapped that page. For a shared memory region accessed by many threads across multiple cores, a single page migration can generate dozens of IPIs simultaneously. Each one causes a mode switch and an execution pause on the receiving core. On a high-core-count server doing frequent NUMA migrations, this overhead is measurable in perf stat as sys time and in /proc/interrupts TLB counters.
# Watch TLB shootdown counts accumulate in real time
watch -n 1 "cat /proc/interrupts | grep TLB"
# See system time vs user time for a running process
perf stat -p <pid> sleep 5
# High sys% relative to user% on a memory-intensive workload is one signal
Atomics and RMW Operations Under NUMA
Read-modify-write (RMW) operations like fetch_add and compare_exchange require exclusive ownership of a cache line before they can complete. On a single-node system, the MESI protocol handles this: the cache line is pulled into the executing core’s cache in exclusive state, the operation completes, done. The round-trip is within the local memory hierarchy.
Under NUMA, the picture changes if the cache line lives on a remote node. Getting exclusive ownership means the coherence message has to cross the NUMA interconnect to reach the remote memory controller, pull the line back, and then the operation can proceed. That interconnect round-trip is the bottleneck. On a two-socket Intel system with UPI, a cache line fetch across NUMA nodes costs roughly 100 to 200 ns. A local L1 hit costs about 4 ns. A global atomic counter being incremented by threads on both nodes will see that 100 to 200 ns latency on nearly every operation from the remote side, because the line bounces between nodes with every successful write.
A global shared counter under heavy NUMA contention is the worst possible case. The cache line for the counter lives on one node. Threads on the other node each need to pull it across the interconnect to increment it, then the line bounces back when the original node’s threads want it. Throughput collapses proportionally to the interconnect round-trip latency, not to the arithmetic.
The practical rules follow directly:
- Hot shared state that all threads write to belongs on one node, with the threads that write it also on that node
- Per-thread or per-node state is always faster than global shared state under NUMA
- Cross-node atomic operations are expensive by construction; the cost is the interconnect, not the operation itself
RCU: The Synchronization Mechanism That Works With NUMA
Read-copy-update (RCU) is a synchronization mechanism designed for workloads where reads dominate and writes are infrequent. It is the mechanism that handles NUMA well in practice, and the Linux kernel uses it extensively for exactly that reason.
The read side of RCU requires no locks, no atomics, and no cross-node coherence traffic. A reader simply marks that it is in a read-side critical section, reads the data, and marks the end. On modern kernels with SRCU or tree RCU, this read marker is either a local per-CPU counter or no operation at all on architectures that guarantee certain memory ordering. Either way, the reader never touches shared mutable state. Threads on node 0 reading RCU-protected data generate no interconnect traffic to node 1.
The update side follows the pattern that gives RCU its name: make a copy of the data structure to update, apply the update to the copy, then atomically publish the new version by swapping a pointer. Then wait for all currently-active readers to finish their critical sections (a “grace period”), and only after that free the old version. The pointer swap is one atomic write. Everything else is either per-thread local state or sequential single-writer work.
Under NUMA, the result is that the common case (reads) is completely free of cross-node overhead, and the update case pays one atomic operation for the pointer swap plus the cost of the grace period. For data that is read by many threads on many nodes but updated infrequently, this profile is far better than a mutex or a conventional lock-free structure where every reader touches shared state.
Per-node RCU takes this further: maintain a separate copy of the protected data per NUMA node, each copy updated independently and read locally. Reads are now both atomic-free and node-local. The tradeoff is memory: N copies of the data for N nodes. For small structures that are hot in the read path, this is often the right call.
The OS Scheduler, Thread Migration, and Memory
The Linux kernel’s automatic NUMA balancing works by tracking which node’s threads are accessing each page and migrating pages toward the accessing threads. This is transparent from the application’s perspective. When it works, memory follows the thread and latency decreases. When it does not, it generates TLB shootdowns without the benefit of actually reducing latency.
The mechanism’s failure mode is thread migration. If the scheduler moves a thread from node 0 to node 1 for load balancing reasons, the thread’s memory is now remote. The kernel’s NUMA balancing will start migrating pages toward node 1. But if the scheduler moves the thread back to node 0 shortly after, those pages start migrating back. Each round of migration costs TLB shootdowns and the CPU time to physically move the page contents across the interconnect. The net result is overhead with no latency benefit.
The fix is explicit binding. numactl handles both CPU affinity and memory policy together:
# Run a process on CPUs of node 0, allocating all its memory from node 0
numactl --cpunodebind=0 --membind=0 ./myprogram
# Interleave memory across all nodes (for workloads with no NUMA locality)
numactl --interleave=all ./myprogram
The difference between taskset and numactl matters here. taskset sets CPU affinity only: the thread stays on specific CPUs but memory still allocates from wherever the kernel decides. numactl sets both CPU affinity and the memory allocation policy. For NUMA-sensitive workloads, taskset alone is not enough.
Thread creation adds another wrinkle. When a thread spawns a child thread, the child’s stack and future heap allocations come from whichever node the scheduler places it on, which may not be the parent’s node. Threads do not inherit the parent’s NUMA node automatically. They inherit the parent’s memory policy only if numa_set_preferred or similar calls were made before the fork. The consequence in a thread pool that was not designed with NUMA in mind: worker threads can end up with memory spread across nodes at creation time, paying remote access costs for the life of the process.
I/O Node Locality
PCIe devices are connected to specific NUMA nodes. A network interface card (NIC), an NVMe drive, or a GPU all attach to one node’s PCIe root complex. Direct memory access (DMA), which is how these devices write data directly to RAM without CPU involvement, is not node-agnostic. When a NIC on node 0 receives a packet and DMA’s it to a buffer on node 1, that DMA transfer crosses the NUMA interconnect. For a 100GbE NIC that is DMA’ing hundreds of thousands of packets per second, this cross-node traffic is measurable and shows up as interconnect saturation before the CPU is even the bottleneck.
The rule for I/O-intensive workloads: the NIC should be on the same NUMA node as the threads processing its traffic, and those threads should be allocating their receive buffers from the same node’s memory. All three aligned to the same node eliminates the cross-node DMA cost entirely.
# Find which NUMA node a network interface is on
cat /sys/class/net/eth0/device/numa_node
# Find which NUMA node a PCI device is on
lspci -v | grep -A 5 "Ethernet" # look for "NUMA node" in the output
# Check the interrupt affinity for the NIC's queues
cat /proc/irq/<irq_number>/smp_affinity_list
For kernel bypass networking (DPDK, RDMA), the binding is explicit and the application controls which node’s memory the receive queues use. For conventional socket I/O, getting the binding right means setting both the thread affinity and the NIC interrupt affinity to the same node’s CPUs.
What You Can Control
The TLB shootdown mechanism cannot be disabled. NUMA migration can be. Here are the levers in order of coarseness:
numactl: pins the entire process to a node. CPU binding prevents the scheduler from moving threads to remote nodes. Memory binding ensures all malloc and stack allocations come from the local node. Stops migration before it starts.
numa_set_localalloc(): sets the current thread’s memory policy to always allocate from whichever node the thread is currently running on. Lighter than a hard bind but still prevents remote allocations as long as the thread stays pinned.
Hugepages: each 2MB huge page covers the same address range as 512 normal pages, but consumes only one TLB entry and requires only one TLB shootdown to migrate. Under NUMA migration, fewer TLB entries means less shootdown traffic for the same amount of memory moved. Transparent hugepages (echo always > /sys/kernel/mm/transparent_hugepage/enabled) let the kernel apply this automatically without code changes. Explicit hugepages give more control.
Disabling automatic NUMA balancing: sysctl -w kernel.numa_balancing=0 turns off the kernel’s transparent page migration entirely. This is the right choice when the application manages its own node binding explicitly, since kernel migration would only add overhead without improving locality.
The libnuma API
The code below requires numa_available() before any other NUMA calls. Without it, libnuma will silently misbehave on systems without NUMA support. Note also that numa_run_on_node pins where the thread runs (CPU affinity to that node’s CPUs) in addition to setting the memory allocation policy. The comment in many examples that calls it “future memory allocations only” understates what it does.
#include <numa.h>
#include <iostream>
#include <cstring>
#include <chrono>
int main() {
/* always check first: returns -1 if NUMA is not supported */
if (numa_available() < 0) {
std::cout << "NUMA not available on this system\n";
return 1;
}
int nodes = numa_num_configured_nodes();
std::cout << "NUMA nodes: " << nodes << "\n";
for (int i = 0; i < nodes; i++) {
long free_size = 0;
long total = numa_node_size(i, &free_size);
std::cout << "Node " << i << ": "
<< total / (1024 * 1024) << " MB total, "
<< free_size / (1024 * 1024) << " MB free\n";
}
/*
numa_run_on_node pins the current thread and its children to the CPUs
of node 0. It also sets the memory allocation policy to prefer node 0.
This affects both where the thread runs and where it allocates, not
just allocations.
*/
numa_run_on_node(0);
/* allocate on node 0 explicitly, freed with numa_free, not free() */
const size_t sz = 64 * 1024 * 1024;
void* buf = numa_alloc_onnode(sz, 0);
if (!buf) {
std::cout << "numa_alloc_onnode failed\n";
return 1;
}
std::memset(buf, 0, sz);
auto t0 = std::chrono::steady_clock::now();
volatile char* p = static_cast<volatile char*>(buf);
for (size_t i = 0; i < sz; i += 64) p[i] = static_cast<char>(i);
auto t1 = std::chrono::steady_clock::now();
std::cout << "Node 0 access (thread on node 0): "
<< std::chrono::duration_cast<std::chrono::milliseconds>(t1 - t0).count()
<< " ms\n";
numa_free(buf, sz);
if (nodes > 1) {
/* allocate on node 1 while thread still runs on node 0 */
void* remote = numa_alloc_onnode(sz, 1);
if (!remote) {
std::cout << "numa_alloc_onnode node 1 failed\n";
return 1;
}
std::memset(remote, 0, sz);
auto t2 = std::chrono::steady_clock::now();
volatile char* q = static_cast<volatile char*>(remote);
for (size_t i = 0; i < sz; i += 64) q[i] = static_cast<char>(i);
auto t3 = std::chrono::steady_clock::now();
std::cout << "Node 1 access (thread on node 0): "
<< std::chrono::duration_cast<std::chrono::milliseconds>(t3 - t2).count()
<< " ms\n";
numa_free(remote, sz);
}
return 0;
}
Build and run:
# install libnuma if needed
sudo apt install libnuma-dev # Debian/Ubuntu
sudo dnf install numactl-devel # Fedora/RHEL
g++ -O2 -std=c++20 numa_demo.cpp -o numa_demo -lnuma
./numa_demo
On a single-node system the local vs remote comparison will not appear and the output just confirms the node count and memory. On a multi-node system the two timings will diverge by however much the interconnect adds to the remote access latency. The numactl --hardware output beforehand tells you what topology to expect.
Further Viewing
Fedor G. Pikus covered NUMA and its performance implications in depth at C++Now 2024 in “Unlocking Modern CPU Power: Next-Gen C++ Optimization Techniques.” The talk goes into measurement difficulties, profiling tool limitations specific to NUMA, and practical debugging experience from enterprise-scale systems. Worth watching after reading this post since it covers the diagnosis side, how to actually find and confirm NUMA-related slowdowns, more than the mechanism side.
https://www.youtube.com/watch?v=wGSSUSeaLgA
Quick Reference
Coming from other languages
The NUMA problem is not language-specific. Any runtime that allocates memory on behalf of the program, whether that is a garbage-collected heap, a managed runtime, or a language-level allocator, is subject to the same underlying hardware constraint. Memory allocated on the wrong node pays a latency penalty regardless of what language requested it. The TLB shootdown mechanism is a kernel-level event with no language visibility at all. The levers available at the application level are the same: explicit node binding, huge pages, and controlling where allocations happen relative to where the threads run.
The 90% mental model
NUMA means not all memory is equally close. Access memory from the node you are running on and you pay local latency. Access it from the wrong node and you pay interconnect latency, often 2 to 3 times higher. Page migration is how the kernel tries to fix this automatically, but every migration requires a TLB shootdown: an interrupt sent to every CPU with a stale address translation, forcing each to pause, flush, and resume. The shootdown cannot be disabled. Migration can, by pinning threads and memory to the same node with numactl. Hugepages reduce the blast radius of shootdowns by replacing 512 individual 4KB TLB entries with one 2MB entry. Cross-node atomics are expensive because exclusive cache line ownership requires a round-trip across the interconnect. Shared mutable state under NUMA should be either node-local or replaced with something like RCU where the read path generates no cross-node traffic.