A normal instruction processes one value: load one, add one, store one. SIMD (single instruction multiple data) loads a batch of values into one wide register and applies the same operation to the whole batch at once. Four floats added to four floats, or eight, or sixteen, in the same instruction that used to handle one. Not a faster core. The instruction itself works on a batch instead of a single value.
This is different from superscalar execution. Superscalar is the CPU running several separate instructions in one cycle, each doing its own job in a different execution unit. More instructions at once. SIMD is one instruction doing more, a single instruction operating on N values simultaneously. A modern CPU does both at the same time, but they solve different problems.
Register width sets the batch size. SSE (Streaming SIMD Extensions) gives 128 bits, four floats at a time. AVX (Advanced Vector Extensions) doubles that to 256 bits, eight floats. AVX-512 doubles it again, sixteen. Same operation, wider register, more values per instruction, nothing about the math changes.
add_normal vs add_simd: The Instruction Gap
Adding N floats element-wise in scalar C++ looks like:
std::vector<float> add_normal(const std::vector<float>& a, const std::vector<float>& b) {
std::vector<float> out(a.size());
for (size_t i = 0; i < a.size(); i++)
out[i] = a[i] + b[i];
return out;
}
The SIMD version using AVX intrinsics loads eight floats at a time into a 256-bit register and adds them in one instruction:
std::vector<float> add_simd(const std::vector<float>& a, const std::vector<float>& b) {
std::vector<float> out(a.size());
size_t i = 0;
for (; i + 8 <= a.size(); i += 8) {
__m256 va = _mm256_loadu_ps(&a[i]);
__m256 vb = _mm256_loadu_ps(&b[i]);
__m256 vsum = _mm256_add_ps(va, vb);
_mm256_storeu_ps(&out[i], vsum);
}
for (; i < a.size(); i++)
out[i] = a[i] + b[i];
return out;
}
_mm256_add_ps is a direct intrinsic: it maps one-to-one to vaddps with a ymm register regardless of optimization level. There is no compiler discretion here. The instruction the intrinsic names is the instruction emitted.
Disassembling both at -O2 confirms the difference:
<add_normal(...)>:
c0: vmovss xmm0,DWORD PTR [rsi+rax*4]
c5: vaddss xmm0,xmm0,DWORD PTR [rdi+rax*4] // scalar, one float per instruction
ca: vmovss DWORD PTR [rcx+rax*4],xmm0
cf: inc rax
d2: cmp rax,rdx
d5: jb c0
<add_simd(...)>:
204: vmovups ymm0,YMMWORD PTR [rsi+rcx*1]
210: vaddps ymm0,ymm0,YMMWORD PTR [rax+rcx*1] // 8-wide AVX add, present at -O2
219: vmovups YMMWORD PTR [rax+rcx*1],ymm0
add_normal has only vaddss, scalar, no ymm register anywhere. add_simd has vaddps ymm0 in its main loop at -O2 already. Intrinsics do not need -O3 to emit vector instructions.
The scalar tail loop inside add_simd, handling the leftover elements that do not divide evenly into 8, is still plain vaddss at -O2. At -O3, the vectorizer picks it up on its own (more on that below).
Auto-Vectorization: What -O2 Actually Does
Adding -fopt-info-vec-optimized makes the compiler report every loop it vectorized:
$ g++ -O2 -std=c++20 -march=native -fopt-info-vec-optimized -c main.cpp -o main-o2.o
[gcc/include/c++/bits/stl_vector.h]:106:4: optimized: basic block part vectorized using 16 byte vectors
At -O2, the only vectorization happening is inside std::vector’s zero-initializing constructor. Nothing mentions add_normal’s loop. Auto-vectorization on this loop does not happen at -O2.
At -O3:
$ g++ -O3 -std=c++20 -march=native -fopt-info-vec-optimized -c main.cpp -o main-o3.o
[gcc/include/c++/bits/stl_vector.h]:106:4: optimized: basic block part vectorized using 16 byte vectors
main.cpp:7:26: optimized: loop vectorized using 32 byte vectors and unroll factor 8
main.cpp:7:26: optimized: loop versioned for vectorization because of possible aliasing
main.cpp:7:26: optimized: epilogue loop vectorized using 16 byte vectors and unroll factor 4
main.cpp:21:14: optimized: loop vectorized using 16 byte vectors and unroll factor 4
main.cpp:21:14: optimized: loop versioned for vectorization because of possible aliasing
main.cpp:21:14: optimized: epilogue loop vectorized using 8 byte vectors and unroll factor 2
Line 7 is add_normal’s loop, now vectorized at 32 bytes (8-wide, ymm), with a runtime alias check inserted first. Line 21 is add_simd’s scalar tail loop, also vectorized at -O3.
The alias check matters. add_normal takes two separate references. The compiler cannot prove at compile time that a and b do not overlap in memory. It generates a runtime check and builds three separate code paths:
<add_normal(...)> at -O3:
120: vmovups ymm0,YMMWORD PTR [rsi+rax*1]
125: vaddps ymm0,ymm0,YMMWORD PTR [rdx+rax*1] // main vectorized path, 8-wide ymm
12a: vmovups YMMWORD PTR [rcx+rax*1],ymm0
...
15c: vmovups xmm0,XMMWORD PTR [rsi+rax*4]
164: vaddps xmm0,xmm0,XMMWORD PTR [rdx+rax*4] // epilogue, 4-wide xmm
171: vmovups XMMWORD PTR [rcx+rax*4],xmm0
...
220: vmovss xmm0,DWORD PTR [rdx+rax*4]
225: vaddss xmm0,xmm0,DWORD PTR [rsi+rax*4] // scalar fallback, taken if alias check fails
22a: vmovss DWORD PTR [rcx+rax*4],xmm0
Three paths in one function: the 8-wide vectorized path, a 4-wide epilogue for the leftover elements after full 8-wide chunks, and a scalar fallback taken only if the runtime alias check determines a and b overlap. The vectorizer report’s “loop versioned for vectorization because of possible aliasing” line describes a code path that actually exists in the binary.
Why the Simple Add Doesn’t Produce a Speedup
Disassembling both functions confirms that intrinsics emit vector instructions where the compiler’s auto-vectorizer does not at -O2. That instruction gap does not translate into a wall-clock speedup for this specific operation.
add_normal and add_simd both load floats from memory, add them, and store the results back. One add per float loaded and stored. The compute step, vaddss or vaddps, executes in one cycle. The memory transfers take orders of magnitude longer. The CPU spends nearly all of its time waiting on memory, not computing. Whether one float or eight floats are added per instruction, the bottleneck is the same: memory bandwidth. Running both modes at every array size tested landed within noise of each other.
The instruction-level story is still correct. One vaddps replaces eight vaddss. But a speedup in wall-clock time requires that the compute step actually be the bottleneck, and for an operation this simple, it is not.
Designing the Benchmark: What Actually Needs Testing
To see the width advantage in timing, the operation needs to be compute-bound. benchmark.cpp uses a multiply-accumulate formula that creates real dependency chains:
x = x * y + a[i]
Each step depends on the previous value of x. The CPU cannot start computing step k+1 before step k is done. This is a serial dependency chain, and a single-chain loop leaves most of the CPU idle waiting on the previous result.
Breaking the dependency requires multiple independent chains. The first implementation used four accumulators:
float acc0 = 0, acc1 = 0, acc2 = 0, acc3 = 0;
for (size_t i = 0; i < n; i++) {
acc0 = acc0 * factor + a[i];
acc1 = acc1 * factor + a[i]; // problem: same a[i] as acc0
acc2 = acc2 * factor + a[i];
acc3 = acc3 * factor + a[i];
}
All four accumulators use the same a[i] and the same formula from the same starting value. They are mathematically identical computations. The compiler proved this and collapsed all four back into one chain. Confirmed in the disassembly: only one arithmetic chain existed where four were intended.
The fix is four independent elements per step:
float acc0 = 0, acc1 = 0, acc2 = 0, acc3 = 0;
for (size_t i = 0; i + 3 < n; i += 4) {
acc0 = acc0 * factor + a[i];
acc1 = acc1 * factor + a[i+1];
acc2 = acc2 * factor + a[i+2];
acc3 = acc3 * factor + a[i+3];
}
a[i], a[i+1], a[i+2], a[i+3] are different values. The compiler cannot prove the results are equal and cannot merge the chains. This is compute_normal.
What the Numbers Show
Three modes, three separate invocations:
$ ./benchmark normal
normal: 3466099 us, checksum=1.17644e+09
$ ./benchmark simd
simd: 1738350 us, checksum=1.17644e+09
$ ./benchmark simd4
simd4: 433877 us, checksum=1.17644e+09
compute_normal is four independent scalar chains (4-way ILP). compute_simd is a single __m256 chain, eight floats wide. compute_simd4 is four independent __m256 chains (4-way ILP at 8-wide vector).
normal vs simd: ~2x. compute_normal already has four-way ILP. compute_simd has one chain. This comparison isolates the effect of switching to vector instructions when ILP is unequal. The 2x does not represent the vector width advantage because the scalar side has more instruction-level parallelism.
normal vs simd4: ~8x. Once ILP is equal on both sides (four independent chains each), the remaining factor is width. compute_simd4 operates on 8x more data per step than compute_normal, and the measured ratio is essentially 8x. This is the vector width advantage once the ILP question is taken off the table.
Neither number is wrong. They answer different questions. Comparing normal against simd and calling it “the SIMD speedup” overstates the ILP contribution and understates the width contribution. The 8x shows what AVX actually buys when the comparison is controlled.
The checksum is identical across all three modes. Every version computed the same answer regardless of how the arithmetic was arranged.
Quick Reference
Coming from other languages
Languages that expose SIMD do so through auto-vectorization, through language-level vector types with compiler support, or through explicit intrinsic wrappers. The pattern here is intrinsics: function calls whose names map directly to individual CPU instructions. There is no runtime overhead from calling them and no abstraction between the function call and the emitted instruction. The cost model is the instruction’s own latency and throughput, which is what makes them useful for performance-critical paths where you need to know exactly what will execute.
The 90% mental model
SIMD loads N values into one wide register and applies one instruction to all of them simultaneously. The width (4, 8, or 16 floats for SSE, AVX, AVX-512) sets how many values are processed per instruction. Auto-vectorization at -O2 is not guaranteed: the compiler may decide not to vectorize, as confirmed directly with -fopt-info-vec-optimized on a simple add loop. Intrinsics bypass that decision entirely, the instruction named by the intrinsic is the instruction emitted. Whether that instruction difference produces a wall-clock speedup depends on whether the operation is compute-bound or memory-bound, simple element-wise operations are typically bandwidth-limited and will not show a speedup despite the correct vector instructions being present. For compute-bound work with real dependency chains and multiple independent lanes, the speedup from 256-bit AVX approaches the theoretical 8x.