This is a follow-up to Making Node.js histograms more useful.
That post covered the first round of analytical APIs I added to the
Histogram class:
CDF, skewness, kurtosis, KS tests, and snapshot diffing.
PR #65416 adds statistical
hypothesis testing methods, streaming statistics, and SLO tools on top of
those. A second PR (#65434) adds
a compact binary exchange format for transmitting histograms between
processes.
But... why?
I do a lot of performance benchmarking in and with Node.js. I originally
added the Histogram API to Node.js to support this and always had the goal
of coming back and expanding it with more analysis tools. Currently, to make
use of any of the data collected you have to either import a number of
dependencies or export the data to R or Python. This has always bugged me and
to give myself a nice break from other work I've been doing, I decided to come
back and finally add the missing pieces.
Node.js itself makes use of several of these analytical functions in its own
benchmarking tools. The benchmark/compare.js script has always required R to
analyze benchmark results. With the new APIs, it can now perform the same
analysis inline without any external dependencies. A bit more batteries included
and a bit more self-contained.
Statistical hypothesis testing
The first post covered ksTest() for distribution comparison. That tells you
whether two distributions differ but not by how much or whether the difference
is statistically significant. The new methods fill that gap.
Welch's t-test
welchTest(other, options)
performs a Welch's t-test
comparing the means of two histograms. It returns a t-statistic, degrees of
freedom, p-value, and a confidence interval for the difference in means. This
is the standard test for "are these two means significantly different?" without
assuming equal variance.
const { createHistogram } = require('node:perf_hooks');
const baseline = createHistogram();
const candidate = createHistogram();
// Record latencies from each build...
for (let i = 0; i < 500; i++) {
baseline.record(10 + Math.ceil(Math.random() * 20));
candidate.record(12 + Math.ceil(Math.random() * 20));
}
const result = baseline.welchTest(candidate);
console.log('p-value:', result.pValue.toFixed(6));
console.log('CI:', result.confidenceInterval);
if (result.pValue < 0.05) {
console.log('Statistically significant difference');
}The implementation uses the regularized incomplete beta function for the
Student's t CDF and a bisection search for the quantile function. All the math
is in C++ using <cmath> primitives. No dependencies.
Mann-Whitney U test
mannWhitneyTest(other)
is a non-parametric alternative. It doesn't assume anything about the
distribution shape. It tests whether one distribution tends to produce larger
values than the other. The implementation works directly on the histogram
bucket counts rather than expanding individual values, so it's efficient even
with millions of recorded samples.
Effect size: Cohen's d and Cliff's delta
A low p-value tells you a difference exists. It doesn't tell you if the difference matters. Effect size metrics answer that question.
cohensD(other)
returns the standardized mean difference. Values around 0.2 are small effects,
0.5 is medium, 0.8 or above is large.
cliffsD(other)
is a non-parametric alternative that returns a value between -1 and 1
representing the probability that a random value from one distribution exceeds
a random value from the other.
const d = baseline.cohensD(candidate);
const cliff = baseline.cliffsD(candidate);
console.log(`Cohen's d: ${d.toFixed(3)}`); // Magnitude of the difference
console.log(`Cliff's delta: ${cliff.toFixed(3)}`); // Directional dominancePercentile confidence intervals
percentileCI(percentile, options)
returns a confidence interval for a given percentile using the exact binomial
method. Your p99 is a point estimate. With 100 samples it's based on a single
observation. With 10,000 samples it's based on 100. The confidence interval
tells you how much to trust the number.
const h = createHistogram();
for (let i = 0; i < 1000; i++) h.record(Math.floor(Math.random() * 100));
const ci = h.percentileCI(99);
console.log(`p99: ${ci.value} [${ci.lower}, ${ci.upper}]`);
// Narrower with more data
const ci99 = h.percentileCI(99, { confidence: 0.99 });The math reuses the regularized incomplete beta function already implemented
for the t-test. The binomial CDF P(X <= k) for X ~ Bin(n, p) is
I_{1-p}(n-k, k+1), which is a single call to the same function. Binary
search finds the lower and upper ranks, then hdr_value_at_percentile maps
them back to histogram values.
Streaming statistics with EWMA
The statistics in the first post (mean, stddev, skewness, etc.) are cumulative. They reflect the entire lifetime of the histogram. For monitoring, you usually want to know what's happening right now, not what's happened on average since the process started.
The subtract() approach from the first post gives you fixed time windows.
EWMA gives you exponential decay: recent values matter more, older values
fade out gradually. The decay rate is controlled by a half-life parameter.
After halfLife recordings, a value's influence has decayed to 50%.
const { createHistogram } = require('node:perf_hooks');
const h = createHistogram({ halfLife: 100 });
// Record latencies...
for (let i = 0; i < 500; i++) h.record(50 + Math.ceil(Math.random() * 10));
console.log('EWMA mean:', h.ewmaMean.toFixed(2));
console.log('EWMA stddev:', h.ewmaStddev.toFixed(2));The EWMA is updated on every record() call inside the same write lock that
updates the histogram. Zero additional overhead when halfLife is not set
(the default). The formulas are standard Welford-style online updates adapted
for exponential weighting:
diff = value - mean
mean += alpha * diff
variance = (1 - alpha) * (variance + alpha * diff^2)
Where alpha = 1 - 2^(-1/halfLife).
When EWMA is active, toJSON() reports the EWMA mean and stddev as the mean
and stddev fields. The cumulative values are still available through the
mean and stddev getters.
SLO burn rate
If you're running a service with an SLO like "99.9% of requests under 200ms", you care about two things: the current error rate (fraction of requests exceeding the threshold) and the burn rate (how fast you're consuming your error budget).
The threshold option enables an EWMA-smoothed error rate tracker. On each
record(), the histogram feeds 1.0 or 0.0 into a binary EWMA depending on
whether the value exceeds the threshold. The result converges to the current
probability of exceeding the threshold.
const { createHistogram } = require('node:perf_hooks');
const h = createHistogram({
halfLife: 100,
threshold: 200_000_000, // 200ms in nanoseconds
});
// Record latencies...
// Current smoothed error rate
console.log(`Error rate: ${(h.ewmaErrorRate * 100).toFixed(2)}%`);
// Burn rate against a 99.9% SLO
// >1 means the error budget is depleting faster than allowed
const rate = h.burnRate(0.999);
console.log(`Burn rate: ${rate.toFixed(2)}x`);burnRate(sloTarget) is just ewmaErrorRate / (1 - sloTarget). A burn rate
of 1 means you'll exactly exhaust your error budget over the SLO window. A
burn rate of 10 means you'll exhaust it 10x faster. This is the same
calculation described in the
Google SRE book for
multi-window burn rate alerting.
The EWMA smoothing matters here. Without it, you'd need to maintain windowed histograms with manual subtract/snapshot cycles. With EWMA, the error rate is always current and available as a simple property read.
Portable histogram exchange
The histogram API has always been able to serialize via toJSON(), but that
output is lossy. It gives you summary statistics and a percentile map. You
can't reconstruct the full histogram from it.
PR #65434 adds export() and
importHistogram() for lossless round-trip serialization. The wire format
is CBOR (RFC 8949), encoded by
a minimal hand-rolled encoder built into Node.js.
const { createHistogram, importHistogram } = require('node:perf_hooks');
const h = createHistogram({ halfLife: 100, threshold: 200_000_000 });
for (let i = 1; i <= 10000; i++) h.record(i);
// Export: histogram → Uint8Array
const data = h.export();
console.log(`${data.length} bytes`);
// Import: Uint8Array → new RecordableHistogram
const h2 = importHistogram(data);
console.log(h2.percentile(99)); // Same as h.percentile(99)
console.log(h2.ewmaMean); // EWMA state preserved
h2.record(99999); // Can record new valuesThe encoding uses delta-encoded sparse bucket counts. Only non-zero buckets
are included, and the bucket indices are delta-encoded (each index is stored
as the difference from the previous one). For a histogram with 1000 recorded
values across ~1000 distinct buckets, the export is about 2 KB. The full
bucket array at figures=3 is 184 KB. The encoding also includes all EWMA
and SLO threshold state.
The output is standards-compliant CBOR. Any CBOR decoder in any language can parse it. This is intentional. If an APM agent collects a histogram in a Node.js process and ships it to a backend written in Go or Java, the backend doesn't need Node.js-specific code to read it. It just needs a CBOR decoder and the key layout (documented in the PR).
Benchmark analysis without R
The benchmark/compare.js tool in the Node.js repo has always required R to
analyze benchmark results. The PR adds a new --analyze flag that performs the
same Welch's t-test inline using the histogram API's welchTest() method.
The output matches R's compare.R at two decimal places.
node benchmark/compare.js \
--old ./node-main --new ./node-pr \
--runs 30 --analyze buffersThe --max-regression flag adds gating. If any statistically significant
regression exceeds the specified percentage, the process exits with code 1.
node benchmark/compare.js \
--old ./node-main --new ./node-pr \
--runs 30 --max-regression 5 buffersThis can replace the R dependency for the common case. R is still useful for plot generation if needed, but for "did this PR regress performance?" you no longer need anything beyond Node.js itself once these PRs land.