Skip to main content

Making Node.js histograms more useful

I first added the Histogram API to Node.js in v11. It let you record values, get min/max/mean/stddev, and pull percentiles. It's been useful but limited. If you wanted to do actual statistical analysis of your latency data you had to export the raw percentile map and do the math yourself, or pipe it into an external system and it's always been something I've wanted to get back to.

Since I've been working on QUIC and Streams stuff so much, I wanted to take an afternoon to work on something different so I decided to work on expanding the histogram API with a set of analytical methods that let you do more with the data without leaving the process. The changes have landed in the main branch and will hopefully hit a release soon.

What's new

The base Histogram class gets:

  • cdf(value) and ccdf(value) for cumulative distribution queries
  • skewness and kurtosis getters for distribution shape
  • ksTest(other) for comparing two histograms
  • percentilesAt(percentiles) for batch percentile queries
  • linearBuckets(stepSize) and logBuckets(firstBucket, base) for rebucketing
  • countAt(value) for bucket-level counts

RecordableHistogram gets:

  • recordCorrected(val, expectedInterval) for coordinated omission correction
  • subtract(other) for computing interval diffs

Under the covers, the Histogram implementation is backed by the HdrHistogram C library.

skewness and kurtosis

The mean and stddev you already had provide the center and spread of your data. They don't tell you what the distribution actually looks like. Two distributions can have identical means and standard deviations but completely different shapes.

skewness measures asymmetry. Latency distributions in real systems are almost always right-skewed: the bulk of the values cluster at the low end, with a long tail to the right. A positive skewness value confirms this. A value near zero would indicate a roughly symmetric distribution. In practice, if you're measuring request latencies and the skewness is near zero or negative, something unusual is going on.

kurtosis (specifically, excess kurtosis) measures the heaviness of the tails relative to a normal distribution. A positive value means the tails are heavier than normal. More extreme outliers. For latency data, high kurtosis tells you that your tail latencies are not just a little worse than the median, they're significantly worse. If you're investigating why your p99.9 is orders of magnitude worse than your p50, the kurtosis value quantifies how extreme that effect is.

const { createHistogram } = require('node:perf_hooks');
 
const h = createHistogram();
 
// Simulate a right-skewed latency distribution.
for (let i = 0; i < 10000; i++) {
  h.record(Math.ceil(Math.random() * 100));
}
// Inject some tail latency outliers.
for (let i = 0; i < 50; i++) {
  h.record(500 + Math.ceil(Math.random() * 2000));
}
 
console.log('Skewness:', h.skewness.toFixed(4));
console.log('Kurtosis:', h.kurtosis.toFixed(4));

A practical use: comparing the shape of your latency distribution across deployments or time windows. If the skewness or kurtosis of your latest release has shifted significantly from your baseline, that tells you the overall character of your latency profile has changed, not just a single percentile.

CDF and CCDF

Percentiles answer the question "what latency value do X% of requests fall under?" The CDF inverts this. It answers "what percentage of requests complete within Y milliseconds?"

This is the question you actually want to ask when you have an SLO. You don't care what the p99 is in the abstract. You care what fraction of your requests finish within your target.

const { createHistogram } = require('node:perf_hooks');
 
const latency = createHistogram();
// Record request latencies in nanoseconds...
 
// "What fraction of requests complete within 100ms?"
const withinSLO = latency.cdf(100_000_000);
console.log(`${(withinSLO * 100).toFixed(2)}% within 100ms SLO`);
 
// "What fraction exceed 500ms?"
const violating = latency.ccdf(500_000_000);
console.log(`${(violating * 100).toFixed(2)}% exceeding 500ms`);

ccdf(value) is 1 - cdf(value). It tells you the probability that a recorded value exceeds the threshold. When you're monitoring SLO violations, the CCDF is the direct answer: if ccdf(your_threshold) returns 0.02, then 2% of your traffic is violating your SLO.

You can use this to build a continuous SLO monitor that runs inside the application process. No need to ship all of your latency data to an external system just to compute a compliance percentage. Record your latencies into a histogram, query the CDF at your SLO boundary, emit a metric or fire an alert.

const { createHistogram, monitorEventLoopDelay } = require('node:perf_hooks');
 
function createSLOMonitor(thresholdNs, targetFraction) {
  const h = createHistogram();
  return {
    record(latencyNs) {
      h.record(latencyNs);
    },
    check() {
      if (h.count < 100) return null; // Not enough data yet.
      const compliance = h.cdf(thresholdNs);
      return {
        compliance,
        passing: compliance >= targetFraction,
        count: h.count,
        p99: h.percentile(99),
      };
    },
    reset() { h.reset(); },
  };
}
 
// 99% of requests must complete within 200ms.
const slo = createSLOMonitor(200_000_000, 0.99);

Comparing distributions with ksTest

The Kolmogorov-Smirnov test compares two empirical distributions and returns a D-statistic between 0 (identical) and 1 (completely disjoint). It tells you whether two sets of measurements came from the same underlying distribution without requiring any assumptions about what that distribution looks like.

This is the tool for regression detection. Record a baseline histogram of your latency profile under known-good conditions. After a deployment, record a new histogram under the same workload. Compare them.

const { createHistogram } = require('node:perf_hooks');
 
const baseline = createHistogram();
const current = createHistogram();
 
// Record baseline latencies from a known-good build...
// Record current latencies from the new build...
 
const d = baseline.ksTest(current);
if (d > 0.1) {
  console.warn(`Distribution shift detected (D=${d.toFixed(4)})`);
}

The threshold depends on your context. A D-statistic of 0.05 might be normal variance. A D-statistic of 0.3 probably means something changed. You'll need to calibrate this for your workload.

The KS test is useful because it captures changes that percentile comparisons miss. If your p99 stays the same but the entire middle of the distribution shifts, individual percentile checks won't catch it. The KS test looks at the full shape.

You can also use this for A/B testing. Run two code paths, record each one's latencies into separate histograms, compare. The D-statistic tells you whether the distributions are meaningfully different. This works inside the process with zero dependencies.

const { createHistogram } = require('node:perf_hooks');
 
const pathA = createHistogram();
const pathB = createHistogram();
 
async function handleRequest(req) {
  const start = process.hrtime.bigint();
  let result;
 
  if (shouldUseNewPath(req)) {
    result = await newCodePath(req);
    pathB.record(Number(process.hrtime.bigint() - start));
  } else {
    result = await oldCodePath(req);
    pathA.record(Number(process.hrtime.bigint() - start));
  }
 
  return result;
}
 
// Periodically check whether the new path has different latency characteristics.
setInterval(() => {
  if (pathA.count > 1000 && pathB.count > 1000) {
    const d = pathA.ksTest(pathB);
    console.log(`A/B divergence: D=${d.toFixed(4)}, ` +
                `A.mean=${pathA.mean.toFixed(0)}, ` +
                `B.mean=${pathB.mean.toFixed(0)}`);
  }
}, 60_000);

Snapshot diffing with subtract

Sometimes you want to analyze your latency profile over a recent time window rather than since process start. The subtract method on RecordableHistogram enables this.

The idea: maintain a running total histogram. Periodically take a snapshot by copying the total into a new histogram, then subtract the previous snapshot. The result contains only the values recorded since the last snapshot.

const { createHistogram } = require('node:perf_hooks');
 
const total = createHistogram();
let previousSnapshot = createHistogram();
 
function recordLatency(ns) {
  total.record(ns);
}
 
// Every 30 seconds, compute stats for just the last window.
setInterval(() => {
  const window = createHistogram();
  window.add(total);
  window.subtract(previousSnapshot);
 
  console.log('Last 30s:', {
    count: window.count,
    mean: window.mean.toFixed(0),
    p99: window.percentile(99),
    skewness: window.skewness.toFixed(4),
  });
 
  // The current total becomes the previous snapshot.
  previousSnapshot = createHistogram();
  previousSnapshot.add(total);
}, 30_000);

This is how you build a sliding-window latency monitor without keeping individual samples in memory. HdrHistogram's memory footprint is fixed regardless of how many values you record. Three histograms (total, previous snapshot, current window) use a constant amount of memory. You could run this for days without growing heap.

Combine this with ksTest and you get automated drift detection: compare each window's distribution to the baseline and alert when the D-statistic exceeds your threshold. That gives you rolling regression detection inside the application with no external infrastructure.

Coordinated omission correction

recordCorrected(val, expectedInterval) addresses a subtle measurement problem called coordinated omission.

The problem: if you're measuring latency by recording one sample per expected interval, and the system stalls for a long time, you only record one high value. But during that stall, many requests would have experienced high latency. Your histogram is now biased; it underrepresents the tail because it only recorded one bad sample where there should have been many.

recordCorrected compensates by backfilling. If you expected to record a sample every 10ms and the actual measured value was 100ms, it records additional samples at 10ms intervals from the expected value up to 100ms. The histogram then reflects what the full population of requests would have experienced.

const { createHistogram } = require('node:perf_hooks');
 
const h = createHistogram();
const expectedInterval = 10_000_000; // 10ms in nanoseconds
 
// In your measurement loop:
function recordSample(measuredNs) {
  // If measuredNs is 100ms but expectedInterval is 10ms,
  // this backfills 9 additional samples at 10ms steps.
  h.recordCorrected(measuredNs, expectedInterval);
}

This matters most when you're building a load generator or a benchmark harness. If you're not correcting for coordinated omission, your tail latency numbers are optimistic. Gil Tene (the author of HdrHistogram) has written extensively about why this matters. The short version: without correction, your p99.9 might look fine while your actual user experience is terrible.

Batch percentiles and bucketing

Two smaller but useful additions.

percentilesAt takes an array of percentile values and computes them in a single pass over the histogram's internal data. This is more efficient than calling percentile() five times, and it's the natural thing to use when you're emitting a standard set of monitoring percentiles (p50, p75, p90, p95, p99, p99.9) on a timer.

const p = h.percentilesAt([50, 75, 90, 95, 99, 99.9]);
// Returns a Map: 50 => value, 75 => value, ...

linearBuckets(stepSize) and logBuckets(firstBucket, base) return the histogram data rebucketed into fixed-width or exponentially-growing intervals. The result is a Map of bucket boundaries to counts. This is what you need if you're building a histogram visualization, exporting to Prometheus histogram format, or just want to see the shape of your distribution as a frequency table.

// Linear buckets: every 10ms
const linear = h.linearBuckets(10_000_000);
for (const [boundary, count] of linear) {
  if (count > 0) {
    console.log(`<= ${boundary / 1e6}ms: ${count}`);
  }
}
 
// Log buckets: starting at 1ms, each bucket 2x wider
const log = h.logBuckets(1_000_000, 2.0);

Log bucketing is usually the better choice for latency data because latency distributions span several orders of magnitude. Linear buckets give you too much resolution at the low end and not enough at the high end, or vice versa. With log buckets, each bucket covers a proportional range of the value space, which matches how latency actually distributes.

Putting it together: a latency analysis toolkit

Here's a more complete example that combines several of the new APIs into a self-contained latency analyzer. This is the kind of thing you might wire into a health check endpoint or a periodic reporting job.

const { createHistogram } = require('node:perf_hooks');
 
class LatencyAnalyzer {
  #total = createHistogram();
  #previous = createHistogram();
  #baseline = null;
 
  record(latencyNs) {
    this.#total.record(latencyNs);
  }
 
  // Capture the current state as the performance baseline.
  setBaseline() {
    this.#baseline = createHistogram();
    this.#baseline.add(this.#total);
  }
 
  // Get the histogram for just the most recent window.
  snapshot() {
    const window = createHistogram();
    window.add(this.#total);
    window.subtract(this.#previous);
 
    this.#previous = createHistogram();
    this.#previous.add(this.#total);
 
    return window;
  }
 
  // Full analysis of the most recent window.
  analyze() {
    const window = this.snapshot();
    if (window.count < 10) return null;
 
    const result = {
      count: window.count,
      mean: window.mean,
      stddev: window.stddev,
      skewness: window.skewness,
      kurtosis: window.kurtosis,
      percentiles: Object.fromEntries(
        window.percentilesAt([50, 90, 95, 99, 99.9])
      ),
    };
 
    // If we have a baseline, check for distribution drift.
    if (this.#baseline && this.#baseline.count >= 100) {
      result.driftScore = this.#baseline.ksTest(window);
    }
 
    return result;
  }
}
 
const analyzer = new LatencyAnalyzer();
 
// In your request handler:
async function handleRequest(req, res) {
  const start = process.hrtime.bigint();
  // ... handle request ...
  analyzer.record(Number(process.hrtime.bigint() - start));
}
 
// After warmup, capture baseline.
setTimeout(() => analyzer.setBaseline(), 60_000);
 
// Report every 30 seconds.
setInterval(() => {
  const report = analyzer.analyze();
  if (report) {
    console.log(JSON.stringify(report));
    if (report.driftScore > 0.15) {
      console.warn('Significant latency distribution change detected');
    }
  }
}, 30_000);

All of this runs in constant memory. There are no arrays of individual samples growing over time. The HdrHistogram data structure uses a fixed allocation based on the configured value range and significant figures, not on the number of recorded values. A histogram configured with the defaults (range 1 to Number.MAX_SAFE_INTEGER, 3 significant figures) uses roughly 180KB. Four histograms for the analyzer above: under 1MB total regardless of how many millions of requests you've served.

Event loop delay: per-iteration sampling

One more change worth mentioning. PR #62935 by Pablo Erhard added a samplePerIteration option to monitorEventLoopDelay(). The existing implementation samples event loop delay on a timer interval. The new option uses uv_prepare_t and uv_check_t handles to measure the delay on every single event loop iteration.

const { monitorEventLoopDelay } = require('node:perf_hooks');
 
const h = monitorEventLoopDelay({ samplePerIteration: true });
h.enable();
 
setTimeout(() => {
  h.disable();
  console.log('Event loop delay (per-iteration):');
  console.log('  mean:', (h.mean / 1e6).toFixed(2), 'ms');
  console.log('  p99:', (h.percentile(99) / 1e6).toFixed(2), 'ms');
  console.log('  skewness:', h.skewness.toFixed(4));
  console.log('  samples:', h.count);
}, 10_000);

Timer-based sampling gives you a statistical sample. Per-iteration sampling gives you every iteration, which means you won't miss short-lived spikes that happen to fall between sampling intervals. The tradeoff is slightly more overhead per event loop iteration (two uv_hrtime() calls), but for diagnosing intermittent stalls it's worth it.