Here's a scenario for this week. A stakeholder walks into a review and the dashboard says average session duration is up 18%. This is good. Someone makes a comment about engagement being strong. Nobody asks a follow-up question. We all move on.

Then you dig in to it. A handful of sessions ran for three hours because users left a tab open. In actuality the median barely moved at all. The "increase" presented was statistical noise dressed up as a trend.

This happens more often than realized, not because analysts are careless, but because certain metric types are structurally deceptive. They look like answers but they're actually questions. This issue breaks down four of the most common offenders and what to do instead.

HIGHLIGHT

The average. Averages are fine when your data is roughly symmetric. They fall apart the moment you have a skewed distribution or meaningful outliers, which is almost always in practice.

The fix: Pair averages with a median and a 90th or 95th percentile. If the average and median diverge significantly, the average is lying. Report both and explain the gap.

The conversion rate. Rates look like they tell the whole story but they tend to drop the denominator. A 12% conversion rate sounds solid until you find out the total volume dropped by half and you're now converting a smaller, self-selected group who were always going to convert. This is how teams convince themselves a product is improving while the actual business is shrinking. The rate went up. The total conversions went down. Both are true. The rate is the one you see on the dashboard.

The fix: Show rates and absolute volumes together, always. A rate without a denominator is a press release, not an analysis.

The aggregate sum. Total revenue, total signups, total page views — these feel like ground truth. But aggregates hide composition shifts. You can hit the same total revenue two months in a row while your product mix shifts dramatically underneath: high-margin products declining, low-margin products picking up slack. The top line doesn't flinch. The P&L eventually does. This one is especially dangerous in BI because aggregate sums are easy to build and easy to present. Executives see them and feel confident. That confidence is often unearned.

The fix: Break aggregates down by dimension, segment, product line, region, cohort. If the breakdown looks different from the total, that difference is the story.

The period-over-period change. MoM and YoY comparisons are among the most trusted formats in any business dashboard. They're also easy to manipulate, intentionally or not, through period selection. Comparing against a period with an anomaly (a product launch, a seasonal spike, a data incident) makes the current period look better or worse than it actually is. Choosing the start date of a trend line strategically can make any story feel credible.

The fix: Use rolling averages or trailing N-period comparisons to smooth noise. Always note what happened in the comparison period. If last year had a one-time event, say so explicitly, don't just let the number speak for itself.

What statistical concept describes the gap between a metric's reported value and the thing you're actually trying to measure?

A. Measurement error
B. Construct validity
C. Sampling bias
D. Overfitting

Answer at the bottom of this issue

Simpson’s Paradox and its Implications in Data Science by Nisha Arya
Simpson's paradox proves the importance of being skeptical when interpreting data, a trend that appears in aggregate can completely reverse when you break it down by subgroup. The Facebook vs. Google conversion rate example they use is a clean, practical illustration your audience will immediately recognize. Read more →

How Mean is The Mean! Beware the Misleading Mean, for Data Analysts by Nam Nguyen
The framing here borrows from an economist's critique of GDP per capita, then extends it to any measurement that involves averaging a total across a number of observations. A bit more academic in tone than the other two, but the core argument is tight and it pairs well with the averages section of this issue. Read more →

Are You Using the Wrong Average? by Dmitri Spiropoulos
The key takeaway is that data reporters should choose the appropriate average based on their dataset's characteristics rather than defaulting to the mean, as using the right measure ensures accurate representation and better decision-making. Read more →

Use VizBuddy in Chrome to quickly summarize any article

Apache Superset #data-visualization
Open-source BI tool with built-in support for percentile charts and distribution views. Good for teams that want more than a bar chart.

FREE Power BI Theme Generator #productivity #data-visualization
A newer generator on the market with a live preview, light/dark mode support, accessibility-friendly color palettes, and the ability to generate palettes from an uploaded image. A solid pick for those who want their reports to look as good as they function.

▶️ This is How Easy It Is to Lie With Statistics by Zach Star (18 mins)
A Zach Star video walking through how statistics can mislead without using a single incorrect number, covering relative vs. absolute risk, the prosecutors fallacy, Simpson's paradox, and truncated bar charts. The through-line is that the math can be technically accurate while the conclusion drawn from it is completely wrong, sometimes with serious real-world consequences like wrongful convictions. Solid watch for anyone who presents data to non-analysts.

What statistical concept describes the gap between a metric's reported value and the thing you're actually trying to measure?

A. Measurement error
B. Construct validity ✅
C. Sampling bias
D. Overfitting

It refers to whether a metric actually measures what it claims to measure. Low construct validity is the formal name for what's happening every time a dashboard metric tells a misleading story.

Newsletter publishing is hard work and it’s just me running the show here. If you ever feel like extending a thanks, idea, or insult you can do that here.
Or email me directly at → [email protected]

If you’re feeling generous, I also keep an Amazon Wishlist of books and tools I’m interested in.