We ran 11,440 backtests. The average 'winning' strategy lost half its edge on data it had never seen

Every trading forum has the same recurring post: a beautiful equity curve, a tuned parameter set, and the caption "this works". We wanted a number for how much of that is real. So we measured it — at scale, with a protocol that doesn't let us cheat, and with the sweep code committed to our repository so anyone can re-run it.

The short version: we tuned 200 strategy-coin combinations on their own history and ran each tuned winner once on data it had never seen. The median attractive winner's Sharpe ratio fell from 1.22 to 0.64 — roughly half the "edge" was memory, not signal. 54% lost more than half their Sharpe, and across all 200 tuned picks, 44% went to zero or negative. What follows is the method, the charts, the numbers that surprised us in both directions, and what actually survives.

The experiment

We took the ten strategy families from our public template library — the same YAML rule sets anyone can run on the site: RSI mean-reversion, EMA crossover, MACD trend, SuperTrend, SMA breakout, Bollinger reversion, ATR breakout, Trend Magic, and two smart-money-concepts templates (order blocks and fair value gaps).

For each family we defined a small, realistic parameter grid — the same knobs the strategy's users actually tune: lookback periods, entry/exit thresholds, band widths, stop distances. No exotic parameters and no thousand-combination grids: between 9 and 64 combinations per family, exactly the kind of "let me try a few settings" exploration every trader does.

Then, for each of 20 liquid USDT pairs (BTC, ETH, and the rest of the large-cap universe), on daily bars from January 2021 to July 2026:

1. **Split the history chronologically: first 70% in-sample, last 30% out-of-sample.** The out-of-sample window is roughly the last 20 months — market conditions the in-sample window doesn't contain.
2. **Run every parameter combination on the in-sample window** and pick the one with the best Sharpe ratio. This is the config a curve-fitter would ship — and, honestly, the config most backtest screenshots show.
3. **Run that winner once on the out-of-sample window.** No re-tuning, no second look, no survivorship: every result is in the data below, including the embarrassing ones.

That's 5,720 configurations, each run on both windows — 11,440 engine runs in total, all through the same event-driven backtester that powers our free backtester, with its default fee and slippage model. The sweep script, the grids, and the analysis code live in our repo; the study is reproducible end to end.

Finding 1: most combinations never looked good in the first place

Before any out-of-sample test, a sobering baseline: even with the freedom to pick the best of dozens of settings, only 37 of 200 strategy-coin combinations (18%) produced an "attractive" in-sample result — a Sharpe of at least 1.0 with at least 10 trades. Three families (RSI mean-reversion, MACD trend, and Bollinger reversion) didn't clear that bar on a *single* coin out of twenty, even after tuning.

Remember that the next time a screenshot shows you the one that worked: you are looking at the survivor of a large, invisible pile.

Finding 2: the out-of-sample haircut

If tuning found pure signal, the dots would hug the diagonal. They don't — the cloud sits systematically below it. For the 37 attractive winners:

• **Median Sharpe fell from 1.22 in-sample to 0.64 out-of-sample.** About half the apparent edge evaporated on contact with unseen data.
• **54% lost more than half their Sharpe.** The **median haircut was 0.69** Sharpe points.
• **16% collapsed outright** to a Sharpe of zero or below.
• Across **all 200** tuned picks (not just the attractive ones), **44% went to zero or negative** out-of-sample.

Finding 3: in-sample ranking is a weak guide — but not worthless

Here's where the data pushed back on the cynical take, and we're reporting that too. Within each strategy-coin grid we ranked every combination by in-sample Sharpe and again by out-of-sample Sharpe, then measured the rank correlation.

The median rank correlation was 0.37 — positive, but weak. And it was wildly unstable: in a quarter of the grids the correlation was zero or negative (picking the in-sample winner was no better than picking at random), while in the best quarter it exceeded 0.60.

Tuning small, structural grids did carry *some* signal: the tuned pick beat the template's factory-default settings out-of-sample in 69% of cells. So the honest conclusion is not "never tune" — it's that in-sample differences are mostly noise with a little signal mixed in, and you cannot tell which grid you're in without holding out data. The bigger the grid, the more the noise wins: every extra combination you try raises the bar your result has to clear — which is exactly what a deflated Sharpe ratio corrects for, and why our robustness scorecard uses it.

Which families held up

The spread is instructive. SuperTrend was the most prolific in-sample winner (13 of 20 coins) — and 8 of those 13 lost more than half their Sharpe out-of-sample. The fair-value-gap template produced 11 attractive winners and held up comparatively well. And the three families that never produced an attractive winner anywhere aren't necessarily *bad* strategies — on daily bars in this period, they simply never had the edge their reputation promises, no matter how the knobs were turned. That is also worth knowing before real money finds it out.

What this doesn't say

An honest study lists its limits: this is one asset class (crypto), one timeframe (daily bars), one split point (70/30), one grid per family, and the engine's default fee model. Sharpe on a ~20-month window is itself a noisy estimate — some "collapses" are bad luck rather than curve-fit, and some survivors are lucky. A long-only family meeting a hostile out-of-sample regime takes an unfair beating. None of that rescues the headline: across ten unrelated strategy designs, in-sample tuning systematically promised roughly twice what it delivered.

And one thing this study explicitly is not: evidence that backtesting is useless. It's evidence that *unvalidated* backtesting manufactures confidence. The fix isn't to stop testing — it's to test the way this study did: hold out data, touch it once, count your attempts, and let the result survive on data it never saw.

Run your own strategy through the same gauntlet

Everything this study did by hand — the held-out data, the robustness scoring, the honest verdict — is what our free backtester runs on any strategy in about a minute. No signup, no card. If your idea is real, it will survive. If it isn't, you just found out for free.

Past performance does not guarantee future results. This study is educational content, not financial advice.