What Should You Ask Before You Believe a Backtest?

Before believing a backtest, ask six things: how much of the history the system never saw, how many ideas were tried before this one, how many trades it contains, the worst drawdown and how long it lasted, where the chart starts and stops, and which costs were charged. A result that cannot answer all six is a description of a window, not evidence.

MasterVP · Systematic Trading Journey

Written by the founder of KenKem, a software engineer of twenty years who traded gold by hand before building the research stack behind MasterVP. As a self taught quant trader I use this checklist on my own work first, because it catches me as reliably as it catches anyone else. This is knowledge sharing, not a recommendation to trade anything. Every figure below is a backtest on real tick data, dated and labelled, and none of it is a live result.

What are the six questions?

How much history, and did the system ever see it? A result on the window the settings were built in describes that window. The useful part is what ran afterwards, on data the settings had never met.

How many things were tried first? One idea that works means something. The best of thousands is partly luck, and the number of attempts tells you how much.

How many trades? A beautiful curve made of forty trades is a coin that landed well. There is no clean threshold, but the count should come before anything else.

What is the worst it got? Not the average. The deepest peak to trough fall and how long it stayed there, because that decides whether a real person could hold on.

Where does the chart start and stop? Both ends are choices.

What was charged? Spread, slippage, commission, financing and the delay between deciding and filling. A test without them is a test of arithmetic, not of trading.

I have tried to make my own gold system answerable on every one. The honest result is that it fails one of them in a way I have published, and that failure is most of this article.

Why does the number of attempts change what a result means?

Because something always comes out on top when you look enough times.

In my quant development process I count the looking. As of the August 2026 write up, the Dquants engine I developed had simulated 7,718 distinct configurations on real ticks to ship three systems, roughly two and a half thousand examined for every survivor. Of 137 pre registered experiments at that point, 122 never became a product. I wrote about how that registry works in what I do when a trading system stops working.

That count works against me in the short term, and it is what lets a reader discount my results correctly. A result should be penalised for how hard its author searched. If the author will not say, the arithmetic cannot be done.

What is the strongest test that a system is not curve fitted?

Make the shipped settings compete against an optimiser that has every advantage except hindsight.

Freeze the settings that ship and score them on every six month block of history in turn. Then run a second version that may re-choose its settings before each block from a stated menu, using only data from before that block. Neither version may look forward. If the re-tuning version wins, the shipped settings were nothing special. If the frozen settings win, they are beating a search that could adapt.

As of August 2026 I ran it on my ten minute gold system across eleven consecutive half year blocks from 2021 to 2026, with 72 candidate configurations on the menu. Eight of the eleven blocks were positive and three lost, the sideways stretches of 2021 and early 2023. Over the nine blocks where both versions existed, the frozen settings finished ahead. Re-tuning every six months would have cost money.

The test contains a trap worth naming. A walk-forward spends its first year building a training set, so the choosing blocks only start in 2022, and 2021 is the worst year in the record. Dropping it would have flattered me by construction, so it was scored under the frozen settings and left on the chart.

Why do the start date and the axis matter so much?

Because both can mislead with a real, unfaked result.

Move the start date one quarter later and a whole bad year can disappear. The trades are real and the picture is false. The axis is subtler: if the vertical scale begins above the lowest balance the account ever reached, the worst moment of the run is simply absent, and an absence gives you nothing to notice.

A logarithmic axis looks like a trick and is not one. When one curve is many times larger than another, a log scale makes equal heights mean equal percentage moves, which is the right choice for comparing things of different size, as long as it is labelled.

When I chart every configuration I would consider releasing, the bottom of the axis sits at the worst balance any of them reached. One of the more aggressive settings spent the whole of 2024 below its starting balance, and it is visible.

What happened when I tested my system on years it had never seen?

It lost money, and I would rather say that first.

In August 2026 I bought three more years of real gold tick history, 2021 to 2023, that did not exist in my research set while the system was built. I ran the ten minute system in the exact configuration that was trading, without touching a parameter. It lost about 13 percent, with a peak to trough drawdown just under 25 percent. The record is five and a half years, and the first three are negative.

Buying history after a strategy exists is the closest a retail developer gets to a genuine out of sample test, because that data could not have leaked into any decision. The measured reason was not mysterious. The edge is a trend following breakout, built inside one of the strongest gold bull markets in decades, and in a sideways market the same logic keeps taking breaks that do not follow through. A system whose edge you cannot name is one you cannot reason about when it starts losing.

What does a fix worth trusting look like?

It pays in the bad regime and costs almost nothing in the good one.

The change I shipped is a volatility floor: when gold's typical range is too small a fraction of its price, the system stands aside. The level that arms it stays private. On the three losing years it turned a loss of about 13 percent into roughly one percent, and cut the drawdown from just under 25 percent to around 14. On the 2024 to 2026 window the net result did not move.

A fix derived from the years that failed will always look good on those years, so that is not clean evidence. The cleaner check was letting the blind walk-forward optimiser choose from a menu that included no floor at all. In all nine blocks where it had a choice, it picked a floor. It still does not make 2021 to 2023 profitable. It makes them survivable, which is a smaller claim, and the one I am making.

Can a validation metric be broken in your favour?

Yes, and it will never tell you, because it keeps agreeing with you.

In August 2026 I rebuilt my release gates. I had been quoting a statistic designed to penalise a result for the size of the search, then found it read near perfect for almost any candidate once the sample was large. A filter that says yes to everything is decoration. It is now non-binding, and no system of mine is called finished on backtest evidence alone. Under those rules my own flagship is graded as a forward test rather than locked.

The same month I found an accounting defect. One broker I test on bills the whole round turn commission on the entry deal, and my per trade export was not expecting it, so rows for that broker sat about 11.5 percent optimistic. I fixed the export, took that broker's headline figures from MetaTrader's own report, and left the old rows in place with a note saying which ones are wrong and by how much.

A bug that makes results worse is found the same day. A bug that makes them better can sit for months. The only defence I know is auditing the things that are going well.

Frequently asked questions

How do I know if a backtest is overfitted? Ask how many configurations were tried, and whether frozen settings beat a walk-forward that could re-tune with no hindsight. In my own August 2026 test, frozen settings finished ahead of the re-tuning version over nine half year blocks.

How many trades does a backtest need? There is no universal number, and it depends on how many ideas were tested to find it. Forty trades is a coin landing well. Ask for the count before looking at the curve.

What is a genuine out of sample test for a retail trader? Data that did not exist in your research set when the system was built. I bought three years of gold ticks after building mine, and the unchanged configuration lost about 13 percent on them.

Why does the start date of an equity curve matter? Moving it one quarter can remove an entire losing year from a real result. Check both dates before the shape.

Is a logarithmic chart a way to hide losses? No, when it is labelled. It makes equal heights mean equal percentage moves, which is correct for comparing curves of very different size. A cropped vertical axis is the real risk.

How can a fix be curve fitted? Any fix derived from the losing years will improve those years. Check that it costs almost nothing in the good years, and whether a blind optimiser would have chosen it too.

Should I trust a validation statistic that always passes? No. A metric that has never disagreed with you is not filtering anything. I made one of mine non-binding in August 2026 for that reason.

Why publish a bug that flattered my own results? Because deleting inconvenient rows hides that they existed, and a flagged number still teaches something. Mine left one broker's derived rows about 11.5 percent optimistic.

If you want to see where this process ended up, MasterVP is my own intraday method built on public volume profile theory. It ships as the Master Volume Profiler indicator on TradingView and MetaTrader 5, which shows the read and leaves the order with you, and as Master Volume Sniper on MQL5 market, an Expert Advisor that executes the rules and risk settings you configure.

---

This article was composed from the believe-a-backtest posts of the MasterVP build log series. Educational purpose only. Not financial advice. Figures cited are backtest results on real tick data, and past performance does not guarantee future results. Every trader remains responsible for their own risk settings and decisions.