How Do You Know a Trading System's Results Are Honest?

You cannot tell from the equity curve. Curves are easy to make smooth. What you can check is whether the author publishes the tests that went against them, names the sample size that actually counts rather than the flattering one, and says which of their own claims they have since withdrawn. An author who has never retracted anything has either been lucky or has not looked.

MasterVP · Systematic Trading Journey

Written by the founder of KenKem, a software engineer of twenty years who traded gold by hand before building the research stack behind MasterVP. In my quant development process the uncomfortable results are the ones worth publishing, because they are the only part a reader cannot get from a sales page. Every figure below is a backtest, a simulation or an audit measurement on real tick data, dated and labelled.

What did a hostile review of my own trading system find?

That the research infrastructure was strong and the research reporting was the liability.

In August 2026 I commissioned an external-eyes audit of Master Volume Profile. The brief was not a friendly read. The reviewer took the stance of a hostile outside quant, instructed to find the weakest part of the process and say so plainly, with every number re-derived from the repository rather than quoted from my own write-ups.

The verdict split in a way I did not expect. On the infrastructure the auditor's words were that they would hire the process: layer separation, a hard parity blocker, negative results preserved and queryable. Then came the other half. The robustness number I led with could not fail. The test that could fail had failed, and I had talked my way past it. The conclusion was not that the edge was fake. It was that my evidence could not demonstrate it was not, while my investor document described it as though it could.

I am publishing what the review found rather than the parts that flattered me. A review I only quote when it agrees with me is marketing with extra steps.

Can a robustness test be impossible to fail?

Yes, and mine was.

For months I led my evidence with the Deflated Sharpe Ratio, which exists specifically to catch a result that came from searching too hard. It scored at the top of its range, so I quoted it as the strongest thing I had.

The audit did the arithmetic instead of reading the verdict. The threshold a result has to beat grows with the spread of outcomes across the trials you ran, and only with the logarithm of how many trials there were. My recorded trial dispersion was roughly an eighth of the result itself. At that dispersion the search would have needed on the order of a hundred trillion trials before the gate could possibly fail. It was not passing. It was arithmetically incapable of doing anything else.

Worse, I had already collected the confirming evidence and read it backwards. An earlier study found that de-duplicating my trial set moved the score by a thousandth. I recorded that as stability. It was a needle that does not move.

Meanwhile the test with genuine discriminating power, the probability of backtest overfitting, came out above one half in eight of nine cells across all three shipped configurations. That says the configuration I picked in-sample lands below median out-of-sample more often than a coin flip would. My override was that those configurations sat high in the out-of-sample percentile range, which is three draws from exactly the distribution the test had just declared unreliable. The question was never whether to ship. It was which test leads the evidence section, and I had put the saturated one first.

How large is a trading sample really?

Smaller than its trade count, and often by an order of magnitude.

I had been quoting a headline of 1,423 trades against a literature minimum of 192, a margin of roughly seven times. The audit measured the research stream properly. On 3,502 trades the distribution carried a skew of 3.49 and an excess kurtosis of 19.5, which is nothing like the bell curve those minimums assume. Converted to a Gaussian-equivalent effective sample, 3,502 trades were worth about 326 observations.

So the real margin was 326 against 192, not 1,423 against 192. Still a margin. Not the one I had been advertising. This is the same failure mode I wrote about in what to ask before you believe a backtest, except the person who needed to ask was me.

What does it mean when thirty trades carry the whole system?

That the average is a much softer number than it looks.

Take the research stream, sort it, and remove the best thirty trades out of 3,502. That is 0.86% of the sample, and it turns the entire system net negative. The top one percent alone accounts for more than the whole net result, which means the remaining ninety-nine percent is collectively negative.

This is not a defect I introduced. It is the shape of breakout trading, where a few large moves pay for a long tail of small losses. But it changes what a profit factor is worth. A ratio quoted on a distribution that lopsided is not a sufficient statistic, and I had been quoting it as though it were.

The counter-evidence belongs here rather than in a footnote, because the audit insisted on it. A twenty-thousand-draw bootstrap of the same stream says the result is robust to resampling that sample. What a bootstrap cannot do is speak to regime risk, since it reshuffles the same two and a half years and assumes the next two and a half contain a comparable population of tail events. The defensible statement is narrower than the one I used to make: the edge is real in the regime I tested, it is carried by a small number of events, and whether it persists is genuinely untested.

Do automated checks catch the bugs that flatter you?

Only if you make them blocking, which is the entire point.

The check I rely on most is unglamorous. Before anything ships, the compiled product replays a frozen reference run and every trade has to match, one for one, byte for byte. It earned its keep on release 1.17, when a defect crept into one of the legs and the reported profit came out higher than the reference, not lower.

Eyeballing the equity curve, I would have nodded and shipped, because the numbers had moved the way I wanted them to move. A parity check does not care which direction an error points. It failed the build. I keep it as a hard blocker rather than a warning because I could not trust myself to override it correctly at the exact moment the number looked good.

The same audit season found six defective presets among my own shipped settings, including two five minute configurations that stopped trading in April 2024 and never restarted. They did not crash. They quietly stood aside, which looks like caution until you check the dates. All six were withdrawn, so release 1.18 shipped with fewer products than the release before it. How one of those protective settings could lock an otherwise recoverable account is in why trading bots blow up accounts.

What did I predict about my own system and get wrong?

How much of it was actually fitted.

The audit flagged that I had around 180 Expert Advisor inputs and 336 engine knobs against roughly 326 effective observations, and that almost none of them had ever been systematically ablated. I predicted that fewer than 25 of those parameters would turn out to matter, then ran 436 replays across the three shipped legs to find out.

Of 231 engine keys, 82 were applied by a preset, 49 changed which trades happen, and 33 shipped at a tuned value. My prediction was wrong, and it was wrong in the direction that costs me: 49, not under 25. That works out to roughly nineteen or twenty effective observations per fitted parameter, which is thin. The honest denominator is 33 rather than the 191 a reader might infer from the input list, and the falsified prediction is recorded as falsified.

Registering the guess before running the test is the only reason I could tell the difference between being right and deciding I had been right afterwards.

Why does a live track record of 62 trades settle nothing?

Because the sample is far too small to carry a conclusion in either direction.

The real money stream is a prop firm account running the five minute system, and at the time of the audit it had produced 62 trades with a t-statistic of minus 0.36. That is not evidence that the system fails. It is not evidence that it works either. It is an honest statement that nothing has been established yet.

The audit's top-ranked recommendation was not another sweep. It was to run the slower legs at minimum size for six months and collect real fills, because everything else in the repository measures the same two and a half years of history. A validated process and a live test that has not yet earned an opinion are two different things, and anyone presenting them as one thing is selling something.

What changed after the audit?

The gate, the ledger schema, and the order the evidence is presented in.

An audit that produces a document and no code change is a performance. A month after this one, the release gate prints a different verdict. Lock records now read "PBO-PASS/DSR-NONBINDING": when the deflation threshold sits below thirty percent of the result, the Deflated Sharpe no longer prints as a pass at all, it prints as non-binding, and the discriminating test leads instead. The experiment ledger gained columns for effective sample size, excess kurtosis and net-without-the-top-thirty, so every future lock states the uncomfortable numbers beside the flattering ones without anyone having to ask.

As of release 1.26 the registry holds 158 pre-declared experiments: 52 rejected outright, 5 stopped mid-campaign, 83 research-only, and 11 that survived to a lock decision. The audit counted 137 in August. A roughly one-in-fourteen lock rate with every failure kept queryable is the number I would want to see from somebody else, which is why it is the number I keep.

It took me quite a while to build my own DQuants framework to significantly speed up the entire development process from hypothesis validation all the way to deployment, and the part of it I would defend hardest is not the speed. It is that the framework makes it expensive to quietly drop a result I did not like.

Frequently asked questions

How can you tell if a backtest is honest? Look for what the author published against themselves: a named worst month, an effective sample size beside the nominal trade count, a list of rejected ideas, and at least one claim they have withdrawn. A presentation with no retractions in it has usually not been audited, only written.

What is the Deflated Sharpe Ratio and why can it be misleading? It estimates the probability that a result is not luck, given how many configurations were tried. It can saturate. The threshold a result must beat scales with the dispersion of outcomes across trials and only logarithmically with the number of trials, so when trial dispersion is small relative to the result, no realistic search width can make the test fail. In my case it would have taken on the order of a hundred trillion trials.

What is the probability of backtest overfitting, or PBO? A test that repeatedly splits the data, picks the best configuration on one half, and checks where it lands on the other. Above one half means the in-sample winner underperforms the median out-of-sample more often than a coin flip. Mine came out above one half in eight of nine cells, and I now lead with it rather than with the test that passed.

Why is the effective sample size smaller than the number of trades? Because the statistics that set minimum sample requirements assume something close to a normal distribution. Trade outcomes are skewed with fat tails. My research stream showed skew 3.49 and excess kurtosis 19.5, which reduced 3,502 nominal trades to roughly 326 Gaussian-equivalent observations.

Is it a problem if a few trades produce most of the profit? It is not automatically a flaw, since breakout systems are shaped that way by design. It does mean averages are unreliable. Removing the best 30 of my 3,502 research trades, under one percent of the sample, turns the whole stream net negative, so a profit factor quoted on that distribution is softer than it appears.

Should a backtest parity check be a warning or a blocker? A blocker. On release 1.17 a defect made the reported profit higher than the frozen reference, not lower, and a warning is exactly what a person talks themselves past when the number moves in their favour.

How many trades does a live track record need before it means anything? More than 62. At the time of my audit the real money sample was 62 trades with a t-statistic of minus 0.36, which is too small to support a conclusion in either direction. Sample size requirements depend on the distribution, which is why effective sample size is the number worth asking for.

What should I ask a developer who sells a trading system? Ask which test in their evidence section is the one that could have failed, and what happened when they ran it. Ask for the effective sample size, not the trade count. Ask what they have withdrawn. The answers are more informative than any curve.

If you want to see how this work became a product, MasterVP is my own intraday method built on public volume profile theory. It ships as the Master Volume Profiler indicator on TradingView and MetaTrader 5, which shows the read and leaves the order with you, and as Master Volume Sniper on MQL5 market, an Expert Advisor that executes the rules and risk settings you configure. Both are documented with the caveats above attached, because those caveats are the reason the rest is worth reading.

---

This article was composed from the self-audit posts of the MasterVP build log series. Educational purpose only. Not financial advice. Figures cited are backtest, simulation and audit results on real tick data, and past performance does not guarantee future results. Every trader remains responsible for their own risk settings and decisions.