What Do You Do When Your Own Backtest Engine Is Wrong?
· #research-method #backtesting #reproducibility #validation #systematic-trading
You stop letting it be the referee, and you say so out loud. When my research engine and MetaTrader 5 disagreed about the same strategy over the same period, I made the broker's own fill stream the only thing allowed to count as a result, re-ran the work underneath it, and wrote down what the old numbers had claimed.
As a self taught quant trader I went quiet for a stretch, and it was not a content gap. It was me re-checking my own homework. Here is what I found, including the parts that went against me.
How did I find out my own engine was wrong?
The same strategy, on the same period, came back with the opposite sign in two places.
It showed up three separate times before I would call it a pattern rather than a fluke. A trailing-stop configuration gained roughly 24 percent in the engine and lost roughly 24 percent in MetaTrader 5. An entry veto based on volume-node absorption looked flat in the engine and came back catastrophic in MetaTrader 5, at about minus 95 percent net. A Bitcoin configuration reported a profit factor near 1.29 in the engine against roughly 1.06 in MetaTrader 5, which is the difference between a small edge and no edge at all. Those figures come from my own robustness note in the DQuants research repo, dated 2026-06, and they are backtest and confirmation runs rather than live results.
Two of three of those were bad news I was about to trust. That is not a footnote. If the measuring stick is wrong, every number ever measured with it is suspect, including the ones I liked.
What exactly was the engine getting wrong?
Exits, and in one direction only.
A research engine is a model of the market, and a model is allowed to be wrong. Mine was wrong in a specific, learnable way. It resolves an intrabar exit along a discrete tick path and cannot know what price did inside a tick, so when a trade could plausibly have been stopped out or trailed out mid-move, the engine tended to give it the better of the two readings. Feed geometry made that worse. On my data the post-breakout continuation rate sits near 45 percent, while on another vendor's feed the same measurement comes out near 94 percent, so the round trips are not even the same shape.
The conclusion I wrote down was narrow on purpose: entry-side numbers from the engine are trustworthy, and exit-side numbers systematically over-credit runners. Flattering exit accounting turns a mediocre system into a beautiful curve on screen. Then a real broker fills you at real prices and the beauty evaporates.
What rule did I adopt instead?
One judge, and it is not my code.
Only the broker's own deal stream on real ticks counts as a result now. The engine proposes and it does not settle. In practice that means entry ideas can still be swept in the engine, where it is fast and honest, while every exit idea goes straight to a MetaTrader 5 A and B test before it is allowed to change anything. It is slower, and it is the only version I can defend later.
Underneath that sits a gate I would recommend to any engineer trading their own code. Before a build can deploy, its per-trade output must be byte-identical to the reference run, checksum and all. That gate once caught a regression that made the reported result better rather than worse. Eyeballing a profit and loss curve passes a defect like that happily. The checksum did not. A hostile external review I commissioned in August 2026 flagged the ordering as the right one, in the reviewer's framing: the platform I do not control overrules the engine I wrote.
Which convenient assumptions are still in my cost model?
Three, and I would rather you hear them from me.
My tick engine currently models spread and optional commission, and it models slippage, latency and swap at zero. On top of that, the gold feed I research on carries a spread roughly ten times tighter than what I actually see on a live account, on the order of 19 points against 189. My own notes list this as the highest-truth-value gap in the whole stack and also the cheapest one to close, which is an uncomfortable sentence to have written and not yet acted on.
A convenient assumption is a loan. The market collects it later, usually with a fee. Publishing the outstanding balance is not modesty, it is just the accurate version.
Why not just show the window where it looked good?
Because a winning window exists for almost everything, if you are willing to hunt for it.
I have a concrete case in my own files. One Bitcoin configuration is net negative across the full test window and net positive across the more recent slice of it. Both statements are true. Reporting only the second one would be marketing with a spreadsheet attached, and it is the easiest lie in this field to tell without ever writing a false sentence. Something similar applies to my thinner gold configuration, where the headline profit factor near 1.43 is carried almost entirely by one strong held-out quarter, and the training window on its own is closer to 1.15.
So the recent, untouched, out-of-sample period is the only fair test I let myself report, and I report it whether it flatters me or not. Honesty in backtesting turns out to be mostly about which numbers you refuse to use.
What was the number that hurt to read?
Two of them, and the second was worse.
The first is live. The only real money my flagship logic has traded is a small prop-account stream of 62 trades at a profit factor of 0.68, which is under break-even and also not statistically significant at that sample size. The honest headline there is not a percentage. It is that there is no live evidence yet, in either direction.
The second came from that external audit. My most-quoted robustness number, a deflated Sharpe ratio of 1.000, turned out to sit on a gate that, at my own recorded trial dispersion, would need something like ten to the fourteenth candidate configurations before it could ever fail. The auditor's line was blunter than anything I would have written: the verdict is not evidence, it is arithmetic. Worse, the test that does discriminate, the probability of backtest overfitting, came back above 0.5 in eight of nine cells and I had let a narrative override it.
None of that proves the edge is fake. It proves the evidence I was leading with could not have shown me if it were.
Why publish any of this?
Because you cannot separate not lying to yourself from not lying to a reader.
A brand that only posts wins is a slot machine with a logo. The research log behind these articles currently holds 137 experiments, of which 45 are rejected outright and 72 are research-only, against 11 that made it to a locked configuration. That is roughly a one-in-twelve hit rate with every failure kept queryable, and it is a far better description of the work than any single curve. It even holds a prediction of mine that failed and is recorded as failed: I guessed that fewer than 25 of about 180 exposed parameters actually changed which trades happen, and the measured answer was 49.
I would rather be slow and correct in public than fast and wrong in private. The audit that embarrasses me today is the credibility I get to spend later.
Frequently asked questions
What does it mean when a backtest engine and MetaTrader 5 disagree? It means one of them is modelling something the other is not, and until you know which, neither number is usable. In my case the disagreement was concentrated on the exit side and was directional rather than random, which is the signature of a modelling bias rather than noise. The platform outside my control gets to be right by default.
Why do backtest engines over-credit exits? Because a bar or a tick is not a path. If a stop and a target both sit inside the same interval, the engine has to guess the order they were touched in, and the convenient guess compounds across every trade with a runner. Entry decisions do not have this problem, since the decision is made at a price the engine can actually see.
How do you stop an engine bug from flattering your results? By making a system you do not control the referee, and by gating deployment on exact reproduction rather than on a similar-looking curve. My deploy gate requires the shipped build's per-trade output to match the reference run byte for byte. It once caught a defect that raised reported profit, which is exactly the kind of bug a human review waves through.
Is a losing out-of-sample window a reason to abandon a strategy? Not on its own, but it is a reason to stop quoting the good window. A configuration of mine came back net negative on the most recent untouched period, and an older, friendlier window exists. That older window overlaps the data the thing was built on, so using it would be selection rather than evidence.
Why is cherry-picking a backtest so hard to spot? Because it needs no false statements. Every number in a cherry-picked report can be individually correct while the report as a whole is misleading, since the choice of window is where the dishonesty lives. The defence is to fix the reporting window before you look at the result, not after.
Which part of MasterVP reads the chart, and which part trades? Both halves exist, and the split matters. The MasterVP Profiler is the indicator, on TradingView and on MetaTrader 5: it shows you the read, where volume actually traded and how the chart is structured, and it leaves the order with you. It is not a signal service. Execution lives on the Expert Advisor side, and it trades only the rules a user configures for themselves.
Is there a live track record behind any of these numbers? No, and I will not imply one. Every figure above is a backtest, a MetaTrader 5 confirmation run on real historical tick data, or a finding from an external audit of my process. The one live stream I have is 62 trades and under break-even, which is too small to conclude anything from. Live results should be expected to be thinner than research, because slippage, latency and swap are not in my cost model yet.
What is a parity gate, in plain terms? It is a rule that a piece of software must reproduce a known reference run exactly before it is allowed anywhere near money. Not approximately, and not on the summary metrics, but trade for trade. It is ordinary practice in software engineering and it is the single habit from my day job that transferred to trading research with no modification at all.
Written by KenKem, a software engineer and founder of twenty years, learning quantitative trading in the open and publishing the process, rejections included.
This article was composed from posts 221 to 230 of the KenKem build-log series. Educational purpose only. Not financial advice. All figures cited are backtest, confirmation-run or audit results, not a live track record. Past performance does not guarantee future results.