How Do You Verify a Backtest When You Wrote the Backtester Yourself?

You build a second implementation and refuse to trust either one until they reproduce each other. That is the short answer. A backtest written by one person and checked by that same person is a number with no independent witness, and the person most motivated to believe it is the one who wrote it.

I am a software engineer before I am a trader, and this is the part of quantitative trading nobody puts in a screenshot. What follows is the verification machinery I actually run, including the time it caught me.

Why can't you trust a backtest you wrote yourself?

Because you wrote the tests too, and you share every blind spot with them by construction. A bug that flatters your results does not announce itself. It arrives as a good equity curve, which is exactly what you were hoping for, so nothing in you wants to look harder.

In a normal engineering job there is a reviewer, a QA function, an incident that embarrasses you in front of colleagues. A solo researcher has none of that. Whatever adversarial pressure exists has to be built into the process deliberately, in advance, at a moment when you have no result yet and therefore no stake in the answer.

That reframes the work. Most of my time is not spent finding edges. It is spent building reasons to disbelieve the ones I find.

What does running two independent engines catch?

Real defects in my own code, which is the only reason the rule survives how annoying it is.

Every strategy goes through two completely separate pieces of software. The first is a C++ laboratory built for width, fast enough to simulate thousands of configurations on real tick data. That matters, because the alternative is testing the three ideas that happened to occur to you on a Tuesday and calling it a search. To ship three systems, that engine simulated 7,718 distinct configurations, roughly two and a half thousand for every one that survived.

The second is MetaTrader 5, also on real ticks, and it is the courtroom. It models the broker, the execution, the fills, all the unglamorous reality that decides whether a theoretical edge survives contact with an actual account. Where the two disagree, MetaTrader wins and my engine is the thing that gets fixed. Behind the three shipped systems sit 178 independent MetaTrader referee runs, each configured and executed by hand.

The important rule comes before either engine is believed: they have to reproduce each other on the same window first. The gate is one percent on net profit and loss, with zero unmatched trades on either side and zero direction or exit mismatches, and a failure is treated as an engine defect rather than an interesting difference of opinion. Even a good parity run is a near match rather than proof, which is why the fast engine is only ever allowed to rank configurations, never to certify one.

That check has caught me properly once. On a reference run my own engine overstated the result by about 48 percent, from three stacked defects. The loader was silently dropping a configuration key, so the engine took trades the live rules never take. A volatility reading was taken from the wrong bar, closed instead of forming, which left every capped stop a few percent too wide. The remaining gap closed to about one percent conservative, which is a pass. Every conclusion I had drawn in the meantime was wrong. I did not find that by being careful, I found it because two implementations refused to agree and I could not make the difference go away by wanting it to. The lasting fix was not the patch, it was making both loaders fail loudly on a key they do not recognise instead of ignoring it.

The same instinct applies when new data finally arrives. Two months of held back history were imported and replayed once, with no tuning. Before reading a single new number, the replay pipeline was proved correct by re running the previous month through it and confirming it reproduced the existing records trade for trade, 119 of 119 on one system and 39 of 39 on the other. Only then were the new results read. Prove the instrument, then take the measurement, and do it in that order, because afterwards you will not be able to tell which you were doing.

Why does the label on a tick file mean nothing?

Because a file can be sold as tick data while being generated, thinned, or full of gaps, and everything tested on it will quietly look better than it should.

Start with why ticks are needed at all. A one minute bar tells you where price opened, where it closed, and the extremes it reached. It does not tell you the order those extremes arrived in. Inside that bar price might have run to your target first and then to your stop, or to your stop first and then your target, and the bar looks identical either way. For anything holding minutes rather than days, that is not a detail. That is the trade.

A bar based backtest has to guess the path, and every guessing convention I know of is generous to the strategy in some situations and brutal in others. The result carries a fudge factor you can neither see nor size. Ticks remove the guess, because the path is the data rather than something reconstructed afterwards.

The second lesson took me longer, and it is measurable. I verify tick history by density before letting any of it near a conclusion. A tester that silently generates its own ticks produces four or fewer per minute. A real broker export of the same market runs between roughly 55 and 660 per minute. The gap is not subtle once you count it, and the count is the only thing that distinguishes the two files. Density also drifts with age on the same feed. Brokers thin their older history, so the same source carries about 66 ticks per minute in early 2024 and about 470 by late 2025, which means a backtest on the older stretch is working with far less information than one on the recent stretch. One year of gold history in my own store is roughly 40 million ticks across 312 trading days. A file claiming to be the same thing with a fraction of that is telling you something.

Almost every scalping backtest I have seen presented publicly was run on bars, and almost none of them mentioned it. That is usually not dishonesty. Most people do not know the path is missing.

What makes a cost model honest?

Charging the toll inside the simulation on every trade, rather than subtracting an estimate at the end.

Every trade pays commission, the spread you cross, and the slippage between the price you wanted and the price you got. The more often a strategy trades, the more those compound, and a system that looks spectacular on paper can flip to a loser once realistic costs are applied. A backtest that ignores costs is not a slightly optimistic version of your strategy. It is a different strategy, one you cannot actually trade. My test conditions charge spread, slippage, commission, financing and a 15 millisecond execution delay on every trade, on real gold ticks, and the numbers I publish come out the far side of that.

Subtracting costs afterwards is worse than it sounds, because costs change which trades happen at all. A position the engine would never have opened under real friction still sits in the frictionless record, dragging its outcome along with it. Modelling the toll from the start is the only reason I could diagnose my worst stretch: across 2024 the spread on gold consumed about 8.6 percent of ATR, against roughly 3 to 4 percent in the surrounding periods. That is visible in a costed simulation and invisible in an uncosted one.

Getting it right is not automatic either, and the failure mode is worth naming. One of my brokers bills the whole round turn commission on the entry deal, and my per trade export missed it, so a set of banked figures sat on an understated cost basis until the defect was found and fixed in August 2026. The costs were charged correctly inside the simulation, the export counted them wrong, and the affected numbers were about 11 and a half percent optimistic until they were restated. That is the kind of thing that only surfaces when two paths to the same number are compared.

The same scepticism applies to the data itself. Historical datasets that quietly drop what died, the bankruptcies and the delistings, describe a past nobody actually lived through. I trade two instruments, gold and bitcoin, so survivorship bias is not my sharpest risk, but the underlying question stays useful: where did the losses go, and would I notice if my data had already removed them?

Why does adding knobs make a strategy worse?

Because with enough of them you can fit any past perfectly and predict nothing, and the fitting feels like progress the whole time.

Give a model enough tuned thresholds and it will describe the random accidents of one particular history in beautiful detail. Those accidents will not repeat. Overfitting does not need a hundred parameters either. It arrives with two, then compounds with every rule added to rescue the last one.

Which is why the number of things you tried matters as much as the quality of what you found. Generate enough candidates and an excellent looking result appears by chance alone. So experiments get registered with a pass and fail criterion written down before the result is known. In the count I published in August 2026 the registry held 137 of them, and 122 never became a product: 45 rejected outright, 5 stopped mid campaign, 72 kept as research only. They stay queryable next to the survivors, so nobody has to take my word for the ratio. Once you have seen a result you can always construct a reason it counts, which is exactly why the rule has to be declared beforehand.

Honesty about out of sample data works the same way, and it is easy to overclaim here. I can say a two month window was held back and read once, because that is what happened and the date is on record. I cannot say the same about an older confirmation period that was already visible when a threshold was chosen, even though the selection rule only looked at the recent window. The defensible sentence is narrower than the impressive one, so the narrower one is what gets published.

Why is my live account running last month's build on purpose?

There is a better version of my system on my laptop than the one trading my money, and it is staying on the laptop, for the same reason a backtest means anything at all.

A backtest is a claim about one specific, frozen configuration. The moment I change something, every result I have accumulated describes a system that no longer exists. Ship each improvement as you find it and the thing handling your money never builds a track record. It builds a run of short, interrupted records belonging to a run of slightly different systems, not one of them long enough to say anything.

So changes wait, and a release has to earn its way out. The rule is that a new build must reproduce the test it was signed off against before it may touch a live account. When the volatility floor shipped in August 2026, the release binary reproduced its reference run byte identically, 1,034 trades matched exactly, which is how you prove the change you made is the only change that happened. Improvements beyond that go into a live demo forward test, in real conditions on the current market, where I cannot peek at the answer and talk myself into shipping early. That forward test is still an outstanding item rather than a finished one, and saying so is part of the same discipline.

Two smaller habits came out of the same reasoning, both of them scars. Release filenames now carry their version, because a deploy that silently overwrites a live binary leaves you unable to say what was running. And every shipped configuration file pins all 62 adjustable inputs instead of leaving a dozen to whatever the dialog happened to remember. That one is a scar because a production build once ignored ten load bearing settings for weeks and ran a configuration that existed in no file and no test. Nothing crashed. It simply was not the system I thought I was running, which is a worse failure than a crash, because a crash tells you.

The uncomfortable part is honest: I am knowingly running something suboptimal, and sometimes I watch the newer build do better while the older one trades. That is the price of being able to say the record belongs to the thing that produced it.

If every edge decays, what is actually worth keeping?

The ability to tell whether something is real, because that outlives any particular thing you build with it.

A glowing ten year backtest can be carried entirely by a market regime that no longer exists. Markets move through phases, calm and violent, reverting and trending, and competition is corrosive. A strategy with permanently frozen parameters is quietly betting the world never changes, and it always does. The polite name is alpha decay.

That sits in tension with freezing a build, and the resolution is worth stating plainly. Freezing is what makes a record mean something over a defined window. Reviewing is what stops that record from becoming an heirloom. A configuration stays frozen while it is on trial, and changes to it are earned through the same evidence chain as everything else, not by a newer idea feeling better. The question is never only whether it worked. It is whether it is still working, and how I would know the moment it stopped.

How does automation make the process more honest, not just faster?

Because a schedule does not get tired, skip a day because the market was boring, or quietly lower its standards in week nine.

Almost every recurring job here runs unattended now. Charts get captured and the session analysis written and delivered. Content gets rendered, checked against its own compliance rules, and published. None of it was free. Every one of those jobs cost an evening and some cost several, because the interesting failures only exist in production: a scheduler that runs fine by hand and dies under a stripped down environment, a publish step that marked a post delivered when only some channels had accepted it, a cleanup that was technically correct and operationally destructive.

The point is not the hours saved. It is that the work continues the same way every time, including the parts of the process whose only job is to make my own results look worse. Consistency is not a character trait I possess. It is an outcome I engineered, because I already know what my consistency looks like when it depends on me remembering.

That is the whole argument, really. Configuration is versioned, so a result traces back to the exact settings that produced it. Failures stay queryable beside the successes. Costs are charged inside the simulation. Two implementations must agree to within a percent before either is believed. Decision rules go in writing before results are seen. Held back data gets read once. None of it improves the strategy by a single basis point. Every item makes my reported results worse, which is the point, because each one closes a specific route by which I could have fooled myself without noticing.

I am not asking anyone to trust my results. I am trying to publish enough of how they were made that trust is not required.

Frequently asked questions

How do you validate a backtesting engine you built yourself? By making a second, independent implementation reproduce it before either is believed. My C++ research engine and MetaTrader 5 have to agree on the same trades over the same window on real tick data, to within one percent on net profit and loss and with no unmatched trades on either side. Where they disagree, the trading platform wins and the engine gets fixed. The rule survives because it works: my own engine once overstated a reference run by about 48 percent, from three stacked defects including a configuration key the loader was silently dropping.

Why use two backtesting engines instead of one good one? Because they answer different questions and neither is sufficient alone. The C++ engine is built for width, fast enough to explore thousands of configurations rather than the handful you happened to think of, and 7,718 of them ran to ship three systems. MetaTrader 5 is built for realism, modelling the broker and the fills, and 178 referee runs were configured by hand. The fast engine is trusted to rank configurations and never to certify one.

Is tick data really necessary for backtesting a scalping strategy? Yes, and the reason is specific rather than a matter of taste. A one minute bar does not record whether price reached your target or your stop first, and for a trade that lives inside that bar, the order is the entire outcome. Bar based engines have to guess the path, and every guessing convention flatters the strategy in some conditions and punishes it in others.

How do you know a tick data file is genuine? By checking density rather than trusting the label. A tester generating its own ticks produces four or fewer per minute, while a real broker export of the same market runs roughly 55 to 660 per minute. Brokers also thin their older history, so the same feed carries about 66 ticks per minute in early 2024 and about 470 by late 2025, which means a backtest on the older period is working with materially less information than one on the recent period.

Should trading costs be subtracted after a backtest or modelled inside it? Inside it, on every trade, because costs change which trades happen at all. Subtracting an estimate afterwards leaves positions in the record that real friction would have prevented. My tests charge spread, slippage, commission, financing and a 15 millisecond execution delay. Modelling the toll from the start is the only reason I could diagnose my worst stretch: across 2024 the spread on gold consumed about 8.6 percent of ATR against roughly 3 to 4 percent nearby, which is arithmetic, not a broken strategy.

How many experiments have failed? 122 of 137, in the count published in August 2026. Each is registered with a pass and fail criterion written down before the result was known: 45 rejected outright, 5 stopped mid campaign, 72 kept as research only. They stay queryable next to the successes, so nobody has to take my word for the ratio.

Why not update a live trading system as soon as you improve it? Because a backtest is a claim about one frozen configuration, and changing it invalidates every result you have collected. Update continuously and the system handling your money never accumulates a record long enough to mean anything. A release here has to reproduce the test it was signed off against before it may go anywhere near a live account, and further improvements wait in a demo forward test, which means I am deliberately running something suboptimal.

What is the point of a parity check if the two engines are never identical? The point is the size and the cause of the difference, not perfection. A one percent band on net profit and loss with no unmatched trades is close enough to treat the remainder as tick timing noise, and anything larger is a defect with a specific explanation waiting to be found. The failures that matter are structural: a trade one engine takes and the other does not, a direction mismatch, an exit for a different reason.

Are these live trading results? No. Every figure here is research output: backtest, parity and validation runs on real broker tick data, plus counts from the experiment registry. Sample sizes in this programme still sit below the minimum track record length I would want before calling anything settled, and I would rather publish that than round it off. I run a live test and monitor it, and I will publish what it shows, but quoting research output as performance would be dishonest.

Which KenKem product does what? The Master Volume Profiler indicator runs on TradingView and on MetaTrader 5, and it shows the read while leaving the order with you. Master Volume Sniper on the MQL5 market is the Expert Advisor side, and it executes only the rules and risk settings a user configures for themselves. Neither is a signal service, and neither removes the need to think.

---

Written by KenKem, a software engineer and founder of twenty years, learning quantitative trading in the open and publishing the process, rejections included.

This article was composed from the research method and engine internals posts of the KenKem build log series. Educational purpose only. Not financial advice. Figures cited are audit, backtest and validation results on real tick data, not a live track record. Past performance does not guarantee future results.