Anything that produces a signal, whether it is a person's hunch, an indicator, a bought "strategy" or a machine-learning model, shares the same weaknesses. Knowing them lets you judge claims, including your own, with the right amount of suspicion.
Past patterns are not future guarantees
A model learns from history. It finds relationships that held in the past. Markets change: volatility regimes shift, participants arrive and leave, rules and conditions move. A relationship that existed last year may be weaker, gone or reversed now. This is called regime change, and it is the main reason good back-tests turn into poor live results.
Overfitting: finding patterns in noise
Give a flexible enough method enough data and it will find rules that fit the past perfectly, including the random bits. A rule like "buy when the 13-period average crosses the 47-period average on Tuesdays after a down candle" might have worked in last year's data purely by accident. The more rules, settings and variations you try, the more likely at least one looks brilliant by chance.
Warning signs of an overfitted result:
- Many tuned settings and conditions for few trades.
- Results that fall apart if a setting changes slightly.
- A smooth, steep profit curve with no bad stretches.
- No simple reason why the rule should work.
A rule that makes sense for a plain reason, and still works with sensible changes to its settings, is more believable.
In-sample versus out-of-sample
The honest test is on data the method has never seen. Build the rule on one period, then test on a later period without changing anything. If the results collapse, the first period was telling you about itself, not about the market. Tuning again after a failed test and testing on the same later period turns it back into in-sample data.
Base rates and the win-rate target
On a fixed payout trade the break-even win rate is 1 ÷ (1 + payout), about 51.3% at a 95% payout. A method has to clear that, with a safety margin, over a large number of trades. A back-test showing a 53% win rate over 60 trades does not do that: ordinary luck on 60 trades swings the result by about 6 points either way.
Always ask three things of any claimed result:
- How many trades are behind it?
- Is it before or after costs, including the payout being under 100%?
- Was it tested on data the method had not seen?
Confidence is not probability
A read's confidence score on this site describes the agreement between its inputs. Most signals from price data are weak and correlated: moving average direction, recent change and tick share often all say the same thing because they are all built from the same closes. Several of them agreeing is closer to one signal than several.
For a figure to be a probability, it would have to be calibrated: of all the times the tool said 70%, it should have been right about 70% of the time, checked on many outcomes. Calibration has to be measured, and even then it holds only while conditions resemble the past. Treat any confidence number accordingly.
Survivorship and selection
Advertised results come from those who survived and chose to share. A screenshot of a winning week tells you nothing about the weeks not shown or about everyone who stopped posting. When you see claims of "consistent profits", ask what you are not seeing.
Using AI sensibly
None of this makes the analyzer useless. It makes it a tool for structure, not for certainty:
- It forces the same set of checks every time, which stops you skipping the dull ones.
- It states an invalidation condition, which many people forget to set.
- It records its reads, so you can compare later.
- It labels data as live, delayed or sample, so you know what you are looking at.
What it cannot do is remove the risk. A system that sounds sure of itself is a reason to look at the evidence harder, not to relax.
A test you can run on any claim
- Is the explanation simple and plausible?
- Is the sample large, and out-of-sample?
- Does it survive costs?
- Would it have been written the same way if the result had been the opposite?
If a claim fails any of these, hold it lightly.
Note: Nothing on this site is a prediction, advice or an offer to trade. Analysis of any kind can be wrong and trading carries a high risk of loss.
