Running a live trading system without a structured review process is like flying a plane without instruments. The system might be working fine, or it might be drifting off course in ways that are too gradual to notice in daily operation. By the time the problem becomes obvious, the damage is done. After about four months of running the signal system, I formalized a monthly review process specifically because I kept discovering, in hindsight, that things had been going wrong for longer than I had realized. This post describes exactly what I review each month, why I review it, and what I've found.
Why I Needed a Formal Process
The trigger for formalizing the review was a specific experience in month four of live operation. I had been tracking performance casually — checking the trade log at the end of each session, noting whether signals had produced good or bad outcomes, forming a general impression of whether the system was working. My general impression in month four was that the system was performing about as well as in month three.
When I sat down to do a more careful analysis, comparing month four's outcome distribution against months two and three, the picture was different from my impression. The raw success rate — signals followed by continuation moves within two hours — had dropped by about 15 percentage points compared to the prior two months. The average size of the continuation moves on winning signals had also declined. The system was still producing positive outcomes overall, but the quality had degraded meaningfully and I hadn't noticed because I had been tracking my general impression rather than measuring specific metrics.
Investigating why the performance had declined led me to discover that the market environment in month four had shifted toward a more volatile, less trend-following regime. The ATR readings on tracked stocks had increased significantly, which meant the ATR-based pullback thresholds had expanded, which meant the system was accepting deeper pullbacks as valid — and deeper pullbacks in a volatile, choppy market are often continuation patterns rather than consolidation patterns. The system was generating technically valid signals in conditions where the underlying pattern was less reliable. Nothing in the rules had changed. The market had changed, and the rules hadn't adapted.
I would have caught this faster with a monthly review process. Instead, I caught it six weeks into the degradation. The monthly review was designed to make six-week delays impossible.
What the Monthly Review Covers
The review takes between two and three hours each month. I run it in the first week of the following month, using data from the month just completed. It has seven components.
Component One: Signal Volume and Distribution
The first thing I look at is how many confirmed signals the system generated in the month, and how they were distributed across the 09:00-10:30 entry window. I'm looking for whether the volume is consistent with prior months and whether the distribution is even across the window or skewed toward particular time slots.
Unusual changes in signal volume are often the first indicator of something changing in the system or the market. In the month I described above, signal volume was actually slightly higher than the prior month — which should have been a warning sign, because higher signal volume usually means the pattern criteria are being met more easily, which can indicate either a favorable environment or a deteriorating signal quality threshold. I had noticed the higher volume at the time and filed it as "good, more opportunities." In retrospect, it was "concerning, the criteria may be producing more noise."
Component Two: Grade Distribution
For each month, I calculate what percentage of confirmed signals were grade A, B, and C. This distribution is informative about both market conditions and signal quality.
In strong trending months, I typically see more A-grade signals because genuine momentum stocks tend to show the kind of simultaneous high trading value, high trade intensity, and high tick acceleration that qualifies for A grade. In choppy or volatile months, grade C signals tend to be more prevalent because fewer stocks are showing all three conditions strongly at the same moment.
A month where grade C signals are unusually dominant is worth investigating — it might mean the market is in a condition where the signal system produces structurally lower-quality opportunities, and adjusting behavior accordingly (being more selective, reducing position size, or temporarily reducing activity) is the appropriate response.
Component Three: Outcome Distribution by Grade
The third component is the one that most directly measures whether the system is working: for each grade category, what percentage of signals produced a meaningful continuation move within two hours, and what was the average size of those moves?
I maintain a specific definition of "meaningful continuation move" to keep this metric consistent: at least 1.5% price advance from the signal confirmation point within a two-hour window, measured against the closing price of the breakout candle. This isn't the only way to measure signal quality, but it's specific enough to be consistent across months.
The key comparison is not whether the percentages are "good" in absolute terms — what matters is whether they've changed materially compared to the previous three months. A 20-percentage-point shift in the continuation rate for A-grade signals is a signal that something has changed, either in the market or in how the system is generating A-grade readings.
Component Four: Stop-Loss Hit Rate and Average Stop Distance
I track the percentage of trades that hit the stop-loss rather than the take-profit target, and the average distance between entry price and stop-loss level across all trades. The second metric is particularly useful for detecting whether the ATR-based pullback thresholds have been drifting — if the average pullback depth is increasing, it means either that the market is becoming more volatile or that the threshold calibration is getting looser.
In month six of live operation, this metric flagged a problem: the average stop distance had increased by about 30% compared to the three-month average, which meant the system was accepting deeper pullbacks before confirming a setup. Deeper pullbacks meant larger stop-losses, which meant larger potential losses per trade. The risk profile of the system had changed without any intentional change to the rules — it had drifted with market conditions. Catching this in the monthly review let me recalibrate the ATR multiplier before the drift produced serious losses.
Component Five: Crossing Count Analysis for Secondary Signals
For stocks that generated secondary signals (the MA120 crossing-based alerts described in earlier posts), I review the distribution of crossing counts and the trading value multiples at each crossing. I'm specifically looking for the proportion of secondary signals where the crossing count exceeded 50 — which, as I described in an earlier post, is typically a consolidation indicator rather than a momentum indicator.
If the proportion of high-count secondary signals is increasing, it suggests that the tracked stocks are spending more time in sideways consolidation rather than trending, which has direct implications for how much weight to give to secondary signals in my manual review process.
Component Six: Market Environment Assessment
The sixth component is qualitative but structured. I assess four aspects of the market environment during the month: the overall trend direction of the KOSPI, the average daily volatility (approximated by the average ATR reading across tracked stocks), the quality of participant flows (were institutional and foreign flows broadly supportive or adverse?), and any significant market events (major earnings cycles, regulatory changes, global macro events) that may have influenced signal quality.
This component exists because some of what the outcome distribution in components three and four measures is market environment rather than system performance. A month where the KOSPI was declining and volatility was elevated will naturally produce worse signal outcomes than a month where the KOSPI was trending upward and volatility was normal. Understanding this distinction prevents me from making changes to the system based on market-driven performance variation rather than genuine system degradation.
Component Seven: Rule Integrity Check
The final component is a spot-check on whether the system's rules are executing as designed. I select ten signals from the month at random, pull their full event logs, and trace through each one to verify that every state transition occurred correctly — that the wave formation was validated with the right number of candles and the right volume confirmation, that the pullback threshold was calculated correctly for that stock's ATR at the time, that the breakout confirmation happened at the right price level, and that the grade was assigned correctly.
This check sounds redundant — if the code is running without errors, why would it be executing incorrectly? But I've found, on three separate occasions over the months of running this system, cases where the rule execution was technically correct according to the code but was producing outcomes inconsistent with the intended design. In each case, the issue was a calibration drift rather than a code bug: a parameter that was set correctly initially had become less appropriate as market conditions changed, and the rule was executing as written while producing results different from what the design intended.
What I Do With the Review Results
The review produces one of three conclusions: the system is performing within expected ranges and no changes are needed; the system is showing performance degradation that appears to be market-driven, in which case I adjust my trading behavior rather than the system itself; or the system is showing performance degradation that appears to be system-driven, in which case I investigate and potentially recalibrate the affected parameters.
The most important discipline in acting on review results is not over-adjusting. A single bad month doesn't necessarily indicate that something is broken — it might indicate bad luck, adverse market conditions, or simply the natural variance of a probabilistic system. The review is designed to distinguish between these possibilities, not to produce an immediate response to every deviation from average performance.
The threshold I use is roughly this: if two consecutive monthly reviews show the same anomaly, or if a single month shows a deviation more than two standard deviations from the prior six-month baseline, I treat it as worth investigating. Below that threshold, I note it and continue watching.
What the Reviews Have Found So Far
Over the eight monthly reviews I've completed, the outcomes have been:
Three months: within expected range, no action taken.
Two months: market-environment-driven degradation, adjusted trading behavior (reduced position sizing, increased selectivity threshold for acting on signals).
Two months: ATR-based threshold drift, recalibrated the ATR multiplier.
One month: discovered the participant breakdown asymmetry described in earlier posts — the month where retail-dominated signals were significantly underperforming institutional-accompanied signals, which led to adding the participant composition check to my manual review process.
The monthly review hasn't prevented every loss or caught every problem early. But it has consistently caught problems that were invisible in daily operation, and it has given me a structured basis for distinguishing genuine system degradation from normal performance variance — which is, ultimately, what any measurement process is for.
Today's Investing Insight — Mean Reversion and the Danger of Recency Bias
One of the most consistent traps in evaluating trading system performance is recency bias — the tendency to weight recent results more heavily than older results when forming an impression of how a system is performing. A system that has had two good months in a row feels like it's working better than average, even if longer-term data suggests the recent performance is simply the high end of normal variance. Conversely, two bad months can trigger significant changes to a system that is actually working correctly but happened to face adverse conditions. Mean reversion — the tendency for performance that is temporarily above or below average to return toward the long-term average — is well-documented in financial markets and in trading system performance. A structured review process that uses multi-month baselines rather than recent impressions is a practical defense against making decisions based on where in the variance cycle the system currently sits.
---
This post documents a personal journey of building and running an algorithmic trading system and reflects personal experience and perspective. It is not a recommendation of any specific strategy or review methodology. All investment decisions and their outcomes are the sole responsibility of the investor.
Comments
Post a Comment