Most writing about algorithmic trading focuses on what happens when the system works correctly. This post is about what happens when it doesn't — specifically, what happens when the server running the signal detection engine goes down in the middle of a trading session, and the trades that are already open have no automated management.
I've experienced this three times over fourteen months of live operation. Each instance was different. Each one taught me something about system design that I hadn't thought carefully enough about beforehand.
The First Crash — A Windows Update Nobody Asked For
The first unplanned shutdown happened on a Wednesday morning about three months into live operation. I was at my desk watching the dashboard when the screen went blank. A moment later, the familiar Windows login screen appeared. The operating system had decided, without asking, that it was time to install updates and restart.
The Python trading engine was gone. The Firebase database still showed the signals that had been active at the moment of shutdown — they didn't disappear, because Firebase persists until explicitly deleted. The dashboard still displayed four stocks in various stages, including two that had confirmed signals and were in active positions. But the engine that would have managed those positions — tracking whether stop-losses or targets had been reached — was offline.
I moved to manual management immediately. Both positions were within their normal price ranges, and I was able to exit one at a modest gain and hold the other, which eventually hit its target later in the session. The outcome was acceptable, but the experience was alarming precisely because it was unexpected. I had thought about what would happen if the system made a wrong decision. I had not thought about what would happen if the system simply stopped.
The immediate fix was obvious: disable automatic Windows updates during market hours. I set the active hours in Windows Update settings to exclude the 09:00-16:00 window. This took about two minutes and should have been done before the system was ever deployed. The fact that it wasn't revealed a gap in my pre-deployment checklist that I immediately corrected.
The Second Crash — A Memory Leak That Built for Three Weeks
The second shutdown was slower and more instructive. Over several weeks, I had noticed that the Python process was consuming increasing amounts of memory — not dramatically, just a steady creep upward. I had noted it, told myself I would investigate it, and continued running the system.
On a Tuesday in month seven, the memory consumption hit a level that caused the system to start responding very slowly, then stop responding altogether. The Python process was still running — it hadn't crashed — but it had consumed enough memory that the operating system was paging aggressively, and the real-time tick processing had slowed to the point where candles were completing with significant delay. The signals being generated were based on stale data by the time they fired.
I only realized what was happening when I noticed that the timestamps on new signal events were running several minutes behind real market time. By the time I identified the cause and restarted the process, approximately 45 minutes of the session had been running on degraded data.
Tracing the memory leak afterward took most of an afternoon. The source was a list that was appending tick timestamps for the tick acceleration calculation but was never being trimmed. Every tick received added an entry to the list. Over a full trading session — tens of thousands of ticks across hundreds of tracked stocks — the lists grew to sizes that consumed significant memory. Over three weeks of daily operation, the cumulative effect became critical.
The fix was a one-line change: limit each list to the most recent fifty entries using a deque with a maxlen parameter instead of a standard list. The memory consumption stabilized immediately. But the real lesson wasn't about the deque — it was about the monitoring gap. I had observed the symptom (increasing memory) for three weeks and done nothing. A proper monitoring setup would have flagged this automatically and prompted action before it caused a problem.
After this, I added a simple memory usage check to the system's startup routine and a periodic log entry during operation that records memory consumption every thirty minutes. If memory exceeds a threshold, the log entry changes format to make it obvious. It's not sophisticated monitoring — it's a basic sanity check. But it's something I check at the start of each session, and it would have caught the leak in its first week rather than its fourth.
The Third Crash — An Internet Outage at the Worst Possible Moment
The third incident didn't involve the Python process at all. The process was running perfectly. The problem was that the internet connection went down for approximately twenty-two minutes during a session when two positions were open.
From the Python engine's perspective, the Firebase write calls were failing silently — the code caught the exceptions and logged them, but continued running. The dashboard, which reads from Firebase, stopped updating because the last known state was still in the database and the new state updates weren't getting through. Someone watching the dashboard would have seen it as frozen rather than obviously broken.
More critically, the Python engine was still receiving tick data from the brokerage API — the API connection uses a local Windows component that maintained its session across the internet interruption. So the engine was correctly tracking prices, updating its state, and attempting to write to Firebase — it just wasn't succeeding. When the internet connection restored after twenty-two minutes, the engine wrote the current state immediately. But twenty-two minutes of state history were lost.
During those twenty-two minutes, one of the two open positions had triggered its stop-loss and the engine had attempted to generate an alert — but the alert hadn't reached the dashboard. I only discovered this when the connection restored and I saw the updated state showing the position had been stopped out several minutes earlier.
This incident revealed a design assumption I had made without examining it: that the dashboard going dark meant something was wrong with the engine. The correct interpretation is that the dashboard going dark means something is wrong with the Firebase connection — which could be the engine, or the network, or Firebase itself. The engine might be running perfectly while the dashboard shows nothing.
After this, I added a local log file that the engine writes to independently of Firebase. Every significant event — state transitions, signal confirmations, stop-loss triggers, take-profit exits — gets written to a local text file regardless of whether the Firebase write succeeds. This log is the ground truth of what the engine actually did during a session. If the dashboard and the log disagree, the log is correct.
What These Three Incidents Revealed About System Resilience
Looking at the three incidents together, they illustrate three different categories of failure that any live system needs to be designed against.
The first category is external interference with the host environment — the Windows update that restarted the machine. The defense is configuration: disable automatic restarts during operating hours, set appropriate power management settings, ensure the operating system is not going to make unilateral decisions during critical windows.
The second category is internal resource degradation — the memory leak that built slowly over weeks. The defense is monitoring: track resource consumption continuously, log it periodically, and have clear thresholds that trigger investigation before the problem becomes critical.
The third category is dependency failure — the internet outage that broke the Firebase connection while the engine continued running. The defense is redundancy and independence: maintain a local record of ground truth that doesn't depend on the external dependency, and design the system to fail gracefully when the dependency is unavailable rather than silently producing degraded outputs.
None of these defenses are sophisticated. They're the basic engineering practices that any production system needs. What surprised me was how long it took to implement them properly — because the system worked correctly in normal conditions, the resilience gaps weren't visible until something went wrong.
What a Pre-Session Checklist Looks Like Now
After the three incidents, I established a pre-session checklist that runs every morning before market open. It takes about three minutes and has prevented several near-misses that would have become incidents.
The checklist covers: confirm the Python process is not running from a prior session and memory consumption from prior processes is clear; verify the internet connection is active and Firebase is reachable; confirm the local log file from the prior session closed cleanly and the final entry is the expected session-end cleanup; check that Windows Update is not scheduled or pending; verify the brokerage platform is connected and the API is responding; confirm the dashboard is displaying the expected cleared state for a new session.
This checklist exists because of the three failures described above. Every item on it corresponds to something that went wrong. Pre-session checklists designed from failure experience tend to be more useful than ones designed from theory, because they contain only the things that have actually broken.
Today's Investing Insight — The Importance of Failure Modes in System Design
In engineering, the discipline of failure mode analysis — identifying in advance the ways a system can fail and designing against the most consequential ones — is standard practice for any system where failure has serious consequences. Financial trading systems are rarely designed with this rigor at the retail level, partly because the consequences of individual failures are limited and partly because the design and operation of retail systems is usually handled by a single person without the review processes that surface failure modes in team environments. The practical equivalent of failure mode analysis for a retail trading system is exactly what this post describes: run the system, observe how it fails, and add specific defenses against each failure mode. This approach is reactive rather than proactive, which means it requires experiencing failures before preventing them. But for a system built and operated by one person without dedicated testing resources, it may be the most realistic path to genuine resilience.
---
This post documents a personal journey of building and running an algorithmic trading system and reflects personal experience and perspective. System failures described here are specific to a particular technical setup and may not be representative of other environments. All investment decisions and their outcomes are the sole responsibility of the investor.
Comments
Post a Comment