The Strategy Variance Test tool lets you run backtests on multiple strategies, symbols, and time frames. It runs multiple backtests based on the options you choose and then presents the results in one spreadsheet. This tool is built for the purpose of exploring variance of your backtest metrics in different circumstances. One can pursue different goals, but analyzing variance is still the key here.
Here are a few examples of how you can use that:
- Test one strategy on different time frames for the same market. This helps you see which time frames look more promising. Also, you can check how metrics of the same strategy change depending on symbols or time frames. Extreme variance in important metrics like drawdown or win rate could indicate that your strategy is too fragile.
- Backtest the same strategy on different symbols but on the same time frame. This helps you identify which market your strategy works best for. At the same time, if you believe that these markets are fundamentally similar, then you should also explore variance of metrics. Just like with p.1 above, extreme variance could indicate that your strategy is too fragile.
- Test different strategies on the same chart, symbol, and time frame. This helps you determine which strategy is the best fit for a given chart.
- Test any combination of the above. Test 3 strategies on 5 symbols on 2 time frames and see what happens.
The Widget
The Strategy Variance Testing widget has four main parts.
- Configuration Panel: This is where you set up your variance test. You choose the strategies, time frames, and symbols you want to test. Since this feature is in beta, keep in mind that you can't exceed more than a total of 30 variants. For example, if you select three strategies and 10 symbols on 1 time frame, the total variants computed will be 30 (3 * 10 * 1).
- Variant Table: This is a spreadsheet displaying all the tabular metrics for each backtest from the list of your variants defined above. The grid helps you assess which variants excel in different aspects.
- Charts: The third part of this widget features three bubble charts. You have the flexibility to customize which metrics represent the horizontal axis, vertical axis, and bubble size. The bubbles are color-coded on a gradient from orange to green. The best combinations of X and Y metrics show as green bubbles, the worst as orange, and the rest with colors in between. While default metrics are set, you can easily change them to suit your analysis.
- Depth. This selector defines the amount of candles to be used in your backtest. This is not a variable factor, you can only pick one depth for all the variants in your experiment.
Variant Table
This table is a straightforward spreadsheet. Each row represents one variant. Each column represents a metric, colored as a heat map. The colors range from green (the best value in a column) to transparent (the worst one). For example, in the R/R column, the highest RR is green, and the lowest is transparent. For Drawdown column, the highest drawdown is transparent and the smallest is green. You can sort and filter columns, as well as choose which metrics do you want to see. Entire row being green is a rare case; most times, specific variants are good in specific areas, requiring your judgment.
Chart 1: Reward/Risk vs Win% vs Positions
The first chart displays Risk-Reward (R/R) against Win Percent. If your bubbles land in the red area, it suggests that with the given R/R and Win Percent, your strategy will eventually hit zero (if you assume that RR and Win Percent persist over time). Red area is a no-go zone on this chart, and it's identical to a corresponding no-go flag in our Tabular Data.
Bubble size on this chart is determined by the amount of positions in the corresponding backtest. We assume that more positions means better statistical significance for the overall backtest.
On this chart, the best bubbles will be big bubbles (a lot of positions) at the top (high Win%) right (high R/R) corner of a chart. Pretty often, you'll see your bubbles sitting at the edge of the red area, which indicates that a given strategy has not managed to extract any alpha at a given market and was simply breaking even.
On the chart above, the vast majority of bubbles are in the red area, which means that they were not viable mathematically: strategies with the combination of R/R and Win% like that will go to zero over time. Strategy 1 at USDCAD,240 and Strategy 2 at USDCAD,M variants are above the red area, with Strategy 1 at USDCAD,240 having more positions than Strategy 2 at USDCAD,M. If used in a straightforward way, this chart implies that Strategy 1 at USDCAD,240 was the best option in terms of R/R and Win%.
Chart 2: Avg.Return vs Return St.Dev vs R/R
The second chart plots average return against standard deviation of returns. It helps identify which variant offers the best average return with the least deviation. Green bubbles indicate high average returns with minimal fluctuation. This chart is handy for assessing if returns are evenly distributed or if there are significant outliers, giving you insights into the consistency of your strategy.
On the chart above, the best average returns were generated by Strategy 1 at EURUSD,W. Strategy 2 at EURUSD,M has the most uneven distribution of returns and average return below zero (which is a no go). Strategy 2 at USDCAD,M was the best option in terms of Avg.Return and Return St.dev.
Chart 3: Drawdown vs Asset Drawdown
The third chart compares the maximum drawdown of your strategy to the maximum drawdown of an asset. It helps you gauge how well your variants handle cases when the asset goes down. Ideally, your strategy's drawdown should be minimal, especially when the asset experiences a drawdown. Look for bubbles where your strategy's drawdown is low while the asset's drawdown may vary. This chart helps you assess how effectively your strategy exits during adverse market conditions.
On the chart above, bubbles in the upper right corner indicate smallest strategy drawdown at smallest asset drawdown. Bubbles in the violet area (I have marked it manually) stand for variants which have done a decent job controlling your equity curve drawdown despite the fact that the corresponding market has had significant drawdown (-22% or worse).
Limitations of Beta version
Variance Explorer is an experimental feature. This means that we have decided to deliver it to you as soon as we could, sacrificing some things and features we otherwise might want to add. The current version is fairly limited; we will work on improving this component as we obtain vision of how traders want to use it. Here's a list of known limitations
- Variance Explorer dialog does not preserve any of its settings. Any customization data (i.e., column visibility data) will be lost as soon as you close the dialog.
- Currently, you can't test more than 30 variants at a time.