Monte Carlo Simulation Basics¶

Runtime: ~8 minutes
Level: Beginner
API Status: ✅ Validated against RustyBT 0.1.2+

This notebook introduces Monte Carlo simulation for strategy robustness testing.

What is Monte Carlo Simulation?¶

Monte Carlo simulation tests strategy robustness by:

• Trade Permutation: Randomly shuffling trade sequences to test if order matters
• Noise Infusion: Adding synthetic noise to price data to test parameter sensitivity
• Distribution Analysis: Understanding the range of possible outcomes

Why Use Monte Carlo?¶

• ✅ Stress Testing: See how strategies perform under different scenarios
• ✅ Confidence Intervals: Understand the range of expected returns
• ✅ Overfitting Detection: Strategies that break down with noise are likely overfit
• ✅ Statistical Significance: Test if results are due to skill or luck

---

📋 Notebook Information

• RustyBT Version: 0.1.2+
• Last Validated: 2025-11-07
• API Compatibility: Verified ✅
• Documentation: Monte Carlo API Reference

---

In [ ]:

# Setup
from rustybt.analytics import setup_notebook
setup_notebook()

import pandas as pd
import numpy as np
from decimal import Decimal
import plotly.graph_objects as go
from plotly.subplots import make_subplots
from datetime import datetime

from rustybt import TradingAlgorithm
from rustybt.optimization.monte_carlo import (
    MonteCarloSimulator,
    MonteCarloResult,
)
from rustybt.optimization.noise_infusion import (
    NoiseInfusionSimulator,
    NoiseInfusionResult,
)
from rustybt.analytics import plot_equity_curve

print("✓ Imports successful")

# Setup from rustybt.analytics import setup_notebook setup_notebook() import pandas as pd import numpy as np from decimal import Decimal import plotly.graph_objects as go from plotly.subplots import make_subplots from datetime import datetime from rustybt import TradingAlgorithm from rustybt.optimization.monte_carlo import ( MonteCarloSimulator, MonteCarloResult, ) from rustybt.optimization.noise_infusion import ( NoiseInfusionSimulator, NoiseInfusionResult, ) from rustybt.analytics import plot_equity_curve print("✓ Imports successful")

1. Define a Simple Strategy¶

Let's start with a basic moving average crossover strategy.

In [ ]:

class MovingAverageCrossover(TradingAlgorithm):
    """
    Simple moving average crossover strategy.
    
    Buy when fast MA crosses above slow MA.
    Sell when fast MA crosses below slow MA.
    """
    
    params = {
        'fast_window': 20,
        'slow_window': 50,
    }
    
    def initialize(self, context):
        context.asset = self.symbol('SPY')
        context.fast_window = self.params['fast_window']
        context.slow_window = self.params['slow_window']
        
        # Schedule daily rebalance
        self.schedule_function(
            self.check_signal,
            date_rules=self.date_rules.every_day(),
            time_rules=self.time_rules.market_open()
        )
    
    def check_signal(self, context, data):
        if not data.can_trade(context.asset):
            return
        
        # Get price history
        prices = data.history(
            context.asset,
            'close',
            context.slow_window + 1,
            '1d'
        )
        
        if len(prices) < context.slow_window + 1:
            return
        
        # Calculate moving averages
        fast_ma = prices[-context.fast_window:].mean()
        slow_ma = prices[-context.slow_window:].mean()
        
        # Previous values for crossover detection
        fast_ma_prev = prices[-(context.fast_window+1):-1].mean()
        slow_ma_prev = prices[-(context.slow_window+1):-1].mean()
        
        current_position = context.portfolio.positions[context.asset].amount
        
        # Golden cross: fast crosses above slow (buy signal)
        if fast_ma > slow_ma and fast_ma_prev <= slow_ma_prev:
            if current_position <= 0:
                self.order_target_percent(context.asset, 1.0)
        
        # Death cross: fast crosses below slow (sell signal)
        elif fast_ma < slow_ma and fast_ma_prev >= slow_ma_prev:
            if current_position > 0:
                self.order_target_percent(context.asset, 0.0)

print("✓ Strategy defined")

class MovingAverageCrossover(TradingAlgorithm): """ Simple moving average crossover strategy. Buy when fast MA crosses above slow MA. Sell when fast MA crosses below slow MA. """ params = { 'fast_window': 20, 'slow_window': 50, } def initialize(self, context): context.asset = self.symbol('SPY') context.fast_window = self.params['fast_window'] context.slow_window = self.params['slow_window'] # Schedule daily rebalance self.schedule_function( self.check_signal, date_rules=self.date_rules.every_day(), time_rules=self.time_rules.market_open() ) def check_signal(self, context, data): if not data.can_trade(context.asset): return # Get price history prices = data.history( context.asset, 'close', context.slow_window + 1, '1d' ) if len(prices) < context.slow_window + 1: return # Calculate moving averages fast_ma = prices[-context.fast_window:].mean() slow_ma = prices[-context.slow_window:].mean() # Previous values for crossover detection fast_ma_prev = prices[-(context.fast_window+1):-1].mean() slow_ma_prev = prices[-(context.slow_window+1):-1].mean() current_position = context.portfolio.positions[context.asset].amount # Golden cross: fast crosses above slow (buy signal) if fast_ma > slow_ma and fast_ma_prev <= slow_ma_prev: if current_position <= 0: self.order_target_percent(context.asset, 1.0) # Death cross: fast crosses below slow (sell signal) elif fast_ma < slow_ma and fast_ma_prev >= slow_ma_prev: if current_position > 0: self.order_target_percent(context.asset, 0.0) print("✓ Strategy defined")

2. Run Baseline Backtest¶

First, run the strategy on historical data to get a baseline and extract trades.

In [ ]:

# Note: Requires data bundle setup (see 02_data_ingestion.ipynb)
#
# from rustybt.utils.run_algo import run_algorithm
#
# # Run backtest
# baseline_result = run_algorithm(
#     start=pd.Timestamp('2020-01-01', tz='utc'),
#     end=pd.Timestamp('2023-12-31', tz='utc'),
#     initialize=MovingAverageCrossover().initialize,
#     handle_data=MovingAverageCrossover().handle_data,
#     capital_base=100000.0,
#     bundle='yfinance',
#     data_frequency='daily',
# )
#
# # Extract key data for Monte Carlo
# trades = baseline_result.transactions  # Polars DataFrame of trades
# baseline_sharpe = baseline_result['sharpe_ratio'].iloc[-1]
# baseline_total_return = baseline_result['total_return'].iloc[-1]
#
# print(f"Baseline Sharpe Ratio: {baseline_sharpe:.3f}")
# print(f"Baseline Total Return: {baseline_total_return:.2%}")
# print(f"Number of trades: {len(trades)}")

print("✓ Backtest pattern defined (uncomment after data bundle setup)")

# Note: Requires data bundle setup (see 02_data_ingestion.ipynb) # # from rustybt.utils.run_algo import run_algorithm # # # Run backtest # baseline_result = run_algorithm( # start=pd.Timestamp('2020-01-01', tz='utc'), # end=pd.Timestamp('2023-12-31', tz='utc'), # initialize=MovingAverageCrossover().initialize, # handle_data=MovingAverageCrossover().handle_data, # capital_base=100000.0, # bundle='yfinance', # data_frequency='daily', # ) # # # Extract key data for Monte Carlo # trades = baseline_result.transactions # Polars DataFrame of trades # baseline_sharpe = baseline_result['sharpe_ratio'].iloc[-1] # baseline_total_return = baseline_result['total_return'].iloc[-1] # # print(f"Baseline Sharpe Ratio: {baseline_sharpe:.3f}") # print(f"Baseline Total Return: {baseline_total_return:.2%}") # print(f"Number of trades: {len(trades)}") print("✓ Backtest pattern defined (uncomment after data bundle setup)")

3. Monte Carlo Trade Permutation¶

Test strategy robustness by randomly shuffling trade sequences.

How it works:

1. Take the trades from your backtest
2. Randomly shuffle their order (permutation method)
3. Recalculate portfolio metrics
4. Repeat 1000+ times to build distribution
5. Compare observed result to distribution

In [ ]:

# Create Monte Carlo simulator
mc_simulator = MonteCarloSimulator(
    n_simulations=1000,  # Run 1000 permutations
    method='permutation',  # or 'bootstrap' for sampling with replacement
    seed=42,  # For reproducibility
    confidence_level=0.95,  # 95% confidence intervals
)

# Run Monte Carlo simulation on trades
# mc_results = mc_simulator.run(
#     trades=trades,  # DataFrame from backtest
#     observed_metrics={
#         'sharpe_ratio': Decimal(str(baseline_sharpe)),
#         'total_return': Decimal(str(baseline_total_return)),
#     },
#     initial_capital=Decimal("100000"),
# )
#
# # View summary
# print(mc_results.get_summary('sharpe_ratio'))
#
# # Check statistical significance
# if mc_results.is_significant['sharpe_ratio']:
#     print("\n✅ Strategy is statistically significant!")
# else:
#     print("\n⚠️  Performance may be due to luck")
#
# # Check robustness (outside 95% CI)
# if mc_results.is_robust['sharpe_ratio']:
#     print("✅ Strategy is robust (outside 95% CI)")

print("✓ Monte Carlo simulator configured")

# Create Monte Carlo simulator mc_simulator = MonteCarloSimulator( n_simulations=1000, # Run 1000 permutations method='permutation', # or 'bootstrap' for sampling with replacement seed=42, # For reproducibility confidence_level=0.95, # 95% confidence intervals ) # Run Monte Carlo simulation on trades # mc_results = mc_simulator.run( # trades=trades, # DataFrame from backtest # observed_metrics={ # 'sharpe_ratio': Decimal(str(baseline_sharpe)), # 'total_return': Decimal(str(baseline_total_return)), # }, # initial_capital=Decimal("100000"), # ) # # # View summary # print(mc_results.get_summary('sharpe_ratio')) # # # Check statistical significance # if mc_results.is_significant['sharpe_ratio']: # print("\n✅ Strategy is statistically significant!") # else: # print("\n⚠️ Performance may be due to luck") # # # Check robustness (outside 95% CI) # if mc_results.is_robust['sharpe_ratio']: # print("✅ Strategy is robust (outside 95% CI)") print("✓ Monte Carlo simulator configured")

Analyze Monte Carlo Results¶

Visualize the distribution of outcomes from Monte Carlo simulations.

In [ ]:

def analyze_monte_carlo_results(mc_results: MonteCarloResult, metric: str = 'sharpe_ratio'):
    """
    Analyze and visualize Monte Carlo simulation results.
    
    Args:
        mc_results: MonteCarloResult from simulator
        metric: Metric to analyze (default: 'sharpe_ratio')
    """
    # Extract data
    observed = float(mc_results.observed_metrics[metric])
    simulated = [float(v) for v in mc_results.simulated_metrics[metric]]
    ci_lower, ci_upper = mc_results.confidence_intervals[metric]
    ci_lower = float(ci_lower)
    ci_upper = float(ci_upper)
    p_value = float(mc_results.p_values[metric])
    percentile = float(mc_results.percentile_ranks[metric])
    
    # Print statistics
    print(f"\n=== Monte Carlo Results ({mc_results.method} method) ===")
    print(f"Number of simulations: {mc_results.n_simulations}")
    print(f"\nMetric: {metric}")
    print(f"  Observed:       {observed:.3f}")
    print(f"  Mean simulated: {np.mean(simulated):.3f}")
    print(f"  Std simulated:  {np.std(simulated):.3f}")
    print(f"  95% CI:         [{ci_lower:.3f}, {ci_upper:.3f}]")
    print(f"  P-value:        {p_value:.4f}")
    print(f"  Percentile:     {percentile:.1f}")
    
    # Interpretation
    print(f"\n=== Interpretation ===")
    pct_positive = sum(1 for s in simulated if s > 0) / len(simulated)
    pct_exceed = sum(1 for s in simulated if s > observed) / len(simulated)
    
    print(f"% of simulations > 0: {pct_positive:.1%}")
    print(f"% exceeding observed: {pct_exceed:.1%}")
    
    if mc_results.is_significant[metric] and mc_results.is_robust[metric]:
        print("\n✅ ROBUST: Statistically significant and outside 95% CI")
    elif mc_results.is_significant[metric]:
        print("\n⚠️  SIGNIFICANT: Significant but close to CI boundary")
    else:
        print("\n❌ NOT ROBUST: Performance may be due to luck")
    
    # Create visualization
    fig = go.Figure()
    
    # Histogram of simulated values
    fig.add_trace(go.Histogram(
        x=simulated,
        nbinsx=50,
        name='Simulated Distribution',
        marker_color='steelblue',
        opacity=0.7
    ))
    
    # Add observed value line
    fig.add_vline(
        x=observed,
        line_dash="solid",
        line_color="red",
        line_width=3,
        annotation_text=f"Observed: {observed:.3f}",
        annotation_position="top"
    )
    
    # Add confidence interval lines
    fig.add_vline(
        x=ci_lower,
        line_dash="dash",
        line_color="green",
        line_width=2,
        annotation_text=f"95% CI Lower: {ci_lower:.3f}"
    )
    fig.add_vline(
        x=ci_upper,
        line_dash="dash",
        line_color="green",
        line_width=2,
        annotation_text=f"95% CI Upper: {ci_upper:.3f}"
    )
    
    fig.update_layout(
        title=f"Monte Carlo Distribution: {metric}",
        xaxis_title=metric.replace('_', ' ').title(),
        yaxis_title="Frequency",
        height=500,
        showlegend=True
    )
    
    return fig

# Example usage:
# fig = analyze_monte_carlo_results(mc_results, 'sharpe_ratio')
# fig.show()

print("✓ Analysis function defined")

def analyze_monte_carlo_results(mc_results: MonteCarloResult, metric: str = 'sharpe_ratio'): """ Analyze and visualize Monte Carlo simulation results. Args: mc_results: MonteCarloResult from simulator metric: Metric to analyze (default: 'sharpe_ratio') """ # Extract data observed = float(mc_results.observed_metrics[metric]) simulated = [float(v) for v in mc_results.simulated_metrics[metric]] ci_lower, ci_upper = mc_results.confidence_intervals[metric] ci_lower = float(ci_lower) ci_upper = float(ci_upper) p_value = float(mc_results.p_values[metric]) percentile = float(mc_results.percentile_ranks[metric]) # Print statistics print(f"\n=== Monte Carlo Results ({mc_results.method} method) ===") print(f"Number of simulations: {mc_results.n_simulations}") print(f"\nMetric: {metric}") print(f" Observed: {observed:.3f}") print(f" Mean simulated: {np.mean(simulated):.3f}") print(f" Std simulated: {np.std(simulated):.3f}") print(f" 95% CI: [{ci_lower:.3f}, {ci_upper:.3f}]") print(f" P-value: {p_value:.4f}") print(f" Percentile: {percentile:.1f}") # Interpretation print(f"\n=== Interpretation ===") pct_positive = sum(1 for s in simulated if s > 0) / len(simulated) pct_exceed = sum(1 for s in simulated if s > observed) / len(simulated) print(f"% of simulations > 0: {pct_positive:.1%}") print(f"% exceeding observed: {pct_exceed:.1%}") if mc_results.is_significant[metric] and mc_results.is_robust[metric]: print("\n✅ ROBUST: Statistically significant and outside 95% CI") elif mc_results.is_significant[metric]: print("\n⚠️ SIGNIFICANT: Significant but close to CI boundary") else: print("\n❌ NOT ROBUST: Performance may be due to luck") # Create visualization fig = go.Figure() # Histogram of simulated values fig.add_trace(go.Histogram( x=simulated, nbinsx=50, name='Simulated Distribution', marker_color='steelblue', opacity=0.7 )) # Add observed value line fig.add_vline( x=observed, line_dash="solid", line_color="red", line_width=3, annotation_text=f"Observed: {observed:.3f}", annotation_position="top" ) # Add confidence interval lines fig.add_vline( x=ci_lower, line_dash="dash", line_color="green", line_width=2, annotation_text=f"95% CI Lower: {ci_lower:.3f}" ) fig.add_vline( x=ci_upper, line_dash="dash", line_color="green", line_width=2, annotation_text=f"95% CI Upper: {ci_upper:.3f}" ) fig.update_layout( title=f"Monte Carlo Distribution: {metric}", xaxis_title=metric.replace('_', ' ').title(), yaxis_title="Frequency", height=500, showlegend=True ) return fig # Example usage: # fig = analyze_monte_carlo_results(mc_results, 'sharpe_ratio') # fig.show() print("✓ Analysis function defined")

4. Noise Infusion Testing¶

Add realistic market noise to test parameter stability.

How it works:

1. Take your original OHLCV data
2. Add synthetic noise (gaussian or bootstrap)
3. Re-run backtest on noisy data
4. Repeat 1000+ times
5. Measure performance degradation

In [ ]:

# Create noise infusion simulator
noise_simulator = NoiseInfusionSimulator(
    n_simulations=1000,  # Number of noise realizations
    std_pct=0.02,  # 2% noise amplitude
    noise_model='gaussian',  # or 'bootstrap'
    seed=42,
    preserve_structure=False,  # Set True to maintain autocorrelation
    confidence_level=0.95,
)

# Run noise infusion test
# Note: This requires passing data and a callable that runs backtest
#
# def run_backtest_on_data(data):
#     """Run strategy on given data and return metrics."""
#     result = run_algorithm(
#         start=pd.Timestamp('2020-01-01', tz='utc'),
#         end=pd.Timestamp('2023-12-31', tz='utc'),
#         initialize=MovingAverageCrossover().initialize,
#         handle_data=MovingAverageCrossover().handle_data,
#         capital_base=100000.0,
#         data=data,  # Use noisy data
#     )
#     return {
#         'sharpe_ratio': Decimal(str(result['sharpe_ratio'].iloc[-1])),
#         'total_return': Decimal(str(result['total_return'].iloc[-1])),
#     }
#
# noise_results = noise_simulator.run(
#     data=original_data,
#     backtest_func=run_backtest_on_data
# )
#
# # View summary
# print(noise_results.get_summary('sharpe_ratio'))
#
# # Check robustness
# if noise_results.is_robust['sharpe_ratio']:
#     print("\n✅ Strategy is robust to noise (degradation < 20%)")
# elif noise_results.is_fragile['sharpe_ratio']:
#     print("\n❌ Strategy is fragile (degradation > 50%, likely overfit)")
# else:
#     print("\n⚠️  Strategy shows moderate noise sensitivity")

print("✓ Noise infusion simulator configured")
print(f"  Noise model: gaussian")
print(f"  Noise level: 2%")

# Create noise infusion simulator noise_simulator = NoiseInfusionSimulator( n_simulations=1000, # Number of noise realizations std_pct=0.02, # 2% noise amplitude noise_model='gaussian', # or 'bootstrap' seed=42, preserve_structure=False, # Set True to maintain autocorrelation confidence_level=0.95, ) # Run noise infusion test # Note: This requires passing data and a callable that runs backtest # # def run_backtest_on_data(data): # """Run strategy on given data and return metrics.""" # result = run_algorithm( # start=pd.Timestamp('2020-01-01', tz='utc'), # end=pd.Timestamp('2023-12-31', tz='utc'), # initialize=MovingAverageCrossover().initialize, # handle_data=MovingAverageCrossover().handle_data, # capital_base=100000.0, # data=data, # Use noisy data # ) # return { # 'sharpe_ratio': Decimal(str(result['sharpe_ratio'].iloc[-1])), # 'total_return': Decimal(str(result['total_return'].iloc[-1])), # } # # noise_results = noise_simulator.run( # data=original_data, # backtest_func=run_backtest_on_data # ) # # # View summary # print(noise_results.get_summary('sharpe_ratio')) # # # Check robustness # if noise_results.is_robust['sharpe_ratio']: # print("\n✅ Strategy is robust to noise (degradation < 20%)") # elif noise_results.is_fragile['sharpe_ratio']: # print("\n❌ Strategy is fragile (degradation > 50%, likely overfit)") # else: # print("\n⚠️ Strategy shows moderate noise sensitivity") print("✓ Noise infusion simulator configured") print(f" Noise model: gaussian") print(f" Noise level: 2%")

Noise Infusion Interpretation¶

What to look for:

• Degradation < 20%: Strategy is robust to parameter variations ✅
• Degradation 20-50%: Moderate sensitivity ⚠️
• Degradation > 50%: Likely overfit to specific data patterns ❌

In [ ]:

def analyze_noise_infusion_results(noise_results: NoiseInfusionResult, metric: str = 'sharpe_ratio'):
    """
    Analyze and visualize noise infusion results.
    
    Args:
        noise_results: NoiseInfusionResult from simulator
        metric: Metric to analyze
    """
    # Extract data
    original = float(noise_results.original_metrics[metric])
    noisy = [float(v) for v in noise_results.noisy_metrics[metric]]
    mean_noisy = float(noise_results.mean_metrics[metric])
    degradation = float(noise_results.degradation_pct[metric])
    worst_case = float(noise_results.worst_case_metrics[metric])
    ci_lower, ci_upper = noise_results.confidence_intervals[metric]
    
    # Print statistics
    print(f"\n=== Noise Infusion Results ({noise_results.noise_model} model) ===")
    print(f"Number of simulations: {noise_results.n_simulations}")
    print(f"Noise amplitude: {float(noise_results.std_pct) * 100:.1f}%")
    print(f"\nMetric: {metric}")
    print(f"  Original (noise-free): {original:.3f}")
    print(f"  Noisy mean:            {mean_noisy:.3f}")
    print(f"  Worst case (5th %ile): {worst_case:.3f}")
    print(f"  Degradation:           {degradation:.1f}%")
    print(f"  95% CI:                [{float(ci_lower):.3f}, {float(ci_upper):.3f}]")
    
    # Interpretation
    print(f"\n=== Interpretation ===")
    if noise_results.is_fragile[metric]:
        print("❌ FRAGILE: Strategy highly sensitive to noise (likely overfit)")
    elif degradation > 25:
        print("⚠️  MODERATE: Strategy shows moderate noise sensitivity")
    elif noise_results.is_robust[metric]:
        print("✅ ROBUST: Strategy tolerates noise well")
    else:
        print("✅ GOOD: Strategy shows good noise tolerance")
    
    # Create visualization
    fig = go.Figure()
    
    # Histogram of noisy values
    fig.add_trace(go.Histogram(
        x=noisy,
        nbinsx=50,
        name='Noisy Distribution',
        marker_color='lightcoral',
        opacity=0.7
    ))
    
    # Add original value line
    fig.add_vline(
        x=original,
        line_dash="solid",
        line_color="green",
        line_width=3,
        annotation_text=f"Original: {original:.3f}",
        annotation_position="top"
    )
    
    # Add mean noisy line
    fig.add_vline(
        x=mean_noisy,
        line_dash="dash",
        line_color="red",
        line_width=2,
        annotation_text=f"Noisy Mean: {mean_noisy:.3f}"
    )
    
    fig.update_layout(
        title=f"Noise Infusion Distribution: {metric} (Degradation: {degradation:.1f}%)",
        xaxis_title=metric.replace('_', ' ').title(),
        yaxis_title="Frequency",
        height=500,
        showlegend=True
    )
    
    return fig

# Example usage:
# fig = analyze_noise_infusion_results(noise_results, 'sharpe_ratio')
# fig.show()

print("✓ Noise analysis function defined")

def analyze_noise_infusion_results(noise_results: NoiseInfusionResult, metric: str = 'sharpe_ratio'): """ Analyze and visualize noise infusion results. Args: noise_results: NoiseInfusionResult from simulator metric: Metric to analyze """ # Extract data original = float(noise_results.original_metrics[metric]) noisy = [float(v) for v in noise_results.noisy_metrics[metric]] mean_noisy = float(noise_results.mean_metrics[metric]) degradation = float(noise_results.degradation_pct[metric]) worst_case = float(noise_results.worst_case_metrics[metric]) ci_lower, ci_upper = noise_results.confidence_intervals[metric] # Print statistics print(f"\n=== Noise Infusion Results ({noise_results.noise_model} model) ===") print(f"Number of simulations: {noise_results.n_simulations}") print(f"Noise amplitude: {float(noise_results.std_pct) * 100:.1f}%") print(f"\nMetric: {metric}") print(f" Original (noise-free): {original:.3f}") print(f" Noisy mean: {mean_noisy:.3f}") print(f" Worst case (5th %ile): {worst_case:.3f}") print(f" Degradation: {degradation:.1f}%") print(f" 95% CI: [{float(ci_lower):.3f}, {float(ci_upper):.3f}]") # Interpretation print(f"\n=== Interpretation ===") if noise_results.is_fragile[metric]: print("❌ FRAGILE: Strategy highly sensitive to noise (likely overfit)") elif degradation > 25: print("⚠️ MODERATE: Strategy shows moderate noise sensitivity") elif noise_results.is_robust[metric]: print("✅ ROBUST: Strategy tolerates noise well") else: print("✅ GOOD: Strategy shows good noise tolerance") # Create visualization fig = go.Figure() # Histogram of noisy values fig.add_trace(go.Histogram( x=noisy, nbinsx=50, name='Noisy Distribution', marker_color='lightcoral', opacity=0.7 )) # Add original value line fig.add_vline( x=original, line_dash="solid", line_color="green", line_width=3, annotation_text=f"Original: {original:.3f}", annotation_position="top" ) # Add mean noisy line fig.add_vline( x=mean_noisy, line_dash="dash", line_color="red", line_width=2, annotation_text=f"Noisy Mean: {mean_noisy:.3f}" ) fig.update_layout( title=f"Noise Infusion Distribution: {metric} (Degradation: {degradation:.1f}%)", xaxis_title=metric.replace('_', ' ').title(), yaxis_title="Frequency", height=500, showlegend=True ) return fig # Example usage: # fig = analyze_noise_infusion_results(noise_results, 'sharpe_ratio') # fig.show() print("✓ Noise analysis function defined")

5. Compare Trade Permutation vs Noise Infusion¶

Both methods test robustness in different ways.

In [ ]:

def compare_robustness_tests(mc_results: MonteCarloResult, noise_results: NoiseInfusionResult, metric: str = 'sharpe_ratio'):
    """
    Compare trade permutation vs noise infusion results.
    """
    print("\n=== Robustness Comparison ===")
    print(f"Metric: {metric}\n")
    
    # Trade permutation results
    mc_significant = mc_results.is_significant[metric]
    mc_robust = mc_results.is_robust[metric]
    mc_percentile = float(mc_results.percentile_ranks[metric])
    
    print("Trade Permutation (Order Matters?):")
    print(f"  Statistically significant: {'✅ Yes' if mc_significant else '❌ No'}")
    print(f"  Outside 95% CI:            {'✅ Yes' if mc_robust else '❌ No'}")
    print(f"  Percentile rank:           {mc_percentile:.1f}")
    
    # Noise infusion results
    noise_robust = noise_results.is_robust[metric]
    noise_fragile = noise_results.is_fragile[metric]
    degradation = float(noise_results.degradation_pct[metric])
    
    print("\nNoise Infusion (Parameter Stability?):")
    print(f"  Robust to noise:           {'✅ Yes' if noise_robust else '❌ No'}")
    print(f"  Fragile (overfit):         {'❌ Yes' if noise_fragile else '✅ No'}")
    print(f"  Performance degradation:   {degradation:.1f}%")
    
    # Overall assessment
    print("\n=== Overall Assessment ===")
    
    if mc_significant and mc_robust and noise_robust:
        print("🌟 EXCELLENT: Strategy passes both robustness tests")
        print("   - Results not due to lucky trade order")
        print("   - Parameters are stable to noise")
    elif mc_significant and noise_robust:
        print("✅ GOOD: Strategy shows good robustness")
        print("   - Statistically significant results")
        print("   - Tolerates parameter noise well")
    elif noise_fragile:
        print("❌ POOR: Strategy appears overfit")
        print("   - High sensitivity to noise (>50% degradation)")
        print("   - Likely curve-fitted to historical data")
    else:
        print("⚠️  MODERATE: Strategy needs improvement")
        print("   - Consider parameter tuning")
        print("   - May benefit from regularization")

# Example usage:
# compare_robustness_tests(mc_results, noise_results, 'sharpe_ratio')

print("✓ Comparison function defined")

def compare_robustness_tests(mc_results: MonteCarloResult, noise_results: NoiseInfusionResult, metric: str = 'sharpe_ratio'): """ Compare trade permutation vs noise infusion results. """ print("\n=== Robustness Comparison ===") print(f"Metric: {metric}\n") # Trade permutation results mc_significant = mc_results.is_significant[metric] mc_robust = mc_results.is_robust[metric] mc_percentile = float(mc_results.percentile_ranks[metric]) print("Trade Permutation (Order Matters?):") print(f" Statistically significant: {'✅ Yes' if mc_significant else '❌ No'}") print(f" Outside 95% CI: {'✅ Yes' if mc_robust else '❌ No'}") print(f" Percentile rank: {mc_percentile:.1f}") # Noise infusion results noise_robust = noise_results.is_robust[metric] noise_fragile = noise_results.is_fragile[metric] degradation = float(noise_results.degradation_pct[metric]) print("\nNoise Infusion (Parameter Stability?):") print(f" Robust to noise: {'✅ Yes' if noise_robust else '❌ No'}") print(f" Fragile (overfit): {'❌ Yes' if noise_fragile else '✅ No'}") print(f" Performance degradation: {degradation:.1f}%") # Overall assessment print("\n=== Overall Assessment ===") if mc_significant and mc_robust and noise_robust: print("🌟 EXCELLENT: Strategy passes both robustness tests") print(" - Results not due to lucky trade order") print(" - Parameters are stable to noise") elif mc_significant and noise_robust: print("✅ GOOD: Strategy shows good robustness") print(" - Statistically significant results") print(" - Tolerates parameter noise well") elif noise_fragile: print("❌ POOR: Strategy appears overfit") print(" - High sensitivity to noise (>50% degradation)") print(" - Likely curve-fitted to historical data") else: print("⚠️ MODERATE: Strategy needs improvement") print(" - Consider parameter tuning") print(" - May benefit from regularization") # Example usage: # compare_robustness_tests(mc_results, noise_results, 'sharpe_ratio') print("✓ Comparison function defined")

Summary¶

Key Takeaways¶

1. Monte Carlo simulation provides confidence intervals for strategy performance
2. Trade permutation tests if results depend on lucky trade order
3. Noise infusion tests sensitivity to parameter variations
4. Tight distributions indicate robust strategies
5. High degradation (>50%) suggests overfitting

Interpretation Guide¶

Robust Strategy:

• ✅ Statistically significant (p < 0.05)
• ✅ Outside 95% confidence interval
• ✅ Noise degradation < 20%
• ✅ >80% of permutations profitable

Overfit Strategy:

• ❌ Not statistically significant
• ❌ Mean << baseline in permutation test
• ❌ Noise degradation > 50%
• ❌ High sensitivity to parameter changes

Best Practices¶

1. Run sufficient simulations: Use at least 1000 simulations
2. Test multiple noise levels: Try 1%, 2%, 5% noise
3. Use both methods: Permutation AND noise infusion
4. Set thresholds: Require <20% degradation, >80% CI
5. Document results: Save distributions for reporting

Workflow Integration¶

Typical validation workflow:

1. Develop strategy (Notebook 03)
2. Optimize parameters (Notebook 05)
3. Monte Carlo validation (this notebook) ← You are here
4. Sensitivity analysis (Notebook 16)
5. Walk-forward validation (Notebook 06)

Next Steps¶

• Notebook 16: Sensitivity Analysis Basics
• Notebook 06: Walk Forward Validation
• Notebook 24: Monte Carlo + Portfolio Optimization (future)

Resources¶

• Monte Carlo Documentation
• Noise Infusion Guide
• Robustness Testing Best Practices