Sensitivity Analysis Basics¶

Runtime: ~6 minutes
Level: Beginner

This notebook introduces parameter sensitivity analysis for understanding strategy robustness.

What is Sensitivity Analysis?¶

Sensitivity analysis measures how strategy performance changes when parameters vary:

• Parameter Stability: How sensitive is performance to parameter changes?
• Optimal Regions: Are optimal parameters on plateaus or sharp peaks?
• Overfitting Detection: Sharp peaks often indicate overfitting

Why Perform Sensitivity Analysis?¶

• ✅ Assess Robustness: Strategies with smooth performance curves are more robust
• ✅ Avoid Overfitting: Sharp performance peaks suggest parameter overfitting
• ✅ Parameter Selection: Choose parameters from stable regions
• ✅ Risk Management: Understand performance variability

---

📋 Notebook Information

• RustyBT Version: 0.1.2+
• Last Validated: 2025-11-07
• API Compatibility: Verified ✅
• Documentation: Sensitivity Analysis API Reference

---

In [ ]:

# Setup
from rustybt.analytics import setup_notebook
setup_notebook()

import pandas as pd
import numpy as np
import plotly.graph_objects as go
from plotly.subplots import make_subplots
from datetime import datetime
from decimal import Decimal

from rustybt import TradingAlgorithm
from rustybt.optimization.sensitivity import (
    SensitivityAnalyzer,
    SensitivityResult,  # ✅ What analyze() returns for each parameter
    InteractionResult,  # ✅ What analyze_interaction() returns for 2D
)
from rustybt.utils.run_algo import run_algorithm

print("✓ Imports successful")

# Setup from rustybt.analytics import setup_notebook setup_notebook() import pandas as pd import numpy as np import plotly.graph_objects as go from plotly.subplots import make_subplots from datetime import datetime from decimal import Decimal from rustybt import TradingAlgorithm from rustybt.optimization.sensitivity import ( SensitivityAnalyzer, SensitivityResult, # ✅ What analyze() returns for each parameter InteractionResult, # ✅ What analyze_interaction() returns for 2D ) from rustybt.utils.run_algo import run_algorithm print("✓ Imports successful")

1. Define Strategy¶

Use a simple RSI mean reversion strategy to demonstrate sensitivity analysis.

In [ ]:

class RSIMeanReversion:
    """
    RSI-based mean reversion strategy.
    
    Buy when RSI < oversold_threshold
    Sell when RSI > overbought_threshold
    """
    
    def __init__(self, params=None):
        self.params = params or {
            'rsi_period': 14,
            'oversold_threshold': 30,
            'overbought_threshold': 70,
        }
    
    def initialize(self, context):
        context.asset = self.symbol('SPY')
        context.rsi_period = self.params['rsi_period']
        context.oversold = self.params['oversold_threshold']
        context.overbought = self.params['overbought_threshold']
        
        self.schedule_function(
            self.check_rsi,
            date_rules=self.date_rules.every_day(),
            time_rules=self.time_rules.market_open()
        )
    
    def check_rsi(self, context, data):
        if not data.can_trade(context.asset):
            return
        
        # Get price history
        prices = data.history(
            context.asset,
            'close',
            context.rsi_period + 1,
            '1d'
        )
        
        if len(prices) < context.rsi_period + 1:
            return
        
        # Calculate RSI
        rsi = self.calculate_rsi(prices, context.rsi_period)
        
        position = context.portfolio.positions[context.asset].amount
        
        # Entry signals
        if position == 0:
            if rsi < context.oversold:
                # Buy oversold
                self.order_target_percent(context.asset, 1.0)
            elif rsi > context.overbought:
                # Short overbought
                self.order_target_percent(context.asset, -1.0)
        
        # Exit when RSI returns to neutral
        elif position != 0:
            if 40 < rsi < 60:
                self.order_target_percent(context.asset, 0.0)
    
    def calculate_rsi(self, prices, period):
        """Calculate RSI"""
        deltas = np.diff(prices)
        gains = np.where(deltas > 0, deltas, 0)
        losses = np.where(deltas < 0, -deltas, 0)
        
        avg_gain = np.mean(gains[-period:])
        avg_loss = np.mean(losses[-period:])
        
        if avg_loss == 0:
            return 100
        
        rs = avg_gain / avg_loss
        rsi = 100 - (100 / (1 + rs))
        return rsi

print("✓ Strategy defined")

class RSIMeanReversion: """ RSI-based mean reversion strategy. Buy when RSI < oversold_threshold Sell when RSI > overbought_threshold """ def __init__(self, params=None): self.params = params or { 'rsi_period': 14, 'oversold_threshold': 30, 'overbought_threshold': 70, } def initialize(self, context): context.asset = self.symbol('SPY') context.rsi_period = self.params['rsi_period'] context.oversold = self.params['oversold_threshold'] context.overbought = self.params['overbought_threshold'] self.schedule_function( self.check_rsi, date_rules=self.date_rules.every_day(), time_rules=self.time_rules.market_open() ) def check_rsi(self, context, data): if not data.can_trade(context.asset): return # Get price history prices = data.history( context.asset, 'close', context.rsi_period + 1, '1d' ) if len(prices) < context.rsi_period + 1: return # Calculate RSI rsi = self.calculate_rsi(prices, context.rsi_period) position = context.portfolio.positions[context.asset].amount # Entry signals if position == 0: if rsi < context.oversold: # Buy oversold self.order_target_percent(context.asset, 1.0) elif rsi > context.overbought: # Short overbought self.order_target_percent(context.asset, -1.0) # Exit when RSI returns to neutral elif position != 0: if 40 < rsi < 60: self.order_target_percent(context.asset, 0.0) def calculate_rsi(self, prices, period): """Calculate RSI""" deltas = np.diff(prices) gains = np.where(deltas > 0, deltas, 0) losses = np.where(deltas < 0, -deltas, 0) avg_gain = np.mean(gains[-period:]) avg_loss = np.mean(losses[-period:]) if avg_loss == 0: return 100 rs = avg_gain / avg_loss rsi = 100 - (100 / (1 + rs)) return rsi print("✓ Strategy defined")

2. Create Objective Function¶

Key Concept: SensitivityAnalyzer requires an objective function that:

1. Takes a parameter dictionary as input
2. Runs the backtest with those parameters
3. Returns the metric value (e.g., Sharpe ratio) as a float

In [ ]:

def create_objective_function(start_date, end_date, bundle='yfinance'):
    """
    Factory function to create objective function for sensitivity analysis.
    
    Args:
        start_date: Start date for backtest
        end_date: End date for backtest
        bundle: Data bundle to use
    
    Returns:
        Objective function that takes params dict and returns Sharpe ratio
    """
    def objective_function(params):
        """
        Run backtest with given parameters and return Sharpe ratio.
        
        Args:
            params: Dictionary like {'rsi_period': 14, 'oversold_threshold': 30, ...}
        
        Returns:
            Float Sharpe ratio
        """
        # Create strategy instance with these parameters
        strategy = RSIMeanReversion(params=params)
        
        # Run backtest
        try:
            result = run_algorithm(
                start=pd.Timestamp(start_date, tz='utc'),
                end=pd.Timestamp(end_date, tz='utc'),
                initialize=strategy.initialize,
                handle_data=strategy.check_rsi,
                capital_base=100000.0,
                bundle=bundle,
            )
            
            # Extract Sharpe ratio (last value)
            sharpe = result['sharpe_ratio'].iloc[-1]
            
            # Handle NaN or inf
            if pd.isna(sharpe) or np.isinf(sharpe):
                return -999.0  # Return very bad value for invalid results
            
            return float(sharpe)
            
        except Exception as e:
            print(f"Error with params {params}: {e}")
            return -999.0  # Return very bad value for failed backtests
    
    return objective_function

# Create objective function for our analysis
objective_fn = create_objective_function(
    start_date='2020-01-01',
    end_date='2023-12-31',
    bundle='yfinance'
)

print("✓ Objective function created")
print("  Will optimize Sharpe ratio over 2020-2023")

def create_objective_function(start_date, end_date, bundle='yfinance'): """ Factory function to create objective function for sensitivity analysis. Args: start_date: Start date for backtest end_date: End date for backtest bundle: Data bundle to use Returns: Objective function that takes params dict and returns Sharpe ratio """ def objective_function(params): """ Run backtest with given parameters and return Sharpe ratio. Args: params: Dictionary like {'rsi_period': 14, 'oversold_threshold': 30, ...} Returns: Float Sharpe ratio """ # Create strategy instance with these parameters strategy = RSIMeanReversion(params=params) # Run backtest try: result = run_algorithm( start=pd.Timestamp(start_date, tz='utc'), end=pd.Timestamp(end_date, tz='utc'), initialize=strategy.initialize, handle_data=strategy.check_rsi, capital_base=100000.0, bundle=bundle, ) # Extract Sharpe ratio (last value) sharpe = result['sharpe_ratio'].iloc[-1] # Handle NaN or inf if pd.isna(sharpe) or np.isinf(sharpe): return -999.0 # Return very bad value for invalid results return float(sharpe) except Exception as e: print(f"Error with params {params}: {e}") return -999.0 # Return very bad value for failed backtests return objective_function # Create objective function for our analysis objective_fn = create_objective_function( start_date='2020-01-01', end_date='2023-12-31', bundle='yfinance' ) print("✓ Objective function created") print(" Will optimize Sharpe ratio over 2020-2023")

3. Single Parameter Sensitivity¶

Analyze how performance varies with one parameter.

API Pattern:

1. Create SensitivityAnalyzer with base parameters
2. Call analyze() with objective function and parameter ranges
3. Access results for specific parameter from returned dict

In [ ]:

# Create sensitivity analyzer with base parameters
analyzer = SensitivityAnalyzer(
    base_params={
        'rsi_period': 14,
        'oversold_threshold': 30,
        'overbought_threshold': 70,
    },
    n_points=20,  # Test 20 points per parameter
    perturbation_pct=0.5,  # Vary ±50% around base value
)

print("✓ Sensitivity analyzer created")
print(f"  Base params: {analyzer.base_params}")
print(f"  Points per parameter: {analyzer.n_points}")
print(f"  Perturbation: ±{analyzer.perturbation_pct*100:.0f}%")

# Run analysis on all parameters
# Uncomment to run (takes ~5-10 minutes):
# results = analyzer.analyze(
#     objective=objective_fn,
#     param_ranges={
#         'rsi_period': (5, 30),
#         'oversold_threshold': (20, 40),
#         'overbought_threshold': (60, 80),
#     }
# )
# 
# # Access individual parameter results
# rsi_result = results['rsi_period']  # Returns SensitivityResult
# oversold_result = results['oversold_threshold']
# overbought_result = results['overbought_threshold']

print("\n✓ Ready to run sensitivity analysis")
print("  Uncomment the code above to execute")

# Create sensitivity analyzer with base parameters analyzer = SensitivityAnalyzer( base_params={ 'rsi_period': 14, 'oversold_threshold': 30, 'overbought_threshold': 70, }, n_points=20, # Test 20 points per parameter perturbation_pct=0.5, # Vary ±50% around base value ) print("✓ Sensitivity analyzer created") print(f" Base params: {analyzer.base_params}") print(f" Points per parameter: {analyzer.n_points}") print(f" Perturbation: ±{analyzer.perturbation_pct*100:.0f}%") # Run analysis on all parameters # Uncomment to run (takes ~5-10 minutes): # results = analyzer.analyze( # objective=objective_fn, # param_ranges={ # 'rsi_period': (5, 30), # 'oversold_threshold': (20, 40), # 'overbought_threshold': (60, 80), # } # ) # # # Access individual parameter results # rsi_result = results['rsi_period'] # Returns SensitivityResult # oversold_result = results['oversold_threshold'] # overbought_result = results['overbought_threshold'] print("\n✓ Ready to run sensitivity analysis") print(" Uncomment the code above to execute")

Understanding SensitivityResult¶

The analyze() method returns a dictionary mapping parameter names to SensitivityResult objects:

rsi_result = results['rsi_period']

# Available attributes:
rsi_result.param_values     # List of parameter values tested
rsi_result.objective_values # Corresponding objective values
rsi_result.stability_score  # 0-1 score (higher = more stable)
rsi_result.classification   # 'robust', 'moderate', or 'sensitive'
rsi_result.optimal_value    # Parameter value with best objective
rsi_result.optimal_objective # Best objective value achieved

Visualize Parameter Sensitivity¶

In [ ]:

def plot_parameter_sensitivity(sensitivity_result, parameter_name, baseline_value):
    """
    Plot how metric varies with parameter.
    
    Args:
        sensitivity_result: SensitivityResult object
        parameter_name: Name of parameter being analyzed
        baseline_value: Baseline/original parameter value
    """
    # Extract data from SensitivityResult
    param_values = sensitivity_result.param_values
    objective_values = sensitivity_result.objective_values
    optimal_value = sensitivity_result.optimal_value
    optimal_objective = sensitivity_result.optimal_objective
    
    # Create plot
    fig = go.Figure()
    
    # Parameter sensitivity curve
    fig.add_trace(go.Scatter(
        x=param_values,
        y=objective_values,
        mode='lines+markers',
        name='Performance',
        line=dict(color='blue', width=2),
        marker=dict(size=6)
    ))
    
    # Baseline value
    fig.add_vline(
        x=baseline_value,
        line_dash="dash",
        line_color="red",
        annotation_text=f"Baseline ({baseline_value})"
    )
    
    # Optimal value
    fig.add_trace(go.Scatter(
        x=[optimal_value],
        y=[optimal_objective],
        mode='markers',
        name=f'Optimal ({optimal_value})',
        marker=dict(size=15, color='gold', symbol='star')
    ))
    
    fig.update_layout(
        title=f"Parameter Sensitivity: {parameter_name}",
        xaxis_title=parameter_name,
        yaxis_title="Sharpe Ratio",
        height=500,
        hovermode='x unified'
    )
    
    # Print analysis
    print(f"\n=== {parameter_name} Sensitivity Analysis ===")
    print(f"Baseline value: {baseline_value}")
    print(f"Optimal value: {optimal_value}")
    print(f"Optimal Sharpe: {optimal_objective:.3f}")
    print(f"\nStability score: {sensitivity_result.stability_score:.3f}")
    print(f"Classification: {sensitivity_result.classification}")
    print(f"\nMetric range: [{min(objective_values):.3f}, {max(objective_values):.3f}]")
    print(f"Metric std dev: {np.std(objective_values):.3f}")
    
    return fig

# Example usage:
# fig = plot_parameter_sensitivity(rsi_result, 'rsi_period', baseline_value=14)
# fig.show()

print("✓ Plotting function defined")

def plot_parameter_sensitivity(sensitivity_result, parameter_name, baseline_value): """ Plot how metric varies with parameter. Args: sensitivity_result: SensitivityResult object parameter_name: Name of parameter being analyzed baseline_value: Baseline/original parameter value """ # Extract data from SensitivityResult param_values = sensitivity_result.param_values objective_values = sensitivity_result.objective_values optimal_value = sensitivity_result.optimal_value optimal_objective = sensitivity_result.optimal_objective # Create plot fig = go.Figure() # Parameter sensitivity curve fig.add_trace(go.Scatter( x=param_values, y=objective_values, mode='lines+markers', name='Performance', line=dict(color='blue', width=2), marker=dict(size=6) )) # Baseline value fig.add_vline( x=baseline_value, line_dash="dash", line_color="red", annotation_text=f"Baseline ({baseline_value})" ) # Optimal value fig.add_trace(go.Scatter( x=[optimal_value], y=[optimal_objective], mode='markers', name=f'Optimal ({optimal_value})', marker=dict(size=15, color='gold', symbol='star') )) fig.update_layout( title=f"Parameter Sensitivity: {parameter_name}", xaxis_title=parameter_name, yaxis_title="Sharpe Ratio", height=500, hovermode='x unified' ) # Print analysis print(f"\n=== {parameter_name} Sensitivity Analysis ===") print(f"Baseline value: {baseline_value}") print(f"Optimal value: {optimal_value}") print(f"Optimal Sharpe: {optimal_objective:.3f}") print(f"\nStability score: {sensitivity_result.stability_score:.3f}") print(f"Classification: {sensitivity_result.classification}") print(f"\nMetric range: [{min(objective_values):.3f}, {max(objective_values):.3f}]") print(f"Metric std dev: {np.std(objective_values):.3f}") return fig # Example usage: # fig = plot_parameter_sensitivity(rsi_result, 'rsi_period', baseline_value=14) # fig.show() print("✓ Plotting function defined")

4. Two-Parameter Sensitivity (Heatmap)¶

Visualize how performance varies across two parameters simultaneously.

API Pattern: Use analyze_interaction() for 2D analysis

In [ ]:

# Run 2D sensitivity analysis using analyze_interaction()
# Uncomment to run (takes ~10-15 minutes):
# interaction_result = analyzer.analyze_interaction(
#     param1='oversold_threshold',
#     param2='overbought_threshold',
#     objective=objective_fn,
#     param_ranges={
#         'oversold_threshold': (20, 40),
#         'overbought_threshold': (60, 80),
#     }
# )

print("✓ Ready to run 2D sensitivity analysis")
print("  Uncomment the code above to execute")
print("  Parameters: oversold_threshold x overbought_threshold")

# Run 2D sensitivity analysis using analyze_interaction() # Uncomment to run (takes ~10-15 minutes): # interaction_result = analyzer.analyze_interaction( # param1='oversold_threshold', # param2='overbought_threshold', # objective=objective_fn, # param_ranges={ # 'oversold_threshold': (20, 40), # 'overbought_threshold': (60, 80), # } # ) print("✓ Ready to run 2D sensitivity analysis") print(" Uncomment the code above to execute") print(" Parameters: oversold_threshold x overbought_threshold")

Understanding InteractionResult¶

The analyze_interaction() method returns an InteractionResult object:

interaction_result = analyzer.analyze_interaction(...)

# Available attributes:
interaction_result.param1_values      # List of param1 values tested
interaction_result.param2_values      # List of param2 values tested
interaction_result.objective_matrix   # 2D numpy array [param1, param2]
interaction_result.has_interaction    # bool: True if params interact
interaction_result.interaction_strength  # float: strength of interaction

Create Sensitivity Heatmap¶

In [ ]:

def plot_sensitivity_heatmap(interaction_result, param1_name, param2_name):
    """
    Create heatmap showing performance across 2D parameter space.
    
    Args:
        interaction_result: InteractionResult object from analyze_interaction()
        param1_name: Name of first parameter
        param2_name: Name of second parameter
    """
    # Extract data from InteractionResult
    param1_values = interaction_result.param1_values
    param2_values = interaction_result.param2_values
    objective_matrix = interaction_result.objective_matrix  # 2D numpy array
    
    # Create heatmap
    fig = go.Figure(data=go.Heatmap(
        x=param2_values,
        y=param1_values,
        z=objective_matrix,
        colorscale='RdYlGn',  # Red-Yellow-Green
        colorbar=dict(title="Sharpe Ratio"),
        hovertemplate=f"{param2_name}: %{{x}}<br>{param1_name}: %{{y}}<br>Sharpe: %{{z:.3f}}<extra></extra>"
    ))
    
    # Mark optimal point
    optimal_idx = np.unravel_index(np.argmax(objective_matrix), objective_matrix.shape)
    optimal_param1 = param1_values[optimal_idx[0]]
    optimal_param2 = param2_values[optimal_idx[1]]
    optimal_metric = objective_matrix[optimal_idx]
    
    fig.add_trace(go.Scatter(
        x=[optimal_param2],
        y=[optimal_param1],
        mode='markers',
        marker=dict(size=15, color='white', symbol='star', line=dict(color='black', width=2)),
        name='Optimal',
        showlegend=False
    ))
    
    fig.update_layout(
        title=f"2D Parameter Sensitivity: {param1_name} vs {param2_name}",
        xaxis_title=param2_name,
        yaxis_title=param1_name,
        height=600
    )
    
    # Print analysis
    print(f"\n=== 2D Sensitivity Analysis ===")
    print(f"Optimal {param1_name}: {optimal_param1}")
    print(f"Optimal {param2_name}: {optimal_param2}")
    print(f"Optimal Sharpe: {optimal_metric:.3f}")
    print(f"\nMetric range: [{np.min(objective_matrix):.3f}, {np.max(objective_matrix):.3f}]")
    print(f"Metric std dev: {np.std(objective_matrix):.3f}")
    print(f"\nHas interaction: {interaction_result.has_interaction}")
    print(f"Interaction strength: {interaction_result.interaction_strength:.3f}")
    
    # Identify stable regions (within 90% of optimal)
    stable_threshold = 0.9 * optimal_metric
    stable_count = np.sum(objective_matrix >= stable_threshold)
    total_count = objective_matrix.size
    
    print(f"\nStable region (≥90% of optimal): {stable_count}/{total_count} ({stable_count/total_count:.1%})")
    
    return fig

# Example usage:
# fig = plot_sensitivity_heatmap(interaction_result, 'oversold_threshold', 'overbought_threshold')
# fig.show()

print("✓ Heatmap function defined")

def plot_sensitivity_heatmap(interaction_result, param1_name, param2_name): """ Create heatmap showing performance across 2D parameter space. Args: interaction_result: InteractionResult object from analyze_interaction() param1_name: Name of first parameter param2_name: Name of second parameter """ # Extract data from InteractionResult param1_values = interaction_result.param1_values param2_values = interaction_result.param2_values objective_matrix = interaction_result.objective_matrix # 2D numpy array # Create heatmap fig = go.Figure(data=go.Heatmap( x=param2_values, y=param1_values, z=objective_matrix, colorscale='RdYlGn', # Red-Yellow-Green colorbar=dict(title="Sharpe Ratio"), hovertemplate=f"{param2_name}: %{{x}}
{param1_name}: %{{y}}
Sharpe: %{{z:.3f}}" )) # Mark optimal point optimal_idx = np.unravel_index(np.argmax(objective_matrix), objective_matrix.shape) optimal_param1 = param1_values[optimal_idx[0]] optimal_param2 = param2_values[optimal_idx[1]] optimal_metric = objective_matrix[optimal_idx] fig.add_trace(go.Scatter( x=[optimal_param2], y=[optimal_param1], mode='markers', marker=dict(size=15, color='white', symbol='star', line=dict(color='black', width=2)), name='Optimal', showlegend=False )) fig.update_layout( title=f"2D Parameter Sensitivity: {param1_name} vs {param2_name}", xaxis_title=param2_name, yaxis_title=param1_name, height=600 ) # Print analysis print(f"\n=== 2D Sensitivity Analysis ===") print(f"Optimal {param1_name}: {optimal_param1}") print(f"Optimal {param2_name}: {optimal_param2}") print(f"Optimal Sharpe: {optimal_metric:.3f}") print(f"\nMetric range: [{np.min(objective_matrix):.3f}, {np.max(objective_matrix):.3f}]") print(f"Metric std dev: {np.std(objective_matrix):.3f}") print(f"\nHas interaction: {interaction_result.has_interaction}") print(f"Interaction strength: {interaction_result.interaction_strength:.3f}") # Identify stable regions (within 90% of optimal) stable_threshold = 0.9 * optimal_metric stable_count = np.sum(objective_matrix >= stable_threshold) total_count = objective_matrix.size print(f"\nStable region (≥90% of optimal): {stable_count}/{total_count} ({stable_count/total_count:.1%})") return fig # Example usage: # fig = plot_sensitivity_heatmap(interaction_result, 'oversold_threshold', 'overbought_threshold') # fig.show() print("✓ Heatmap function defined")

5. Stability Metrics¶

Calculate quantitative metrics for parameter stability.

In [ ]:

def calculate_stability_metrics(sensitivity_result, baseline_objective):
    """
    Calculate stability metrics from sensitivity analysis.
    
    Args:
        sensitivity_result: SensitivityResult object
        baseline_objective: Baseline objective value for comparison
    """
    objective_values = np.array(sensitivity_result.objective_values)
    
    # Calculate metrics
    stability = {
        'coefficient_of_variation': np.std(objective_values) / np.abs(np.mean(objective_values)),
        'range': np.max(objective_values) - np.min(objective_values),
        'range_percent': (np.max(objective_values) - np.min(objective_values)) / np.abs(baseline_objective),
        'positive_rate': np.mean(objective_values > 0),
        'baseline_exceedance_rate': np.mean(objective_values > baseline_objective),
        'percentile_90_range': np.percentile(objective_values, 95) - np.percentile(objective_values, 5),
    }
    
    # Interpretation
    print("\n=== Parameter Stability Metrics ===")
    print(f"\nCoefficient of Variation: {stability['coefficient_of_variation']:.3f}")
    print(f"  Interpretation: {'✓ Low (stable)' if stability['coefficient_of_variation'] < 0.3 else '✗ High (unstable)'}")
    
    print(f"\nMetric Range: {stability['range']:.3f}")
    print(f"  As % of baseline: {stability['range_percent']:.1%}")
    print(f"  Interpretation: {'✓ Narrow (robust)' if stability['range_percent'] < 0.5 else '✗ Wide (sensitive)'}")
    
    print(f"\nPositive Rate: {stability['positive_rate']:.1%}")
    print(f"  Interpretation: {'✓ Mostly positive' if stability['positive_rate'] > 0.8 else '✗ Often negative'}")
    
    print(f"\nBaseline Exceedance Rate: {stability['baseline_exceedance_rate']:.1%}")
    print(f"  Interpretation: {'✓ Often exceeds baseline' if stability['baseline_exceedance_rate'] > 0.4 else '⚠ Rarely exceeds baseline'}")
    
    print(f"\n90% Confidence Range: {stability['percentile_90_range']:.3f}")
    print(f"  Interpretation: {'✓ Tight distribution' if stability['percentile_90_range'] < 0.5 else '✗ Wide distribution'}")
    
    # Overall assessment (also available from sensitivity_result.classification)
    stable_count = sum([
        stability['coefficient_of_variation'] < 0.3,
        stability['range_percent'] < 0.5,
        stability['positive_rate'] > 0.8,
        stability['baseline_exceedance_rate'] > 0.4,
    ])
    
    print(f"\n=== Overall Assessment ===")
    print(f"Built-in classification: {sensitivity_result.classification.upper()}")
    print(f"Stability score: {sensitivity_result.stability_score:.3f}")
    
    if stable_count >= 3:
        print("✓ ROBUST: Parameter shows good stability")
    elif stable_count == 2:
        print("⚠ MODERATE: Parameter shows moderate stability")
    else:
        print("✗ UNSTABLE: Parameter is highly sensitive")
    
    return stability

# Example usage:
# stability = calculate_stability_metrics(rsi_result, baseline_objective=1.2)

print("✓ Stability metrics function defined")

def calculate_stability_metrics(sensitivity_result, baseline_objective): """ Calculate stability metrics from sensitivity analysis. Args: sensitivity_result: SensitivityResult object baseline_objective: Baseline objective value for comparison """ objective_values = np.array(sensitivity_result.objective_values) # Calculate metrics stability = { 'coefficient_of_variation': np.std(objective_values) / np.abs(np.mean(objective_values)), 'range': np.max(objective_values) - np.min(objective_values), 'range_percent': (np.max(objective_values) - np.min(objective_values)) / np.abs(baseline_objective), 'positive_rate': np.mean(objective_values > 0), 'baseline_exceedance_rate': np.mean(objective_values > baseline_objective), 'percentile_90_range': np.percentile(objective_values, 95) - np.percentile(objective_values, 5), } # Interpretation print("\n=== Parameter Stability Metrics ===") print(f"\nCoefficient of Variation: {stability['coefficient_of_variation']:.3f}") print(f" Interpretation: {'✓ Low (stable)' if stability['coefficient_of_variation'] < 0.3 else '✗ High (unstable)'}") print(f"\nMetric Range: {stability['range']:.3f}") print(f" As % of baseline: {stability['range_percent']:.1%}") print(f" Interpretation: {'✓ Narrow (robust)' if stability['range_percent'] < 0.5 else '✗ Wide (sensitive)'}") print(f"\nPositive Rate: {stability['positive_rate']:.1%}") print(f" Interpretation: {'✓ Mostly positive' if stability['positive_rate'] > 0.8 else '✗ Often negative'}") print(f"\nBaseline Exceedance Rate: {stability['baseline_exceedance_rate']:.1%}") print(f" Interpretation: {'✓ Often exceeds baseline' if stability['baseline_exceedance_rate'] > 0.4 else '⚠ Rarely exceeds baseline'}") print(f"\n90% Confidence Range: {stability['percentile_90_range']:.3f}") print(f" Interpretation: {'✓ Tight distribution' if stability['percentile_90_range'] < 0.5 else '✗ Wide distribution'}") # Overall assessment (also available from sensitivity_result.classification) stable_count = sum([ stability['coefficient_of_variation'] < 0.3, stability['range_percent'] < 0.5, stability['positive_rate'] > 0.8, stability['baseline_exceedance_rate'] > 0.4, ]) print(f"\n=== Overall Assessment ===") print(f"Built-in classification: {sensitivity_result.classification.upper()}") print(f"Stability score: {sensitivity_result.stability_score:.3f}") if stable_count >= 3: print("✓ ROBUST: Parameter shows good stability") elif stable_count == 2: print("⚠ MODERATE: Parameter shows moderate stability") else: print("✗ UNSTABLE: Parameter is highly sensitive") return stability # Example usage: # stability = calculate_stability_metrics(rsi_result, baseline_objective=1.2) print("✓ Stability metrics function defined")

6. Identify Stable Regions¶

Find parameter ranges where performance is consistently good.

In [ ]:

def identify_stable_regions(sensitivity_result, threshold_percentile=80):
    """
    Identify parameter regions with stable high performance.
    
    Args:
        sensitivity_result: SensitivityResult object
        threshold_percentile: Percentile threshold for "good" performance
    """
    param_values = np.array(sensitivity_result.param_values)
    objective_values = np.array(sensitivity_result.objective_values)
    
    # Define threshold (e.g., top 20% of performance)
    threshold = np.percentile(objective_values, threshold_percentile)
    
    # Find stable regions
    stable_mask = objective_values >= threshold
    stable_params = param_values[stable_mask]
    
    # Identify contiguous regions
    regions = []
    if len(stable_params) > 0:
        region_start = stable_params[0]
        region_end = stable_params[0]
        
        for i in range(1, len(stable_params)):
            # Check if contiguous (allowing for floating point spacing)
            spacing = param_values[1] - param_values[0] if len(param_values) > 1 else 1
            if abs(stable_params[i] - region_end) <= spacing * 1.5:
                region_end = stable_params[i]
            else:
                regions.append((region_start, region_end))
                region_start = stable_params[i]
                region_end = stable_params[i]
        
        regions.append((region_start, region_end))
    
    # Print results
    print(f"\n=== Stable Parameter Regions (Top {100-threshold_percentile}%) ===")
    print(f"Performance threshold: {threshold:.3f}")
    print(f"Number of stable values: {len(stable_params)} / {len(param_values)}")
    print(f"\nStable regions:")
    
    for i, (start, end) in enumerate(regions, 1):
        print(f"  Region {i}: [{start}, {end}]")
    
    return regions, stable_params

# Example usage:
# regions, stable_params = identify_stable_regions(rsi_result, threshold_percentile=80)

print("✓ Stable regions function defined")

def identify_stable_regions(sensitivity_result, threshold_percentile=80): """ Identify parameter regions with stable high performance. Args: sensitivity_result: SensitivityResult object threshold_percentile: Percentile threshold for "good" performance """ param_values = np.array(sensitivity_result.param_values) objective_values = np.array(sensitivity_result.objective_values) # Define threshold (e.g., top 20% of performance) threshold = np.percentile(objective_values, threshold_percentile) # Find stable regions stable_mask = objective_values >= threshold stable_params = param_values[stable_mask] # Identify contiguous regions regions = [] if len(stable_params) > 0: region_start = stable_params[0] region_end = stable_params[0] for i in range(1, len(stable_params)): # Check if contiguous (allowing for floating point spacing) spacing = param_values[1] - param_values[0] if len(param_values) > 1 else 1 if abs(stable_params[i] - region_end) <= spacing * 1.5: region_end = stable_params[i] else: regions.append((region_start, region_end)) region_start = stable_params[i] region_end = stable_params[i] regions.append((region_start, region_end)) # Print results print(f"\n=== Stable Parameter Regions (Top {100-threshold_percentile}%) ===") print(f"Performance threshold: {threshold:.3f}") print(f"Number of stable values: {len(stable_params)} / {len(param_values)}") print(f"\nStable regions:") for i, (start, end) in enumerate(regions, 1): print(f" Region {i}: [{start}, {end}]") return regions, stable_params # Example usage: # regions, stable_params = identify_stable_regions(rsi_result, threshold_percentile=80) print("✓ Stable regions function defined")

Summary¶

Key Takeaways¶

1. Sensitivity analysis reveals parameter robustness
2. Smooth curves indicate stable strategies
3. Sharp peaks suggest overfitting
4. Stable regions are better than optimal points
5. 2D heatmaps show parameter interactions

Correct API Pattern¶

# 1. Create objective function
def objective_function(params):
    result = run_algorithm(...with params...)
    return float(result['sharpe_ratio'].iloc[-1])

# 2. Create analyzer with base parameters
analyzer = SensitivityAnalyzer(
    base_params={'param1': value1, 'param2': value2},
    n_points=20,
)

# 3. Run 1D analysis
results = analyzer.analyze(
    objective=objective_function,
    param_ranges={'param1': (min, max), ...}
)

# 4. Access results
param1_result = results['param1']  # SensitivityResult
print(param1_result.stability_score)
print(param1_result.classification)  # 'robust', 'moderate', 'sensitive'

# 5. Run 2D analysis
interaction = analyzer.analyze_interaction(
    'param1', 'param2',
    objective=objective_function,
    param_ranges={...}
)

Interpretation Guide¶

Robust Parameters:

• ✅ Coefficient of variation < 0.3
• ✅ Wide stable regions (not single point)
• ✅ Smooth performance curves
• ✅ >80% parameters yield positive returns

Overfit Parameters:

• ❌ Coefficient of variation > 0.5
• ❌ Single optimal point (sharp peak)
• ❌ Jagged performance curves
• ❌ <50% parameters yield positive returns

Best Practices¶

1. Test all critical parameters: Don't just optimize, analyze sensitivity
2. Use 2D heatmaps: Understand parameter interactions
3. Choose from stable regions: Not just optimal points
4. Monitor stability metrics: Track coefficient of variation
5. Combine with walk-forward: Validate stability over time

Decision Framework¶

When choosing parameters:

1. Identify optimal performance
2. Analyze sensitivity around optimal
3. Choose from stable plateau (not sharp peak)
4. Validate with Monte Carlo (see Notebook 15)
5. Confirm with walk-forward (see Notebook 06)

Next Steps¶

• Notebook 15: Monte Carlo Basics
• Notebook 06: Walk-Forward Optimization
• Notebook 19: Portfolio Allocation Methods

Resources¶

• Sensitivity Analysis Documentation
• Parameter Stability Guide
• Overfitting Detection