Comprehensive temporal analysis of Low-Rank Adaptation patterns
Template PreviewInline Data
Checkpoints
0
Components
0
Data Points
0
Parameters
0
Anomalies
0
Phase Transitions
0
Trajectory Analysis
Explore the temporal evolution of LoRA components across training checkpoints.
Select components to compare their trajectories and identify patterns.
Component Trajectories
Component List
Component
Layer
Type
Initial
Final
Change
Volatility
Actions
Phase Transition Detection
Identifying critical points where LoRA weights undergo significant changes in behavior,
indicating potential training phase transitions or learning regime shifts.
Transition Timeline
Transition Distribution
Layer-wise Transition Heatmap
Detected Transitions
Checkpoint
Component
Layer
Metric
Before
After
Change
Significance
Component Clustering Analysis
Components are grouped based on their evolution patterns, helping identify similar behaviors
and potential functional relationships between different parts of the model.
Cluster Visualization (PCA)
Cluster Statistics
Cluster Trajectory Patterns
Cluster Summary
Cluster
Size
Dominant Layer
Dominant Type
Avg Frobenius
Avg Rank
Characteristics
Gradient Flow Analysis
Examining the rate of change in LoRA weights between checkpoints to understand
learning dynamics and identify periods of rapid adaptation.
Gradient Magnitude Over Time
Layer-wise Gradient Flow
Gradient Acceleration Analysis
Anomaly Detection Results
Components and checkpoints showing unusual patterns that deviate significantly from
expected behavior. These may indicate training issues or interesting phenomena.
Anomaly Distribution
Anomaly Timeline
Detected Anomalies
Component
Checkpoint
Type
Metric
Value
Z-Score
Severity
Convergence Analysis
Evaluating how LoRA weights stabilize over training, including convergence rates,
stability metrics, and identification of components that have reached steady states.
Convergence Rates
Stability Analysis
Convergence Quality Heatmap
Cross-Component Correlation Analysis
Identifying relationships between different LoRA components based on their
co-evolution patterns throughout training.
Correlation Matrix
Correlation Network
Advanced Analysis Tools
Additional research-oriented visualizations and metrics for in-depth exploration
of LoRA weight dynamics.
3D Visualization Settings
3D Metric Space Exploration
Mutual Information Analysis
Rank Dynamics
DoRA (Weight-Decomposed Low-Rank Adaptation) Analysis
Analyze magnitude vectors and directional components unique to DoRA checkpoints.
Magnitude Vector Analysis
Magnitude Distribution by Layer
Magnitude Evolution Over Time
Magnitude Value Histogram
Directional Diversity Score
Magnitude Heatmap
Directional Component Analysis
Direction Stability Scores
Direction Drift Over Time
Principal Direction Analysis
LoRA vs DoRA Comparison
This feature requires both LoRA and DoRA checkpoints to be present in the dataset.
DoRAscope is a comprehensive analysis tool for exploring the temporal evolution of Low-Rank Adaptation (LoRA) and DoRA weights during model training. It provides deep insights into how adaptation parameters change across checkpoints, helping researchers understand training dynamics and optimize their models.
Key Features
Temporal Analysis: Track how LoRA weights evolve across training checkpoints
Multi-metric Evaluation: Analyze weights using 14+ mathematical metrics
Anomaly Detection: Automatically identify unusual patterns and outliers
Component Clustering: Group similar adaptation patterns together
Interactive Visualizations: Explore data through various chart types and 3D plots
Performance Optimized: Handles large-scale analyses with caching and parallel processing
Quick Start
Use the Overview tab to get a high-level view of your LoRA weights
Apply filters (layers, modules, projections) to focus on specific components
Switch between metrics using the dropdown to explore different aspects
Navigate to specialized tabs for detailed analyses
Export your findings using the Export button
Core Concepts
LoRA (Low-Rank Adaptation)
LoRA adapts large language models by learning two low-rank matrices (A and B) that approximate weight updates. The adaptation is computed as W' = W + α(BA), where α is a scaling factor.
DoRA (Weight-Decomposed Low-Rank Adaptation)
DoRA extends LoRA by decomposing the adaptation into magnitude and direction components. It includes a magnitude vector that scales the adaptation: W' = W + m⊙(BA), where m is the learned magnitude vector.
Components Structure
Layers: Transformer layers (0, 1, 2, ...)
Module Types:
Attention: Self-attention mechanisms (q, k, v, o projections)
MLP: Feed-forward networks (gate, up, down projections)
Projections:
q_proj: Query projection in attention
k_proj: Key projection in attention
v_proj: Value projection in attention
o_proj: Output projection in attention
gate_proj: Gating mechanism in MLP
up_proj: Upward projection in MLP
down_proj: Downward projection in MLP
Metrics Explained
Magnitude Metrics
Frobenius Norm
Overall magnitude of the weight matrix. Higher values indicate stronger adaptations. Formula: ||W||_F = √(Σᵢⱼ w²ᵢⱼ)
Spectral Norm
Largest singular value. Indicates the maximum amplification factor. Important for stability analysis.
Nuclear Norm
Sum of all singular values. Measures the total "energy" in the adaptation.
Rank & Dimensionality Metrics
Effective Rank
Shannon entropy-based measure of how many dimensions are effectively used. Formula: exp(H(σ²/||σ||²))
Stable Rank
Ratio of Frobenius norm squared to spectral norm squared. More stable than traditional rank.
Rank Utilization
Percentage of the allocated rank that's actively used (effective_rank / actual_rank).
Distribution Metrics
Spectral Entropy (A/B)
Entropy of singular value distribution. High values indicate even distribution across dimensions.
LoRA Entropy (A/B)
Shannon entropy of weight values. Measures randomness/structure in the weights.
Ratio of largest to smallest eigenvalue. High values indicate potential numerical instability.
Participation Ratio
How many eigenmodes contribute significantly. Formula: (Σλᵢ)²/Σλᵢ²
Mutual Information Proxy
Estimate of information shared between A and B matrices.
Navigation Guide
Tab Functions
Overview
High-level visualizations of metrics across all components. Best starting point for exploration.
Trajectories
Track individual component evolution over time. Compare multiple components simultaneously.
Clustering
Groups components with similar evolution patterns. Uses PCA for dimensionality reduction.
Anomalies
Identifies components with unusual behavior (>3.5σ from mean). Critical for debugging.
Advanced
3D visualizations and specialized analyses for deep exploration.
Filtering System
Layer Filter: Use ranges (0-10) or comma-separated values (5,10,15)
Module Type: Focus on attention or MLP components
Projection Type: Analyze specific weight matrices
Keyboard Shortcuts
Esc - Close active modal
? - Toggle help (this window)
E - Open export dialog
P - Print current view
Frequently Asked Questions
Analysis & Interpretation
What does high Frobenius norm indicate?
High Frobenius norm suggests strong adaptations. This could mean the component is learning significant features or potentially overfitting. Compare with rank utilization for better insights.
Why are some components marked as anomalies?
Components are flagged when their metrics deviate >3.5 standard deviations from the mean. This could indicate initialization issues, dead neurons, or components learning unique patterns.
What's a healthy rank utilization percentage?
Typically 60-90% indicates efficient use of allocated capacity. Below 50% suggests over-parameterization; above 95% might benefit from higher rank.
How do I identify convergence issues?
Look for components with high volatility in the Trajectories tab, or check for increasing condition numbers over time. Stable convergence shows plateauing values with low variance.
Technical & Performance
Why is my analysis taking long?
Analysis time depends on checkpoint count, component count, and enabled metrics. You can:
Set LORA_SAMPLE_RATE=0.1 to analyze 10% of components
Limit workers with LORA_MAX_WORKERS=4
Disable advanced metrics by setting ADVANCED_METRICS=False
Can I analyze non-LoRA weights?
The tool expects safetensor files with LoRA naming conventions (lora_A, lora_B). Custom naming patterns would require modifying the extraction logic.
Memory usage is high. What can I do?
Large analyses can be memory-intensive. Try:
Process checkpoints in batches
Clear the cache directory (.lora_analysis_cache/)
Reduce component sampling rate
Export & Sharing
What's included in JSON exports?
JSON exports include raw data, computed metrics, analysis results (anomalies, clusters, etc.), and metadata. Enable "Include metadata" for complete exports.
How do I create publication-ready figures?
Plotly charts can be saved as high-resolution images using the camera icon in the chart toolbar. For LaTeX integration, export data as CSV and recreate plots with your preferred tools.
Troubleshooting
Charts show "No data available"
This typically means:
Your filters exclude all data (check layer ranges)
The selected metric isn't available for your model type
Analysis failed - check the Python console for errors
Browser is unresponsive
Large datasets can strain the browser. Try:
Closing other tabs to free memory
Using Chrome/Edge for better performance
Filtering to fewer components before complex operations
Help & Documentation
Overview
This dashboard provides comprehensive analysis of LoRA (Low-Rank Adaptation) weight evolution during training.
Key Metrics
Frobenius Norm: Overall magnitude of the adaptation
Effective Rank: Dimensionality of the weight space being used
Spectral Entropy: Distribution of singular values
Rank Utilization: Efficiency of rank usage
Navigation
Use the tabs to explore different aspects of the analysis. Each tab focuses on specific patterns and behaviors in the LoRA weights.
Filtering
Use the control panel to filter data by layers, module types, and projection types. Layer ranges can be specified as "0-10" or comma-separated values like "5,10,15".