High-Dimensional Inference under Dynamics and Complexity: Applications in Biology and Finance

Abstract

This thesis focuses on advancing statistical inference methods and theoretical understanding for modeling and optimization problems in high-dimensional settings, specifically targeting scenarios where the number of variables substantially exceeds available observations. The central contributions include the development of a robust method for detecting change points in graphical models and an in-depth theoretical analysis of the "double descent" phenomenon in mean-variance portfolio optimization.

In Chapter 2, we propose the Factor-Augmented Graphical Model (FAGM), a novel methodology for detecting structural changes in piecewise-constant graphical models. Unlike traditional approaches that assume independent observations and static network structures, FAGM explicitly incorporates unobserved, temporally correlated common factors. This allows the method to isolate true shifts in conditional dependencies by analyzing error components, which are assumed to be independently and piecewise-identically distributed. We rigorously establish the theoretical consistency of a regularized estimator and demonstrate its effectiveness through extensive simulation studies and a real-world application involving gene expression data.

Chapter 3 investigates the empirically observed but theoretically ambiguous "double descent" phenomenon within the framework of mean-variance portfolio optimization. By explicitly modeling complexity through the number of assets, we uncover a distinctive "double ascent" pattern in the out-of-sample Sharpe ratio. Initially, increased portfolio complexity enhances performance, followed by deterioration due to estimation errors, and eventually a renewed improvement when complexity surpasses the number of observations. Our theoretical results provide transparency into the causal mechanisms behind this phenomenon, quantifying how economic benefits (theoretical Sharpe ratio) and statistical accuracy (estimation precision) jointly drive this counterintuitive behavior. This analysis reveals that over-parameterization can significantly outperform simpler models in high-dimensional portfolio selection contexts.

Collectively, this thesis provides rigorous theoretical foundations, complemented by extensive simulation and empirical studies, clearly demonstrating the practical utility and robustness of the proposed statistical methodologies and theoretical insights. Show more