Feature-Weighted Elastic Net for Competing Risk Outcomes

Lukas Burk

Context

The elastic net objective function is given as:

\[ J(\beta_0, \beta) = \frac 1 2 || \mathbf{y} - \beta_0\mathbf{1} - \mathbf{X}\beta||^2_2 + {\color{green}\lambda} \sum_{j=1}^p \left( {\color{red}\alpha} |\beta_j| + \frac{1 - {\color{red}\alpha}}{2} \beta^2_j \right) \]

  • \(\color{green}{\lambda} \in \mathbb{R}_+\) controls overall sparsity
  • Higher \(\Rightarrow\) larger penalty on all \(\beta_j\), equally
  • \(\color{red}{\alpha} \in [0, 1]\) is the mixing parameter
  • \(1 \Rightarrow\) only \(\ell^1\) penalty (LASSO)
  • \(0 \Rightarrow\) only \(\ell^2_2\) penalty

(Zou and Hastie 2005)

Feature-Weighted Elastic Net (fwelnet)

  • Introduced by Tay et al. (2020)
  • Motivation: Using external information & feature-grouping
  • Adjust penalization weights on individual or groups of coefficients
  • Assign groups via matrix \(\mathbf{Z} \in \mathbb{R}_{\geq 0}^{p \times K}\)

Feature Weighting: Groups

Example for \(p = 4\) features \(X_{1,2,3,4}\) and \(K = 2\) groups

\[ \mathbf{Z} = \begin{pmatrix} 1 & 0 \\ 1 & 0 \\ 0 & 1 \\ 0 & 1 \\ \end{pmatrix} \]

  • \(X_1, X_2\) ➔ group 1
  • \(X_3, X_4\) ➔ group 2

Now imagine this, but with \(p >> 1000\) and e.g. genetic data.

Feature-Weighting

\[ J(\beta_0, \beta) = \frac{1}{2} || \mathbf{y} - \beta_0\mathbf{1} - \mathbf{X}\beta||^2_2 + \lambda \sum_{j=1}^p {\color{blue}w_j(\theta)}\left( \alpha |\beta_j| + \frac{1 - \alpha}{2} \beta^2_j \right) \]

\[ {\color{blue}w_j(\theta)} = \frac{\sum_{l=1}^p \exp(\mathbf{z}_l^T \theta)}{p \exp(\mathbf{z}_j^T \theta)} \]

  • Defines the penalization weight of \(\beta_j\) based on its corresponding value in \(\mathbf{Z}\) and hyper-parameter \(\theta \in \mathbb{R}^{K \times 1}\)
  • \(\mathbf{z}_j\): Row \(j\) of \(\mathbf{Z}\) as column vector

Penalization Weights

  • \(w_j(\theta)\) is chosen heuristically, suggested by the authors for desirable properties
  • \(\mathbf{z}_j^T \theta\) functions as a score
    • \(= 0\), reduces to original elastic net
    • Higher score \(\rightarrow\) lower \(w_j\), feature is “more important”

  • In the “feature grouping” setting, this just allows group-specific penalization weights.

  • Related to the “group lasso” (Jacob, Obozinski, and Vert 2009)

Feature-Weighting: Single Group

  • \(\mathbf{Z}\) can also be \(p \times 1\): No groups, just weights
  • Simulation example from Tay et al. (2020): \(\mathbf{Z}\) set to noisy version of true \(|\beta|\)
  • Higher \(|\beta_j| \Rightarrow\) lower penalization for \(\hat{\beta}_j\)
  • \(|\beta_j| \approx 0 \Rightarrow\) higher penalization for \(\hat{\beta}_j\)

Application for Multi-Task Learning

  • Authors suggest multi-task learning algorithm, outline:

  1. Set \(\beta_1^{(0)}, \beta_2^{(0)}\) to glmnet solution for \((\mathbf{X}, \mathbf{y}_1), (\mathbf{X}, \mathbf{y}_2)\) respectively

  2. For \(k = 0, 1, \ldots\):

    1. \(\mathbf{Z}_2 = \left|\beta_1^{(k)}\right|\). Fit fwelnet with \((\mathbf{X}, \mathbf{y}_2, \mathbf{Z}_2)\)
      • Set \(\left|\beta_2^{(k+1)}\right|\) to solution with optimal lambda
    2. \(\mathbf{Z}_1 = \left|\beta_2^{(k+1)}\right|\). Fit fwelnet with \((\mathbf{X}, \mathbf{y}_1, \mathbf{Z}_1)\)
      • Set \(\left|\beta_1^{(k+1)}\right|\) to solution with optimal lambda
  • Side note: \(\beta_2^{(0)}\) is never used. Weird.

Transfer to Competing Risks

  • Setting with two outcomes/causes: \((\mathbf{t}_1, \mathbf{y}_1), (\mathbf{t}_2, \mathbf{y}_2)\)
  • Assumption: Shared information for both causes:
    • If \(X_j\) is important for cause 1, may also be relevant for cause 2
    • \(\Rightarrow\) lower its penalty in cause-specific models
  • Basic idea: Adapt previous algorithm to Cox regression
  • Multi-task \(\Rightarrow\) “Multi-cause”

What We Have

  • 1: Adapt fwelnet for Coxnet/Surv endpoint via glmnet
  • 2: Implement multi-task algorithm for Cox/CR setting
  • 3: Apply algorithm to some “easy” simulated data settings:
    • Large effects (\(\beta_j = 1\) or \(0.25\) for “small effect”)
    • \(N = 1000\)
    • 11-13 noise variables
  • First goal: See if we find some improvement over cause-specific glmnet
  • Caveat: Prediction NYI, so only looking at \(\hat{\beta}_j\)

What Comes Next

  • Wider set of simulation cases
  • Evaluate predictive performance
    • (non-trivial in CR/censored setting)
  • Try on real data

References

Jacob, Laurent, Guillaume Obozinski, and Jean-Philippe Vert. 2009. “Group Lasso with Overlap and Graph Lasso.” In Proceedings of the 26th Annual International Conference on Machine Learning - ICML ’09, 1–8. ACM Press. https://doi.org/10.1145/1553374.1553431.
Tay, J. Kenneth, Nima Aghaeepour, Trevor Hastie, and Robert Tibshirani. 2020. “Feature-Weighted Elastic Net: Using ‘Features of Features’ for Better Prediction.” arXiv:2006.01395 [Cs, Stat], June. http://arxiv.org/abs/2006.01395.
Zou, Hui, and Trevor Hastie. 2005. “Regularization and Variable Selection via the Elastic Net.” Journal of the Royal Statistical Society. Series B (Statistical Methodology) 67 (2): 301–20. https://www.jstor.org/stable/3647580.