Feature Selection and Model Evaluation

Understanding Feature Selection Techniques

Feature selection is a cornerstone of modern data science, especially when dealing with high‑dimensional data. Selecting the right subset of variables improves model interpretability, reduces overfitting, and often leads to faster training times. In this module we explore the most common strategies—filter, wrapper, and embedded methods—while highlighting their strengths and pitfalls.

Filter Methods vs. Wrapper Methods vs. Embedded Methods

Three broad families of feature selection exist:

• Filter methods evaluate each feature independently of any learning algorithm, using statistical scores such as correlation, chi‑square, or mutual information.
• Wrapper methods treat the learning algorithm as a black box and assess subsets of features by training a model on each candidate set. While powerful, they are computationally intensive and can overfit on small datasets.
• Embedded methods incorporate feature selection directly into the model training process (e.g., Lasso, Elastic Net, tree‑based algorithms).

Common Misconception About Wrapper Methods

It is often mistakenly believed that wrapper methods are cheaper than filter methods. In reality, the opposite is true: wrappers require repeated model fitting, making them the most computationally demanding approach. This misconception is highlighted by the following false statement:

"They are computationally cheaper than filter methods."

Understanding this nuance helps data scientists choose the appropriate technique based on dataset size and computational resources.

Regularization for High‑Dimensional Data

When the number of predictors exceeds the number of observations—as in the classic "p > n" scenario—regularization becomes essential. Regularization adds a penalty term to the loss function, shrinking coefficient estimates and, in some cases, forcing them to zero.

Lasso (L1) vs. Ridge (L2) vs. Elastic Net

Consider a dataset with 500 features and only 200 observations. The goal is to obtain a parsimonious model while handling correlated predictors. The most suitable technique is Elastic Net Regularization, which combines the strengths of both L1 and L2 penalties:

• Lasso (L1) can set coefficients exactly to zero, performing variable selection, but struggles when predictors are highly correlated.
• Ridge (L2) shrinks coefficients toward zero without eliminating any, making it effective for multicollinearity but not for sparsity.
• Elastic Net balances these effects, encouraging sparsity while stabilizing groups of correlated variables.

When multicollinearity is present, the regularization method that both reduces coefficient variance and sets some coefficients to zero is Lasso Regression (L1 penalty). However, if you also need to retain groups of correlated features, Elastic Net is often the better choice.

Evaluating Models on Imbalanced Data

Class imbalance—where one class dominates the dataset—poses a challenge for traditional accuracy metrics. In a binary classification problem with severe imbalance, the most reliable metric for assessing the minority class is Recall of the minority class. Recall (also called sensitivity) measures the proportion of actual positive instances correctly identified, directly reflecting the model’s ability to capture the rare events.

Why Not Accuracy?

Overall accuracy can be misleading because a naïve model that always predicts the majority class may achieve high accuracy while completely ignoring the minority class. Therefore, practitioners turn to metrics that focus on the minority class performance:

• Recall (Sensitivity) – captures true positives.
• Precision – captures false positives.
• F1‑score – the harmonic mean of precision and recall, useful when both false positives and false negatives are costly.

F1‑Score Explained

The F1‑score is defined as:

F1 = 2 * (Precision * Recall) / (Precision + Recall)

It is preferred when you need a single metric that balances the trade‑off between precision and recall, especially in domains such as fraud detection or medical diagnosis where both types of errors have serious consequences.

Dimensionality Reduction with Principal Component Analysis (PCA)

PCA is a powerful unsupervised technique that transforms correlated variables into a set of orthogonal components. After standardizing variables, the first eigenvalue represents the proportion of variance explained by the first principal component. This value indicates how much of the original data’s variability can be captured by a single linear combination of features.

Interpreting Eigenvalues

Each eigenvalue corresponds to a principal component:

• Higher eigenvalues mean the component captures more variance.
• The sum of all eigenvalues equals the total variance of the standardized data (which is equal to the number of variables).
• Choosing components with eigenvalues greater than 1 (Kaiser’s criterion) or using a scree plot are common strategies for dimensionality reduction.

Feature Selection Algorithms in Practice

When dealing with ultra‑high dimensional data—such as 10,000 features—algorithmic choices matter. Forward Selection, a greedy wrapper method, adds features one at a time based on improvement in model performance. However, its stepwise nature can cause it to miss the globally optimal feature set because it never revisits earlier decisions.

Key Drawback of Forward Selection

The most likely issue is:

"The method may miss optimal feature combinations because it adds variables step by step."

In contrast, exhaustive search evaluates all possible subsets but is computationally infeasible for thousands of features. Hybrid approaches—such as combining filter methods to pre‑screen variables before applying a wrapper—can mitigate this limitation.

Linear Discriminant Analysis (LDA) Objective

Linear Discriminant Analysis seeks a linear combination of features that best separates classes. The objective function maximized in LDA is the ratio of between‑class scatter to within‑class scatter. By maximizing this ratio, LDA ensures that projected class means are far apart while keeping the variance within each class as small as possible.

Mathematical Formulation

Let S_B denote the between‑class scatter matrix and S_W the within‑class scatter matrix. LDA solves:

argmax_w (w^T S_B w) / (w^T S_W w)

The solution involves eigen‑decomposition of S_W^{-1} S_B, yielding discriminant vectors that can be used for classification or dimensionality reduction.

Putting It All Together: A Practical Workflow

Below is a recommended step‑by‑step workflow for a typical data‑science project involving feature selection and model evaluation:

1. Data Exploration: Examine distributions, missing values, and correlations.
2. Pre‑processing: Standardize or normalize features, especially before applying regularization or PCA.
3. Initial Filtering: Use filter methods (e.g., variance threshold, mutual information) to drop clearly irrelevant features.
4. Dimensionality Reduction (Optional): Apply PCA if you need to compress information while preserving variance.
5. Embedded Selection: Fit an Elastic Net model to simultaneously handle multicollinearity and enforce sparsity.
6. Wrapper Evaluation: If computational budget permits, run a wrapper (e.g., recursive feature elimination) on the reduced set to fine‑tune the feature subset.
7. Model Training: Choose an algorithm appropriate for the problem (e.g., logistic regression for binary classification).
8. Performance Metrics: For imbalanced data, prioritize recall, precision, and F1‑score; for balanced data, consider accuracy and AUC.
9. Cross‑Validation: Use stratified k‑fold CV to obtain reliable estimates and guard against overfitting.
10. Interpretation & Reporting: Present the selected features, their coefficients, and the chosen evaluation metrics.

Following this structured approach ensures that you address multicollinearity, maintain model parsimony, and evaluate performance with metrics that truly reflect the business or research objectives.