Dimensionality Reduction - PCA and t-SNE

What Is It?

What Is Dimensionality Reduction?

In machine learning, each feature in your dataset is a dimension. A dataset with 100 features is a 100-dimensional space. Dimensionality reduction transforms high-dimensional data into a lower-dimensional representation while preserving as much useful information as possible.

Imagine you have a dataset of customer profiles with 200 features (demographics, purchase history, browsing behavior, etc.). Many of these features are correlated or redundant. Dimensionality reduction distills these 200 features down to, say, 20 components that capture most of the variation in the data.

Two Approaches

• Feature selection: Choose a subset of the original features and discard the rest. The selected features are unchanged. Example: pick the 10 most important features out of 200.
• Feature extraction: Create new features by combining existing ones. The new features are mathematical transformations, not the original columns. Example: PCA creates new components that are linear combinations of all original features.

Why Do We Need It?

Reducing dimensions helps with visualization (you can plot 2D or 3D), speeds up training, reduces overfitting, and removes noise. It is a critical preprocessing step in many ML pipelines.

Why Does It Matter?

Why Is Dimensionality Reduction Important?

1. The Curse of Dimensionality

As the number of features increases, the volume of the feature space grows exponentially. With 10 binary features, there are 2^10 = 1,024 possible regions. With 100 binary features, there are 2^100 regions -- more than atoms in the universe. Your data becomes incredibly sparse in this high-dimensional space.

This sparsity causes several problems:

• Distance becomes meaningless: In high dimensions, the distance between any two points converges to roughly the same value. Algorithms like KNN and K-Means that rely on distance stop working effectively.
• More data is needed: To cover a 100-dimensional space with reasonable density, you need exponentially more data points. This is rarely feasible.
• Overfitting increases: With many features and limited data, models memorize noise instead of learning patterns.

2. Visualization

Humans can visualize at most 3 dimensions. If your data has 784 features (like MNIST images, which are 28x28 pixels), you cannot plot it directly. Dimensionality reduction to 2D or 3D lets you see cluster structure, outliers, and class separation.

3. Faster Training

Fewer features mean smaller matrices, faster distance computations, and shorter training times. Reducing 1,000 features to 50 can make algorithms like SVM or KNN orders of magnitude faster.

4. Noise Removal

Not all features carry useful signal. Some are pure noise. PCA, for example, can separate the signal (top principal components) from the noise (bottom components), effectively denoising the data.

5. Multicollinearity

When features are highly correlated, linear models become unstable (coefficients blow up). PCA creates uncorrelated components, solving multicollinearity completely.

Detailed Explanation

Detailed Explanation

1. Principal Component Analysis (PCA)

PCA is the most widely used dimensionality reduction technique. It finds new axes (principal components) that capture the maximum variance in the data.

Intuition

Imagine a cloud of data points in 3D. If the points mostly lie on a flat plane (with little thickness), you can project them onto that plane (2D) without losing much information. PCA finds this plane automatically. The first principal component (PC1) is the direction of maximum variance. The second (PC2) is perpendicular to PC1 and captures the next most variance. And so on.

Mathematical Foundation

1. Standardize the data: Center each feature to zero mean (and optionally scale to unit variance).
2. Compute the covariance matrix: This captures how each pair of features varies together.
3. Compute eigenvectors and eigenvalues: Eigenvectors are the directions (principal components). Eigenvalues tell you how much variance each direction captures.
4. Sort by eigenvalue: The eigenvector with the largest eigenvalue is PC1, the next is PC2, etc.
5. Project the data: Multiply the original data by the top k eigenvectors to get k-dimensional data.

Explained Variance Ratio

Each principal component has an explained variance ratio: the proportion of total variance it captures. If PC1 explains 60% and PC2 explains 25%, together they explain 85% of the variance. You typically choose enough components to capture 90-95% of the total variance.

Choosing the Number of Components

Plot the cumulative explained variance ratio against the number of components. Choose the number where the cumulative variance reaches your threshold (e.g., 95%). Alternatively, set PCA(n_components=0.95) in sklearn to automatically select the number of components that explain 95% of the variance.

2. t-SNE (t-Distributed Stochastic Neighbor Embedding)

t-SNE is a non-linear dimensionality reduction technique designed specifically for visualization. Unlike PCA, it can preserve complex, non-linear relationships.

How t-SNE Works (Intuition)

1. In the high-dimensional space, compute the probability that point A would pick point B as its neighbor (based on distance). Nearby points have high probability; distant points have low probability.
2. In the low-dimensional space (2D), define a similar probability using a Student's t-distribution (which has heavier tails than a Gaussian, preventing the "crowding problem").
3. Optimize the low-dimensional positions to minimize the difference (KL divergence) between the high-dimensional and low-dimensional probabilities.

Key Parameter: Perplexity

Perplexity roughly corresponds to the number of effective nearest neighbors. Typical values are 5-50. Low perplexity focuses on local structure (tight clusters but may miss global relationships). High perplexity considers more neighbors (better global structure but may blur local details). Try multiple values.

t-SNE Limitations

• Visualization only: Do not use t-SNE for feature extraction before training a classifier. It is non-parametric (no transform method for new data), slow, and the components have no interpretable meaning.
• Non-deterministic: Different runs with different random seeds produce different layouts. Always set random_state for reproducibility.
• Cluster sizes and distances are not meaningful: t-SNE distorts global distances. Clusters that appear large or far apart may not actually be.
• Slow on large datasets: O(n^2) time and memory. Use PCA first to reduce to 50 dimensions, then apply t-SNE.

3. UMAP (Uniform Manifold Approximation and Projection)

UMAP is a newer alternative to t-SNE that is generally faster and better at preserving global structure. It works by constructing a topological representation of the high-dimensional data and finding a low-dimensional projection that preserves that topology.

Key advantages over t-SNE: faster computation, better preservation of global structure, can be used for feature extraction (has a transform() method for new data), and can produce embeddings in any number of dimensions (not just 2 or 3).

4. When to Use PCA

• Before KNN/SVM: These algorithms suffer badly from high dimensionality. PCA reduces dimensions and removes noise, improving both speed and accuracy.
• For multicollinearity: PCA creates uncorrelated components, which is ideal for linear regression and logistic regression.
• As preprocessing: Reduce from 1000 to 50-100 components before training, significantly speeding up computation.
• NOT for tree-based models: Decision trees, random forests, and gradient boosting handle high dimensions natively and select features internally. PCA can actually hurt performance because it creates components that are harder for trees to split on.

5. Feature Selection vs Feature Extraction

Code Examples

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris

# Load Iris dataset: 4 features, 150 samples
iris = load_iris()
X = iris.data
y = iris.target
print(f"Original shape: {X.shape}")  # (150, 4)

# Step 1: Standardize (important for PCA)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Step 2: Apply PCA to reduce from 4D to 2D
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
print(f"Reduced shape: {X_pca.shape}")  # (150, 2)

# Step 3: Check explained variance
print(f"\nExplained variance ratio: {pca.explained_variance_ratio_}")
print(f"Cumulative variance: {pca.explained_variance_ratio_.cumsum()}")
print(f"Total variance explained by 2 components: {pca.explained_variance_ratio_.sum():.4f}")

# Step 4: The principal components (loadings)
print(f"\nPC1 loadings: {pca.components_[0].round(3)}")
print(f"PC2 loadings: {pca.components_[1].round(3)}")
print(f"Feature names: {iris.feature_names}")

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_wine

# Wine dataset: 13 features
wine = load_wine()
X = StandardScaler().fit_transform(wine.data)
print(f"Original features: {X.shape[1]}")

# Fit PCA with all components to see variance distribution
pca_full = PCA()
pca_full.fit(X)

print("\nVariance explained by each component:")
cum_var = 0
for i, var in enumerate(pca_full.explained_variance_ratio_):
    cum_var += var
    marker = " <-- 95% threshold" if 0.94 < cum_var < 0.96 else ""
    print(f"  PC{i+1}: {var:.4f} (cumulative: {cum_var:.4f}){marker}")

# Method 1: Manual threshold
n_95 = np.argmax(pca_full.explained_variance_ratio_.cumsum() >= 0.95) + 1
print(f"\nComponents for 95% variance: {n_95}")

# Method 2: Let sklearn choose automatically
pca_auto = PCA(n_components=0.95)
X_reduced = pca_auto.fit_transform(X)
print(f"Auto-selected components: {pca_auto.n_components_}")
print(f"Reduced shape: {X_reduced.shape}")
print(f"Actual variance captured: {pca_auto.explained_variance_ratio_.sum():.4f}")

import numpy as np
from sklearn.manifold import TSNE
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler

# Load MNIST-like digits dataset: 64 features (8x8 images)
digits = load_digits()
X = digits.data
y = digits.target
print(f"Shape: {X.shape}")  # (1797, 64)
print(f"Classes: {sorted(set(y))}")

# Apply t-SNE to reduce from 64D to 2D
tsne = TSNE(n_components=2, perplexity=30, random_state=42, n_iter=1000)
X_tsne = tsne.fit_transform(X)
print(f"\nt-SNE output shape: {X_tsne.shape}")

# Check how well classes are separated in 2D
print("\nCluster centers in t-SNE space (per digit):")
for digit in range(10):
    mask = y == digit
    center = X_tsne[mask].mean(axis=0)
    spread = X_tsne[mask].std(axis=0).mean()
    print(f"  Digit {digit}: center=({center[0]:.1f}, {center[1]:.1f}), spread={spread:.1f}")

print(f"\nKL divergence: {tsne.kl_divergence_:.4f}")
print("Lower KL divergence = better embedding")

import numpy as np
from sklearn.decomposition import PCA
from sklearn.neighbors import KNeighborsClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import accuracy_score
import time

# Load digits: 64 features
digits = load_digits()
X = digits.data
y = digits.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale
scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc = scaler.transform(X_test)

# KNN without PCA (64 features)
start = time.time()
knn_full = KNeighborsClassifier(n_neighbors=5)
knn_full.fit(X_train_sc, y_train)
acc_full = accuracy_score(y_test, knn_full.predict(X_test_sc))
time_full = time.time() - start
print(f"KNN (64 features): accuracy={acc_full:.4f}, time={time_full:.4f}s")

# KNN with PCA (reduce to ~95% variance)
pca = PCA(n_components=0.95)
X_train_pca = pca.fit_transform(X_train_sc)
X_test_pca = pca.transform(X_test_sc)

start = time.time()
knn_pca = KNeighborsClassifier(n_neighbors=5)
knn_pca.fit(X_train_pca, y_train)
acc_pca = accuracy_score(y_test, knn_pca.predict(X_test_pca))
time_pca = time.time() - start

print(f"KNN ({pca.n_components_} PCA components): accuracy={acc_pca:.4f}, time={time_pca:.4f}s")
print(f"\nDimension reduction: {64} -> {pca.n_components_}")
print(f"Variance retained: {pca.explained_variance_ratio_.sum():.4f}")
print(f"Accuracy change: {acc_pca - acc_full:+.4f}")

import numpy as np
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
from sklearn.datasets import load_digits
from sklearn.preprocessing import StandardScaler

# Load digit images
digits = load_digits()
X = digits.data  # 1797 samples, 64 features
y = digits.target
print(f"Original shape: {X.shape}")

# Standardize
X_scaled = StandardScaler().fit_transform(X)

# Method 1: PCA to 2D
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X_scaled)
print(f"\nPCA 2D shape: {X_pca.shape}")
print(f"PCA variance explained: {pca.explained_variance_ratio_.sum():.4f}")

# Method 2: PCA to 30D, then t-SNE to 2D (recommended pipeline)
pca_30 = PCA(n_components=30)
X_pca30 = pca_30.fit_transform(X_scaled)
print(f"PCA 30D variance: {pca_30.explained_variance_ratio_.sum():.4f}")

tsne = TSNE(n_components=2, perplexity=30, random_state=42)
X_tsne = tsne.fit_transform(X_pca30)
print(f"t-SNE 2D shape: {X_tsne.shape}")

# Compare separation: calculate mean inter-class distance
def class_separation(X_2d, y):
    centers = {}
    for label in sorted(set(y)):
        centers[label] = X_2d[y == label].mean(axis=0)
    total_dist = 0
    count = 0
    for i in centers:
        for j in centers:
            if i < j:
                total_dist += np.linalg.norm(centers[i] - centers[j])
                count += 1
    return total_dist / count

pca_sep = class_separation(X_pca, y)
tsne_sep = class_separation(X_tsne, y)

print(f"\nClass separation (higher is better):")
print(f"  PCA 2D:  {pca_sep:.2f}")
print(f"  t-SNE 2D: {tsne_sep:.2f}")
print(f"\nt-SNE provides much better class separation for visualization.")
print("But remember: t-SNE is for visualization only, not for ML features.")

import numpy as np
from sklearn.manifold import TSNE
from sklearn.datasets import load_digits

# Load data
digits = load_digits()
X = digits.data[:500]  # Use subset for speed
y = digits.target[:500]

# Try different perplexity values
perplexities = [5, 15, 30, 50]
print("Effect of perplexity on t-SNE:")
print(f"{'Perplexity':<12} {'KL Divergence':<16} {'Spread (std)':<14}")
print("-" * 42)

for perp in perplexities:
    tsne = TSNE(n_components=2, perplexity=perp, random_state=42, n_iter=1000)
    X_2d = tsne.fit_transform(X)
    spread = X_2d.std()
    kl = tsne.kl_divergence_
    print(f"{perp:<12} {kl:<16.4f} {spread:<14.2f}")

print("\nLow perplexity (5): Tight local clusters, may miss global structure.")
print("High perplexity (50): Better global structure, may blur local details.")
print("Typical range: 5-50. Start with 30 and adjust.")

Common Mistakes

from sklearn.decomposition import PCA
import numpy as np

# Features with very different scales
X = np.array([[25000, 25], [30000, 30], [80000, 35], [85000, 40]])
pca = PCA(n_components=1)
X_pca = pca.fit_transform(X)
print(pca.explained_variance_ratio_)  # PC1 captures only income variance

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
import numpy as np

X = np.array([[25000, 25], [30000, 30], [80000, 35], [85000, 40]])
X_scaled = StandardScaler().fit_transform(X)
pca = PCA(n_components=1)
X_pca = pca.fit_transform(X_scaled)
print(pca.explained_variance_ratio_)  # Both features contribute

from sklearn.manifold import TSNE
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.data, digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

# WRONG: t-SNE on training data, then... how to transform test data?
tsne = TSNE(n_components=2)
X_train_2d = tsne.fit_transform(X_train)
# X_test_2d = tsne.transform(X_test)  # ERROR: t-SNE has no transform()!

from sklearn.decomposition import PCA
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_digits

digits = load_digits()
X, y = digits.data, digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

# CORRECT: Use PCA for dimensionality reduction in ML pipelines
pca = PCA(n_components=30)
X_train_pca = pca.fit_transform(X_train)
X_test_pca = pca.transform(X_test)  # PCA has transform()!

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train_pca, y_train)
print(f"Accuracy: {knn.score(X_test_pca, y_test):.4f}")

# After t-SNE, Priya concludes:
# "Cluster A is far from Cluster B, so they are very different."
# "Cluster C is small, so those points are very similar."
# BOTH conclusions are WRONG!

# Correct interpretation of t-SNE:
# - Points within the SAME cluster are likely similar (local structure is preserved)
# - The DISTANCE between clusters is NOT meaningful
# - The SIZE of clusters is NOT meaningful
# - Only the NUMBER of clusters and their MEMBERSHIP are useful

# For meaningful distances, use PCA or UMAP instead
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)  # Distances are meaningful in PCA space

from sklearn.decomposition import PCA
from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

digits = load_digits()
X, y = digits.data, digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

scaler = StandardScaler()
X_train_sc = scaler.fit_transform(X_train)
X_test_sc = scaler.transform(X_test)

pca = PCA(n_components=20)
X_train_pca = pca.fit_transform(X_train_sc)
X_test_pca = pca.transform(X_test_sc)

rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train_pca, y_train)
print(f"RF with PCA: {rf.score(X_test_pca, y_test):.4f}")
# Often WORSE than without PCA for tree-based models!

from sklearn.ensemble import RandomForestClassifier
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

digits = load_digits()
X, y = digits.data, digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)

# Trees handle high dimensions natively - no PCA needed
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
print(f"RF without PCA: {rf.score(X_test, y_test):.4f}")
# Trees don't need scaling or PCA!

Summary