XzhLoongSkills/skills/data-science/scikit-learn/references/unsupervised_learning.md

Unsupervised Learning Reference

Overview

Unsupervised learning discovers patterns in unlabeled data through clustering, dimensionality reduction, and density estimation.

Clustering

K-Means

KMeans (sklearn.cluster.KMeans)

• Partition-based clustering into K clusters
• Key parameters:
  • n_clusters: Number of clusters to form
  • init: Initialization method ('k-means++', 'random')
  • n_init: Number of initializations (default=10)
  • max_iter: Maximum iterations
• Use when: Know number of clusters, spherical cluster shapes
• Fast and scalable

• Example:

  from sklearn.cluster import KMeans
  
  model = KMeans(n_clusters=3, init='k-means++', n_init=10, random_state=42)
  labels = model.fit_predict(X)
  centers = model.cluster_centers_
  
  # Inertia (sum of squared distances to nearest center)
  print(f"Inertia: {model.inertia_}")
  

MiniBatchKMeans

• Faster K-Means using mini-batches
• Use when: Large datasets, need faster training
• Slightly less accurate than K-Means

• Example:

  from sklearn.cluster import MiniBatchKMeans
  
  model = MiniBatchKMeans(n_clusters=3, batch_size=100, random_state=42)
  labels = model.fit_predict(X)
  

Density-Based Clustering

DBSCAN (sklearn.cluster.DBSCAN)

• Density-Based Spatial Clustering
• Key parameters:
  • eps: Maximum distance between two samples to be neighbors
  • min_samples: Minimum samples in neighborhood to form core point
  • metric: Distance metric
• Use when: Arbitrary cluster shapes, presence of noise/outliers
• Automatically determines number of clusters
• Labels noise points as -1

• Example:

  from sklearn.cluster import DBSCAN
  
  model = DBSCAN(eps=0.5, min_samples=5, metric='euclidean')
  labels = model.fit_predict(X)
  
  # Number of clusters (excluding noise)
  n_clusters = len(set(labels)) - (1 if -1 in labels else 0)
  n_noise = list(labels).count(-1)
  print(f"Clusters: {n_clusters}, Noise points: {n_noise}")
  

HDBSCAN (sklearn.cluster.HDBSCAN)

• Hierarchical DBSCAN with adaptive epsilon
• More robust than DBSCAN
• Key parameter: min_cluster_size
• Use when: Varying density clusters

• Example:

  from sklearn.cluster import HDBSCAN
  
  model = HDBSCAN(min_cluster_size=10, min_samples=5)
  labels = model.fit_predict(X)
  

OPTICS (sklearn.cluster.OPTICS)

• Ordering points to identify clustering structure
• Similar to DBSCAN but doesn't require eps parameter
• Key parameters: min_samples, max_eps
• Use when: Varying density, exploratory analysis

• Example:

  from sklearn.cluster import OPTICS
  
  model = OPTICS(min_samples=5, max_eps=0.5)
  labels = model.fit_predict(X)
  

Hierarchical Clustering

AgglomerativeClustering

• Bottom-up hierarchical clustering
• Key parameters:
  • n_clusters: Number of clusters (or use distance_threshold)
  • linkage: 'ward', 'complete', 'average', 'single'
  • metric: Distance metric
• Use when: Need dendrogram, hierarchical structure important

• Example:

  from sklearn.cluster import AgglomerativeClustering
  
  model = AgglomerativeClustering(n_clusters=3, linkage='ward')
  labels = model.fit_predict(X)
  
  # Create dendrogram using scipy
  from scipy.cluster.hierarchy import dendrogram, linkage
  Z = linkage(X, method='ward')
  dendrogram(Z)
  

Other Clustering Methods

MeanShift

• Finds clusters by shifting points toward mode of density
• Automatically determines number of clusters
• Key parameter: bandwidth
• Use when: Don't know number of clusters, arbitrary shapes

• Example:

  from sklearn.cluster import MeanShift, estimate_bandwidth
  
  # Estimate bandwidth
  bandwidth = estimate_bandwidth(X, quantile=0.2, n_samples=500)
  model = MeanShift(bandwidth=bandwidth)
  labels = model.fit_predict(X)
  

SpectralClustering

• Uses graph-based approach with eigenvalues
• Key parameters: n_clusters, affinity ('rbf', 'nearest_neighbors')
• Use when: Non-convex clusters, graph structure

• Example:

  from sklearn.cluster import SpectralClustering
  
  model = SpectralClustering(n_clusters=3, affinity='rbf', random_state=42)
  labels = model.fit_predict(X)
  

AffinityPropagation

• Finds exemplars by message passing
• Automatically determines number of clusters
• Key parameters: damping, preference
• Use when: Don't know number of clusters

• Example:

  from sklearn.cluster import AffinityPropagation
  
  model = AffinityPropagation(damping=0.9, random_state=42)
  labels = model.fit_predict(X)
  n_clusters = len(model.cluster_centers_indices_)
  

BIRCH

• Balanced Iterative Reducing and Clustering using Hierarchies
• Memory efficient for large datasets
• Key parameters: n_clusters, threshold, branching_factor
• Use when: Very large datasets

• Example:

  from sklearn.cluster import Birch
  
  model = Birch(n_clusters=3, threshold=0.5)
  labels = model.fit_predict(X)
  

Clustering Evaluation

Metrics when ground truth is known:

from sklearn.metrics import adjusted_rand_score, normalized_mutual_info_score
from sklearn.metrics import adjusted_mutual_info_score, fowlkes_mallows_score

# Compare predicted labels with true labels
ari = adjusted_rand_score(y_true, y_pred)
nmi = normalized_mutual_info_score(y_true, y_pred)
ami = adjusted_mutual_info_score(y_true, y_pred)
fmi = fowlkes_mallows_score(y_true, y_pred)

Metrics without ground truth:

from sklearn.metrics import silhouette_score, calinski_harabasz_score
from sklearn.metrics import davies_bouldin_score

# Silhouette: [-1, 1], higher is better
silhouette = silhouette_score(X, labels)

# Calinski-Harabasz: higher is better
ch_score = calinski_harabasz_score(X, labels)

# Davies-Bouldin: lower is better
db_score = davies_bouldin_score(X, labels)

Elbow method for K-Means:

from sklearn.cluster import KMeans
import matplotlib.pyplot as plt

inertias = []
K_range = range(2, 11)
for k in K_range:
    model = KMeans(n_clusters=k, random_state=42)
    model.fit(X)
    inertias.append(model.inertia_)

plt.plot(K_range, inertias, 'bo-')
plt.xlabel('Number of clusters')
plt.ylabel('Inertia')
plt.title('Elbow Method')

Dimensionality Reduction

Principal Component Analysis (PCA)

PCA (sklearn.decomposition.PCA)

• Linear dimensionality reduction using SVD
• Key parameters:
  • n_components: Number of components (int or float for explained variance)
  • whiten: Whiten components to unit variance
• Use when: Linear relationships, want to explain variance

• Example:

  from sklearn.decomposition import PCA
  
  # Keep components explaining 95% variance
  pca = PCA(n_components=0.95)
  X_reduced = pca.fit_transform(X)
  
  print(f"Original dimensions: {X.shape[1]}")
  print(f"Reduced dimensions: {X_reduced.shape[1]}")
  print(f"Explained variance ratio: {pca.explained_variance_ratio_}")
  print(f"Total variance explained: {pca.explained_variance_ratio_.sum()}")
  
  # Or specify exact number of components
  pca = PCA(n_components=2)
  X_2d = pca.fit_transform(X)
  

IncrementalPCA

• PCA for large datasets that don't fit in memory
• Processes data in batches
• Key parameter: n_components, batch_size

• Example:

  from sklearn.decomposition import IncrementalPCA
  
  pca = IncrementalPCA(n_components=50, batch_size=100)
  X_reduced = pca.fit_transform(X)
  

KernelPCA

• Non-linear dimensionality reduction using kernels
• Key parameters: n_components, kernel ('linear', 'poly', 'rbf', 'sigmoid')
• Use when: Non-linear relationships

• Example:

  from sklearn.decomposition import KernelPCA
  
  pca = KernelPCA(n_components=2, kernel='rbf', gamma=0.1)
  X_reduced = pca.fit_transform(X)
  

Manifold Learning

t-SNE (sklearn.manifold.TSNE)

• t-distributed Stochastic Neighbor Embedding
• Excellent for 2D/3D visualization
• Key parameters:
  • n_components: Usually 2 or 3
  • perplexity: Balance between local and global structure (5-50)
  • learning_rate: Usually 10-1000
  • n_iter: Number of iterations (min 250)
• Use when: Visualizing high-dimensional data
• Note: Slow on large datasets, no transform() method

• Example:

  from sklearn.manifold import TSNE
  
  tsne = TSNE(n_components=2, perplexity=30, learning_rate=200, n_iter=1000, random_state=42)
  X_embedded = tsne.fit_transform(X)
  
  # Visualize
  import matplotlib.pyplot as plt
  plt.scatter(X_embedded[:, 0], X_embedded[:, 1], c=labels, cmap='viridis')
  plt.title('t-SNE visualization')
  

UMAP (not in scikit-learn, but compatible)

• Uniform Manifold Approximation and Projection
• Faster than t-SNE, preserves global structure better
• Install: uv pip install umap-learn

• Example:

  from umap import UMAP
  
  reducer = UMAP(n_components=2, n_neighbors=15, min_dist=0.1, random_state=42)
  X_embedded = reducer.fit_transform(X)
  

Isomap

• Isometric Mapping
• Preserves geodesic distances
• Key parameters: n_components, n_neighbors
• Use when: Non-linear manifolds

• Example:

  from sklearn.manifold import Isomap
  
  isomap = Isomap(n_components=2, n_neighbors=5)
  X_embedded = isomap.fit_transform(X)
  

Locally Linear Embedding (LLE)

• Preserves local neighborhood structure
• Key parameters: n_components, n_neighbors

• Example:

  from sklearn.manifold import LocallyLinearEmbedding
  
  lle = LocallyLinearEmbedding(n_components=2, n_neighbors=10)
  X_embedded = lle.fit_transform(X)
  

MDS (Multidimensional Scaling)

• Preserves pairwise distances
• Key parameter: n_components, metric (True/False)

• Example:

  from sklearn.manifold import MDS
  
  mds = MDS(n_components=2, metric=True, random_state=42)
  X_embedded = mds.fit_transform(X)
  

Matrix Factorization

NMF (Non-negative Matrix Factorization)

• Factorizes into non-negative matrices
• Key parameters: n_components, init ('nndsvd', 'random')
• Use when: Data is non-negative (images, text)
• Interpretable components

• Example:

  from sklearn.decomposition import NMF
  
  nmf = NMF(n_components=10, init='nndsvd', random_state=42)
  W = nmf.fit_transform(X)  # Document-topic matrix
  H = nmf.components_  # Topic-word matrix
  

TruncatedSVD

• SVD for sparse matrices
• Similar to PCA but works with sparse data
• Use when: Text data, sparse matrices

• Example:

  from sklearn.decomposition import TruncatedSVD
  
  svd = TruncatedSVD(n_components=100, random_state=42)
  X_reduced = svd.fit_transform(X_sparse)
  print(f"Explained variance: {svd.explained_variance_ratio_.sum()}")
  

FastICA

• Independent Component Analysis
• Separates multivariate signal into independent components
• Key parameter: n_components
• Use when: Signal separation (e.g., audio, EEG)

• Example:

  from sklearn.decomposition import FastICA
  
  ica = FastICA(n_components=10, random_state=42)
  S = ica.fit_transform(X)  # Independent sources
  A = ica.mixing_  # Mixing matrix
  

LatentDirichletAllocation (LDA)

• Topic modeling for text data
• Key parameters: n_components (number of topics), learning_method ('batch', 'online')
• Use when: Topic modeling, document clustering

• Example:

  from sklearn.decomposition import LatentDirichletAllocation
  
  lda = LatentDirichletAllocation(n_components=10, random_state=42)
  doc_topics = lda.fit_transform(X_counts)  # Document-topic distribution
  
  # Get top words for each topic
  feature_names = vectorizer.get_feature_names_out()
  for topic_idx, topic in enumerate(lda.components_):
  top_words = [feature_names[i] for i in topic.argsort()[-10:]]
  print(f"Topic {topic_idx}: {', '.join(top_words)}")
  

Outlier and Novelty Detection

Outlier Detection

IsolationForest

• Isolates anomalies using random trees
• Key parameters:
  • contamination: Expected proportion of outliers
  • n_estimators: Number of trees
• Use when: High-dimensional data, efficiency important

• Example:

  from sklearn.ensemble import IsolationForest
  
  model = IsolationForest(contamination=0.1, random_state=42)
  predictions = model.fit_predict(X)  # -1 for outliers, 1 for inliers
  

LocalOutlierFactor

• Measures local density deviation
• Key parameters: n_neighbors, contamination
• Use when: Varying density regions

• Example:

  from sklearn.neighbors import LocalOutlierFactor
  
  lof = LocalOutlierFactor(n_neighbors=20, contamination=0.1)
  predictions = lof.fit_predict(X)  # -1 for outliers, 1 for inliers
  outlier_scores = lof.negative_outlier_factor_
  

One-Class SVM

• Learns decision boundary around normal data
• Key parameters: nu (upper bound on outliers), kernel, gamma
• Use when: Small training set of normal data

• Example:

  from sklearn.svm import OneClassSVM
  
  model = OneClassSVM(nu=0.1, kernel='rbf', gamma='auto')
  model.fit(X_train)
  predictions = model.predict(X_test)  # -1 for outliers, 1 for inliers
  

EllipticEnvelope

• Assumes Gaussian distribution
• Key parameter: contamination
• Use when: Data is Gaussian-distributed

• Example:

  from sklearn.covariance import EllipticEnvelope
  
  model = EllipticEnvelope(contamination=0.1, random_state=42)
  predictions = model.fit_predict(X)
  

Gaussian Mixture Models

GaussianMixture

• Probabilistic clustering with mixture of Gaussians
• Key parameters:
  • n_components: Number of mixture components
  • covariance_type: 'full', 'tied', 'diag', 'spherical'
• Use when: Soft clustering, need probability estimates

• Example:

  from sklearn.mixture import GaussianMixture
  
  gmm = GaussianMixture(n_components=3, covariance_type='full', random_state=42)
  gmm.fit(X)
  
  # Predict cluster labels
  labels = gmm.predict(X)
  
  # Get probability of each cluster
  probabilities = gmm.predict_proba(X)
  
  # Information criteria for model selection
  print(f"BIC: {gmm.bic(X)}")  # Lower is better
  print(f"AIC: {gmm.aic(X)}")  # Lower is better
  

Choosing the Right Method

Clustering:

• Know K, spherical clusters: K-Means
• Arbitrary shapes, noise: DBSCAN, HDBSCAN
• Hierarchical structure: AgglomerativeClustering
• Very large data: MiniBatchKMeans, BIRCH
• Probabilistic: GaussianMixture

Dimensionality Reduction:

• Linear, variance explanation: PCA
• Non-linear, visualization: t-SNE, UMAP
• Non-negative data: NMF
• Sparse data: TruncatedSVD
• Topic modeling: LatentDirichletAllocation

Outlier Detection:

• High-dimensional: IsolationForest
• Varying density: LocalOutlierFactor
• Gaussian data: EllipticEnvelope