Kernel Density Estimator / Classifier

A small Python implementation of a non-parametric classifier based on Gaussian kernel density estimation (KDE).

The classifier estimates a density for each class from the training samples and predicts the class with the highest estimated density at the query point. The implementation supports classification in N dimensions; decision-boundary plotting is limited to two input features.

Features

• Gaussian kernel density classification implemented directly in Python/NumPy
• N-dimensional classification
• Separate training and testing datasets loaded from CSV files
• Leave-one-out training-accuracy calculation
• Held-out testing-accuracy calculation
• 2-D decision-boundary visualization for two-class data
• User-controlled kernel bandwidth

How classification works

For each class, the classifier evaluates an isotropic Gaussian kernel centered on every training sample in that class and averages those values to estimate the class-conditional density at the query point.

The predicted class is the class with the largest estimated density.

The implementation does not separately learn or apply class priors, and it does not automatically choose the kernel bandwidth.

Data format

Input data is read from numeric CSV files. The final column is treated as the class label; all preceding columns are treated as features.

For example, a two-feature dataset has rows of the form:

feature_1,feature_2,class

The bundled example also uses a higher-dimensional dataset; classification is not restricted to two dimensions.

Installation

The code depends on NumPy and Matplotlib:

python -m pip install numpy matplotlib

Example

from kerneldensityestimator import KDE

kde = KDE()
kde.bandwidth = 0.5

kde.training_model.load("train.csv")
print(f"Training Accuracy: {kde.get_training_accuracy()}%")

kde.testing_model.load("test.csv")
print(f"Testing Accuracy: {kde.get_testing_accuracy()}%")

kde.plot_class_boundaries(kde.testing_model)

See example.py for the complete example, including use with the bundled higher-dimensional dataset.

Evaluation

get_training_accuracy() performs leave-one-out evaluation: each training point is removed from the active training data before that point is classified.

get_testing_accuracy() classifies each point in the testing dataset using the full training dataset.

Decision-boundary plotting

plot_class_boundaries() creates a grid across a two-feature dataset, classifies each grid point, and plots the resulting class regions.

The current plotting function is intended for two-feature, two-class data using class labels 0 and 1. The classifier itself is not limited to two features.

Limitations

• Kernel bandwidth is supplied by the user; there is no automatic bandwidth selection.
• Features are not normalized or standardized automatically.
• Class priors are not modeled separately from the per-class density estimates.
• Classification evaluates the query point against the training samples directly, so runtime grows with dataset size.
• Decision-boundary visualization is limited to two input features and the current plot implementation expects classes 0 and 1.

Files

• kerneldensityestimator.py — classifier, data loading, evaluation, and plotting
• example.py — example usage
• train.csv / test.csv — bundled example data
• zipcode_train.csv / zipcode_test.csv — bundled higher-dimensional example data

License

MIT. See LICENSE.