Transformer-Based Hyperspectral Image Classification with Spectral Embedding and Permutation-Invariant Neighborhood-Set Attention

Abstract

Direct adoption of generic vision-transformer tokenization is often inefficient for hyperspectral image (HSI) classification because the data are high-dimensional, spectrally redundant, spatially structured, and label-scarce. This paper presents a multi-benchmark empirical study built on two ideas: linear spectral embedding of per-pixel descriptors reduced by principal component analysis (PCA), and patch-restricted, order-free self-attention confined to a 15 × 15 neighborhood, realized as a compact one-block, four-head transformer encoder with fewer than ten thousand trainable parameters, implemented end-to-end in MATLAB. No positional or center-token encoding is used; the model aggregates an unordered neighborhood set and therefore cannot identify which token originates at the labeled center pixel. Archived outputs report overall accuracies (OA) of 96.85% on Indian Pines, 98.86% on Salinas, and 99.17% on Pavia University, with kappa values of 0.9642, 0.9873, and 0.9890. Under the archived protocol of Section IV, these values place the model above the support vector machine (SVM) and SpectralFormer baselines on all three scenes and numerically close to the archived HybridSN and SSFTT values on Salinas and Pavia University; model-size ratios were not re-measured, and the exact split identity could not be independently recovered. Because the split policy is random pixel-wise rather than spatially disjoint, the study emphasizes archived empirical behavior under a partially documented protocol rather than strict state-of-the-art claims, and identifies grouped spectral tokenization, explicit local-window attention, and leakage-resistant sampling as the key directions for fair comparison.