Phillip Pramberger

A Pipelined FPGA Architecture for Banded Sparse Matrix Dense Matrix Multiplication in Longformer

Sparse attention mechanisms have become increasingly important for transformer models processing long input sequences due to their lower computational and memory complexity compared to full self-attention. Longformer achieves this through a sliding-window attention mechanism that produces a structured banded sparse att...