Towards Generalization of Block Attention via Automatic Segmentation and Block Distillation
This work constructs SemanticSeg, a large and diverse semantic segmentation dataset, and proposes block distillation, a training framework that is more efficient than block fine-tuning, which uses a frozen full-attention teacher model to guide the block-attention student.