계층적 중복성과 이상치 토큰을 고려한 비전 트랜스포머 프루닝 연구

초록

The Vision Transformer achieves high accuracy in visual tasks but suffers from high computational cost as depth increases. To improve efficiency, this study explores pruning strategies that consider both hierarchical redundancy and outlier tokens. Inter-layer similarity is quantified using Centered Kernel Alignment, and attention maps are analyzed to track outlier token behavior. Results show that shallow models lack global representation, while deeper ones form outlier tokens that compactly encode global context. The decrease in redundancy in later layers indicates that unnecessary layers can be removed without performance loss. These findings suggest an efficient pruning direction for Vision Transformers.