Utku Mert Topçuoğlu, Efficient Pretraining of Vision Transformers: A Layer-Freezing Approach with Local Masked Image Modeling
This thesis explores efficient pretraining methods for Vision Transformers by integrating progressive layer freezing with local masked image modeling. The study assesses the computational demands and extended training periods typical of self-supervised learning methods for ViTs. Key innovations include implementing the FreezeOut method within the LocalMIM architecture to significantly enhance training efficiency. Experimental results show a reduction in training time by about 12.5% while maintaining competitive accuracy, demonstrating the effectiveness of strategic layer freezing combined with tailored learning rate scheduling. This approach promotes more accessible self-supervised learning on constrained computational resources.
Date: 03.09.2024 / 09:30 Place: B-116









