Topic in Deep Learning: Transformers (Winter 048010)

The Transformer is a type of network architecture that is highly efficient in representing time series. It is widely used in large language models like BERT and ChatGPT, and in state-of-the-art automatic speech recognition models like wav2vec 2.0, HuBERT, and Whisper. The Transformer has also proven to be effective in the field of vision.

This course aims to provide a detailed understanding of the fundamental concepts behind the development of Transformers and the Transformer model. We'll explore the most significant research conducted in this field, both theoretically and practically. Moreover, we'll explore how Transformers are used in natural language processing (NLP), speech processing, and computer vision. In the latter part of the course, students will also have the opportunity to present significant papers in this field.