Transformers Are So Dusty