results highlight the importance of previously overlooked design choices, and raise questions about the sourceNevertheless, in the vocabulary size growth in RoBERTa allows to encode almost any word or subword without using the unknown token, compared to BERT. This gives a considerable advant