If you choose this second option, there are three possibilities you can use to gather all the input Tensors The original BERT uses a subword-level tokenization with the vocabulary size of 30K which is learned after input preprocessing and using several heuristics. RoBERTa uses byte