Preparing Practice Exam…
Please wait a moment while the system generates the exam questions, answers, and study session.
Please wait a moment while the system generates the exam questions, answers, and study session.
Question 18 of 35
Free Preview · 35 of 116 · questions 1–35 of 116 are free. Sign up free to sync your progress across devices.
Select one answer.
Written by other learners. Comments can disagree with the marked answer — check the official documentation when in doubt.
- Tokenization splits raw text into tokens — sub-word units that the model processes. Common algorithms include Byte-Pair Encoding (BPE, used by GPT), WordPiece (BERT), and SentencePiece (T5, Llama). Tokenization affects context window usage, latency, and even multilingual fairness because some languages tokenize less efficiently than others.