COMPARATIVE ANALYSIS OF SUBWORD TOKENIZATION ALGORITHMS FOR THE KARAKALPAK LANGUAGE

Authors

  • Nietbay Uteulievich Uteuliev

    Nukus State Technical University image/svg+xml

  • Gulistan Rafikovna Abdalieva

    Nukus State Technical University image/svg+xml

  • Nurbek Adilbaevich Asenbaev

    Nukus State Technical University image/svg+xml

  • Jaqsilik Abdimuratovich Orazbaev

    Nukus State Technical University image/svg+xml

  • Dalloris Adilbayena Asenbaeva

    Tashkent University of Information Technology image/svg+xml

Keywords: Karakalpak language, tokenization, subword, BPE, WordPiece, Unigram, Byte-Level BPE

Abstract

Twelve BPE, WordPiece, Unigram, and Byte-Level BPE configurations were compared for Karakalpak. The tokenizers were trained on 22,913 records and evaluated on 234 documents. Unigram achieved 1.443 tokens per word at 8K; BPE achieved 1.279 and 1.205 at 16K and 24K. Considering sequence length, vocabulary utilization, and embedding size, BPE + Metaspace 16K was selected as the balanced candidate.

References

1. Sennrich R., Haddow B., Birch A. Neural Machine Translation of Rare Words with Subword Units. // Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – Berlin, Germany: Association for Computational Linguistics, 2016. – P. 1715-1725.

2. Kudo T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. // Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – Melbourne, Australia: Association for Computational Linguistics, 2018. – P. 66-75.

3. Kudo T., Richardson, J. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. // Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. – Brussels, Belgium: Association for Computational Linguistics, 2018. – P. 66-71.

4. Devlin J., Chang, M.-W., Lee, K., Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. // Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1. – Minneapolis, Minnesota: Association for Computational Linguistics, 2019. – P. 4171-4186.

5. Radford A., Wu J., Child R., Luan D., Amodei D., Sutskever I. Language Models are Unsupervised Multitask Learners. – OpenAI Technical Report. – OpenAI, 2019.

6. Joshi P., Santy S., Budhiraja A., Bali K., Choudhury M. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. – Association for Computational Linguistics, 2020. – P. 6282-6293.

7. Rust P., Pfeiffer J., Vulić I., Ruder S., Gurevych I. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. // Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). – Association for Computational Linguistics, 2021. – P. 3118-3135.

8. Ali M., Fromm M., Thellmann K., et al. Tokenizer Choice For LLM Training: Negligible or Crucial? // Findings of the Association for Computational Linguistics: NAACL 2024. – Mexico City, Mexico: Association for Computational Linguistics, 2024. – P. 3907-3924.