COMPARATIVE ANALYSIS OF SUBWORD TOKENIZATION ALGORITHMS FOR THE KARAKALPAK LANGUAGE
Authors
-
Nietbay Uteulievich Uteuliev
-
Gulistan Rafikovna Abdalieva
-
Nurbek Adilbaevich Asenbaev
-
Jaqsilik Abdimuratovich Orazbaev
-
Dalloris Adilbayena Asenbaeva
Keywords: Karakalpak language, tokenization, subword, BPE, WordPiece, Unigram, Byte-Level BPE
Abstract
References
1. Sennrich R., Haddow B., Birch A. Neural Machine Translation of Rare Words with Subword Units. // Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – Berlin, Germany: Association for Computational Linguistics, 2016. – P. 1715-1725.
2. Kudo T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. // Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – Melbourne, Australia: Association for Computational Linguistics, 2018. – P. 66-75.
3. Kudo T., Richardson, J. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. // Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. – Brussels, Belgium: Association for Computational Linguistics, 2018. – P. 66-71.
4. Devlin J., Chang, M.-W., Lee, K., Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. // Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1. – Minneapolis, Minnesota: Association for Computational Linguistics, 2019. – P. 4171-4186.
5. Radford A., Wu J., Child R., Luan D., Amodei D., Sutskever I. Language Models are Unsupervised Multitask Learners. – OpenAI Technical Report. – OpenAI, 2019.
6. Joshi P., Santy S., Budhiraja A., Bali K., Choudhury M. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. – Association for Computational Linguistics, 2020. – P. 6282-6293.
7. Rust P., Pfeiffer J., Vulić I., Ruder S., Gurevych I. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. // Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). – Association for Computational Linguistics, 2021. – P. 3118-3135.
8. Ali M., Fromm M., Thellmann K., et al. Tokenizer Choice For LLM Training: Negligible or Crucial? // Findings of the Association for Computational Linguistics: NAACL 2024. – Mexico City, Mexico: Association for Computational Linguistics, 2024. – P. 3907-3924.