QORAQALPOQ TILI UChUN SUBWORD TOKENIZATSIYA ALGORITMLARINING QIYOSIY TAHLILI

Avtorlar

  • Uteuliev Nietbay Uteulievich

    Nukus State Technical University image/svg+xml

  • Abdalieva Gulistan Rafikovna

    Nukus State Technical University image/svg+xml

  • Asenbaev Nurbek Adilbaevich

    Nukus State Technical University image/svg+xml

  • Orazbaev Jaqsilik Abdimuratovich

    Nukus State Technical University image/svg+xml

  • Asenbaeva Dalloris Adilbay qizi

    Tashkent University of Information Technology image/svg+xml

Gilt sózler: qoraqalpoq tili, tokenizatsiya, subword, BPE, WordPiece, Unigram, Byte-Level BPE

Annotaciya

Maqolada qoraqalpoq tili uchun BPE + Metaspace, WordPiece, Unigram + Metaspace va Byte-Level BPE algoritmlarining 12 konfiguratsiyasi taqqoslandi. Tokenizatorlar 22 913 ta train yozuvida oʻqitilib, 234 validation hujjatida baholandi. 8K lugʻatda Unigram 1,443 token/soʻz, 16K va 24K lugʻatlarda BPE 1,279 va 1,205 natija berdi. Ketma-ketlik uzunligi, lugʻatdan foydalanish va embedding hajmi boʻyicha BPE + Metaspace 16K muvozanatli nomzod sifatida tanlandi.

Paydalanılǵan ádebiyatlar

1. Sennrich R., Haddow B., Birch A. Neural Machine Translation of Rare Words with Subword Units. // Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – Berlin, Germany: Association for Computational Linguistics, 2016. – P. 1715-1725.

2. Kudo T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. // Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – Melbourne, Australia: Association for Computational Linguistics, 2018. – P. 66-75.

3. Kudo T., Richardson, J. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. // Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. – Brussels, Belgium: Association for Computational Linguistics, 2018. – P. 66-71.

4. Devlin J., Chang, M.-W., Lee, K., Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. // Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1. – Minneapolis, Minnesota: Association for Computational Linguistics, 2019. – P. 4171-4186.

5. Radford A., Wu J., Child R., Luan D., Amodei D., Sutskever I. Language Models are Unsupervised Multitask Learners. – OpenAI Technical Report. – OpenAI, 2019.

6. Joshi P., Santy S., Budhiraja A., Bali K., Choudhury M. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. – Association for Computational Linguistics, 2020. – P. 6282-6293.

7. Rust P., Pfeiffer J., Vulić I., Ruder S., Gurevych I. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. // Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). – Association for Computational Linguistics, 2021. – P. 3118-3135.

8. Ali M., Fromm M., Thellmann K., et al. Tokenizer Choice For LLM Training: Negligible or Crucial? // Findings of the Association for Computational Linguistics: NAACL 2024. – Mexico City, Mexico: Association for Computational Linguistics, 2024. – P. 3907-3924.