СРАВНИТЕЛЬНЫЙ АНАЛИЗ АЛГОРИТМОВ СУБСЛОВНОЙ ТОКЕНИЗАЦИИ ДЛЯ КАРАКАЛПАКСКОГО ЯЗЫКА

Авторы

  • Утеулиев Ниетбай Утеулиевич

    Nukus State Technical University image/svg+xml

  • Абдалиева Гулистан Рафиковна

    Nukus State Technical University image/svg+xml

  • Асенбаев Нурбек Адилбаевич

    Nukus State Technical University image/svg+xml

  • Оразбаев Жаксылык Абдимуратович

    Nukus State Technical University image/svg+xml

  • Асенбаева Даллорис Адилбаевна

    Ташкентский Университет Информационных Технологий image/svg+xml

Ключевые слова: каракалпакский язык, токенизация, субсловные единицы, BPE, WordPiece, Unigram, Byte-Level BPE

Аннотация

Сравнены 12 конфигураций BPE + Metaspace, WordPiece, Unigram + Metaspace и Byte-Level BPE для каракалпакского языка. Токенизаторы обучались на 22 913 записях и оценивались на 234 документах. Unigram показал 1,443 токена на слово при 8K, а BPE — 1,279 и 1,205 при 16K и 24K. С учётом длины последовательности, использования словаря и размера embedding-слоя BPE + Metaspace 16K выбран как сбалансированный кандидат.

Библиографические ссылки

1. Sennrich R., Haddow B., Birch A. Neural Machine Translation of Rare Words with Subword Units. // Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – Berlin, Germany: Association for Computational Linguistics, 2016. – P. 1715-1725.

2. Kudo T. Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. // Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). – Melbourne, Australia: Association for Computational Linguistics, 2018. – P. 66-75.

3. Kudo T., Richardson, J. SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. // Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. – Brussels, Belgium: Association for Computational Linguistics, 2018. – P. 66-71.

4. Devlin J., Chang, M.-W., Lee, K., Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. // Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1. – Minneapolis, Minnesota: Association for Computational Linguistics, 2019. – P. 4171-4186.

5. Radford A., Wu J., Child R., Luan D., Amodei D., Sutskever I. Language Models are Unsupervised Multitask Learners. – OpenAI Technical Report. – OpenAI, 2019.

6. Joshi P., Santy S., Budhiraja A., Bali K., Choudhury M. The State and Fate of Linguistic Diversity and Inclusion in the NLP World. // Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. – Association for Computational Linguistics, 2020. – P. 6282-6293.

7. Rust P., Pfeiffer J., Vulić I., Ruder S., Gurevych I. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. // Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). – Association for Computational Linguistics, 2021. – P. 3118-3135.

8. Ali M., Fromm M., Thellmann K., et al. Tokenizer Choice For LLM Training: Negligible or Crucial? // Findings of the Association for Computational Linguistics: NAACL 2024. – Mexico City, Mexico: Association for Computational Linguistics, 2024. – P. 3907-3924.