OʻZBEK TILIDA TRANSFORMER–BILSTM ASOSIDA PUNKTUATSIYANI TIKLASh UChUN PUNKTUATSION TEGLANGAN KORPUSNI AVTOMATIK ShAKLLANTIRISh

Mualliflar

  • Sharipov Maqsud Siddiqovich

    Tashkent University of Information Technology image/svg+xml

  • Adinayev Xushnudbek Saylboyevich

    Urgench State University image/svg+xml

Kalit soʻzlar: punktuatsiyani tiklash, oʻzbek tili, Transformer, BiLSTM, tabiiy tilni qayta ishlash

Annotatsiya

Ushbu tadqiqotda oʻzbek tili uchun Transformer–BiLSTM modellarini oʻqitishga moʻljallangan punktuatsion teglangan maʼlumotlar toʻplamini avtomatik shakllantirish algoritmi taklif etiladi. Jarayon matnni normallashtirish, punktuatsiyaga sezgir tokenizatsiya, token darajasida teglash, maʼlumotlarni ajratish va CoNLL formatiga oʻtkazishni oʻz ichiga oladi. Jami 11 ta teg qoʻllanilib, 11 888 213 ta tokenli korpus shakllantirilgan. Taklif etilgan usul punktuatsiyani tiklash uchun katta hajmdagi oʻzbekcha teglangan maʼlumotlarni tayyorlashga xizmat qiladi.

Foydalanilgan adabiyotlar

1. Tilk O., Alumäe T. LSTM for punctuation restoration in speech transcripts. Proceedings of Interspeech 2015.

2. Tilk O., Alumäe T. Bidirectional recurrent neural network with attention mechanism for punctuation restoration. // Proceedings of Interspeech 2016. – P. 3047-3051.

3. Courtland M., Faulkner, A., McElvain, G. Efficient automatic punctuation restoration using bidirectional Transformers with robust inference. 2020, Proceedings of the 17th International Conference on Spoken Language Translation. – P. 272-279.

4. Alam T., Khan A., Alam, F. Punctuation restoration using Transformer models for high – and low-resource languages. Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020).

5. Nagy A., Bial B., Ács J. Automatic punctuation restoration with BERT models. arxiv preprint arxiv: 2101.07343.

6. Sharipov M.S., Adinaev H.S., Kuriyozov E.R. Rule-Based Punctuation Algorithm for the Uzbek Language. 2024 IEEE 25th International Conference of Young Professionals in Electron Devices and Materials (EDM), 2024. – P. 2410-2414.

7. Sharipov M.S., Adinaev H.S. Development of Models for Punctuation Analysis of Uzbek Language Texts. // Bulletin of TUIT: Management and Communication Technologies, 2025, 1(4).

8. Adinaev H.S. Punctuation Analysis of Uzbek Texts Based on the N-Gram Model. // Electronic Journal of Actual Problems of Modern Science, Education and Training, 2025, № 2. – Р. 104-10.

9. Sharipov M.S., Adinayev H.S., Yusupova M.M. Oʻzbek tili matnlarida punktuatsion tahlil qilish uchun korpus yaratish. // Development of Science, 2025, Vol. 2, №3, 128-133-b.

10. Sharipov M., Adinayev H. Oʻzbek tili matnlarida soʻroq gaplarni aniqlashning qoidaga asoslangan algoritmini ishlab chiqish. // Management and Future Technologies, 2025, Vol. 2, Issue 1, 182-89-b.

11. Sharipov M.S., Adinayev H.S. Shartli tasodifiy maydonlar modeli asosida oʻzbek tili matnlarini punktuatsion tahlil qilish. // Al-Fargʻoniy avlodlari, 2025, Vol. 1, Issue 2.

Sharipov M.S., Adinayev H.S. Parallel hisoblash va Apache Spark asosida katta hajmdagi matnlarni punktuatsion tahlil qilish. // Oʻzbekiston milliy axborot agentligi – Oʻza ilm-fan boʻlimi (elektron jurnal), Vol. 11(73), November 2025, 243-247-b.