{"entry_id": "71b461dc-650f-4544-a42f-39ab386a06ac", "infos": [{"txt": "UNIBERT:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 0, "bbox": [[0.249, 0.124, 0.379, 0.14]]}]}, "tags": ["title"], "label": "article_title", "web_segment_id": 1, "global_sentence_id": 0, "edu_l1_label": "BOT"}, {"txt": " ADVERSARIAL TRAINING FOR", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 0, "bbox": [[0.379, 0.124, 0.748, 0.14]]}]}, "tags": ["title"], "label": "article_title", "web_segment_id": 1, "global_sentence_id": 1, "edu_l1_label": "BOT"}, {"txt": "LANGUAGE-UNIVERSAL REPRESENTATIONS", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 0, "bbox": [[0.234, 0.149, 0.761, 0.165]]}]}, "tags": ["title"], "label": "article_title", "web_segment_id": 2, "global_sentence_id": 2, "edu_l1_label": "BOT"}, {"txt": "Andrei-Marius Avram1,3, Marian Lupascu2,3, Dumitru-Clementin Cercel¹,", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 100000, "bbox": [[0.24, 0.234, 0.759, 0.243]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 3, "global_sentence_id": 3, "edu_l1_label": "EDU_O"}, {"txt": "Ionuț Mironicǎ3, Stefan Trǎusan-Matu¹,4", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 200000, "bbox": [[0.357, 0.249, 0.642, 0.253]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 4, "global_sentence_id": 4, "edu_l1_label": "EDU_O"}, {"txt": "National University of Science and Technology POLITEHNICA Bucharest, Bucharest, Romania", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 300000, "bbox": [[0.187, 0.262, 0.818, 0.272]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 5, "global_sentence_id": 5, "edu_l1_label": "EDU_O"}, {"txt": "2University of Bucharest, Bucharest, Romania", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 400000, "bbox": [[0.349, 0.277, 0.65, 0.286]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 6, "global_sentence_id": 6, "edu_l1_label": "EDU_O"}, {"txt": "3Adobe Research, Bucharest, Romania", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 500000, "bbox": [[0.373, 0.292, 0.627, 0.301]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 7, "global_sentence_id": 7, "edu_l1_label": "EDU_O"}, {"txt": "4Academy of Romanian Scientists (AOSR), Bucharest, Romania", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 600000, "bbox": [[0.288, 0.305, 0.709, 0.316]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 8, "global_sentence_id": 8, "edu_l1_label": "EDU_O"}, {"txt": "ABSTRACT", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 700000, "bbox": [[0.452, 0.342, 0.547, 0.352]]}]}, "tags": ["title"], "label": "abstract", "web_segment_id": 9, "global_sentence_id": 9, "edu_l1_label": "BOS"}, {"txt": "This paper presents UniBERT, a compact multilingual language model that uses an innovative train-ing framework that integrates three components:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 700000, "bbox": [[0.175, 0.369, 0.826, 0.374], [0.176, 0.383, 0.499, 0.39]]}]}, "tags": ["text"], "label": "abstract", "web_segment_id": 10, "global_sentence_id": 10, "edu_l1_label": "IOS"}, {"txt": " masked language modeling, adversarial training,and knowledge distillation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 700000, "bbox": [[0.499, 0.383, 0.828, 0.391], [0.176, 0.398, 0.358, 0.404]]}]}, "tags": ["text"], "label": "abstract", "web_segment_id": 10, "global_sentence_id": 11, "edu_l1_label": "IOS"}, {"txt": "Pre-trained on a meticulously curated Wikipedia corpus spanning 107languages, UniBERT is designed to reduce the computational demands of large-scale models while maintaining competitive performance across various natural language processing tasks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 700000, "bbox": [[0.358, 0.398, 0.824, 0.404], [0.176, 0.41, 0.823, 0.418], [0.181, 0.426, 0.763, 0.432]]}]}, "tags": ["text"], "label": "abstract", "web_segment_id": 10, "global_sentence_id": 12, "edu_l1_label": "IOS"}, {"txt": "Compre-hensive evaluations on four tasks-named entity recognition, natural language inference, question answering,and semantic textual similarity-demonstrate that our multilingual training strategy en-hanced by an adversarial objective significantly improves cross-lingual generalization.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 700000, "bbox": [[0.763, 0.426, 0.826, 0.43], [0.176, 0.438, 0.819, 0.446], [0.175, 0.453, 0.825, 0.459], [0.175, 0.465, 0.743, 0.473]]}]}, "tags": ["text"], "label": "abstract", "web_segment_id": 10, "global_sentence_id": 13, "edu_l1_label": "IOS"}, {"txt": "Specifically,UniBERT models show an average relative improvement of 7.72% over traditional baselines, which achieved an average relative improvement of only 1.17%, and statistical analysis confirms the signifi-cance of these gains (p-value = 0.0181).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 700000, "bbox": [[0.743, 0.465, 0.827, 0.475], [0.176, 0.478, 0.823, 0.487], [0.176, 0.494, 0.827, 0.499], [0.176, 0.509, 0.439, 0.515]]}]}, "tags": ["text"], "label": "abstract", "web_segment_id": 10, "global_sentence_id": 14, "edu_l1_label": "IOS"}, {"txt": "This work highlights the benefits of combining adversarial training and knowledge distillation to build scalable and robust language models, thus advancing the field of multilingual and cross-lingual natural language processing.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 700000, "bbox": [[0.439, 0.509, 0.823, 0.515], [0.176, 0.522, 0.823, 0.529], [0.176, 0.534, 0.614, 0.542]]}]}, "tags": ["text"], "label": "abstract", "web_segment_id": 10, "global_sentence_id": 15, "edu_l1_label": "IOS"}, {"txt": "S707ndy8[TO.so] C4809Z1:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 800000, "bbox": [[0.029, 0.275, 0.052, 0.598]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 11, "global_sentence_id": 16, "edu_l1_label": "EDU_O"}, {"txt": "SOgZAIXTe", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 800000, "bbox": [[0.052, 0.275, 0.054, 0.7]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 11, "global_sentence_id": 17, "edu_l1_label": "EDU_O"}, {"txt": "1 Introduction", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 900000, "bbox": [[0.117, 0.566, 0.252, 0.578]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 12, "global_sentence_id": 18, "edu_l1_label": "BOS"}, {"txt": "Natural language processing (NLP) has significantly advanced in recent years, primarily due to breakthroughs in deep learning and Transformer-based architectures [1].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1000000, "bbox": [[0.117, 0.597, 0.881, 0.606], [0.117, 0.612, 0.479, 0.621]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 19, "edu_l1_label": "IOS"}, {"txt": "Models such as Bidirectional Encoder Representations from Transformers (BERT) [2] and its successors [3, 4] have redefined the state of the art in tasks such as machine translation,sentiment analysis, summarization, and question answering by learning contextualized representations that capture both syntactic and semantic nuances.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1000000, "bbox": [[0.479, 0.612, 0.874, 0.62], [0.118, 0.626, 0.886, 0.634], [0.118, 0.641, 0.882, 0.647], [0.118, 0.655, 0.328, 0.661]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 20, "edu_l1_label": "IOS"}, {"txt": "This evolution has not only elevated task-specific performance,but also opened new avenues for exploring the complexities of human language.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1000000, "bbox": [[0.328, 0.655, 0.881, 0.66], [0.117, 0.669, 0.502, 0.675]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 21, "edu_l1_label": "IOS"}, {"txt": "At the same time, there has been a surge of interest in developing multilingual systems capable of processing and understanding multiple languages simultaneously [5,6,7].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1100000, "bbox": [[0.117, 0.686, 0.88, 0.696], [0.117, 0.703, 0.507, 0.71]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 22, "edu_l1_label": "IOS"}, {"txt": "Multilingual training leverages shared linguistic features across diverse languages, a benefit that is especially pronounced in low-resource settings.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1100000, "bbox": [[0.507, 0.703, 0.882, 0.71], [0.117, 0.717, 0.721, 0.722]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 23, "edu_l1_label": "IOS"}, {"txt": "In addition, adversarial training [8] has emerged as a powerful mechanism to encourage the learning of language-invariant representations, thus reducing language-specific biases and improving cross-lingual generalization in a variety of tasks [9,10].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1100000, "bbox": [[0.721, 0.717, 0.882, 0.722], [0.117, 0.729, 0.881, 0.737], [0.117, 0.744, 0.803, 0.751]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 24, "edu_l1_label": "IOS"}, {"txt": "However, developing robust multilingual models poses several challenges, including managing heterogeneous data distributions,accommodating language-specific nuances, and needing scalability without compromising performance.Adversarial training addresses these difficulties by introducing a competitive dynamic that forces the model to focus on features common to all languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1200000, "bbox": [[0.117, 0.763, 0.881, 0.771], [0.117, 0.778, 0.886, 0.785], [0.117, 0.79, 0.881, 0.799], [0.118, 0.804, 0.342, 0.812]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 15, "global_sentence_id": 25, "edu_l1_label": "IOS"}, {"txt": "This approach not only boosts overall performance but also diminishes the model's reliance on language-specific signals to enable better generalization across many linguistic domains [11].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1200000, "bbox": [[0.342, 0.804, 0.882, 0.812], [0.117, 0.819, 0.801, 0.826]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 15, "global_sentence_id": 26, "edu_l1_label": "IOS"}, {"txt": "In this work, we introduce UniBERT, a compact and efficient BERT-based model specifically designed for multilingual processing, available in three open-source versions:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1300000, "bbox": [[0.117, 0.838, 0.882, 0.847], [0.117, 0.855, 0.455, 0.861]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 16, "global_sentence_id": 27, "edu_l1_label": "IOS"}, {"txt": " (1) UniBERT-Small2, (2) UniBERT-XSmall3, and (3) UniBERT-", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1300000, "bbox": [[0.455, 0.855, 0.885, 0.859]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 16, "global_sentence_id": 28, "edu_l1_label": "IOS"}, {"txt": "*Corresponding author:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.136, 0.874, 0.279, 0.882]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 17, "global_sentence_id": 29, "edu_l1_label": "EDU_O"}, {"txt": " dumitru.cercel@upb.ro.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.279, 0.874, 0.422, 0.882]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 17, "global_sentence_id": 30, "edu_l1_label": "EDU_O"}, {"txt": "2https://huggingface.co/avramandrei/unibert-small", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.137, 0.886, 0.512, 0.895]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 18, "global_sentence_id": 31, "edu_l1_label": "EDU_O"}, {"txt": "3https://huggingface.co/avramandrei/unibert-xsmall", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.138, 0.898, 0.52, 0.908]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 19, "global_sentence_id": 32, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.737, 0.045, 0.882, 0.056]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 33, "edu_l1_label": "EDU_O"}, {"txt": "XXSmall4.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.116, 0.095, 0.191, 0.104]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 34, "edu_l1_label": "IOS"}, {"txt": "UniBERT was pre-trained on a curated Wikipedia corpus spanning 107 languages and combines three key training components: masked language modeling [2], adversarial training, and knowledge distillation [12].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.191, 0.095, 0.881, 0.103], [0.117, 0.109, 0.85, 0.117]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 35, "edu_l1_label": "IOS"}, {"txt": "This integrated methodology enables the model to achieve robust performance while significantly reducing the parameter count relative to larger models, making it an attractive solution for resource-constrained applications.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.85, 0.109, 0.882, 0.117], [0.118, 0.123, 0.882, 0.131], [0.117, 0.138, 0.776, 0.145]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 36, "edu_l1_label": "IOS"}, {"txt": "We perform extensive evaluations of UniBERT on four NLP tasks:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1500000, "bbox": [[0.116, 0.157, 0.554, 0.165]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 37, "edu_l1_label": "IOS"}, {"txt": " (1) named entity recognition, (2) natural language inference, (3) question answering, and (4) semantic textual similarity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1500000, "bbox": [[0.554, 0.157, 0.882, 0.165], [0.117, 0.171, 0.598, 0.179]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 38, "edu_l1_label": "IOS"}, {"txt": "The results demonstrate that our models,fine-tuned jointly on all languages, obtain better results than when they are fine-tuned on each language individually.Concretely, on average, UniBERT-Small improves the performance from 47.10% to 48.87%, UniBERT-XSmall from 40.20% to 44.35%, and UniBERT-XXSmall from 34.51% to 37.64% when trained jointly in all languages,yielding an average relative improvement of 7.72%.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1500000, "bbox": [[0.598, 0.171, 0.884, 0.181], [0.118, 0.185, 0.886, 0.193], [0.117, 0.198, 0.873, 0.206], [0.117, 0.211, 0.881, 0.221], [0.117, 0.228, 0.413, 0.234]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 39, "edu_l1_label": "IOS"}, {"txt": "In contrast, larger language models show an improvement of 1.17%,underscoring the significant benefits of the joint multilingual training approach enhanced by adversarial objectives.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1500000, "bbox": [[0.413, 0.228, 0.886, 0.235], [0.117, 0.241, 0.867, 0.247]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 40, "edu_l1_label": "IOS"}, {"txt": "We also performed a comprehensive statistical comparative analysis to confirm the strengths of our approach.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1600000, "bbox": [[0.117, 0.259, 0.849, 0.269]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 41, "edu_l1_label": "IOS"}, {"txt": "The student's t-test yielded a p-value of 0.0181, demonstrating that gains in UniBERT performance are significant at a significance threshold ofα=0.05.It also shows that the combination of joint multilingual training and adversarial training effectively compensates for the reduced capacity of compact models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1600000, "bbox": [[0.849, 0.259, 0.881, 0.269], [0.118, 0.276, 0.882, 0.283], [0.118, 0.289, 0.281, 0.295], [0.285, 0.29, 0.882, 0.296], [0.118, 0.302, 0.623, 0.309]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 42, "edu_l1_label": "IOS"}, {"txt": "In summary, the main contributions of this paper are as follows:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1700000, "bbox": [[0.117, 0.321, 0.533, 0.33]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 43, "edu_l1_label": "IOS"}, {"txt": "·We propose UniBERT, a compact, novel multilingual language model that integrates masked language modeling, adversarial training, and knowledge distillation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1800000, "bbox": [[0.161, 0.35, 0.881, 0.357], [0.185, 0.364, 0.557, 0.371]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 44, "edu_l1_label": "IOS"}, {"txt": "·We show that joint multilingual training, combined with an adversarial objective, significantly enhances cross-lingual performance, particularly in resource-constrained settings.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1900000, "bbox": [[0.161, 0.382, 0.881, 0.39], [0.175, 0.396, 0.645, 0.403]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 45, "edu_l1_label": "IOS"}, {"txt": "·We provide extensive evaluations on multipleNLP tasks, establishing the competitive performance of UniBERT compared to larger baseline models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2000000, "bbox": [[0.161, 0.416, 0.882, 0.422], [0.175, 0.429, 0.413, 0.436]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 46, "edu_l1_label": "IOS"}, {"txt": "·We offer a detailed comparative analysis supported by statistical evidence, highlighting the efficacy of our approach in achieving language-invariant representations.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2100000, "bbox": [[0.161, 0.449, 0.88, 0.456], [0.176, 0.463, 0.552, 0.469]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 47, "edu_l1_label": "IOS"}, {"txt": "2 Related Work", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2200000, "bbox": [[0.116, 0.494, 0.262, 0.506]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 10, "global_sentence_id": 48, "edu_l1_label": "BOS"}, {"txt": "2.1 Multilingual Language Models", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2300000, "bbox": [[0.117, 0.526, 0.368, 0.535]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 11, "global_sentence_id": 49, "edu_l1_label": "IOS"}, {"txt": "The development of multilingual sentence embeddings has evolved from initial methods such as LASER [13], which leveraged a shared long short-term memory [14]encoder-decoder framework to align sentence representations across languages using parallel data.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2400000, "bbox": [[0.116, 0.552, 0.881, 0.56], [0.118, 0.566, 0.881, 0.574], [0.117, 0.58, 0.321, 0.588]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 50, "edu_l1_label": "IOS"}, {"txt": "Subsequent models, such as SBERT-distill [15], utilized knowledge distillation to transfer monolingual sentence embedding capabilities into a multilingual space, allowing robust performance with fewer parallel sentences.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2400000, "bbox": [[0.321, 0.58, 0.881, 0.588], [0.116, 0.594, 0.88, 0.602], [0.118, 0.609, 0.242, 0.615]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 51, "edu_l1_label": "IOS"}, {"txt": "LaBSE [16] expanded these ideas by introducing additive margin softmax,which improved the discriminative power for translation pair similarity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2400000, "bbox": [[0.242, 0.609, 0.881, 0.615], [0.117, 0.621, 0.451, 0.629]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 52, "edu_l1_label": "IOS"}, {"txt": "Despite their effectiveness, these methods often require extensive computational resources and large-scale parallel data, making them less accessible for low-resource languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2400000, "bbox": [[0.451, 0.621, 0.881, 0.629], [0.117, 0.637, 0.838, 0.644]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 53, "edu_l1_label": "IOS"}, {"txt": "Multilingual pre-trained language models have also made significant strides in cross-lingual understanding.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2500000, "bbox": [[0.117, 0.655, 0.837, 0.664]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 54, "edu_l1_label": "IOS"}, {"txt": "Cross-lingual language models (XLM) [17] introduced the objective of translation language modeling(TLM),extending masked language modeling to leverage parallel data and align representations between languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2500000, "bbox": [[0.837, 0.655, 0.883, 0.664], [0.118, 0.669, 0.881, 0.679], [0.118, 0.684, 0.769, 0.691]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 55, "edu_l1_label": "IOS"}, {"txt": "Building on this,XLM-RoBERTa (XLM-R) [5] scaled these methods to 100 languages by pretraining on more than two terabytes of Common Crawl data [18], significantly improving performance on benchmarks for low-resource languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2500000, "bbox": [[0.769, 0.684, 0.886, 0.692], [0.117, 0.696, 0.882, 0.704], [0.117, 0.71, 0.816, 0.719]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 56, "edu_l1_label": "IOS"}, {"txt": "XLM-R's success underscored the importance of large-scale data and careful language sampling strategies,particularly for balancing performance across high-and low-resource languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2500000, "bbox": [[0.816, 0.71, 0.883, 0.718], [0.118, 0.727, 0.882, 0.733], [0.117, 0.738, 0.54, 0.746]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 57, "edu_l1_label": "IOS"}, {"txt": "mT5 [6], which extended the T5 framework [19] to multilingual contexts, introduced further innovations.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2600000, "bbox": [[0.117, 0.76, 0.798, 0.766]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 58, "edu_l1_label": "IOS"}, {"txt": "By adopting a text-to-text paradigm and pre-training using a large multilingual dataset covering 101 languages, mT5 achieved state-of-the-art results on various multilingual tasks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2600000, "bbox": [[0.798, 0.76, 0.881, 0.768], [0.117, 0.774, 0.88, 0.781], [0.118, 0.788, 0.471, 0.795]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 59, "edu_l1_label": "IOS"}, {"txt": "Balanced training was ensured by methods such as language sampling with controlled scaling and the use of large vocabularies.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2600000, "bbox": [[0.471, 0.788, 0.881, 0.795], [0.118, 0.802, 0.556, 0.808]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 60, "edu_l1_label": "IOS"}, {"txt": "In conjunction with a span-corruption pretraining objective, the model's encoder-decoder design enabled flexibility across tasks ranging from classification to sequence generation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2600000, "bbox": [[0.556, 0.802, 0.881, 0.81], [0.117, 0.815, 0.881, 0.822], [0.117, 0.83, 0.192, 0.836]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 61, "edu_l1_label": "IOS"}, {"txt": "Recent efforts, including EMS [20] and mSimCSE [21], have emphasized efficiency by reducing the reliance on large-scale parallel datasets and employing streamlined architectures.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2700000, "bbox": [[0.118, 0.848, 0.881, 0.856], [0.117, 0.862, 0.569, 0.871]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 15, "global_sentence_id": 62, "edu_l1_label": "IOS"}, {"txt": "EMS's integration of token-level generative and sentence-level contrastive objectives exemplifies this shift, achieving a well-aligned multilingual embedding space", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2700000, "bbox": [[0.569, 0.862, 0.88, 0.871], [0.118, 0.878, 0.881, 0.884]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 15, "global_sentence_id": 63, "edu_l1_label": "IOS"}, {"txt": "4https://huggingface.co/avramandrei/unibert-xxsmall", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2800000, "bbox": [[0.139, 0.9, 0.527, 0.908]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 16, "global_sentence_id": 64, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2800000, "bbox": [[0.494, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 17, "global_sentence_id": 65, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2800000, "bbox": [[0.737, 0.045, 0.882, 0.056]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 66, "edu_l1_label": "EDU_O"}, {"txt": "with minimal computational overhead.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2800000, "bbox": [[0.117, 0.097, 0.374, 0.104]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 67, "edu_l1_label": "IOS"}, {"txt": "Similarly, unsupervised approaches such as those used in SimCSE [22] have been adapted for multilingual scenarios, providing competitive alternatives that further diminish the dependence on supervised data.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2800000, "bbox": [[0.374, 0.097, 0.881, 0.103], [0.118, 0.108, 0.881, 0.117], [0.118, 0.125, 0.225, 0.131]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 68, "edu_l1_label": "IOS"}, {"txt": "These methods collectively push the boundaries of multilingual NLP, addressing challenges such as data scarcity and computational resource constraints.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2800000, "bbox": [[0.225, 0.125, 0.881, 0.131], [0.117, 0.137, 0.463, 0.145]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 69, "edu_l1_label": "IOS"}, {"txt": "Despite these advances, it is still difficult to scale these models to include underrepresented languages and optimize their performance in low-resource settings.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2900000, "bbox": [[0.117, 0.156, 0.883, 0.165], [0.117, 0.173, 0.367, 0.179]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 70, "edu_l1_label": "IOS"}, {"txt": "Research into strategies such as improved sampling distributions, multilingual adversarial training, better representation of underrepresented data, and the integration of knowledge distillation continues to enhance the accessibility and efficacy of multilingual models [11].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2900000, "bbox": [[0.367, 0.173, 0.882, 0.179], [0.118, 0.186, 0.881, 0.193], [0.117, 0.2, 0.633, 0.207]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 71, "edu_l1_label": "IOS"}, {"txt": "2.2 Adversarial Training", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3000000, "bbox": [[0.117, 0.229, 0.3, 0.24]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 4, "global_sentence_id": 72, "edu_l1_label": "IOS"}, {"txt": "Adversarial training has become a cornerstone technique in machine learning, especially for improving model robustness and enhancing generalization to unseen data distributions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3100000, "bbox": [[0.116, 0.256, 0.882, 0.264], [0.117, 0.271, 0.489, 0.277]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 73, "edu_l1_label": "IOS"}, {"txt": "Initially introduced in the context of adversarial examples for image classification [8], adversarial training has since been adapted to a variety of tasks, such as named entity recognition [23],text classification [24], and multiword expression detection [25] in NLP.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3100000, "bbox": [[0.489, 0.271, 0.881, 0.278], [0.117, 0.284, 0.88, 0.292], [0.117, 0.296, 0.636, 0.306]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 74, "edu_l1_label": "IOS"}, {"txt": "A central idea in adversarial training is the minimax optimization framework, which pits a discriminator against a feature extractor to encourage learning task-relevant and domain-invariant features.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3100000, "bbox": [[0.636, 0.296, 0.881, 0.306], [0.12, 0.311, 0.88, 0.32], [0.118, 0.325, 0.408, 0.332]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 75, "edu_l1_label": "IOS"}, {"txt": "One influential approach involves the incorporation of gradient reversal layers to align feature spaces between domains without requiring explicit labels in the target domain.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3100000, "bbox": [[0.408, 0.325, 0.882, 0.332], [0.117, 0.338, 0.774, 0.346]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 76, "edu_l1_label": "IOS"}, {"txt": "Techniques such as the domain-adversarial neural netwwork [26] exemplify this principle, achieving significant success in cross-domain applications [27].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3100000, "bbox": [[0.774, 0.338, 0.88, 0.347], [0.117, 0.354, 0.881, 0.36], [0.118, 0.368, 0.231, 0.374]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 77, "edu_l1_label": "IOS"}, {"txt": "Loss reversal [28] is a novel variation of the adversarial training paradigm that has demonstrated improved performance for domain-specific tasks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3200000, "bbox": [[0.117, 0.386, 0.881, 0.395], [0.118, 0.401, 0.284, 0.408]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 78, "edu_l1_label": "IOS"}, {"txt": "Instead of reversing the gradient at the input to the feature extractor, this method simply flips the sign of the domain discriminator's loss before optimization.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3200000, "bbox": [[0.284, 0.401, 0.881, 0.408], [0.118, 0.414, 0.526, 0.422]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 79, "edu_l1_label": "IOS"}, {"txt": "As proposed in conjunction with the HistNERo dataset for historical NER in Romanian, loss reversal encourages the model to produce more distinct embeddings between domains,boosting performance on the task.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3200000, "bbox": [[0.526, 0.414, 0.883, 0.422], [0.117, 0.428, 0.88, 0.436], [0.117, 0.443, 0.415, 0.45]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 80, "edu_l1_label": "IOS"}, {"txt": "Empirical results show that integrating loss reversal with pre-trained Romanian language models, namely BERT-ro [29] and DistilRoBERT [30], leads to substantial gains in strict F1-scores,with increases of over 10% compared to the baseline non-adversarial approaches.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3200000, "bbox": [[0.415, 0.443, 0.881, 0.45], [0.118, 0.455, 0.886, 0.465], [0.118, 0.47, 0.648, 0.477]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 81, "edu_l1_label": "IOS"}, {"txt": "In addition to classical adversarial setups, recent research has explored integrating adversarial training with multitask learning and self-supervised pre-training [31, 32].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3300000, "bbox": [[0.117, 0.489, 0.881, 0.497], [0.118, 0.504, 0.447, 0.511]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 82, "edu_l1_label": "IOS"}, {"txt": "These hybrid models aim to simultaneously optimize for multiple objectives, such as predicting token-level labels and aligning domain embeddings.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3300000, "bbox": [[0.447, 0.504, 0.88, 0.511], [0.117, 0.518, 0.66, 0.525]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 83, "edu_l1_label": "IOS"}, {"txt": "For instance, topic-aware domain adaptation frameworks have incorporated latent topic modeling into adversarial setups, allowing models to learn domain-invariant features and topic-specific representations jointly.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3300000, "bbox": [[0.66, 0.518, 0.88, 0.526], [0.117, 0.533, 0.881, 0.539], [0.117, 0.545, 0.553, 0.552]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 84, "edu_l1_label": "IOS"}, {"txt": "This combination has proven particularly effective for scientific keyphrase identification [33], where incorporating adversarial examples further enhances performance.Such approaches demonstrate the versatility of adversarial techniques in addressing domain-specific challenges across a wide range of tasks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3300000, "bbox": [[0.553, 0.545, 0.881, 0.553], [0.118, 0.558, 0.886, 0.566], [0.118, 0.572, 0.881, 0.581], [0.117, 0.588, 0.248, 0.595]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 85, "edu_l1_label": "IOS"}, {"txt": "Moreover, adversarial training has expanded beyond text classification and NER to address multilingual and cross-lingual tasks [34,35].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.118, 0.607, 0.883, 0.614], [0.117, 0.62, 0.265, 0.634]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 86, "edu_l1_label": "IOS"}, {"txt": "In these applications, adversarial discriminators are designed to promote language-invariant representations, enabling zero-shot or few-shot learning for low-resource languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.265, 0.62, 0.88, 0.629], [0.117, 0.636, 0.687, 0.642]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 87, "edu_l1_label": "IOS"}, {"txt": "In addition, techniques such as lateral inhibition layers have been integrated with adversarial setups to enhance the separation of embeddings for specific tasks such as multi-word expression identification [9].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.687, 0.636, 0.88, 0.642], [0.117, 0.65, 0.882, 0.657], [0.118, 0.664, 0.522, 0.67]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 88, "edu_l1_label": "IOS"}, {"txt": "These advancements highlight the ongoing evolution of adversarial training, with each iteration introducing improved mechanisms to control the interaction between feature extraction and task-specific optimization.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.522, 0.664, 0.882, 0.67], [0.117, 0.678, 0.881, 0.684], [0.117, 0.691, 0.386, 0.698]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 89, "edu_l1_label": "IOS"}, {"txt": "3 UniBERT", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3500000, "bbox": [[0.116, 0.723, 0.229, 0.734]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 9, "global_sentence_id": 90, "edu_l1_label": "BOS"}, {"txt": "3.1 Training Data", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3600000, "bbox": [[0.116, 0.755, 0.252, 0.765]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 10, "global_sentence_id": 91, "edu_l1_label": "IOS"}, {"txt": "The UniBERT models were pre-trained on the latest dump of the Wikipedia corpus [36],which was thoroughly filtered to ensure a high-quality dataset by employing the following heuristics:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3700000, "bbox": [[0.116, 0.78, 0.882, 0.79], [0.116, 0.796, 0.578, 0.803]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 92, "edu_l1_label": "IOS"}, {"txt": "·Removed all the HTML artifacts left in the corpus using a simple regex.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3800000, "bbox": [[0.161, 0.824, 0.646, 0.83]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 93, "edu_l1_label": "IOS"}, {"txt": "·Removed all samples that contained only URLs or ISBNs.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3900000, "bbox": [[0.162, 0.843, 0.556, 0.85]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 94, "edu_l1_label": "IOS"}, {"txt": "·Removed all samples for which the count of digits represents more than 70% of the length of the whole text.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4000000, "bbox": [[0.162, 0.863, 0.882, 0.869]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 95, "edu_l1_label": "IOS"}, {"txt": "·Removed all the hidden characters from the text.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4100000, "bbox": [[0.162, 0.883, 0.493, 0.89]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 15, "global_sentence_id": 96, "edu_l1_label": "IOS"}, {"txt": "·Removed all samples with less than five tokens using the multilingual BERT (mBERT) [2] tokenizer.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4200000, "bbox": [[0.161, 0.902, 0.834, 0.908]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 16, "global_sentence_id": 97, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4300000, "bbox": [[0.494, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 17, "global_sentence_id": 98, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4300000, "bbox": [[0.735, 0.045, 0.883, 0.055]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 99, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4300000, "bbox": [[0.116, 0.093, 0.874, 0.275]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 2, "global_sentence_id": 100, "edu_l1_label": "EDU_O"}, {"txt": "Figure 1: The amount of data in GB (log-scale) for each of the 107 languages that appear in the Wikipedia corpus was used to pre-train the UniBERT models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4400000, "bbox": [[0.115, 0.287, 0.885, 0.312]]}]}, "tags": ["figure_caption"], "label": "figure_title", "web_segment_id": 3, "global_sentence_id": 101, "edu_l1_label": "EDU_O"}, {"txt": "Table 1: Comparison of our UniBERT models and the other multilingual BERT-based models available in the literature regarding their number of layers, number of hidden units on each layer, number of heads, vocabulary dimension, number of parameters, and model size.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4500000, "bbox": [[0.115, 0.337, 0.885, 0.378]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 4, "global_sentence_id": 102, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4600000, "bbox": [[0.144, 0.379, 0.851, 0.508]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 5, "global_sentence_id": 103, "edu_l1_label": "EDU_O"}, {"txt": "The resulting dataset contained 82GB of textual data, with the English language being the most representative (i.e.,9.8GB of data) and the Swati language the least representative (i.e., 802K of data), as depicted in Figure 1.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4700000, "bbox": [[0.116, 0.54, 0.886, 0.549], [0.117, 0.554, 0.827, 0.562]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 104, "edu_l1_label": "IOS"}, {"txt": "We also measured the average number of tokens in the dataset using the mBERT tokenizer and found that each sample in the dataset had approximately 110 tokens on average, with the maximum number of tokens being approximately 10K tokens5.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4700000, "bbox": [[0.827, 0.554, 0.881, 0.562], [0.118, 0.569, 0.881, 0.576], [0.117, 0.581, 0.881, 0.59], [0.117, 0.596, 0.171, 0.604]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 105, "edu_l1_label": "IOS"}, {"txt": "3.2 Model Architecture", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4800000, "bbox": [[0.117, 0.627, 0.29, 0.636]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 7, "global_sentence_id": 106, "edu_l1_label": "IOS"}, {"txt": "In this work, we introduce three distinct UniBERT models:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4900000, "bbox": [[0.117, 0.653, 0.527, 0.661]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 107, "edu_l1_label": "IOS"}, {"txt": " UniBERT-Small, UniBERT-XSmall, and UniBERT-XXSmall.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4900000, "bbox": [[0.527, 0.653, 0.885, 0.66], [0.116, 0.666, 0.185, 0.675]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 108, "edu_l1_label": "IOS"}, {"txt": "These models provide varied configurations to address different computational needs and efficiency requirements.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4900000, "bbox": [[0.185, 0.666, 0.881, 0.677], [0.117, 0.682, 0.207, 0.689]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 109, "edu_l1_label": "IOS"}, {"txt": "Each variant is designed to minimize the number of parameters and the overall size of the model, allowing efficient multilingual processing without sacrificing the core model performances.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4900000, "bbox": [[0.207, 0.682, 0.881, 0.691], [0.116, 0.696, 0.651, 0.703]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 110, "edu_l1_label": "IOS"}, {"txt": "Unlike prominent multilingual models such as Distil-mBERT-base [37], mBERT-base [2], and the XLM-RoBERTa series [17], our UniBERT models offer a streamlined architecture.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5000000, "bbox": [[0.117, 0.715, 0.881, 0.723], [0.117, 0.732, 0.565, 0.737]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 111, "edu_l1_label": "IOS"}, {"txt": "Table 1 shows that the UniBERT-Small model includes 6 layers with 512 hidden units per layer and 8 attention heads, which make it comparable in layer count to Distil-mBERT-base but with fewer hidden units and heads, resulting in a reduced number of parameters (87M versus 135M) and a smaller model size (334M versus 543M).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5000000, "bbox": [[0.565, 0.732, 0.882, 0.738], [0.118, 0.744, 0.882, 0.751], [0.118, 0.756, 0.881, 0.765], [0.118, 0.77, 0.464, 0.778]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 112, "edu_l1_label": "IOS"}, {"txt": "This reduction is achieved while maintaining a similar vocabulary dimension of 120K, indicating that UniBERT-Small retains sufficient linguistic flexibility for multilingual tasks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5000000, "bbox": [[0.464, 0.77, 0.882, 0.781], [0.117, 0.785, 0.847, 0.793]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 113, "edu_l1_label": "IOS"}, {"txt": "The UniBERT-XSmall and UniBERT-XXSmall configurations further emphasize parameter efficiency.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5100000, "bbox": [[0.117, 0.805, 0.81, 0.814]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 114, "edu_l1_label": "IOS"}, {"txt": "UniBERT-XSmall, with 4 layers, 256 hidden units per layer, and 8 heads, holds only 39M parameters and has a model size of 148M, nearly a quarter of the size of Distil-mBERT-base.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5100000, "bbox": [[0.81, 0.805, 0.884, 0.818], [0.116, 0.818, 0.882, 0.827], [0.117, 0.832, 0.487, 0.84]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 115, "edu_l1_label": "IOS"}, {"txt": "UniBERT-XXSmall, the most compact variant, is constructed with 4 layers, 128 hidden units, and 4 heads, resulting in just 19M parameters and a model size of 74M.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5100000, "bbox": [[0.487, 0.832, 0.881, 0.84], [0.117, 0.848, 0.786, 0.854]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 116, "edu_l1_label": "IOS"}, {"txt": "This reduction in architectural complexity allows UniBERT-XXSmall to be highly resource-efficient, suiting applications with limited computational resources.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5100000, "bbox": [[0.786, 0.848, 0.88, 0.855], [0.118, 0.861, 0.882, 0.868], [0.117, 0.876, 0.28, 0.882]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 117, "edu_l1_label": "IOS"}, {"txt": "5This number is truncated during pre-training and fine-tuning depending on the model maximum context window.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5200000, "bbox": [[0.14, 0.9, 0.806, 0.909]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 11, "global_sentence_id": 118, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5200000, "bbox": [[0.493, 0.936, 0.506, 0.948]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 12, "global_sentence_id": 119, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5200000, "bbox": [[0.736, 0.045, 0.882, 0.056]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 120, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5200000, "bbox": [[0.149, 0.093, 0.85, 0.376]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 2, "global_sentence_id": 121, "edu_l1_label": "EDU_O"}, {"txt": "Figure 2: UniBERT pre-training methodology. Given a sequence of tokens, both UniBERT and mBERT produce a corresponding set of embeddings. The UniBERT embeddings are first used in the masked language modeling task,as in the original mBERT pre-training. Then, those embeddings are averaged out, and a feed-forward layer is used to predict the language of the input sentence. The learning signal from this classifier is used to train the UniBERT model to produce language-independent embeddings adversarially. Finally, the knowledge distillation technique is used to align the probability distributions of the two models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5300000, "bbox": [[0.114, 0.39, 0.886, 0.47]]}]}, "tags": ["figure_caption"], "label": "figure_title", "web_segment_id": 3, "global_sentence_id": 122, "edu_l1_label": "EDU_O"}, {"txt": "3.3 Training Process", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5400000, "bbox": [[0.117, 0.497, 0.27, 0.506]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 4, "global_sentence_id": 123, "edu_l1_label": "IOS"}, {"txt": "Figure 2 shows the general pre-training methodology.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5500000, "bbox": [[0.117, 0.523, 0.478, 0.532]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 124, "edu_l1_label": "IOS"}, {"txt": "The training framework involves a novel approach to train a smaller, more efficient BERT-based language model, called UniBERT, through three main techniques: (1)masked language modeling as for the mBERT model, (2) adversarial training using a language discriminator [38] with the loss reversal technique [28], and (3) knowledge distillation with the mBERT model serving as the teacher model.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5500000, "bbox": [[0.478, 0.523, 0.882, 0.532], [0.118, 0.54, 0.882, 0.546], [0.117, 0.551, 0.883, 0.559], [0.117, 0.566, 0.822, 0.573]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 125, "edu_l1_label": "IOS"}, {"txt": "3.3.1 Masked Language Modeling", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5600000, "bbox": [[0.116, 0.593, 0.364, 0.604]]}]}, "tags": ["title"], "label": "title3", "web_segment_id": 6, "global_sentence_id": 126, "edu_l1_label": "IOS"}, {"txt": "Masked language modeling (MLM) [2] is a core component of BERT-based models that enables them to learn bidirectional representations of language.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5700000, "bbox": [[0.115, 0.616, 0.886, 0.71]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 127, "edu_l1_label": "IOS"}, {"txt": "In MLM, a percentage of tokens within the input sequence are masked randomly, and the model is trained to predict these masked tokens based on the surrounding context.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5700000, "bbox": [[0.115, 0.616, 0.886, 0.71]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 128, "edu_l1_label": "IOS"}, {"txt": "Formally,let $x=\\left(x_{1}x_{2}x_{3}\\cdots x_{}\\right)$be an input sequence of tokens,where$x_{i}$represents each token.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5700000, "bbox": [[0.115, 0.616, 0.886, 0.71]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 129, "edu_l1_label": "IOS"}, {"txt": "A subset of these tokens,calledxmasked$\\subset $,is replaced by a special token [MASK], and the model optimizes the probability of recovering the original tokens$P\\left(x_{\\text {mk}}\\vert x_{\\backslash \\text {m}}\\right.$ $k)$given the context.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5700000, "bbox": [[0.115, 0.616, 0.886, 0.71]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 130, "edu_l1_label": "IOS"}, {"txt": "This choice can be achieved by minimizing the cross-entropy loss over the mased token,s shown in Equation 1:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5700000, "bbox": [[0.115, 0.616, 0.886, 0.71]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 131, "edu_l1_label": "IOS"}, {"txt": "$$\\mathcal {L}_{\\mathrm {MLM}}=-\\sum _{x_{\\text {masked}}}\\log P\\left(x_{\\text {masked}}\\vert x_{\\backslash \\text {masked}};\\theta _{\\text {student}}\\right)\\tag{1}$$", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5800000, "bbox": [[0.342, 0.729, 0.885, 0.761]]}]}, "tags": ["equation"], "label": "formula", "web_segment_id": 8, "global_sentence_id": 132, "edu_l1_label": "IOS"}, {"txt": "where$\\theta _{\\text {student}}$ represents the parameters of the student model.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5900000, "bbox": [[0.114, 0.765, 0.885, 0.818]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 133, "edu_l1_label": "IOS"}, {"txt": "The MLM objective enables the model to capture contextual dependencies on both sides of a token, effectively learning a bidirectional understanding of language structure.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5900000, "bbox": [[0.114, 0.765, 0.885, 0.818]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 134, "edu_l1_label": "IOS"}, {"txt": "In UniBERT, MLM serves as a foundational pre-training task, helping the model learn syntactic and semantic representations that are robust and generalizable across different applications.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5900000, "bbox": [[0.114, 0.765, 0.885, 0.818]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 135, "edu_l1_label": "IOS"}, {"txt": "3.3.2 Adversarial Training", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6000000, "bbox": [[0.117, 0.835, 0.312, 0.845]]}]}, "tags": ["title"], "label": "title3", "web_segment_id": 10, "global_sentence_id": 136, "edu_l1_label": "IOS"}, {"txt": "In UniBERT, adversarial training [38] involves training a language classifier that learns to distinguish between different language inputs, using the average of the embeddings$\\overline {e_{i}}$generated by the student model.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6100000, "bbox": [[0.115, 0.859, 0.885, 0.91]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 137, "edu_l1_label": "IOS"}, {"txt": "To encourage language invariance, a loss reversal technique is applied, which involves simply reversing the sign of the loss$\\mathcal {L}_{\\text {isc}}$provided by the language discriminator, as shown in Equation 3:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6100000, "bbox": [[0.115, 0.859, 0.885, 0.91]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 138, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6200000, "bbox": [[0.494, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 12, "global_sentence_id": 139, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6200000, "bbox": [[0.735, 0.045, 0.883, 0.055]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 140, "edu_l1_label": "EDU_O"}, {"txt": "$$y_{i}=FF\\left(\\bar {e}_{i}\\right)\\tag{2}$$", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6300000, "bbox": [[0.453, 0.112, 0.885, 0.128]]}]}, "tags": ["equation"], "label": "formula", "web_segment_id": 2, "global_sentence_id": 141, "edu_l1_label": "IOS"}, {"txt": "$$\\mathcal {L}_{\\mathrm {adv}}=-\\sum _{i}y_{i}\\log P\\left(y_{i}\\vert \\bar {e}_{i};\\theta _{\\mathrm {disc}}\\right)\\tag{3}$$", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6400000, "bbox": [[0.387, 0.161, 0.885, 0.188]]}]}, "tags": ["equation"], "label": "formula", "web_segment_id": 3, "global_sentence_id": 142, "edu_l1_label": "IOS"}, {"txt": "where $FF$ is the feed-forward layer that produce the language probabilities $y_{i}$  and $\\theta _{\\text {disc}}$ are the parameters of the language discriminator model.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6500000, "bbox": [[0.115, 0.202, 0.885, 0.227]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 143, "edu_l1_label": "IOS"}, {"txt": "3.3.3 Knowledge Distillation", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6600000, "bbox": [[0.117, 0.25, 0.324, 0.259]]}]}, "tags": ["title"], "label": "title3", "web_segment_id": 5, "global_sentence_id": 144, "edu_l1_label": "IOS"}, {"txt": "Knowledge distillation [12] is a model compression technique in which a smaller model (knownas the student) learns from a larger, pre-trained model (known as the teacher) by approximating its predictions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6700000, "bbox": [[0.118, 0.275, 0.881, 0.285], [0.118, 0.29, 0.714, 0.298]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 145, "edu_l1_label": "IOS"}, {"txt": "In UniBERT, knowledge distillation is implemented by using mBERT as the teacher model, allowing UniBERT to inherit multilingual capabilities from mBERT while maintaining a more compact and efficient structure.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6700000, "bbox": [[0.714, 0.29, 0.882, 0.299], [0.118, 0.304, 0.881, 0.312], [0.118, 0.318, 0.592, 0.325]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 146, "edu_l1_label": "IOS"}, {"txt": "The objective of knowledge distillation is to minimize the Kullback-Leibler (KL) divergence between the teacher and student model outputs, as shown in Equation 4:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6700000, "bbox": [[0.592, 0.318, 0.881, 0.325], [0.126, 0.333, 0.881, 0.34], [0.117, 0.344, 0.13, 0.354]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 147, "edu_l1_label": "IOS"}, {"txt": "$$\\mathcal {L}_{\\mathrm {KD}}=\\mathrm {KL}\\left(z_{\\text {teacher}}\\\\ vertz_{\\text {student}}\\right)=-\\sum _{i}z_{\\text {teacher}}^{(i)}\\log \\frac {z_{\\text {teacher}}^{(i)}}{z_{\\text {student}}^{(i)}}\\tag{4}$$", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6800000, "bbox": [[0.319, 0.382, 0.885, 0.421]]}]}, "tags": ["equation"], "label": "formula", "web_segment_id": 7, "global_sentence_id": 148, "edu_l1_label": "IOS"}, {"txt": "where$z_{\\text {teacher}}$ and $z_{\\text {student}}$ represent the logits of the teacher and student models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6900000, "bbox": [[0.114, 0.429, 0.885, 0.47]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 149, "edu_l1_label": "IOS"}, {"txt": "This objective aligns the output distribution of the student model with that of the teacher, effectively transferring knowledge without requiring the student to have the same computational complexity as the teacher.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6900000, "bbox": [[0.114, 0.429, 0.885, 0.47]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 150, "edu_l1_label": "IOS"}, {"txt": "3.3.4 Training Objective", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7000000, "bbox": [[0.117, 0.492, 0.297, 0.501]]}]}, "tags": ["title"], "label": "title3", "web_segment_id": 9, "global_sentence_id": 151, "edu_l1_label": "IOS"}, {"txt": "The general training objective for UniBERT combines the objectives of MLM, adversarial training, and knowledge distillation,represented as the weighted sum of the three individual losses, as shown in Equation 5:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7100000, "bbox": [[0.116, 0.518, 0.882, 0.527], [0.117, 0.532, 0.762, 0.54]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 152, "edu_l1_label": "IOS"}, {"txt": "$$\\mathcal {L}_{\\text {total}}=λ_{\\mathrm {MLM}}\\mathcal {L}_{\\mathrm {MLM}}-λ_{\\mathrm {adv}}\\mathcal {L}_{\\mathrm {adv}}+λ_{\\mathrm {KD}}\\mathcal {L}_{\\mathrm {KD}}\\tag{5}$$", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7200000, "bbox": [[0.358, 0.569, 0.885, 0.587]]}]}, "tags": ["equation"], "label": "formula", "web_segment_id": 11, "global_sentence_id": 153, "edu_l1_label": "IOS"}, {"txt": "where $\\mathcal {L}_{\\mathrm {MLM}}$,$\\mathcal {L}_{\\mathrm {adv}},$ $\\mathcal {L}_{\\mathrm {KD}}$ are the objectives of MLM, adversarial, and knowledge distillation, respectively, whereas $λ_{\\mathrm {MLM}}$，$λ_{\\mathrm {adv}}$，and 入kD are their corresponding weights. The MLM loss$\\mathcal {L}_{\\mathrm {MLM}}$ enables UniBERT to learn semantic representations by predicting masked tokens from context. In contrast, the adversarial loss$\\mathcal {L}_{\\mathrm {adv}}$encourages the model to be language-agnostic by reversing the gradient of the language classifier. Finally, the knowledge distillation loss $\\mathcal {L}_{\\mathrm {KD}}$allows UniBERT to mimic a larger multilingual BERT model, preserving multilingual capabilities in a mmore compact form. Together, these objectives ensure that UniBERT is both efficient and capable of handling cross-lingual tasks with a high degree of generalization.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7300000, "bbox": [[0.115, 0.595, 0.885, 0.691]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 154, "edu_l1_label": "IOS"}, {"txt": "3.4 Training Hyperparameters", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7400000, "bbox": [[0.117, 0.714, 0.342, 0.724]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 13, "global_sentence_id": 155, "edu_l1_label": "IOS"}, {"txt": "The UniBERT model was pre-trained for a total of 1M steps with a batchsize of 128, utilizing a learning rate of 5e-4with a linear scheduling approach and an initial warmup period spanning the first 50,000 steps.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7500000, "bbox": [[0.116, 0.741, 0.881, 0.75], [0.117, 0.758, 0.722, 0.765]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 156, "edu_l1_label": "IOS"}, {"txt": "We selected the AdamW optimizer [39] to optimize the stability of the training.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7500000, "bbox": [[0.722, 0.758, 0.881, 0.765], [0.117, 0.772, 0.468, 0.778]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 157, "edu_l1_label": "IOS"}, {"txt": "In addition, gradient clipping [40] was applied with a maximum gradient norm set to 5, effectively preventing large gradient updates.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7500000, "bbox": [[0.468, 0.772, 0.881, 0.778], [0.118, 0.785, 0.577, 0.791]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 158, "edu_l1_label": "IOS"}, {"txt": "A weight decay of 0.01 was used to focus on optimization and improve the primary parameters without penalizing their magnitudes over time.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7500000, "bbox": [[0.577, 0.785, 0.881, 0.792], [0.117, 0.799, 0.749, 0.806]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 159, "edu_l1_label": "IOS"}, {"txt": "We set the training objective coefficients for the MLM loss $λ_{\\mathrm {MLM}}$ to 0.5, for the knowledge distillation loss $λ_{\\mathrm {KD}}$ to 0.4,and for the adversarial loss$λ_{\\mathrm {adv}}$to 0.1.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.115, 0.817, 0.885, 0.912]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 15, "global_sentence_id": 160, "edu_l1_label": "IOS"}, {"txt": "Furthermore, to refine UniBERT's language understanding for the MLM objective, we randomly selected $15\\%$ of tokens for masking, the probability being applied uniformly, ensuring an even distribution between tokens.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.115, 0.817, 0.885, 0.912]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 15, "global_sentence_id": 161, "edu_l1_label": "IOS"}, {"txt": "The MLM loss incorporated label smoothing, set to a factor of 0.7, which helped alleviate overconfidence in predictions by distributing some probability mass across non-target classes.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.115, 0.817, 0.885, 0.912]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 15, "global_sentence_id": 162, "edu_l1_label": "IOS"}, {"txt": "Finally, a softmax temperature of 2 was utilized to adjust the output distribution's sharpness, effectively balancing the confidence of the UniBERT models across potential predictions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.115, 0.817, 0.885, 0.912]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 15, "global_sentence_id": 163, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7700000, "bbox": [[0.494, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 16, "global_sentence_id": 164, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7700000, "bbox": [[0.735, 0.045, 0.883, 0.055]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 165, "edu_l1_label": "EDU_O"}, {"txt": "4 Evaluation Datasets", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7800000, "bbox": [[0.116, 0.092, 0.312, 0.103]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 2, "global_sentence_id": 166, "edu_l1_label": "BOS"}, {"txt": "To evaluate the UniBERT models, we considered two scenarios:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7900000, "bbox": [[0.116, 0.128, 0.537, 0.136]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 167, "edu_l1_label": "IOS"}, {"txt": " (1) \"each\" - measuring their performance by training and evaluating each of the individual languages in the dataset, and (2) \"all\" -training the model in all languages and calcuulating the average performance in each language.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7900000, "bbox": [[0.537, 0.128, 0.881, 0.137], [0.117, 0.144, 0.881, 0.15], [0.117, 0.156, 0.469, 0.163]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 168, "edu_l1_label": "IOS"}, {"txt": "Experiments were carried out on four key NLP tasks as follows:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7900000, "bbox": [[0.469, 0.156, 0.884, 0.163]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 169, "edu_l1_label": "IOS"}, {"txt": "·Named Entity Recognition (NER), whose goal is to identify and categorize specific entities within the text, such as people, organizations, and locations.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8000000, "bbox": [[0.161, 0.188, 0.881, 0.195], [0.176, 0.201, 0.511, 0.208]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 170, "edu_l1_label": "IOS"}, {"txt": "We evaluated UniBERT on the CoNNLL-2002[41](i.e.,for the Spanish and Dutch languages) and CoNLL-2003 [42] (i.e., the English language6)datasets,which contain entities labeled as PER (person), ORG (organization), LOC (location), and MISC (miscellaneous).The CoNLL-2002 dataset contains approximately 12K sentences in Spanish and 24K sentences in Dutch,while CoNLL-2003 includes around 21K sentences in English.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8000000, "bbox": [[0.511, 0.201, 0.884, 0.209], [0.176, 0.214, 0.775, 0.222], [0.779, 0.213, 0.88, 0.222], [0.176, 0.23, 0.886, 0.235], [0.176, 0.241, 0.886, 0.251], [0.176, 0.257, 0.578, 0.264]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 171, "edu_l1_label": "IOS"}, {"txt": "Performance was measured using the F1-score,which balances precision and recall for entity categorization.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8000000, "bbox": [[0.578, 0.257, 0.886, 0.265], [0.175, 0.27, 0.572, 0.277]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 172, "edu_l1_label": "IOS"}, {"txt": "·Natural Language Inference (NLI), whose goal is to assess the model's ability to understand relationships between pairs of sentences, classifying each as entailment, contradiction, or neutral.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8100000, "bbox": [[0.161, 0.295, 0.882, 0.302], [0.175, 0.307, 0.712, 0.316]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 173, "edu_l1_label": "IOS"}, {"txt": "We used the Cross-lingual Natural Language Inference (XNLI) dataset [43], which includes about 400K sentence pairs for each of the 15available languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8100000, "bbox": [[0.712, 0.307, 0.88, 0.316], [0.176, 0.32, 0.881, 0.329], [0.175, 0.336, 0.312, 0.343]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 174, "edu_l1_label": "IOS"}, {"txt": "We used the F1-score as the primary evaluation metric, which balances precision and recall for sentence classification.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8100000, "bbox": [[0.312, 0.336, 0.881, 0.343], [0.176, 0.35, 0.39, 0.357]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 175, "edu_l1_label": "IOS"}, {"txt": "·Question Answering (QA, which measures the ability of the models to provide answers based on a provided context.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8200000, "bbox": [[0.161, 0.374, 0.882, 0.38], [0.176, 0.388, 0.232, 0.4]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 176, "edu_l1_label": "IOS"}, {"txt": "We evaluated UniBERT models on the MultiLingual Question Answering (MLQA) dataset [44],which consists of more than 90K context-question-answer triplets in seven languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8200000, "bbox": [[0.232, 0.388, 0.884, 0.395], [0.175, 0.402, 0.739, 0.408]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 177, "edu_l1_label": "IOS"}, {"txt": "Each input includes a context paragraph and a question, with the goal of identifying the correct text span as the answer.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8200000, "bbox": [[0.739, 0.402, 0.881, 0.408], [0.175, 0.416, 0.794, 0.422]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 178, "edu_l1_label": "IOS"}, {"txt": "We measured performance using F1-scores for partial matches.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8200000, "bbox": [[0.794, 0.416, 0.882, 0.422], [0.175, 0.429, 0.495, 0.435]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 179, "edu_l1_label": "IOS"}, {"txt": "·Semantic Textual Similarity (STS), which measures the degree of semantic similarity between sentence pairs, rated on a scale from 0 (no similarity) to 5 (high similarity).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8300000, "bbox": [[0.159, 0.45, 0.886, 0.558]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 180, "edu_l1_label": "IOS"}, {"txt": "We used the SemEval-2022 Task 8(STS22) dataset [45], which provides over 9K sentence pairs with human-annotated similarity scores in ten languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8300000, "bbox": [[0.159, 0.45, 0.886, 0.558]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 181, "edu_l1_label": "IOS"}, {"txt": "Additionally, because thisdataset contains samples with the two sentences in different languages,we differentiate between fine-tuning the model on the whole dataset, which includes the samples in different languages (case called$\"a$1\" in this paper) and fine-tuning the model only on the samples that contain sentences in the same language (case called \"mono\" in this paper).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8300000, "bbox": [[0.159, 0.45, 0.886, 0.558]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 182, "edu_l1_label": "IOS"}, {"txt": "We evaluated the UniBERT performance using the Pearson correlation coefficient [46], reflecting the alignment between predicted scores and human judgments.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8300000, "bbox": [[0.159, 0.45, 0.886, 0.558]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 183, "edu_l1_label": "IOS"}, {"txt": "Furthermore, we compared the performance of the UniBERT models in these tasks with well-known baselines in the literature:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8400000, "bbox": [[0.117, 0.578, 0.881, 0.586], [0.117, 0.592, 0.184, 0.6]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 184, "edu_l1_label": "IOS"}, {"txt": " mBERT-base, Distil-mBERT-base, and XLM-RoBERTa (base and large variants).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8400000, "bbox": [[0.184, 0.592, 0.718, 0.6]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 185, "edu_l1_label": "IOS"}, {"txt": "5 Results", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8500000, "bbox": [[0.116, 0.632, 0.207, 0.643]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 9, "global_sentence_id": 186, "edu_l1_label": "BOS"}, {"txt": "5.1 Named Entity Recognition", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8600000, "bbox": [[0.116, 0.667, 0.338, 0.676]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 10, "global_sentence_id": 187, "edu_l1_label": "IOS"}, {"txt": "Table 2 summarizes the F1-scores achieved on the CoNLL-2002 and CoNLL-2003 datasets.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8700000, "bbox": [[0.114, 0.694, 0.886, 0.774]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 188, "edu_l1_label": "IOS"}, {"txt": "The results are somewhat mixed.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8700000, "bbox": [[0.114, 0.694, 0.886, 0.774]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 189, "edu_l1_label": "IOS"}, {"txt": "Among the baseline models, XLM-RoBERTa-large achieves the highest performance when individually fine-tuned,reaching F1-scores of$92.70\\%$ (en),$90.05\\%$ (es), and $93.35\\%$ (nl), for an overall average of$92.03\\%$.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8700000, "bbox": [[0.114, 0.694, 0.886, 0.774]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 190, "edu_l1_label": "IOS"}, {"txt": "In contrast,when trained jointly in all languages, its performance drops slightly (i.e., to $91.82\\%$,$88.45\\%$, and 91.77 for an average of $90.67\\%$).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8700000, "bbox": [[0.114, 0.694, 0.886, 0.774]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 191, "edu_l1_label": "IOS"}, {"txt": "Similarly, mBERT-base and XLM-RoBERTa-base show marginally lower scores in the \"all\" configuration than their \"each\" counterparts.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8700000, "bbox": [[0.114, 0.694, 0.886, 0.774]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 192, "edu_l1_label": "IOS"}, {"txt": "The proposed UniBERT models, while inherently more compact, display an interesting trend.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8800000, "bbox": [[0.115, 0.783, 0.886, 0.864]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 193, "edu_l1_label": "IOS"}, {"txt": "For instance, UniBERT-Small achieves an average F1-score of $83.25\\%$ when individually fine-tuned, but this improves to $84.40\\%$ under joint multilingual training.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8800000, "bbox": [[0.115, 0.783, 0.886, 0.864]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 194, "edu_l1_label": "IOS"}, {"txt": "A similar improvement is observed for the even smaller UniBERT-XSmall and UniBERT-XXSmall models, with average scores increasing from $77.25\\%$ to $78.80\\%$ and from $71.64\\%$ to $74.00\\%$, respectively.These gains suggest that the joint training strategy effectively mitigates some of the capacity limitations inherent to more compact architectures, particularly for non-English languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8800000, "bbox": [[0.115, 0.783, 0.886, 0.864]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 195, "edu_l1_label": "IOS"}, {"txt": "6The German subset is not available for public usage, so we did not evaluate the performance of the UniBERT models on this language.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8900000, "bbox": [[0.137, 0.885, 0.884, 0.896], [0.117, 0.901, 0.174, 0.909]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 13, "global_sentence_id": 196, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8900000, "bbox": [[0.493, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 14, "global_sentence_id": 197, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8900000, "bbox": [[0.737, 0.045, 0.882, 0.055]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 198, "edu_l1_label": "EDU_O"}, {"txt": "Table 2: Results of NER on the CoNLL-2002 and CoNLL-2003 datasets. We report the F1-score of the models that were fine-tuned for each language (i.e., case \"each\") and for all languages at the same time (i.e.,case\"all\"),evaluated on each language, together with the average F1-score.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9000000, "bbox": [[0.115, 0.09, 0.885, 0.128]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 199, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9100000, "bbox": [[0.255, 0.142, 0.739, 0.4]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 3, "global_sentence_id": 200, "edu_l1_label": "EDU_O"}, {"txt": "Table 3: Results on XNLI. We report the F1-score of the models that were fine-tuned for each language (i.e., case \"each\") and for all languages at the same time (i.e., case \"all\"), evaluated on each language, together with the average F1-score.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9200000, "bbox": [[0.114, 0.444, 0.885, 0.483]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 4, "global_sentence_id": 201, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9300000, "bbox": [[0.12, 0.485, 0.88, 0.653]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 5, "global_sentence_id": 202, "edu_l1_label": "EDU_O"}, {"txt": "5.2 Natural Language Inference", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9400000, "bbox": [[0.116, 0.706, 0.35, 0.716]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 6, "global_sentence_id": 203, "edu_l1_label": "IOS"}, {"txt": "Table 3 presents the F1-scores on the XNLI dataset.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9500000, "bbox": [[0.116, 0.741, 0.458, 0.75]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 204, "edu_l1_label": "IOS"}, {"txt": "The results are somewhat mixed on the NLI task, with the larger models leading overall performance.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9500000, "bbox": [[0.458, 0.741, 0.883, 0.75], [0.117, 0.758, 0.364, 0.765]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 205, "edu_l1_label": "IOS"}, {"txt": "For instance, XLM-RoBERTa-large achieves an average F1-score of 82.29%when fine-tuned independently in each language, which further increases to 83.30% in joint multilingual training.Likewise,XLM-RoBERTa-base shows a notable improvement from 75.18% to 79.04% when switching from individual to joint training.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9500000, "bbox": [[0.364, 0.758, 0.881, 0.764], [0.117, 0.771, 0.886, 0.778], [0.116, 0.783, 0.882, 0.792], [0.117, 0.798, 0.23, 0.806]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 206, "edu_l1_label": "IOS"}, {"txt": "In contrast, mBERT-base exhibits a moderate increase of an average of 72.64% to 73.37%.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9500000, "bbox": [[0.23, 0.798, 0.844, 0.806]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 207, "edu_l1_label": "IOS"}, {"txt": "Also,Distil-mBERT-base remains relatively stable with averages of 67.68% (i.e., case \"each\"):and 67.74% (i.e.,case \"all\").", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9500000, "bbox": [[0.844, 0.798, 0.886, 0.806], [0.117, 0.811, 0.696, 0.818], [0.694, 0.812, 0.882, 0.819]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 208, "edu_l1_label": "IOS"}, {"txt": "Our UniBERT variants, although more compact, benefit appreciably from the joint training strategy.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9600000, "bbox": [[0.118, 0.831, 0.797, 0.84]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 209, "edu_l1_label": "IOS"}, {"txt": "The average F1-score of UniBERT-Small increases from 61.52% in the \"each\" setting to 65.69% when all languages are considered at once.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9600000, "bbox": [[0.797, 0.831, 0.88, 0.84], [0.117, 0.845, 0.884, 0.854], [0.117, 0.861, 0.154, 0.867]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 210, "edu_l1_label": "IOS"}, {"txt": "The smaller UniBERT-XSmall and UniBERT-XXSmall modelsexperience even more pronounced improvements,with their average F1-scores rising from 54.36% to 61.28%, and from 48.07% to 54.45%, respectively.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9600000, "bbox": [[0.154, 0.861, 0.886, 0.868], [0.118, 0.874, 0.798, 0.881]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 211, "edu_l1_label": "IOS"}, {"txt": "These gains,which range roughly between 4% and 7%, suggest that sharing cross-lingual representations during fine-tuning is especially beneficial for smaller models, helping compensate for their reduced capacity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9600000, "bbox": [[0.798, 0.874, 0.886, 0.882], [0.118, 0.888, 0.882, 0.895], [0.117, 0.902, 0.689, 0.909]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 212, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9700000, "bbox": [[0.494, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 213, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9700000, "bbox": [[0.737, 0.045, 0.882, 0.056]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 214, "edu_l1_label": "EDU_O"}, {"txt": "Table 4: Results on MLQA. We report the F1-score of the models that were fine-tuned for each language (i.e.,case \"each\") and for all languages at the same time (i.e., case \"all\"), evaluated on each language, together with the average F1-score.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9800000, "bbox": [[0.115, 0.098, 0.885, 0.136]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 215, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9900000, "bbox": [[0.145, 0.14, 0.852, 0.396]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 3, "global_sentence_id": 216, "edu_l1_label": "EDU_O"}, {"txt": "5.3 Question Answering", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10000000, "bbox": [[0.116, 0.444, 0.295, 0.455]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 4, "global_sentence_id": 217, "edu_l1_label": "IOS"}, {"txt": "Table 4 reports the F1-scores on the MLQA dataset for each model.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10100000, "bbox": [[0.115, 0.477, 0.885, 0.544]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 218, "edu_l1_label": "IOS"}, {"txt": "The baseline models consistently outperform in absolute terms.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10100000, "bbox": [[0.115, 0.477, 0.885, 0.544]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 219, "edu_l1_label": "IOS"}, {"txt": "For example, XLM-RoBERTa-large achieves an average F1-score of $63.51\\%$ when individually fine-tuned, which further improves to $64.85\\%$under joint training.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10100000, "bbox": [[0.115, 0.477, 0.885, 0.544]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 220, "edu_l1_label": "IOS"}, {"txt": "Similarly, XLM-RoBERTa-base records average scores of $58.93\\%$(i.e.,case \"each\") and $59.97\\%$(i.e., case \"all\"), mBERT-base reaches $57.86\\%$ (i.e., case \"each\") and $59.21\\%$(i.e., case \"all\"), while Distil-mBERT-base achieves $51.95\\%$(i.e., case \"each\") and $55.21\\%$ (i.e., case \"all\").", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10100000, "bbox": [[0.115, 0.477, 0.885, 0.544]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 221, "edu_l1_label": "IOS"}, {"txt": "Our proposed UUniBERT models, despite their reduced parameter count, benefit noticeably from joint multilingual fine-tuning.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10200000, "bbox": [[0.114, 0.553, 0.886, 0.621]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 222, "edu_l1_label": "IOS"}, {"txt": "UniBERT-Small, for example, sees its average F1-score rise from $42.87\\%$ to $44.59\\%$.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10200000, "bbox": [[0.114, 0.553, 0.886, 0.621]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 223, "edu_l1_label": "IOS"}, {"txt": "The improvements are even more pronounced for smaller variants: UniBERT-XSmall increases from $28.43\\%$ to $36.53\\%$, and UniBERT-XXSmall from $17.57\\%$ to $21.33\\%$.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10200000, "bbox": [[0.114, 0.553, 0.886, 0.621]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 224, "edu_l1_label": "IOS"}, {"txt": "These enhancements, which range from approximately $2\\%$ to more than $8\\%$,highlight the advantages of using cross-lingual data during training, particularly for models with limited capacity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10200000, "bbox": [[0.114, 0.553, 0.886, 0.621]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 225, "edu_l1_label": "IOS"}, {"txt": "5.4 Semantic Textual Similarity", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10300000, "bbox": [[0.117, 0.66, 0.348, 0.672]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 7, "global_sentence_id": 226, "edu_l1_label": "IOS"}, {"txt": "Table 5 presents the Pearson correlation coefficients obtained on the STS22 dataset.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10400000, "bbox": [[0.116, 0.694, 0.65, 0.703]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 227, "edu_l1_label": "IOS"}, {"txt": "In this evaluation, we consider three training configurations: fine-tuning on each language individually (i.e., case \"each\"), on all the monolingual subsets simultaneously (i.e., case \"mono\"), and on the entire dataset that includes cross-lingual samples (i.e., case\"all\").Note that results are provided only for the \"all\" and \"mono\" settings for Chinese, Russian, and Italian (i.e., languages for which just a test set is available).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10400000, "bbox": [[0.65, 0.694, 0.882, 0.703], [0.117, 0.708, 0.883, 0.715], [0.118, 0.723, 0.801, 0.729], [0.807, 0.722, 0.881, 0.729], [0.118, 0.735, 0.882, 0.744], [0.117, 0.751, 0.332, 0.757]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 228, "edu_l1_label": "IOS"}, {"txt": "Among the baseline models, mBERT-base achieves an average coefficient of 0.674 when fine-tuned on each language individually.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10500000, "bbox": [[0.114, 0.769, 0.887, 0.836]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 229, "edu_l1_label": "IOS"}, {"txt": "Its performance improves to 0.723 with monolingual training and settles to 0.710 when including cross-lingual data.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10500000, "bbox": [[0.114, 0.769, 0.887, 0.836]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 230, "edu_l1_label": "IOS"}, {"txt": "Similarly, XLM-RoBERTa-large obtains average scores of 0.628, 0.737, and 0.743 in the \"each\",\"mono\",and$\"a$settings,respectively,suggesting that joint training, especially when leveraging cross-lingual samples,can boost semantic similarity performance.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10500000, "bbox": [[0.114, 0.769, 0.887, 0.836]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 231, "edu_l1_label": "IOS"}, {"txt": "In contrast, the UniBERT variants, while more compact, show lower overall correlations.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10600000, "bbox": [[0.117, 0.845, 0.687, 0.854]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 232, "edu_l1_label": "IOS"}, {"txt": "For example, UniBERT-Small records an average of 0.547 in the \"each\" configuration, increasing to 0.614 with monolingual training and 0.624when cross-lingual examples are included.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10600000, "bbox": [[0.687, 0.845, 0.882, 0.854], [0.117, 0.861, 0.881, 0.867], [0.117, 0.874, 0.399, 0.881]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 233, "edu_l1_label": "IOS"}, {"txt": "UniBERT-XSmall and UniBERT-XXSmall follow a similar pattern, with their average coefficients increasing by approximately 0.07-0.08 points when moving from individual to joint training strategies.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10600000, "bbox": [[0.399, 0.874, 0.88, 0.881], [0.117, 0.887, 0.881, 0.896], [0.118, 0.902, 0.185, 0.909]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 234, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10700000, "bbox": [[0.493, 0.936, 0.506, 0.948]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 11, "global_sentence_id": 235, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10700000, "bbox": [[0.735, 0.045, 0.883, 0.055]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 236, "edu_l1_label": "EDU_O"}, {"txt": "Table 5: Results on the STS22 dataset. We report the Pearson correlation coefficients of the models fine-tuned on each language (i.e., case \"each\"), on all the monolingual subsets at the same time (i.e., case \"mono\"), and on the whole dataset, which includes cross-lingual samples (i.e., case \"all\"), evaluated on each language, together with the average Pearson correlation coefficient. The Italian, Russian, and Chinese languages provide only the test set, so we do not give any results on the \"each\" category for them.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10800000, "bbox": [[0.115, 0.119, 0.885, 0.188]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 237, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10900000, "bbox": [[0.117, 0.186, 0.877, 0.497]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 3, "global_sentence_id": 238, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10900000, "bbox": [[0.159, 0.559, 0.85, 0.845]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 4, "global_sentence_id": 239, "edu_l1_label": "EDU_O"}, {"txt": "Figure 3: Relative mean improvements in UniBERT and other models when trained on individual languages (i.e.\"each\")as opposed to all languages (i.e.\"all\").", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11000000, "bbox": [[0.115, 0.861, 0.885, 0.887]]}]}, "tags": ["figure_caption"], "label": "figure_title", "web_segment_id": 5, "global_sentence_id": 240, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11100000, "bbox": [[0.492, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 6, "global_sentence_id": 241, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11100000, "bbox": [[0.737, 0.045, 0.883, 0.055]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 242, "edu_l1_label": "EDU_O"}, {"txt": "5.5 Discussion", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11200000, "bbox": [[0.116, 0.094, 0.222, 0.103]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 2, "global_sentence_id": 243, "edu_l1_label": "IOS"}, {"txt": "The results indicate that the multilingual training strategy, in conjunction with the adversarial objective, has a pronounced impact on overall performance.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11300000, "bbox": [[0.114, 0.12, 0.886, 0.225]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 244, "edu_l1_label": "IOS"}, {"txt": "Figure 3 illustrates the relative improvements achieved by each model when transitioning from individual language fine-tuning (denoted as \"each\") to joint multilingual fine-tuning (denoted as$\"a\")7$.mBERT-base, Distil-mBERT-base, and the XLM-RoBERTa versions exhibit a moderate average relative improvement of approximately $1.17\\%$: $1.51\\%$ for Distil-mBERT-base,$0.62\\%$ for mBERT-base,$2.11\\%$ for XLM-RoBERTa-base,and $0.44\\%$ for XLM-RoBERTa-large.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11300000, "bbox": [[0.114, 0.12, 0.886, 0.225]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 245, "edu_l1_label": "IOS"}, {"txt": "In contrast, UniBERT models indicate significantly greater gains, with an average relative improvement of $7.72\\%$: $3.76\\%$ for UniBERT-small,$10.33\\%$ for uniBERT-XSmall, and $9.08\\%$ for UniBERT-XXSmall.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11300000, "bbox": [[0.114, 0.12, 0.886, 0.225]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 246, "edu_l1_label": "IOS"}, {"txt": "This significant difference suggests that the adversarial training component, which promotes the learning of language-agnostic representations, is particularly effective when combined with joint multilingual training.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11400000, "bbox": [[0.114, 0.236, 0.886, 0.317]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 247, "edu_l1_label": "IOS"}, {"txt": "For example, the UniBERT-XSmall model improves from $40.20\\%$ in the \"each\" configuration to $44.35\\%$ in the \"\"all\"\"configuration,and the UniBERT-XXSmall model similarly increases from $34.51\\%$ to $37.64\\%$.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11400000, "bbox": [[0.114, 0.236, 0.886, 0.317]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 248, "edu_l1_label": "IOS"}, {"txt": "Such improvements are notably higher than those observed in larger models, where the increase is relatively modest (e.g., XLM-RoBERTa-base and XLM-RoBERTa-large improve by less than 1.2 and 0.3 percentage points, respectively).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11400000, "bbox": [[0.114, 0.236, 0.886, 0.317]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 249, "edu_l1_label": "IOS"}, {"txt": "Furthermore, a statistical t-test resulted in a t-statistic of 2.715 with a p-value of 0.0181 when comparing the relative improvements between the UniBERT models (12 entries, average improvement of $7.72\\%$) and the other models (16entries, average improvement of $1.17\\%$).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11500000, "bbox": [[0.115, 0.326, 0.885, 0.407]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 250, "edu_l1_label": "IOS"}, {"txt": "The observed differences are statistically significant because the $P$ value is lower than the significance threshold of$α=0$ $05$.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11500000, "bbox": [[0.115, 0.326, 0.885, 0.407]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 251, "edu_l1_label": "IOS"}, {"txt": "This analysis underscores that combining joint multilingual training with the adversarial objective enhances absolute performance and significantly benefits models with limited capacity by effectively leveraging cross-lingual representations.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11500000, "bbox": [[0.115, 0.326, 0.885, 0.407]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 252, "edu_l1_label": "IOS"}, {"txt": "6 Conclusion", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.117, 0.431, 0.239, 0.442]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 6, "global_sentence_id": 253, "edu_l1_label": "BOS"}, {"txt": "In this paper, we provide UniBERT, a multilingual language model that employs a novel training framework consisting of masked language modeling, adversarial training, and knowledge distillation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11700000, "bbox": [[0.115, 0.462, 0.886, 0.584]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 254, "edu_l1_label": "IOS"}, {"txt": "Our extensive evaluations of four NLP tasks (i.e., named entity recognition, natural language inference, question answering, and semantic textual similarity) demonstrate that the proposed approach achieves competitive performance relative to established baselines and benefits significantly from joint multilingual training.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11700000, "bbox": [[0.115, 0.462, 0.886, 0.584]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 255, "edu_l1_label": "IOS"}, {"txt": "The UniBERT variants, for example, obtained an average relative improvement of$7.72\\%$ between tasks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11700000, "bbox": [[0.115, 0.462, 0.886, 0.584]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 256, "edu_l1_label": "IOS"}, {"txt": "However, the larger baseline models showed an improvement of$1.17\\%$.Integrating the adversarial objective has proven particularly effective in fostering language-invariant representations,enhancing cross-lingual generalization, and producing statistically significant gains (i.e., p-valu$\\mathrm {}=0$.0181)over models trained in individual languages when trained jointly on multilingual data.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11700000, "bbox": [[0.115, 0.462, 0.886, 0.584]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 257, "edu_l1_label": "IOS"}, {"txt": "The promising results of UniBERT underscore the potential of our methodology in addressing the challenges associated with multilingual processing in resource-constrained settings.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11800000, "bbox": [[0.116, 0.594, 0.881, 0.602], [0.117, 0.609, 0.511, 0.616]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 258, "edu_l1_label": "IOS"}, {"txt": "Although our model achieves a favorable balance between efficiency and performance, there remains ample scope for future work, including further refinement of adversarial strategies, exploration of alternative distillation techniques, and extension to additional languages and domains.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11800000, "bbox": [[0.511, 0.609, 0.881, 0.616], [0.117, 0.623, 0.882, 0.629], [0.118, 0.637, 0.83, 0.643]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 259, "edu_l1_label": "IOS"}, {"txt": "Overall,the contributions of this work provide a robust foundation for fuiture research in efficient multilingual modeling and adaptable and scalable NLP systems.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11800000, "bbox": [[0.83, 0.637, 0.886, 0.648], [0.118, 0.65, 0.882, 0.657], [0.117, 0.665, 0.359, 0.672]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 260, "edu_l1_label": "IOS"}, {"txt": "7 Limitations", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11900000, "bbox": [[0.116, 0.697, 0.243, 0.708]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 9, "global_sentence_id": 261, "edu_l1_label": "BOS"}, {"txt": "Despite the promising results achieved by UniBERT, our approach has certain limitations, primarily due to computational constraints that influenced our model design choices.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.117, 0.729, 0.882, 0.738], [0.117, 0.746, 0.478, 0.752]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 262, "edu_l1_label": "IOS"}, {"txt": "To ensure feasibility with available resources, we opted for compact architectures with fewer parameters than the larger models used for comparison.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.478, 0.746, 0.882, 0.753], [0.117, 0.759, 0.699, 0.765]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 263, "edu_l1_label": "IOS"}, {"txt": "Although this design allows efficiency and scalability, it limits the model's capacity to learn and retain complex linguistic representations.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.699, 0.759, 0.881, 0.765], [0.117, 0.772, 0.85, 0.779]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 264, "edu_l1_label": "IOS"}, {"txt": "As a result, UniBERT exhibits lower absolute performance, particularly in tasks that require deep contextual understanding,such as question answering and semantic textual similarity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.85, 0.772, 0.881, 0.78], [0.117, 0.787, 0.886, 0.794], [0.118, 0.801, 0.519, 0.807]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 265, "edu_l1_label": "IOS"}, {"txt": "However, our findings indicate that joint multilingual training and adversarial learning effectively compensate for reduced capacity, allowing UniBERT to achieve competitive results despite its smaller size.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.519, 0.801, 0.882, 0.807], [0.118, 0.814, 0.881, 0.821], [0.117, 0.828, 0.317, 0.835]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 266, "edu_l1_label": "IOS"}, {"txt": "Although adversarial training and multilingual learning help mitigate the drawbacks of a compact model, they do not eliminate the inherent limitations of reduced parameterization.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12100000, "bbox": [[0.117, 0.847, 0.881, 0.855], [0.117, 0.862, 0.566, 0.869]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 267, "edu_l1_label": "IOS"}, {"txt": "Larger models retain an advantage in absolute performance due to their ability to capture more intricate linguistic structures.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12100000, "bbox": [[0.566, 0.862, 0.881, 0.869], [0.117, 0.877, 0.638, 0.883]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 268, "edu_l1_label": "IOS"}, {"txt": "However, ouir methodology remains", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12100000, "bbox": [[0.638, 0.877, 0.882, 0.883]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 269, "edu_l1_label": "IOS"}, {"txt": "⁷Since the score on the STS isbetween 0 and 1, we normalized it before calculating the mean relative improvement.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12200000, "bbox": [[0.137, 0.898, 0.818, 0.909]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 270, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.492, 0.938, 0.508, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 13, "global_sentence_id": 271, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.737, 0.045, 0.882, 0.055]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 272, "edu_l1_label": "EDU_O"}, {"txt": "fundamentally sound, demonstrating that efficient multilingual training strategies can yield strong cross-lingual generalization even with fewer resources.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.117, 0.095, 0.882, 0.102], [0.117, 0.111, 0.402, 0.117]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 273, "edu_l1_label": "IOS"}, {"txt": "This balance between efficiency and performance makes UniBERT a compellingsolution for resource-constrained applications.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.402, 0.111, 0.881, 0.117], [0.117, 0.124, 0.492, 0.131]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 274, "edu_l1_label": "IOS"}, {"txt": "Future work could explore scaling UniBERT with additional computational resources to close the performance gap further while maintaining its efficiency benefits.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.492, 0.124, 0.879, 0.131], [0.117, 0.138, 0.786, 0.145]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 275, "edu_l1_label": "IOS"}, {"txt": "Acknowledgements", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12400000, "bbox": [[0.117, 0.17, 0.279, 0.181]]}]}, "tags": ["title"], "label": "content", "web_segment_id": 3, "global_sentence_id": 276, "edu_l1_label": "EDU_O"}, {"txt": "This work was supported by the National University of Science and Technology POLITEHNICA Bucharest through the PubArt program.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12500000, "bbox": [[0.116, 0.203, 0.88, 0.211], [0.118, 0.217, 0.256, 0.225]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 277, "edu_l1_label": "EDU_O"}, {"txt": "DCC is funded by the National Program for Research of the National Association of Technical Universities (GNAC ARUT 2023).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12500000, "bbox": [[0.256, 0.217, 0.882, 0.225], [0.118, 0.23, 0.345, 0.239]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 278, "edu_l1_label": "EDU_O"}, {"txt": "References", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12600000, "bbox": [[0.117, 0.265, 0.208, 0.275]]}]}, "tags": ["title"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 279, "edu_l1_label": "EDU_O"}, {"txt": "[1] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12700000, "bbox": [[0.13, 0.297, 0.881, 0.306], [0.153, 0.31, 0.262, 0.319]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 280, "edu_l1_label": "EDU_O"}, {"txt": "Attention is all you need.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12700000, "bbox": [[0.262, 0.31, 0.433, 0.319]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 281, "edu_l1_label": "EDU_O"}, {"txt": "Advances in neural information processing systems, 30,2017.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12700000, "bbox": [[0.433, 0.31, 0.841, 0.319]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 282, "edu_l1_label": "EDU_O"}, {"txt": "[2] Jacob Devlin, Ming-Wei Chang,Kenton Lee, and Kristina Toutanova.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12800000, "bbox": [[0.127, 0.33, 0.613, 0.34]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 283, "edu_l1_label": "EDU_O"}, {"txt": "BERT: Pre-training of deep bidirectional transformers for language understanding.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12800000, "bbox": [[0.613, 0.33, 0.882, 0.34], [0.153, 0.347, 0.417, 0.354]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 284, "edu_l1_label": "EDU_O"}, {"txt": "In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186,Minneapolis, Minnesota, June 2019.Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12800000, "bbox": [[0.417, 0.347, 0.882, 0.353], [0.152, 0.36, 0.88, 0.367], [0.152, 0.372, 0.881, 0.381], [0.152, 0.386, 0.476, 0.395]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 285, "edu_l1_label": "EDU_O"}, {"txt": "[3] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen,Omer Levy,Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12900000, "bbox": [[0.126, 0.405, 0.881, 0.414], [0.152, 0.419, 0.391, 0.427]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 286, "edu_l1_label": "EDU_O"}, {"txt": "Roberta: A robustly optimized bert pretraining approach.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12900000, "bbox": [[0.391, 0.419, 0.782, 0.427]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 287, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:1907.11692,2019.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12900000, "bbox": [[0.782, 0.419, 0.882, 0.428], [0.152, 0.436, 0.318, 0.442]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 288, "edu_l1_label": "EDU_O"}, {"txt": "[4] Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13000000, "bbox": [[0.125, 0.452, 0.641, 0.461]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 289, "edu_l1_label": "EDU_O"}, {"txt": "Electra: Pre-training text encoders as discriminators rather than generators.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13000000, "bbox": [[0.641, 0.452, 0.882, 0.461], [0.152, 0.467, 0.398, 0.475]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 290, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2003.10555,2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13000000, "bbox": [[0.398, 0.467, 0.664, 0.475]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 291, "edu_l1_label": "EDU_O"}, {"txt": "[5] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán,Edouard Grave, Myle Ott,Luke Zettlemoyer, and Veselin Stoyanov.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.129, 0.485, 0.884, 0.496], [0.153, 0.5, 0.602, 0.508]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 292, "edu_l1_label": "EDU_O"}, {"txt": "Unsupervised cross-lingual representation learning at scale.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.602, 0.5, 0.881, 0.508], [0.153, 0.514, 0.266, 0.522]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 293, "edu_l1_label": "EDU_O"}, {"txt": "In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451,Online,July 2020.Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.266, 0.514, 0.881, 0.523], [0.153, 0.528, 0.884, 0.537], [0.152, 0.542, 0.433, 0.55]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 294, "edu_l1_label": "EDU_O"}, {"txt": "[6] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13200000, "bbox": [[0.124, 0.561, 0.88, 0.57], [0.153, 0.575, 0.245, 0.584]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 11, "global_sentence_id": 295, "edu_l1_label": "EDU_O"}, {"txt": "mT5: A massively multilingual pre-trained text-to-text transformer.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13200000, "bbox": [[0.245, 0.575, 0.721, 0.584]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 11, "global_sentence_id": 296, "edu_l1_label": "EDU_O"}, {"txt": "In Kristina Toutanova,Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty,and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498,Online,June 2021.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13200000, "bbox": [[0.721, 0.575, 0.884, 0.585], [0.152, 0.59, 0.881, 0.599], [0.152, 0.602, 0.883, 0.612], [0.152, 0.617, 0.881, 0.625], [0.153, 0.63, 0.191, 0.639]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 11, "global_sentence_id": 297, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13200000, "bbox": [[0.191, 0.63, 0.476, 0.639]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 11, "global_sentence_id": 298, "edu_l1_label": "EDU_O"}, {"txt": "[7] Chihiro Yano, Akihiko Fukuchi, Shoko Fukasawa, Hideyuki Tachibana, and Yotaro Watanabe.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13300000, "bbox": [[0.127, 0.649, 0.792, 0.658]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 12, "global_sentence_id": 299, "edu_l1_label": "EDU_O"}, {"txt": "Multilingual sentence-t5: Scalable sentence encoders for multilingual applications.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13300000, "bbox": [[0.792, 0.649, 0.882, 0.657], [0.153, 0.666, 0.611, 0.672]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 12, "global_sentence_id": 300, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 2024 Joint Interna-tional Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024),pages 11849-11858,2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13300000, "bbox": [[0.611, 0.666, 0.883, 0.672], [0.152, 0.678, 0.883, 0.687], [0.153, 0.694, 0.328, 0.699]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 12, "global_sentence_id": 301, "edu_l1_label": "EDU_O"}, {"txt": "[8] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.125, 0.71, 0.536, 0.719]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 13, "global_sentence_id": 302, "edu_l1_label": "EDU_O"}, {"txt": "Explaining and harnessing adversarial examples.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.536, 0.71, 0.864, 0.719]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 13, "global_sentence_id": 303, "edu_l1_label": "EDU_O"}, {"txt": "In International Conference on Learning Representations, 2015.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.864, 0.71, 0.881, 0.72], [0.152, 0.724, 0.555, 0.734]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 13, "global_sentence_id": 304, "edu_l1_label": "EDU_O"}, {"txt": "[9] Andrei-Marius Avram, Verginica Barbu Mititelu, Vasile Pǎis, Dumitru-Clementin Cercel, and Stefan Trǎusan-Matu.Multilingual multiword expression identification using lateral inhibition and domain adaptation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.125, 0.744, 0.883, 0.753], [0.153, 0.758, 0.791, 0.766]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 14, "global_sentence_id": 305, "edu_l1_label": "EDU_O"}, {"txt": "Mathematics,11(11):2548,2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.791, 0.758, 0.884, 0.768], [0.152, 0.771, 0.278, 0.78]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 14, "global_sentence_id": 306, "edu_l1_label": "EDU_O"}, {"txt": "[10] Andrei-Marius Avram, Mircea Timpuriu, Andreea Iuga, Vlad-Cristian Matei, Iulian-Marius Taiatu, Tudor Gǎinǎ,Dumitru1-Clementin Cercel, Mihaela-Claudia Cercel, and Florin Pop.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.116, 0.791, 0.886, 0.801], [0.162, 0.806, 0.593, 0.814]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 15, "global_sentence_id": 307, "edu_l1_label": "EDU_O"}, {"txt": "Rolargesum: A large dialect-aware romanian news dataset for summary, headline, and keyword generation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.593, 0.806, 0.88, 0.814], [0.152, 0.821, 0.547, 0.828]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 15, "global_sentence_id": 308, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 31st International Conference on Computational Linguistics, pages 2049-2066,2025.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.547, 0.821, 0.881, 0.828], [0.152, 0.835, 0.515, 0.842]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 15, "global_sentence_id": 309, "edu_l1_label": "EDU_O"}, {"txt": "[11] Yuemei Xu,Ling Hu, Jiayi Zhao,Zihan Qiu,Kexin XU,Yuqi Ye,and Hanwen Gu.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13500000, "bbox": [[0.116, 0.852, 0.684, 0.861]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 16, "global_sentence_id": 310, "edu_l1_label": "EDU_O"}, {"txt": "A survey on multilingual large language models: Corpora, alignment, and bias.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13500000, "bbox": [[0.684, 0.852, 0.882, 0.862], [0.153, 0.866, 0.466, 0.876]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 16, "global_sentence_id": 311, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2404.00929,2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13500000, "bbox": [[0.466, 0.866, 0.732, 0.876]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 16, "global_sentence_id": 312, "edu_l1_label": "EDU_O"}, {"txt": "[12]Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.116, 0.886, 0.471, 0.895]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 17, "global_sentence_id": 313, "edu_l1_label": "EDU_O"}, {"txt": "Distilling the knowledge in a neural network.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.471, 0.886, 0.781, 0.895]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 17, "global_sentence_id": 314, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:1503.02531,2015.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.781, 0.886, 0.883, 0.895], [0.152, 0.902, 0.317, 0.909]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 17, "global_sentence_id": 315, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13700000, "bbox": [[0.492, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 18, "global_sentence_id": 316, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13700000, "bbox": [[0.737, 0.045, 0.882, 0.055]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 317, "edu_l1_label": "EDU_O"}, {"txt": "[13] Vishrav Chaudhary,Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13800000, "bbox": [[0.116, 0.094, 0.743, 0.104]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 318, "edu_l1_label": "EDU_O"}, {"txt": "Low-resource corpus filtering using multilingual sentence embeddings.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13800000, "bbox": [[0.743, 0.094, 0.882, 0.104], [0.153, 0.109, 0.467, 0.117]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 319, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 261-266,2019.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13800000, "bbox": [[0.467, 0.109, 0.88, 0.117], [0.151, 0.122, 0.56, 0.131]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 320, "edu_l1_label": "EDU_O"}, {"txt": "[14] Felix A Gers, Jürgen Schmidhuber, and Fred Cummins.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13900000, "bbox": [[0.117, 0.141, 0.507, 0.149]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 321, "edu_l1_label": "EDU_O"}, {"txt": "Learning to forget: Continual prediction with lstm.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13900000, "bbox": [[0.507, 0.141, 0.836, 0.149]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 322, "edu_l1_label": "EDU_O"}, {"txt": "Neural computation,12(10):2451-2471,2000.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13900000, "bbox": [[0.836, 0.141, 0.884, 0.149], [0.152, 0.157, 0.408, 0.163]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 323, "edu_l1_label": "EDU_O"}, {"txt": "[15] Nils Reimers and Iryna Gurevych.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14000000, "bbox": [[0.116, 0.173, 0.385, 0.182]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 324, "edu_l1_label": "EDU_O"}, {"txt": "Making monolingual sentence embeddings multilingual using knowledge distillation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14000000, "bbox": [[0.385, 0.173, 0.882, 0.183], [0.153, 0.189, 0.23, 0.196]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 325, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512-4525,2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14000000, "bbox": [[0.23, 0.189, 0.881, 0.198], [0.151, 0.2, 0.385, 0.21]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 326, "edu_l1_label": "EDU_O"}, {"txt": "[16] Mikel Artetxe and Holger Schwenk.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.116, 0.219, 0.398, 0.229]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 327, "edu_l1_label": "EDU_O"}, {"txt": "Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.398, 0.219, 0.882, 0.228], [0.151, 0.234, 0.288, 0.242]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 328, "edu_l1_label": "EDU_O"}, {"txt": "Transactions of the Association for Computational Linguistics, 7:597-610,2019.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.288, 0.234, 0.819, 0.242]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 329, "edu_l1_label": "EDU_O"}, {"txt": "[17] Alexis Conneau and Guillaume Lample.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.116, 0.252, 0.408, 0.261]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 330, "edu_l1_label": "EDU_O"}, {"txt": "Cross-lingual language model pretraining.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.408, 0.252, 0.681, 0.261]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 331, "edu_l1_label": "EDU_O"}, {"txt": "Advances in neural information processing systems,32,2019.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.681, 0.252, 0.881, 0.261], [0.151, 0.268, 0.346, 0.274]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 332, "edu_l1_label": "EDU_O"}, {"txt": "[18] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.116, 0.284, 0.88, 0.294], [0.151, 0.298, 0.341, 0.307]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 333, "edu_l1_label": "EDU_O"}, {"txt": "CCNet: Extracting high quality monolingual datasets from web crawl data.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.341, 0.298, 0.861, 0.307]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 334, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003-4012,Marseille,France,May 2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.861, 0.298, 0.881, 0.307], [0.152, 0.313, 0.884, 0.323], [0.153, 0.326, 0.225, 0.335]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 335, "edu_l1_label": "EDU_O"}, {"txt": "European Language Resources Association.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.225, 0.326, 0.516, 0.335]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 336, "edu_l1_label": "EDU_O"}, {"txt": "[19] Colin Raffel,Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14300000, "bbox": [[0.116, 0.344, 0.883, 0.353], [0.153, 0.359, 0.282, 0.367]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 337, "edu_l1_label": "EDU_O"}, {"txt": "Exploring the limits of transfer learning with a unified text-to-text transformer.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14300000, "bbox": [[0.282, 0.359, 0.81, 0.367]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 338, "edu_l1_label": "EDU_O"}, {"txt": "Journal of machine learning research,21(140):1-67,2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14300000, "bbox": [[0.81, 0.359, 0.883, 0.369], [0.152, 0.374, 0.469, 0.381]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 339, "edu_l1_label": "EDU_O"}, {"txt": "[20] Zhuoyuan Mao,Chenhui Chu, and Sadao Kurohashi.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.116, 0.391, 0.493, 0.4]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 340, "edu_l1_label": "EDU_O"}, {"txt": "Ems: Efficient and effective massively multilingual sentence embedding learning.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.493, 0.391, 0.881, 0.4], [0.152, 0.407, 0.288, 0.414]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 341, "edu_l1_label": "EDU_O"}, {"txt": "IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.288, 0.407, 0.794, 0.414]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 342, "edu_l1_label": "EDU_O"}, {"txt": "[21] Yaushian Wang, Ashley Wu, and Graham Neubig.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14500000, "bbox": [[0.116, 0.423, 0.481, 0.432]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 343, "edu_l1_label": "EDU_O"}, {"txt": "English contrastive learning can learn universal cross-lingual sentence embeddings.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14500000, "bbox": [[0.481, 0.423, 0.882, 0.432], [0.153, 0.44, 0.304, 0.446]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 344, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9122-9133,2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14500000, "bbox": [[0.304, 0.44, 0.882, 0.446], [0.152, 0.451, 0.392, 0.46]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 345, "edu_l1_label": "EDU_O"}, {"txt": "[22] Tianyu Gao, Xingcheng Yao, and Danqi Chen.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14600000, "bbox": [[0.116, 0.469, 0.458, 0.479]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 11, "global_sentence_id": 346, "edu_l1_label": "EDU_O"}, {"txt": "Simcse: Simple contrastive learning of sentence embeddings.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14600000, "bbox": [[0.458, 0.469, 0.866, 0.484]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 11, "global_sentence_id": 347, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 2021 Conference on Empirical MMethods in Natural Language Processing, pages 6894-6910,2021.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14600000, "bbox": [[0.866, 0.469, 0.881, 0.478], [0.152, 0.484, 0.884, 0.493], [0.153, 0.497, 0.19, 0.506]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 11, "global_sentence_id": 348, "edu_l1_label": "EDU_O"}, {"txt": "[23] Buqing Cai, Shengwei Tian, Long Yu, Jun Long, Tiejun Zhou, and Bo Wang.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14700000, "bbox": [[0.116, 0.516, 0.651, 0.525]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 12, "global_sentence_id": 349, "edu_l1_label": "EDU_O"}, {"txt": "Atbbc: Named entity recognition in emergency domains based on joint bert-bilstm-crf adversarial training.Journal of Intelligent & Fuzzy Systems,(Preprint):1-14,2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14700000, "bbox": [[0.651, 0.516, 0.882, 0.525], [0.152, 0.532, 0.884, 0.54], [0.151, 0.544, 0.299, 0.552]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 12, "global_sentence_id": 350, "edu_l1_label": "EDU_O"}, {"txt": "[24] Xia Lina, Askar Hamdulla, Mijit Ablimit, and Wang Sijie.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.119, 0.562, 0.547, 0.571]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 13, "global_sentence_id": 351, "edu_l1_label": "EDU_O"}, {"txt": "Adversarial training for uncertainty estimation in cross-lingual text classification.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.547, 0.562, 0.882, 0.571], [0.152, 0.579, 0.358, 0.585]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 13, "global_sentence_id": 352, "edu_l1_label": "EDU_O"}, {"txt": "In 2024 International Joint Conference on Neural Networks (IJCNN),pages 1-7.IEEE,2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.358, 0.579, 0.884, 0.585], [0.153, 0.59, 0.234, 0.599]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 13, "global_sentence_id": 353, "edu_l1_label": "EDU_O"}, {"txt": "[25] Andrei Avram, Verginica Barbu MMititelu, and Dumitru-Clementin Cercel.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.118, 0.608, 0.654, 0.617]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 14, "global_sentence_id": 354, "edu_l1_label": "EDU_O"}, {"txt": "Romanian multiword expression detection using multilingual adversarial training and lateral inhibition.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.654, 0.608, 0.882, 0.617], [0.152, 0.623, 0.613, 0.631]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 14, "global_sentence_id": 355, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 19th Workshop on Multiword Expressions (MWE 2023),pages 7-13,2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.613, 0.623, 0.881, 0.631], [0.151, 0.636, 0.519, 0.645]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 14, "global_sentence_id": 356, "edu_l1_label": "EDU_O"}, {"txt": "[26] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.118, 0.655, 0.882, 0.664], [0.153, 0.669, 0.356, 0.677]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 15, "global_sentence_id": 357, "edu_l1_label": "EDU_O"}, {"txt": "Domain-adversarial training of neural networks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.356, 0.669, 0.686, 0.677]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 15, "global_sentence_id": 358, "edu_l1_label": "EDU_O"}, {"txt": "Journal of machine learning research,17(59):1-35,2016.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.686, 0.669, 0.882, 0.678], [0.153, 0.685, 0.342, 0.691]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 15, "global_sentence_id": 359, "edu_l1_label": "EDU_O"}, {"txt": "[27] Anthony Sicilia, Xingchen Zhao, and Seong Jae Hwang.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.119, 0.701, 0.545, 0.71]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 16, "global_sentence_id": 360, "edu_l1_label": "EDU_O"}, {"txt": "Domain adversarial neural networks for domain generalization: When it works and how to improve.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.545, 0.701, 0.881, 0.71], [0.153, 0.718, 0.491, 0.723]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 16, "global_sentence_id": 361, "edu_l1_label": "EDU_O"}, {"txt": "Machine Learning, 112(7):2685-2721,2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.491, 0.718, 0.791, 0.723]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 16, "global_sentence_id": 362, "edu_l1_label": "EDU_O"}, {"txt": "[28] Andrei-Marius Avram, Andreea Iuga, George-Vlad Manolache, Vlad-Cristian Matei, Rǎzvan-Gabriel Micliuş,Vlad-Andrei Muntean, Manuel-Petru Sorlescu, Drago-Andrei Şerban, Adrian-Dinu Urse,Vasile Pǎiş, et al.Histnero:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15200000, "bbox": [[0.118, 0.733, 0.886, 0.743], [0.153, 0.747, 0.886, 0.756], [0.153, 0.761, 0.217, 0.77]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 17, "global_sentence_id": 363, "edu_l1_label": "EDU_O"}, {"txt": " Historical named entity recognition for the romanian language.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15200000, "bbox": [[0.217, 0.761, 0.66, 0.77]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 17, "global_sentence_id": 364, "edu_l1_label": "EDU_O"}, {"txt": "In International Conference on Document Analysis and Recognition, pages 126-144.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15200000, "bbox": [[0.66, 0.761, 0.881, 0.77], [0.152, 0.775, 0.502, 0.784]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 17, "global_sentence_id": 365, "edu_l1_label": "EDU_O"}, {"txt": "Springer,2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15200000, "bbox": [[0.502, 0.775, 0.605, 0.784]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 17, "global_sentence_id": 366, "edu_l1_label": "EDU_O"}, {"txt": "[29] Stefan Dumitrescu, Andrei-Marius Avram, and Sampo Pyysalo.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15300000, "bbox": [[0.116, 0.794, 0.576, 0.802]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 18, "global_sentence_id": 367, "edu_l1_label": "EDU_O"}, {"txt": "The birth of romanian bert.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15300000, "bbox": [[0.576, 0.794, 0.761, 0.802]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 18, "global_sentence_id": 368, "edu_l1_label": "EDU_O"}, {"txt": "In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4324-4328,2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15300000, "bbox": [[0.761, 0.794, 0.882, 0.803], [0.15, 0.808, 0.697, 0.816]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 18, "global_sentence_id": 369, "edu_l1_label": "EDU_O"}, {"txt": "[30] Andrei-Marius Avram, Darius Catrina, Dumitru-Clementin Cercel, Mihai Dascalu, Traian Rebedea, Vasile Pǎis,and Dan Tufis.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.117, 0.826, 0.884, 0.836], [0.153, 0.843, 0.257, 0.849]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 19, "global_sentence_id": 370, "edu_l1_label": "EDU_O"}, {"txt": "Distilling the knowledge of romanian berts using multiple teachers.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.257, 0.843, 0.727, 0.849]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 19, "global_sentence_id": 371, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 374-384,2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.727, 0.843, 0.881, 0.849], [0.152, 0.854, 0.693, 0.862]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 19, "global_sentence_id": 372, "edu_l1_label": "EDU_O"}, {"txt": "[31] Rǎzvan-Alexandru Smǎdu, Ion-Robert Dinicǎ, Andrei-Marius Avram, Dumitru-Clementin Cercel, Florin Pop, and Mihaela-Claudia Cercel.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.119, 0.872, 0.882, 0.881], [0.153, 0.886, 0.316, 0.895]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 20, "global_sentence_id": 373, "edu_l1_label": "EDU_O"}, {"txt": "Legal named entity recognition with multi-task domain adaptation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.316, 0.886, 0.761, 0.895]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 20, "global_sentence_id": 374, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 305-321,2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.761, 0.886, 0.883, 0.896], [0.152, 0.9, 0.668, 0.909]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 20, "global_sentence_id": 375, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 21, "global_sentence_id": 376, "edu_l1_label": "EDU_O"}, {"txt": "header", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.737, 0.045, 0.882, 0.056]]}]}, "tags": ["header"], "label": "O", "web_segment_id": 1, "global_sentence_id": 377, "edu_l1_label": "EDU_O"}, {"txt": "[32] Sara Meftah, Nasredine Semmar, Mohamed-Ayoub Tahiri, Youssef Tamaazousti, Hassane Essafi, and Fatiha Sadat.Multi-task supervised pretraining for neural domain adaptation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.117, 0.094, 0.886, 0.104], [0.153, 0.109, 0.561, 0.117]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 378, "edu_l1_label": "EDU_O"}, {"txt": "In The Eighth International Workshop on Natural Language Processing for Social Media, pages 61-71.Association for Computational Linguistics, 2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.561, 0.109, 0.882, 0.117], [0.152, 0.122, 0.828, 0.131]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 379, "edu_l1_label": "EDU_O"}, {"txt": "[33] Rǎzvan-Alexandru Smǎdu, George-Eduard Zaharia, Andrei-Marius Avram, Dumitru-Clementin Cercel, Mihai Dascalu, and Florin Pop.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.118, 0.141, 0.882, 0.15], [0.154, 0.155, 0.328, 0.163]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 380, "edu_l1_label": "EDU_O"}, {"txt": "Ta-da: topic-aware domain adaptation for scientific keyphrase identification and classification (student abstract).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.328, 0.155, 0.882, 0.164], [0.152, 0.169, 0.364, 0.177]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 381, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37,pages 16334-16335,2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.364, 0.169, 0.884, 0.178], [0.153, 0.184, 0.328, 0.191]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 382, "edu_l1_label": "EDU_O"}, {"txt": "[34] Khoi M Le, Trinh Pham, Tho Quan, and Anh Tuan Luu.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15900000, "bbox": [[0.117, 0.201, 0.51, 0.21]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 383, "edu_l1_label": "EDU_O"}, {"txt": "Lampat: Low-rank adaption for multilingual paraphrasing using adversarial training.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15900000, "bbox": [[0.51, 0.201, 0.882, 0.211], [0.153, 0.217, 0.328, 0.223]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 384, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38,pages 18435-18443,2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15900000, "bbox": [[0.328, 0.217, 0.881, 0.223], [0.153, 0.229, 0.288, 0.237]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 385, "edu_l1_label": "EDU_O"}, {"txt": "[35] Qichen Zhang, Shuai Wang, and Jingmei Li.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.116, 0.247, 0.435, 0.256]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 386, "edu_l1_label": "EDU_O"}, {"txt": "Multilingual mixture attention interaction framework with adversarial training for cross-lingual slu.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.435, 0.247, 0.882, 0.256], [0.153, 0.263, 0.343, 0.27]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 387, "edu_l1_label": "EDU_O"}, {"txt": "Neural Computing and Applications, 36(4):1915-1930,2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.343, 0.263, 0.75, 0.27]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 388, "edu_l1_label": "EDU_O"}, {"txt": "[36] Wikimedia Foundation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.118, 0.28, 0.309, 0.289]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 389, "edu_l1_label": "EDU_O"}, {"txt": "Wikimedia downloads, https://dumps.wikimedia.org [accessed: March 2025].", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.309, 0.28, 0.819, 0.289]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 390, "edu_l1_label": "EDU_O"}, {"txt": "[37] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.118, 0.299, 0.591, 0.307]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 391, "edu_l1_label": "EDU_O"}, {"txt": "Distilbert, a distilled version of bert: smaller,faster,cheaper and lighter.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.591, 0.299, 0.886, 0.307], [0.153, 0.313, 0.327, 0.321]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 392, "edu_l1_label": "EDU_O"}, {"txt": "ArXiv,abs/1910.01108,2019.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.327, 0.313, 0.525, 0.321]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 393, "edu_l1_label": "EDU_O"}, {"txt": "[38] Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.118, 0.331, 0.692, 0.345]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 394, "edu_l1_label": "EDU_O"}, {"txt": "Adversarial deep averaging networks for cross-lingual sentiment classification.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.692, 0.331, 0.881, 0.342], [0.153, 0.348, 0.48, 0.354]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 395, "edu_l1_label": "EDU_O"}, {"txt": "Tranisactions of the Association for Computational Linguistics,6:557-570,2018.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.48, 0.348, 0.884, 0.355], [0.152, 0.36, 0.267, 0.368]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 396, "edu_l1_label": "EDU_O"}, {"txt": "[39] Ilya Loshchilov and Frank Hutter.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16400000, "bbox": [[0.118, 0.378, 0.386, 0.386]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 397, "edu_l1_label": "EDU_O"}, {"txt": "Decoupled weight decay regularization.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16400000, "bbox": [[0.386, 0.378, 0.665, 0.386]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 398, "edu_l1_label": "EDU_O"}, {"txt": "In International Conference on Learning Representations, 2019.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16400000, "bbox": [[0.665, 0.378, 0.881, 0.386], [0.152, 0.391, 0.366, 0.4]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 399, "edu_l1_label": "EDU_O"}, {"txt": "[40] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16500000, "bbox": [[0.117, 0.41, 0.507, 0.419]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 400, "edu_l1_label": "EDU_O"}, {"txt": "On the difficulty of training recurrent neural networks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16500000, "bbox": [[0.507, 0.41, 0.865, 0.419]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 401, "edu_l1_label": "EDU_O"}, {"txt": "In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 1310-1318,Atlanta,Georgia,USA,17-19 Jun 2013.PMLR.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16500000, "bbox": [[0.865, 0.41, 0.881, 0.419], [0.153, 0.424, 0.88, 0.432], [0.152, 0.438, 0.884, 0.448], [0.152, 0.451, 0.312, 0.461]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 402, "edu_l1_label": "EDU_O"}, {"txt": "[41] Erik F. Tjong Kim Sang.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16600000, "bbox": [[0.116, 0.47, 0.323, 0.479]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 11, "global_sentence_id": 403, "edu_l1_label": "EDU_O"}, {"txt": "Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16600000, "bbox": [[0.323, 0.47, 0.883, 0.48], [0.153, 0.487, 0.234, 0.493]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 11, "global_sentence_id": 404, "edu_l1_label": "EDU_O"}, {"txt": "In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002),2002.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16600000, "bbox": [[0.234, 0.487, 0.864, 0.493]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 11, "global_sentence_id": 405, "edu_l1_label": "EDU_O"}, {"txt": "[42] Erik F. Tjong Kim Sang and Fien De Meulder.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16700000, "bbox": [[0.117, 0.503, 0.483, 0.511]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 12, "global_sentence_id": 406, "edu_l1_label": "EDU_O"}, {"txt": "Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16700000, "bbox": [[0.483, 0.503, 0.883, 0.511], [0.153, 0.518, 0.404, 0.526]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 12, "global_sentence_id": 407, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147,2003.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16700000, "bbox": [[0.404, 0.518, 0.882, 0.527], [0.152, 0.533, 0.443, 0.539]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 12, "global_sentence_id": 408, "edu_l1_label": "EDU_O"}, {"txt": "[43] Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16800000, "bbox": [[0.116, 0.549, 0.88, 0.558], [0.153, 0.563, 0.268, 0.571]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 13, "global_sentence_id": 409, "edu_l1_label": "EDU_O"}, {"txt": "Xnli: Evaluating cross-lingual sentence representations.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16800000, "bbox": [[0.268, 0.563, 0.631, 0.571]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 13, "global_sentence_id": 410, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475-2485,2018.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16800000, "bbox": [[0.631, 0.563, 0.881, 0.572], [0.152, 0.58, 0.682, 0.586]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 13, "global_sentence_id": 411, "edu_l1_label": "EDU_O"}, {"txt": "[44] Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16900000, "bbox": [[0.116, 0.596, 0.677, 0.604]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 14, "global_sentence_id": 412, "edu_l1_label": "EDU_O"}, {"txt": "Mlqa: Evaluating cross-lingual extractive question answering.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16900000, "bbox": [[0.677, 0.596, 0.882, 0.605], [0.152, 0.612, 0.353, 0.619]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 14, "global_sentence_id": 413, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315-7330,2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16900000, "bbox": [[0.353, 0.612, 0.881, 0.619], [0.152, 0.622, 0.392, 0.631]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 14, "global_sentence_id": 414, "edu_l1_label": "EDU_O"}, {"txt": "[45] Xi Chen, Ali Zeynali, Chico Camargo, Fabian Flöck, Devin Gaffney,Przemyslaw Grabowicz, Scott Hale, David Jurgens, and Mattia Samory.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17000000, "bbox": [[0.117, 0.641, 0.88, 0.65], [0.152, 0.657, 0.344, 0.664]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 15, "global_sentence_id": 415, "edu_l1_label": "EDU_O"}, {"txt": "Semeval-2022 task 8: Multilingual news article similarity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17000000, "bbox": [[0.344, 0.657, 0.736, 0.664]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 15, "global_sentence_id": 416, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1094-1106,2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17000000, "bbox": [[0.736, 0.657, 0.881, 0.665], [0.151, 0.668, 0.776, 0.678]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 15, "global_sentence_id": 417, "edu_l1_label": "EDU_O"}, {"txt": "[46] Joseph Lee Rodgers and W. Alan Nicewander.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.116, 0.688, 0.453, 0.697]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 16, "global_sentence_id": 418, "edu_l1_label": "EDU_O"}, {"txt": "Thirteen ways to look at the correlation coefficient.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.453, 0.688, 0.788, 0.698]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 16, "global_sentence_id": 419, "edu_l1_label": "EDU_O"}, {"txt": "The American Statistician, 42(1):59-66,1988.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.788, 0.688, 0.881, 0.698], [0.152, 0.702, 0.36, 0.711]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 16, "global_sentence_id": 420, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17200000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 17, "global_sentence_id": 421, "edu_l1_label": "EDU_O"}], "type": "PDF"}
# UNIBERT: ADVERSARIAL TRAINING FORLANGUAGE-UNIVERSAL REPRESENTATIONS
## ABSTRACT
## 1 Introduction
## 2 Related Work
### 2.1 Multilingual Language Models
### 2.2 Adversarial Training
## 3 UniBERT
### 3.1 Training Data
### 3.2 Model Architecture
### 3.3 Training Process
#### 3.3.1 Masked Language Modeling
#### 3.3.2 Adversarial Training
#### 3.3.3 Knowledge Distillation
#### 3.3.4 Training Objective
### 3.4 Training Hyperparameters
## 4 Evaluation Datasets
## 5 Results
### 5.1 Named Entity Recognition
### 5.2 Natural Language Inference
### 5.3 Question Answering
### 5.4 Semantic Textual Similarity
### 5.5 Discussion
## 6 Conclusion
## 7 Limitations
