{"entry_id": "0eca2177-b722-4d68-99f0-21e50efdf01c", "infos": [{"txt": "Kaleidoscope:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 0, "bbox": [[0.195, 0.09, 0.417, 0.109]]}]}, "tags": ["title"], "label": "article_title", "web_segment_id": 1, "global_sentence_id": 0, "edu_l1_label": "BOT"}, {"txt": " In-language Exams for", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 0, "bbox": [[0.417, 0.09, 0.805, 0.109]]}]}, "tags": ["title"], "label": "article_title", "web_segment_id": 1, "global_sentence_id": 1, "edu_l1_label": "BOT"}, {"txt": "Massively Multilingual Vision Evaluation", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 0, "bbox": [[0.167, 0.121, 0.831, 0.14]]}]}, "tags": ["title"], "label": "article_title", "web_segment_id": 2, "global_sentence_id": 2, "edu_l1_label": "BOT"}, {"txt": "Israfel Salazar*2, Manuel Fernández Burda*3, Shayekh Bin Islam*4,Arshia Soltani Moakhar*4, Shivalika Singh*1,Fabian Farestam*17,Angelika Romanou*, Danylo Boiko4,9, Dipika Khullar4, Mike Zhang10,Dominik Krzemiński4, Jekaterina Novikova4, Luísa Shimabucoro18,Joseph Marvin Imperial19,26, Rishabh Maheshwary4, Sharad Duwal4,Alfonso Amayuelas5, Swati Rajwal6, Jebish Purbey4,7,Ahmed Ruby25,Nicholas Popovič11,12, Marek Suppa22,23, Azmine Toushik Wasi4,Ram Mohan Rao Kadiyala7,8, Olga Tsymboi13,28, Maksim Kostritsya15,16,Bardia Soltani Moakhar4, Gabriel da Costa Merlin18, Otávio Ferracioli Coletti18,Maral Jabbari Shiviari4, MohammadAmin farahani fard4, Silvia Fernandez$4$,María Grandury$21$, Dmitry Abulkhanov4, Drishti Sharma$4,7$,Andre Guarnier De Mitri18, Leticia Bossatto Marchezi$20$, Setayesh Heydari4,Johan Obando-Ceron$4,24$4, Nazar Kohut14, Beyza Ermis1, Desmond Elliott$2,27$,Enzo Ferrante*3, Sara Hooker*1,and Marzieh Fadaee 1Cohere For AI,$2$ Department of Computer Science, University of Copenhagen,$3$Institute of Computer Sciences, CONICET & Universidad de Buenos Aires, $4$Cohere For AI Community,${}^{5}\\mathrm {U}$niversity of California, Santa Barbara,$6$Emory University,$7$M2ai.in,$8$Traversaal.ai,${}^{9}\\mathrm {}$ras Shevchenko National University of Kyiv,${}^{10}\\mathrm {Aa}$lborg University,${}^{11}\\mathrm {\\sim K}$rlsruhe Institute of Technology,Germany,$12$ScaDS.AI,TU Dresden,Germany,$13$T-Tech, $14$Lviv Polytechnic National University, $15$HSE University (Higher School of Economics),$16$RAFT,${}^{17}\\mathrm {E}^{\\prime }$TH Zürich,$18$University of São Paulo,${}^{19}\\mathrm {N}$tional University Philippines,20Federal University of São Carlos,$21$SomosNLP, $22$Cisco,$23$Comenius University in Bratislava,$24Mia$University of Montreal,$25$Uppsala University,${}^{26}\\mathrm {Ur}$niversity of Bath,${}^{27}\\mathrm {P}$ioneer Center for AI,${}^{28}\\mathrm {M}$oscow", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 100000, "bbox": [[0.12, 0.172, 0.881, 0.538]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 3, "global_sentence_id": 3, "edu_l1_label": "EDU_O"}, {"txt": "IT0.3· ZACLOL0．寸0い", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 200000, "bbox": [[0.029, 0.408, 0.052, 0.629]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 4, "global_sentence_id": 4, "edu_l1_label": "EDU_O"}, {"txt": "Institute of Physics and Technology", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 300000, "bbox": [[0.372, 0.54, 0.627, 0.551]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 5, "global_sentence_id": 5, "edu_l1_label": "EDU_O"}, {"txt": "Corresponding authors:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 400000, "bbox": [[0.117, 0.562, 0.283, 0.571]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 6, "global_sentence_id": 6, "edu_l1_label": "EDU_O"}, {"txt": " Israfel Salazar <israfel.salazar@di.ku.dk>,Manuel Fernández Burda <mburda@dc.uba.ar>,Marzieh Fadaee <marzieh@cohere.com>", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 400000, "bbox": [[0.283, 0.562, 0.809, 0.572], [0.118, 0.58, 0.568, 0.586]]}]}, "tags": ["text"], "label": "author", "web_segment_id": 6, "global_sentence_id": 7, "edu_l1_label": "EDU_O"}, {"txt": "The evaluation of vision-language models (VLMs) has mainly relied on English-language bench-marks, leaving significant gaps in both muItilingual and multicultural coverage.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 500000, "bbox": [[0.118, 0.636, 0.88, 0.646], [0.117, 0.656, 0.752, 0.664]]}]}, "tags": ["text"], "label": "abstract", "web_segment_id": 7, "global_sentence_id": 8, "edu_l1_label": "BOS"}, {"txt": "While multilin-gual benchmarks have expanded, both in size and languages, many rely on translations of English datasets, failing to capture cultural nuances.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 500000, "bbox": [[0.752, 0.656, 0.881, 0.663], [0.118, 0.674, 0.88, 0.68], [0.118, 0.687, 0.462, 0.698]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 9, "edu_l1_label": "IOS"}, {"txt": "In this work, we propose KALEIDOSCOPE, as the most comprehensive exam benchmark to date for the multilingual evaluation of vision-language models.KALEIDOSCOPE is a large-scale, in-language multimodal benchmark designed to evaluate VLMs across diverse languages and visual inputs.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 500000, "bbox": [[0.462, 0.687, 0.88, 0.697], [0.117, 0.708, 0.882, 0.714], [0.118, 0.721, 0.881, 0.731], [0.117, 0.742, 0.458, 0.749]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 10, "edu_l1_label": "IOS"}, {"txt": "KALEIDOSCOPE covers 18 languages and 14 different subjects,amounting to a total of 20,911 multiple-choice questions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 500000, "bbox": [[0.458, 0.742, 0.881, 0.749], [0.116, 0.759, 0.64, 0.766]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 11, "edu_l1_label": "IOS"}, {"txt": "Built through an open science collaboration with a diverse group of researchers worldwide, KALEIDOSCOPE ensures linguisticand cultural authenticity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 500000, "bbox": [[0.64, 0.759, 0.881, 0.765], [0.117, 0.777, 0.881, 0.783], [0.117, 0.794, 0.281, 0.8]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 12, "edu_l1_label": "IOS"}, {"txt": "WVe evaluate top-performing multilingual vision-language models and find that they perform poorly on low-resource languages and in complex multimodal scenarios.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 500000, "bbox": [[0.281, 0.794, 0.88, 0.801], [0.117, 0.808, 0.788, 0.818]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 13, "edu_l1_label": "IOS"}, {"txt": "Our results highlight the need for progress on culturally inclusive multimodal evaluation frameworks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 500000, "bbox": [[0.788, 0.808, 0.881, 0.817], [0.117, 0.824, 0.81, 0.835]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 14, "edu_l1_label": "IOS"}, {"txt": "website:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 600000, "bbox": [[0.118, 0.862, 0.182, 0.869]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 15, "edu_l1_label": "EDU_O"}, {"txt": "dataset:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 700000, "bbox": [[0.117, 0.876, 0.181, 0.886]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 16, "edu_l1_label": "EDU_O"}, {"txt": "*First authors.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 800000, "bbox": [[0.149, 0.901, 0.25, 0.908]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 17, "edu_l1_label": "EDU_O"}, {"txt": "*Principal senior adviors.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 800000, "bbox": [[0.25, 0.901, 0.429, 0.908]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 18, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 900000, "bbox": [[0.115, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 11, "global_sentence_id": 19, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 900000, "bbox": [[0.115, 0.095, 0.397, 0.334]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 1, "global_sentence_id": 20, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 900000, "bbox": [[0.679, 0.083, 0.886, 0.33]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 2, "global_sentence_id": 21, "edu_l1_label": "EDU_O"}, {"txt": "(a)", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1000000, "bbox": [[0.244, 0.332, 0.263, 0.343]]}]}, "tags": ["figure_caption"], "label": "O", "web_segment_id": 3, "global_sentence_id": 22, "edu_l1_label": "EDU_O"}, {"txt": "(c)", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1000000, "bbox": [[0.774, 0.334, 0.792, 0.34]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 4, "global_sentence_id": 23, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1100000, "bbox": [[0.4, 0.086, 0.678, 0.33]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 5, "global_sentence_id": 24, "edu_l1_label": "EDU_O"}, {"txt": "(b)", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1100000, "bbox": [[0.55, 0.333, 0.569, 0.34]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 6, "global_sentence_id": 25, "edu_l1_label": "EDU_O"}, {"txt": "Figure 1: Overview of the KALEIDOSCOPE Benchmark. (a) Multilingual-Multimodal MCQ Samples (b) Language and Multimodal Samples Distribution. (c) Exam Category Breakdown.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1200000, "bbox": [[0.115, 0.355, 0.885, 0.385]]}]}, "tags": ["figure_caption"], "label": "figure_title", "web_segment_id": 7, "global_sentence_id": 26, "edu_l1_label": "EDU_O"}, {"txt": "1 Introduction", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1300000, "bbox": [[0.117, 0.415, 0.294, 0.429]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 8, "global_sentence_id": 27, "edu_l1_label": "BOS"}, {"txt": "Evaluations are the backbone of measuring progress in machine learning, yet many benchmarks-especially for language models - continue to mirror an English and Western-centric worldview (Joshi et al., 2020;", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.118, 0.449, 0.882, 0.456], [0.117, 0.47, 0.881, 0.476], [0.117, 0.487, 0.207, 0.496]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 28, "edu_l1_label": "IOS"}, {"txt": " Fan et al., 2020; Dodge et al., 2021; Liu et al., 2021; Chung et al., 2022; Gehrmann et al.,2022; Lucy et al., 2024).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.207, 0.487, 0.882, 0.496], [0.118, 0.501, 0.301, 0.511]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 29, "edu_l1_label": "IOS"}, {"txt": "This imbalance becomes even more striking at the cutting edge of AI,where generative models are rapidly expanding into multimodal territory (OpenAI et al., 2024; Google et a1., 2024; Anthropic, 2024; Deitke et al., 2024; Yue et al., 2025; Qwen-Team, 2025), seeking to represent a richer world made up of different modalities such as image, text, sound.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.301, 0.501, 0.881, 0.511], [0.117, 0.521, 0.881, 0.527], [0.117, 0.537, 0.881, 0.545], [0.117, 0.556, 0.759, 0.562]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 30, "edu_l1_label": "IOS"}, {"txt": "In recent years,the community has made promising strides toward broader multilingual text evaluation (Ahuja et al., 2023; Singh et al., 2024b;a; Aakanksha et al., 2024; Pozzobon et al., 2024; Romanou et al.,2024;Singh et al., 2025; Adelani et al., 2024), and multimodal benchmarks are starting to take shape (Bugliarello et al., 2022; Fu et al., 2023; Yue et al., 2024a;b; Li et al., 2024a; Xu et al.,2025).Yet reliable evaluation at the intersection of multilingual and multimodal tasks remains rare.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.759, 0.556, 0.882, 0.564], [0.117, 0.57, 0.881, 0.58], [0.117, 0.589, 0.883, 0.598], [0.118, 0.604, 0.88, 0.613], [0.117, 0.622, 0.882, 0.631], [0.118, 0.638, 0.843, 0.648]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 31, "edu_l1_label": "IOS"}, {"txt": "This gap is precisely what motivates our work.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1400000, "bbox": [[0.843, 0.638, 0.881, 0.647], [0.117, 0.658, 0.44, 0.664]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 32, "edu_l1_label": "IOS"}, {"txt": "One common but imperfect solution is translating English benchmarks into other languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1500000, "bbox": [[0.117, 0.689, 0.832, 0.699]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 33, "edu_l1_label": "IOS"}, {"txt": "While convenient, this approach often falls short of capturing cultural context and nuance.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1500000, "bbox": [[0.832, 0.689, 0.881, 0.698], [0.117, 0.709, 0.791, 0.716]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 34, "edu_l1_label": "IOS"}, {"txt": "Translated datasets can easily reinforce Western-centric knowledge and assumptions (van Miltenburg et al.,2017;Frank et al., 2018; Singh et al., 2025; Longpre et al., 2025) limiting their ability to truly assess model performance across diverse settings.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1500000, "bbox": [[0.791, 0.709, 0.881, 0.715], [0.117, 0.723, 0.882, 0.735], [0.118, 0.741, 0.881, 0.75], [0.117, 0.76, 0.446, 0.767]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 35, "edu_l1_label": "IOS"}, {"txt": "Moreover, automated data curation pipelines frequently amplify existing quality issues (Luccioni & Viviano, 2021; Caswell et al., 2020; Kreutzer et al., 2022),with translation artifacts such as translationese muddying the waters even further (Koppel & Ordan,2011; Zhang & Toral, 2019; Bizzoni et al., 2020; Vanmassenhove et al., 2021).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1500000, "bbox": [[0.446, 0.76, 0.881, 0.769], [0.117, 0.778, 0.882, 0.787], [0.118, 0.794, 0.882, 0.804], [0.118, 0.809, 0.71, 0.82]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 36, "edu_l1_label": "IOS"}, {"txt": "While translated data has its place, especially for some particularly low-resource tasks (Zhou et al., 2021; Thapliyal et al.,2022; Qiu et al., 2022; Ramos et al., 2024; Geigle et al., 2025; Dang et al., 2024; Üstün et al., 2024;Aakanksha et al., 2024), it is an imperfect substitute for genuinely diverse, in-language benchmarks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1500000, "bbox": [[0.71, 0.809, 0.88, 0.819], [0.117, 0.825, 0.882, 0.838], [0.118, 0.843, 0.882, 0.855], [0.118, 0.86, 0.882, 0.869]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 37, "edu_l1_label": "IOS"}, {"txt": "In this work, we introduce the largest benchmark of real-world, in-language exam questions that", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1600000, "bbox": [[0.117, 0.894, 0.881, 0.903]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 38, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1700000, "bbox": [[0.494, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 12, "global_sentence_id": 39, "edu_l1_label": "EDU_O"}, {"txt": "blend image and text modalities.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1700000, "bbox": [[0.117, 0.094, 0.377, 0.105]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 40, "edu_l1_label": "IOS"}, {"txt": "Our dataset pushes beyond simple captioning tasks, challenging models to reason about visual content in various topics, the way humans are evaluated in exams worldwide.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1700000, "bbox": [[0.377, 0.094, 0.881, 0.106], [0.117, 0.115, 0.881, 0.122], [0.118, 0.131, 0.203, 0.139]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 41, "edu_l1_label": "IOS"}, {"txt": "Through a large-scale open science effort across 18 languages, we construct KALEIDO-SCOPE(see Figure 1), featuring a diverse selection of knowledge domains across 14 subjects.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1700000, "bbox": [[0.203, 0.131, 0.881, 0.138], [0.117, 0.148, 0.836, 0.156]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 42, "edu_l1_label": "IOS"}, {"txt": "With 55% of the total 20,911 questions requiring image understanding for accurate resolution, our work aims to establish a comprehensive, and inclusive evaluation framework for multimodal language models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1700000, "bbox": [[0.836, 0.148, 0.88, 0.156], [0.117, 0.163, 0.88, 0.173], [0.117, 0.183, 0.881, 0.19], [0.117, 0.201, 0.179, 0.208]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 43, "edu_l1_label": "IOS"}, {"txt": "We evaluate a wide range of state-of-the-art models on KALEIDOSCOPE, including Claude 3.5 Sonnet (Anthropic, 2024), GPT-4o (OpenAI et al., 2024),and Gemini-V (Google et al., 2024),as well as smaller open-weight VLMs, such as Aya-Vision model family (Cohere-For-AI-Team,2025),Molmo (Deitke et al., 2024) Pangea (Yue et al., 2025), and Qwen2.5-VL model family (Qwen-Team,2025).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1700000, "bbox": [[0.179, 0.201, 0.882, 0.208], [0.117, 0.215, 0.881, 0.224], [0.118, 0.235, 0.882, 0.244], [0.118, 0.249, 0.883, 0.261], [0.118, 0.266, 0.17, 0.277]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 44, "edu_l1_label": "IOS"}, {"txt": "Our key contributions and findings are highlighted here:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1700000, "bbox": [[0.17, 0.266, 0.607, 0.276]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 45, "edu_l1_label": "IOS"}, {"txt": "·KALEIDOSCOPE Benchmark:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1800000, "bbox": [[0.143, 0.318, 0.402, 0.325]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 46, "edu_l1_label": "IOS"}, {"txt": " We present the largest multilingual multimodal exam set,covering high resouirce (e.g., English, Spanish) to underrepresented languages (e.g.,Bengali,Telugu) across diverse subjects from sociology to STEM.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1800000, "bbox": [[0.402, 0.318, 0.883, 0.327], [0.161, 0.335, 0.883, 0.345], [0.162, 0.349, 0.605, 0.359]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 47, "edu_l1_label": "IOS"}, {"txt": "Most languages (10/18) include5+topics, with the rest focusing on multi-subtopics like mathematics or engineering.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1800000, "bbox": [[0.605, 0.349, 0.854, 0.359], [0.859, 0.35, 0.879, 0.359], [0.161, 0.368, 0.801, 0.378]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 48, "edu_l1_label": "IOS"}, {"txt": "Questions emphasize vision grounded reasoning through tasks like interpreting graphs, pictures, and region-specific diagrams, supported by fine-grained metadata for model diagnostics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1800000, "bbox": [[0.801, 0.368, 0.881, 0.376], [0.161, 0.386, 0.881, 0.393], [0.161, 0.404, 0.811, 0.41]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 49, "edu_l1_label": "IOS"}, {"txt": "·Modality-Specific Performance Disparities:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1900000, "bbox": [[0.143, 0.431, 0.542, 0.439]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 50, "edu_l1_label": "IOS"}, {"txt": " All models perform substantially better on text-only questions, revealing a clear disparity across modalities.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1900000, "bbox": [[0.542, 0.431, 0.881, 0.439], [0.161, 0.448, 0.676, 0.456]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 51, "edu_l1_label": "IOS"}, {"txt": "The gap widens in larger modelsl; for instance, GPT-4o shows a 21.6% difference between text-only and multimodal performance, while smaller models like Molmo exhibit a much narrower gap of 3.69%.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1900000, "bbox": [[0.676, 0.448, 0.88, 0.456], [0.167, 0.466, 0.882, 0.473], [0.162, 0.484, 0.837, 0.491]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 52, "edu_l1_label": "IOS"}, {"txt": "(Sec-tion 4.1).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1900000, "bbox": [[0.837, 0.484, 0.882, 0.488], [0.162, 0.498, 0.233, 0.508]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 53, "edu_l1_label": "IOS"}, {"txt": "Furthermore, multimodal performance varies significantly by visual data type: mod-els are more capable of answering questions about tables (76.5%) and photographs (81.5%)compared to diagrams (62.9%).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 1900000, "bbox": [[0.233, 0.498, 0.882, 0.505], [0.161, 0.517, 0.88, 0.527], [0.161, 0.535, 0.404, 0.543]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 54, "edu_l1_label": "IOS"}, {"txt": "·Domain-Specific Performance Disparities:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2000000, "bbox": [[0.143, 0.562, 0.537, 0.57]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 55, "edu_l1_label": "IOS"}, {"txt": " We observe a significant performance gap between questions requiring knowledge of Humanities & Social Sciences and those focused on STEM subjects (Section 4.4).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2000000, "bbox": [[0.537, 0.562, 0.881, 0.573], [0.162, 0.577, 0.881, 0.586], [0.161, 0.594, 0.396, 0.605]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 56, "edu_l1_label": "IOS"}, {"txt": "On average, models present accuracy of 83.7% for humanities versus 59.2% for STEM (based on the best scores across models).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2000000, "bbox": [[0.396, 0.594, 0.881, 0.604], [0.161, 0.614, 0.67, 0.621]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 57, "edu_l1_label": "IOS"}, {"txt": "Models struggle more with STEM questions,suggesting that while they can often recognize visual content and retrieve related knowledge, they lack the reasoning capabilities needed to arrive at the correct answers in STEM domains.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2000000, "bbox": [[0.67, 0.614, 0.88, 0.621], [0.161, 0.629, 0.881, 0.638], [0.161, 0.648, 0.881, 0.656], [0.162, 0.664, 0.308, 0.673]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 58, "edu_l1_label": "IOS"}, {"txt": "·Crosslingual Performance Disparities:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2100000, "bbox": [[0.143, 0.694, 0.496, 0.701]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 59, "edu_l1_label": "IOS"}, {"txt": " Model performance varies across languages, with noticeably better results in high-resource languages and weaker performance in mid- and low-resource ones (Section 4.3).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2100000, "bbox": [[0.496, 0.694, 0.88, 0.701], [0.161, 0.711, 0.881, 0.717], [0.161, 0.729, 0.378, 0.735]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 60, "edu_l1_label": "IOS"}, {"txt": "Crosslingual transfer appears to play a role, as models perform better on average in languages using Latin scripts compared to those with non-Latin scripts.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2100000, "bbox": [[0.378, 0.729, 0.874, 0.735], [0.162, 0.742, 0.881, 0.752]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 61, "edu_l1_label": "IOS"}, {"txt": "2 The KALEIDOSCOPE Benchmark", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2200000, "bbox": [[0.117, 0.794, 0.507, 0.807]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 6, "global_sentence_id": 62, "edu_l1_label": "BOS"}, {"txt": "The KALEIDOSCOPE Benchmark is a global collection of multiple-choice questions sourced from real-world exams, with the goal of evaluating multimodal and multilingual understanding in VLMs.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2300000, "bbox": [[0.118, 0.826, 0.882, 0.834], [0.117, 0.847, 0.848, 0.854]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 63, "edu_l1_label": "IOS"}, {"txt": "The collected exams are in a Multiple-choice question answering (MCQA) format which provides a struc-tured framework for evaluation by prompting models with predefined answer choices (Hendrycks et al., 2021; Lu et al., 2023; Wang et al., 2024a; Yue et al., 2024a; Romero et al., 2024; Romanou", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2300000, "bbox": [[0.848, 0.847, 0.881, 0.854], [0.117, 0.865, 0.881, 0.869], [0.117, 0.88, 0.881, 0.889], [0.117, 0.898, 0.881, 0.906]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 64, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2400000, "bbox": [[0.494, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 8, "global_sentence_id": 65, "edu_l1_label": "EDU_O"}, {"txt": "et al., 2024),closely mimicking conventional human testing methodologies.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2400000, "bbox": [[0.117, 0.098, 0.688, 0.105]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 66, "edu_l1_label": "IOS"}, {"txt": "Our work is built around three core design principles that guide the selection, curation, processing, and addition of exams:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2400000, "bbox": [[0.688, 0.098, 0.88, 0.104], [0.117, 0.113, 0.871, 0.121]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 67, "edu_l1_label": "IOS"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2500000, "bbox": [[0.132, 0.153, 0.156, 0.169]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 2, "global_sentence_id": 68, "edu_l1_label": "EDU_O"}, {"txt": "Multimodality:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2600000, "bbox": [[0.172, 0.157, 0.297, 0.167]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 69, "edu_l1_label": "IOS"}, {"txt": " Images are central to KALEIDOSCOPE, as we aim to evaluate how VLMs integrate and reason about visual information to answer questions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2600000, "bbox": [[0.297, 0.157, 0.882, 0.167], [0.161, 0.174, 0.682, 0.184]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 70, "edu_l1_label": "IOS"}, {"txt": "We prioritize multimodal questions with diverse image types, complemented by a similar proportion of text-only ques-tions for a complete assessment and comparison.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2600000, "bbox": [[0.682, 0.174, 0.882, 0.184], [0.161, 0.194, 0.88, 0.2], [0.161, 0.209, 0.538, 0.218]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 71, "edu_l1_label": "IOS"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2700000, "bbox": [[0.133, 0.234, 0.153, 0.249]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 4, "global_sentence_id": 72, "edu_l1_label": "EDU_O"}, {"txt": "Multilinguality:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2800000, "bbox": [[0.162, 0.235, 0.3, 0.246]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 73, "edu_l1_label": "IOS"}, {"txt": " The benchmark contains questions in 18 languages, with a focus on under-represented mid-and low-resource languages (e.g., Nepali, Lithuanian) alongside high-resource languages (e.g., English, Spanish) for a thorough evaluation across a broad range of languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2800000, "bbox": [[0.3, 0.235, 0.88, 0.246], [0.161, 0.256, 0.881, 0.263], [0.161, 0.27, 0.882, 0.28]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 74, "edu_l1_label": "IOS"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 2900000, "bbox": [[0.132, 0.294, 0.154, 0.312]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 6, "global_sentence_id": 75, "edu_l1_label": "EDU_O"}, {"txt": "Diversity:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3000000, "bbox": [[0.163, 0.298, 0.248, 0.308]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 76, "edu_l1_label": "IOS"}, {"txt": " Our goal is to collect exams covering as wide a range of topics as possible ranging from Mathematics and Sociology, to Medicine and Driving Licenses, ensuring comprehensive evaluation across various domains.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3000000, "bbox": [[0.248, 0.298, 0.881, 0.309], [0.162, 0.316, 0.881, 0.326], [0.161, 0.336, 0.434, 0.342]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 77, "edu_l1_label": "IOS"}, {"txt": "The final collection includes exams from 14 different do-mains, collected from 18 countries and with varying educational levels (from high school to professional exams), allowing detailed clustering and comprehensive evaluation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3000000, "bbox": [[0.434, 0.336, 0.88, 0.342], [0.163, 0.353, 0.881, 0.36], [0.162, 0.37, 0.779, 0.376]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 78, "edu_l1_label": "IOS"}, {"txt": "2.1 Global Collaboration", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3100000, "bbox": [[0.117, 0.412, 0.353, 0.422]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 8, "global_sentence_id": 79, "edu_l1_label": "IOS"}, {"txt": "Our work entailed an extensive, open science process to manually collect data by working directly with native speakers of different languages (Elliott et al., 2016;", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3200000, "bbox": [[0.117, 0.441, 0.881, 0.453], [0.117, 0.461, 0.617, 0.472]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 80, "edu_l1_label": "IOS"}, {"txt": " Liu et al.,2021;Thapliyal et al.,2022; Li et al., 2024c; Üstün et al., 2024; Singh et al., 2024b).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3200000, "bbox": [[0.617, 0.461, 0.882, 0.472], [0.118, 0.477, 0.596, 0.485]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 81, "edu_l1_label": "IOS"}, {"txt": "This is acutely needed in the field of machine learning, where recent studies have highlighted that dataset creators remain predominantly Western-centric (Longpre et al., 2025).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3200000, "bbox": [[0.596, 0.477, 0.882, 0.486], [0.117, 0.496, 0.881, 0.504], [0.118, 0.509, 0.426, 0.52]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 82, "edu_l1_label": "IOS"}, {"txt": "The manual curation of datasets is a costly process that requuires careful attention to detail in every language to ensure high-quality, contextually relevant content for evaluation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3200000, "bbox": [[0.426, 0.509, 0.88, 0.52], [0.117, 0.53, 0.88, 0.537], [0.117, 0.548, 0.301, 0.555]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 83, "edu_l1_label": "IOS"}, {"txt": "In this work, we engage in a large-scale open science collection process,which brings together contributors spanning 20 nations across four continents to ensure linguistic and cultural authenticity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3200000, "bbox": [[0.301, 0.548, 0.882, 0.557], [0.118, 0.564, 0.88, 0.572], [0.117, 0.582, 0.317, 0.589]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 84, "edu_l1_label": "IOS"}, {"txt": "For related participatory research see Section 6.3.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3200000, "bbox": [[0.317, 0.582, 0.708, 0.589]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 85, "edu_l1_label": "IOS"}, {"txt": "2.2 Data Pipeline", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3300000, "bbox": [[0.117, 0.621, 0.292, 0.631]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 10, "global_sentence_id": 86, "edu_l1_label": "IOS"}, {"txt": "Collection:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.114, 0.651, 0.886, 0.838]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 87, "edu_l1_label": "IOS"}, {"txt": " We collected KALEIDOSCOPE with guidelines detailing information about the type of exams and questions required, formatting, specifications, and quality control measures.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.114, 0.651, 0.886, 0.838]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 88, "edu_l1_label": "IOS"}, {"txt": "The data was collected through a global call for contributions and distributed across global communities, with the majority of contributors being independent researchers in the Cohere for AI$(\\mathrm {C}4\\mathrm {AI})^{1}$open sci-ence community.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.114, 0.651, 0.886, 0.838]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 89, "edu_l1_label": "IOS"}, {"txt": "This effort resulted in a collection of 20,911 questions from 18 different countries and 18 languages, all sourced in their original languages, avoiding translations to maintain linguistic authenticity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.114, 0.651, 0.886, 0.838]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 90, "edu_l1_label": "IOS"}, {"txt": "We prioritized original, domain-expert-written questions (e.g.,from teachers),ensur-ing real-world relevance and quality.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.114, 0.651, 0.886, 0.838]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 91, "edu_l1_label": "IOS"}, {"txt": "The exams were gathered from various repositories, including official government websites, question banks, and other publicly available repositories with educa-tional materials.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.114, 0.651, 0.886, 0.838]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 92, "edu_l1_label": "IOS"}, {"txt": "Throughout the process, contributors also annotated associated licenses with each dataset to allow for documentation of data provenance (Longpre et al., 2024).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3400000, "bbox": [[0.114, 0.651, 0.886, 0.838]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 93, "edu_l1_label": "IOS"}, {"txt": "Processing:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3500000, "bbox": [[0.118, 0.857, 0.217, 0.868]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 94, "edu_l1_label": "IOS"}, {"txt": " The annotation process consists of two stages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3500000, "bbox": [[0.217, 0.857, 0.573, 0.867]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 95, "edu_l1_label": "IOS"}, {"txt": "In the first stage, we perform automated parsing and extraction.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3500000, "bbox": [[0.573, 0.857, 0.881, 0.867], [0.117, 0.878, 0.296, 0.884]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 96, "edu_l1_label": "IOS"}, {"txt": "For directly parsable text, we use PDF or web parsers, while for non-parsable", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3500000, "bbox": [[0.296, 0.878, 0.881, 0.884]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 97, "edu_l1_label": "IOS"}, {"txt": "1https://cohere.com/research", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3600000, "bbox": [[0.142, 0.898, 0.355, 0.909]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 13, "global_sentence_id": 98, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3600000, "bbox": [[0.494, 0.936, 0.507, 0.948]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 14, "global_sentence_id": 99, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3600000, "bbox": [[0.118, 0.09, 0.884, 0.473]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 100, "edu_l1_label": "EDU_O"}, {"txt": "Table 1: Statistics of the KALEIDOSCOPE Dataset. Breakdown of subjects (Subjects), total questions (Total), multimodal questions (Visual), and text-only questions (Text) per language. Lan-guages are covered by multiple sources with single-subject cases containing specialized subdomains.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3700000, "bbox": [[0.115, 0.487, 0.883, 0.536]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 101, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3800000, "bbox": [[0.115, 0.539, 0.138, 0.553]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 3, "global_sentence_id": 102, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3800000, "bbox": [[0.114, 0.556, 0.135, 0.571]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 4, "global_sentence_id": 103, "edu_l1_label": "EDU_O"}, {"txt": ":", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3900000, "bbox": [[0.137, 0.544, 0.146, 0.551]]}]}, "tags": ["text"], "label": "figure_title", "web_segment_id": 5, "global_sentence_id": 104, "edu_l1_label": "EDU_O"}, {"txt": "Supports evaluation of both multimodal (image+text) and unimodal (text-only) capabilities.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3900000, "bbox": [[0.146, 0.544, 0.882, 0.551]]}]}, "tags": ["text"], "label": "figure_title", "web_segment_id": 5, "global_sentence_id": 105, "edu_l1_label": "EDU_O"}, {"txt": ":Languages are classified by resource level (high/mid/low) following Joshi et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3900000, "bbox": [[0.136, 0.561, 0.774, 0.568]]}]}, "tags": ["text"], "label": "figure_title", "web_segment_id": 5, "global_sentence_id": 106, "edu_l1_label": "EDU_O"}, {"txt": "(2019); Singh et al.(2024b).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3900000, "bbox": [[0.774, 0.561, 0.881, 0.568], [0.117, 0.578, 0.259, 0.585]]}]}, "tags": ["text"], "label": "figure_title", "web_segment_id": 5, "global_sentence_id": 107, "edu_l1_label": "EDU_O"}, {"txt": ":Enables granular analysis of model performance across modalities,languages,and subject domains.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 3900000, "bbox": [[0.259, 0.578, 0.882, 0.588], [0.117, 0.595, 0.283, 0.602]]}]}, "tags": ["text"], "label": "figure_title", "web_segment_id": 5, "global_sentence_id": 108, "edu_l1_label": "EDU_O"}, {"txt": "text, we employ OCR API's, such as Mathpix2, along with vision-language models such as GPT-4o.These tools allow us to extract both text and image elements from exam source formats, which are then converted into structured outputs in LaTeX, Markdown, and JSON formats, as required.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4000000, "bbox": [[0.117, 0.638, 0.882, 0.646], [0.118, 0.653, 0.881, 0.664], [0.117, 0.672, 0.837, 0.68]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 109, "edu_l1_label": "IOS"}, {"txt": "Since automated parsing can sometimes result in misaligned images and text, the second stage involves refining the extracted text.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4000000, "bbox": [[0.837, 0.672, 0.881, 0.68], [0.117, 0.691, 0.881, 0.698], [0.117, 0.708, 0.335, 0.714]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 110, "edu_l1_label": "IOS"}, {"txt": "WVe apply heuristic rules, as well as high-performing LLMs,such as Claude 3.5 Sonnet and GPT-4o, to restructure the output, ensuring proper alignment of questions,text, and answer choices.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4000000, "bbox": [[0.335, 0.708, 0.881, 0.715], [0.118, 0.721, 0.882, 0.734], [0.116, 0.74, 0.314, 0.749]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 111, "edu_l1_label": "IOS"}, {"txt": "This stage was followed by human verification, ensuring that images are correctly linked to the corresponding questions, and checking that extracted formulas match the expected equation format.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4000000, "bbox": [[0.314, 0.74, 0.881, 0.749], [0.117, 0.759, 0.881, 0.765], [0.118, 0.776, 0.321, 0.783]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 112, "edu_l1_label": "IOS"}, {"txt": "Quality Assessment:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4100000, "bbox": [[0.118, 0.807, 0.298, 0.817]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 113, "edu_l1_label": "IOS"}, {"txt": " Maintaining reliable and high quality data is essential, especially given the large-scale international collaboration involved in this project.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4100000, "bbox": [[0.298, 0.807, 0.88, 0.817], [0.117, 0.826, 0.647, 0.834]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 114, "edu_l1_label": "IOS"}, {"txt": "To ensure data integrity,we include manual validation in three stages of the collection and annotation pipeline.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4100000, "bbox": [[0.647, 0.826, 0.881, 0.835], [0.117, 0.842, 0.776, 0.852]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 115, "edu_l1_label": "IOS"}, {"txt": "First, at the end of the collection stage, two independent annotators validate each exam to ensure conformity", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4100000, "bbox": [[0.776, 0.842, 0.881, 0.851], [0.117, 0.862, 0.88, 0.87]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 116, "edu_l1_label": "IOS"}, {"txt": "2https://mathpix.com/convert", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4200000, "bbox": [[0.143, 0.884, 0.355, 0.893]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 8, "global_sentence_id": 117, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4200000, "bbox": [[0.494, 0.936, 0.506, 0.948]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 118, "edu_l1_label": "EDU_O"}, {"txt": "with the guidelines.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4200000, "bbox": [[0.118, 0.097, 0.275, 0.105]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 119, "edu_l1_label": "IOS"}, {"txt": "Part of this verification includes a strict revision to confirm compliance with the distribution license requirements.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4200000, "bbox": [[0.275, 0.097, 0.88, 0.105], [0.117, 0.113, 0.411, 0.121]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 120, "edu_l1_label": "IOS"}, {"txt": "Only exams approved by both independent annotators are included in the dataset.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4200000, "bbox": [[0.411, 0.113, 0.881, 0.121], [0.117, 0.129, 0.299, 0.138]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 121, "edu_l1_label": "IOS"}, {"txt": "Next, following the annotation process, a validation script checks for JSON formatting errors, duplicate questions, and malformed strings that do not conform to identified entry specifications (see Appendix A.1).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4200000, "bbox": [[0.299, 0.129, 0.88, 0.138], [0.117, 0.146, 0.881, 0.155], [0.117, 0.166, 0.431, 0.173]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 122, "edu_l1_label": "IOS"}, {"txt": "Finally, at the last stage, two separate validators perform a final manual review of the collected files before merging them into KALEIDOSCOPE.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4200000, "bbox": [[0.431, 0.166, 0.874, 0.172], [0.117, 0.183, 0.78, 0.19]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 123, "edu_l1_label": "IOS"}, {"txt": "Quality control extends to the evaluation phase as well, where we analyze the most prominent failure modes.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4300000, "bbox": [[0.118, 0.215, 0.882, 0.224], [0.117, 0.235, 0.175, 0.241]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 124, "edu_l1_label": "IOS"}, {"txt": "During model inference, suspicious outputs, such as ambiguous answers, no response, or consistent failures across all models, are flagged for manual review.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4300000, "bbox": [[0.175, 0.235, 0.881, 0.242], [0.117, 0.252, 0.631, 0.258]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 125, "edu_l1_label": "IOS"}, {"txt": "For example, if all models fail in a specific question, we investigate further and may discover issues like missing images or incorrect labels.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4300000, "bbox": [[0.631, 0.252, 0.881, 0.258], [0.117, 0.269, 0.879, 0.275], [0.117, 0.283, 0.168, 0.292]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 126, "edu_l1_label": "IOS"}, {"txt": "If an issue is identified, the entire exam that contains the problematic question is reviewed for correction or removal.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4300000, "bbox": [[0.168, 0.283, 0.881, 0.292], [0.118, 0.3, 0.317, 0.309]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 127, "edu_l1_label": "IOS"}, {"txt": "This process guarantees that any errors in the benchmark questions are identified and addressed, further enhancing the reliability of the dataset.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4300000, "bbox": [[0.317, 0.3, 0.882, 0.31], [0.117, 0.318, 0.681, 0.327]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 128, "edu_l1_label": "IOS"}, {"txt": "2.3 Data Statistics", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4400000, "bbox": [[0.117, 0.359, 0.303, 0.371]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 3, "global_sentence_id": 129, "edu_l1_label": "IOS"}, {"txt": "The final KALEIDOSCOPE benchmark contains 20,911 questions across 18 languages belonging to 8 language families.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4500000, "bbox": [[0.118, 0.39, 0.881, 0.4], [0.117, 0.407, 0.277, 0.417]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 130, "edu_l1_label": "IOS"}, {"txt": "A total of 11,459 questions require an image to be answered (55%), while the remaining 9,452 (45%) are text-only.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4500000, "bbox": [[0.277, 0.407, 0.881, 0.417], [0.117, 0.426, 0.439, 0.434]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 131, "edu_l1_label": "IOS"}, {"txt": "The dataset covers 14 different subjects, grouped into 6broad domains.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4500000, "bbox": [[0.439, 0.426, 0.881, 0.434], [0.117, 0.441, 0.241, 0.451]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 132, "edu_l1_label": "IOS"}, {"txt": "Figure 1 presents an overview of the dataset; detailed statistics can be found in Table 1.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4500000, "bbox": [[0.241, 0.441, 0.881, 0.451], [0.118, 0.458, 0.184, 0.469]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 133, "edu_l1_label": "IOS"}, {"txt": "The majority of questions in KALEIDOSCOPE are multimodal, with the exact proportion varyring across languages, ranging from 50% to 100%, with some languages always requiring images for resolution.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4500000, "bbox": [[0.184, 0.458, 0.881, 0.468], [0.117, 0.479, 0.88, 0.486], [0.117, 0.493, 0.226, 0.503]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 134, "edu_l1_label": "IOS"}, {"txt": "Each exam question contains 17 fields, including source country, language, license, educational level,category, and multimodal information.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4600000, "bbox": [[0.118, 0.527, 0.882, 0.537], [0.117, 0.544, 0.478, 0.554]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 135, "edu_l1_label": "IOS"}, {"txt": "These fields are detailed in Appendix A.1.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4600000, "bbox": [[0.478, 0.544, 0.839, 0.554]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 136, "edu_l1_label": "IOS"}, {"txt": "The questions are formatted in MCQA format with 4 options and a single correct answer.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4600000, "bbox": [[0.839, 0.544, 0.881, 0.554], [0.117, 0.564, 0.784, 0.571]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 137, "edu_l1_label": "IOS"}, {"txt": "The subject is labeled in both English (category_en) and the source language(category_source_lang).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4600000, "bbox": [[0.784, 0.564, 0.88, 0.571], [0.118, 0.58, 0.845, 0.59]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 138, "edu_l1_label": "IOS"}, {"txt": "The educational level (e.g., high school, university entrance, professional licensing) is also included to ensure diverse representation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4600000, "bbox": [[0.845, 0.58, 0.881, 0.588], [0.117, 0.598, 0.881, 0.605], [0.118, 0.616, 0.352, 0.622]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 139, "edu_l1_label": "IOS"}, {"txt": "Multimodal questions additionally specify the type of image, such as graphs, tables, or diagrams.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4600000, "bbox": [[0.352, 0.616, 0.879, 0.622], [0.117, 0.633, 0.361, 0.64]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 140, "edu_l1_label": "IOS"}, {"txt": "Additionally, each entry includes metadata such as source details,licensing status, and ISO 639-1 language codes.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4600000, "bbox": [[0.361, 0.633, 0.882, 0.643], [0.117, 0.646, 0.488, 0.657]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 141, "edu_l1_label": "IOS"}, {"txt": "For a fine-grained analysis, each question includes detailed metadata, with examples provided in Appendix A.2.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4600000, "bbox": [[0.488, 0.646, 0.881, 0.657], [0.118, 0.664, 0.598, 0.674]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 142, "edu_l1_label": "IOS"}, {"txt": "The metadata allows us to evaluate how visual and textual elements interact in multimodal reasoning tasks, making the benchmark valuable for evaluating models across diverse scenarios.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4600000, "bbox": [[0.598, 0.664, 0.881, 0.674], [0.117, 0.681, 0.882, 0.691], [0.117, 0.701, 0.546, 0.708]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 143, "edu_l1_label": "IOS"}, {"txt": "KALEIDOSCOPE covers a wide range of languages, including low- and mid-resource languages such as Nepali, Lithuanian, Bengali, Telugu, Persian, Ukrainian, Croatian, Serbian, and Hungarian, as well as high-resource languages such as English,Spanish, Portuguese, Russian, French, German, Arabic,Hindi,and Dutch.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4700000, "bbox": [[0.118, 0.731, 0.882, 0.742], [0.117, 0.749, 0.882, 0.759], [0.117, 0.77, 0.882, 0.778], [0.124, 0.783, 0.257, 0.794]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 144, "edu_l1_label": "IOS"}, {"txt": "This selection allows us to evaluate how performance is affected by the amount of resources available for a given language.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4700000, "bbox": [[0.257, 0.783, 0.882, 0.794], [0.117, 0.804, 0.424, 0.812]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 145, "edu_l1_label": "IOS"}, {"txt": "The dataset spans 8 different language families, providing a broad linguistic range.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4700000, "bbox": [[0.424, 0.804, 0.88, 0.811], [0.117, 0.818, 0.289, 0.828]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 146, "edu_l1_label": "IOS"}, {"txt": "The number of questions per language varies significantly,from 126 for Nepali to 2000 for Portuguese, Serbian, and Persian.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4700000, "bbox": [[0.289, 0.818, 0.882, 0.828], [0.117, 0.836, 0.479, 0.845]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 147, "edu_l1_label": "IOS"}, {"txt": "The linguistic diversity present in KALEIDOSCOPE enables a robust evaluation of models across both widely spoken and underrepresented languages,making the dataset suitable for comprehensive multilingualassessment.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4700000, "bbox": [[0.479, 0.836, 0.882, 0.845], [0.117, 0.855, 0.882, 0.864], [0.118, 0.872, 0.672, 0.879]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 148, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4800000, "bbox": [[0.494, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 7, "global_sentence_id": 149, "edu_l1_label": "EDU_O"}, {"txt": "3 Experimental Setup", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 4900000, "bbox": [[0.117, 0.093, 0.377, 0.11]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 1, "global_sentence_id": 150, "edu_l1_label": "BOS"}, {"txt": "3.1 Models", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5000000, "bbox": [[0.117, 0.129, 0.232, 0.14]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 2, "global_sentence_id": 151, "edu_l1_label": "IOS"}, {"txt": "We benchmark both open-weights and closed multimodal vision-language models on KALEIDO-SCOPE, focusing on lighter open-weight models and larger closed models to assess performance across a wide range of model sizes.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5100000, "bbox": [[0.118, 0.16, 0.881, 0.167], [0.117, 0.179, 0.88, 0.186], [0.117, 0.197, 0.333, 0.204]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 152, "edu_l1_label": "IOS"}, {"txt": "The open-weight models3 include Aya-Vision-8B and 32B (Cohere-For-AI-Team, 2025),Molmo-7B-D (Deitke et al., 2024),PPangea-7B (Yue et al., 2025), and all sizes of Qwen2.5-VL-Instruct (Qwen-Team, 2025) (3B,7B,32B, and 72B) to analyze the impact of model scale on KALEIDOSCOPE.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5100000, "bbox": [[0.333, 0.197, 0.881, 0.201], [0.118, 0.211, 0.882, 0.22], [0.117, 0.229, 0.883, 0.239], [0.117, 0.249, 0.32, 0.256]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 153, "edu_l1_label": "IOS"}, {"txt": "All models have image and multilingual support; Aya-Vision supports 23 languages, Qwen2.5-VL supports 29 languages, and Pangea was trained on a dataset spanning 39 different languages, making them strong candidates for multimodal and multilingual evalua-tion.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5100000, "bbox": [[0.32, 0.249, 0.881, 0.256], [0.117, 0.263, 0.881, 0.274], [0.117, 0.28, 0.881, 0.287], [0.117, 0.298, 0.154, 0.307]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 154, "edu_l1_label": "IOS"}, {"txt": "For the closed models, we evaluate GPT-4o (OpenAI et al., 2024) (2024/08/06), Claude 3.5Sonnet (Anthropic, 2024) (2024/10/22), and Gemini 1.5 Pro (Google et al., 2024).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5100000, "bbox": [[0.154, 0.298, 0.88, 0.306], [0.117, 0.314, 0.757, 0.323]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 155, "edu_l1_label": "IOS"}, {"txt": "3.2 Evaluation Setup", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5200000, "bbox": [[0.117, 0.357, 0.323, 0.37]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 4, "global_sentence_id": 156, "edu_l1_label": "IOS"}, {"txt": "We designed two distinct evaluation setups to accommodate VLMs' varying reasoning and instruction-following capabilities.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5300000, "bbox": [[0.117, 0.387, 0.891, 0.397], [0.117, 0.404, 0.283, 0.414]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 157, "edu_l1_label": "IOS"}, {"txt": "For closed models, we designed zero-shot prompts using the Chain-of-Thought (CoT) method (Wei et al., 2022).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5300000, "bbox": [[0.283, 0.404, 0.879, 0.414], [0.117, 0.42, 0.384, 0.431]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 158, "edu_l1_label": "IOS"}, {"txt": "The prompt instructs the model to reason through the correct answer and to write the chosen option within specific <ANSWER> </ANSWER> tags.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5300000, "bbox": [[0.384, 0.42, 0.88, 0.431], [0.117, 0.442, 0.761, 0.448]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 159, "edu_l1_label": "IOS"}, {"txt": "This approach is natural and aligns the real-world application of MCQs, encouraging the model to generate a step-by-step reasoning before selecting the final answer.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5300000, "bbox": [[0.761, 0.442, 0.88, 0.448], [0.117, 0.455, 0.881, 0.465], [0.117, 0.475, 0.556, 0.482]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 160, "edu_l1_label": "IOS"}, {"txt": "We define a common template,ensuring equal evaluation conditions for all models (see Appendix A.5).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5300000, "bbox": [[0.556, 0.475, 0.881, 0.484], [0.117, 0.493, 0.603, 0.5]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 161, "edu_l1_label": "IOS"}, {"txt": "The instructions were translated to all the evaluated languages, creating a fully in-language setup, following the methodology proposed by Romanou et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5300000, "bbox": [[0.603, 0.493, 0.881, 0.5], [0.117, 0.51, 0.881, 0.517], [0.117, 0.524, 0.264, 0.534]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 162, "edu_l1_label": "IOS"}, {"txt": "(2024).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5300000, "bbox": [[0.264, 0.524, 0.326, 0.535]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 163, "edu_l1_label": "IOS"}, {"txt": "The selected choice is extracted using string matching of the tags.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5300000, "bbox": [[0.326, 0.524, 0.843, 0.534]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 164, "edu_l1_label": "IOS"}, {"txt": "For the smaller open-weight models, which typically have limited capacity for complex reasoning,CoT prompting proved less effective in our preliminary experiments (see Appendix A.7 for details).Therefore, we implemented a direct answer generation approach, instructing the models to produce a JSON output containing their choice within a predefined 'choice' field.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5400000, "bbox": [[0.117, 0.557, 0.883, 0.569], [0.118, 0.575, 0.882, 0.585], [0.118, 0.592, 0.881, 0.602], [0.117, 0.612, 0.716, 0.619]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 165, "edu_l1_label": "IOS"}, {"txt": "The instruction was always in English, independent of the question language (see Appendix A.5).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5400000, "bbox": [[0.716, 0.612, 0.881, 0.619], [0.117, 0.629, 0.717, 0.636]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 166, "edu_l1_label": "IOS"}, {"txt": "This setup simplifies the task,reducing errors related to multi-step reasoning or formatting inconsistencies, and ensuring a straightforward answer extraction.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5400000, "bbox": [[0.717, 0.629, 0.881, 0.636], [0.117, 0.645, 0.881, 0.655], [0.117, 0.664, 0.404, 0.671]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 167, "edu_l1_label": "IOS"}, {"txt": "Further discussion about model output error analysis can be found in section 5.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5400000, "bbox": [[0.404, 0.664, 0.881, 0.67], [0.117, 0.678, 0.262, 0.688]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 168, "edu_l1_label": "IOS"}, {"txt": "3.3 Evaluation Metrics", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5500000, "bbox": [[0.117, 0.72, 0.338, 0.731]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 7, "global_sentence_id": 169, "edu_l1_label": "IOS"}, {"txt": "Given the multiple-choice nature of the task, we use accuracy as the primary evaluation metric.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5600000, "bbox": [[0.117, 0.751, 0.852, 0.761]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 170, "edu_l1_label": "IOS"}, {"txt": "We report overall accuracy across all questions, as well as accuracy on the subset of questions where the model produces valid responses.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5600000, "bbox": [[0.852, 0.751, 0.881, 0.761], [0.117, 0.771, 0.881, 0.778], [0.117, 0.788, 0.365, 0.795]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 171, "edu_l1_label": "IOS"}, {"txt": "A respponse is considered valid if the model successfully provides an answer in the expected format and selects a valid option (i.e., one of the letters A, B, C, D).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5600000, "bbox": [[0.365, 0.788, 0.88, 0.795], [0.117, 0.806, 0.825, 0.812]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 172, "edu_l1_label": "IOS"}, {"txt": "Invalid responses typically result from missing the selected choice, selecting an invalid option, or refusal to answer.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5600000, "bbox": [[0.825, 0.806, 0.881, 0.812], [0.117, 0.823, 0.881, 0.83], [0.125, 0.84, 0.178, 0.846]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 173, "edu_l1_label": "IOS"}, {"txt": "To quantify these cases, we report the Format Error Rate, which measures the proportion", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5600000, "bbox": [[0.178, 0.84, 0.88, 0.847]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 174, "edu_l1_label": "IOS"}, {"txt": "3All open-weight models are evaluated locally using1xNVIDIA Ampere A100 GPU with 64GB of memory for models up to 8B, and 4xA100 for models on the range 32B-72B.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5700000, "bbox": [[0.142, 0.861, 0.486, 0.872], [0.493, 0.864, 0.883, 0.867], [0.117, 0.88, 0.549, 0.886]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 9, "global_sentence_id": 175, "edu_l1_label": "EDU_O"}, {"txt": "To ensure a consistent evaluation environment,we set the temperature to 0.7, the maximum token generation to 1024, and the image size to 512x512 for all models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5700000, "bbox": [[0.549, 0.88, 0.881, 0.886], [0.118, 0.893, 0.871, 0.899]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 9, "global_sentence_id": 176, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5700000, "bbox": [[0.494, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 10, "global_sentence_id": 177, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5700000, "bbox": [[0.118, 0.092, 0.879, 0.328]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 178, "edu_l1_label": "EDU_O"}, {"txt": "Table 2: Performance Evaluation on KALEIDOSCOPE. Results are reported as macro-averaged accuracy ($\\%$) across all languages (equal weight per language). Acc.: Accuracy over all samples;F.E.: Format Error rate (invalid responses); Valid Acc.: Accuracy excluding invalid responses.Metrics are shown for the full dataset (Overall), multimodal inputs (Multimodal), and text-only inputs (Text-only).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5800000, "bbox": [[0.115, 0.346, 0.885, 0.427]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 179, "edu_l1_label": "EDU_O"}, {"txt": "of questions for which the model fails to generate a valid answer.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5900000, "bbox": [[0.117, 0.462, 0.615, 0.469]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 180, "edu_l1_label": "IOS"}, {"txt": "For grouped results, we report the macro average across languages, i.e.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5900000, "bbox": [[0.615, 0.462, 0.881, 0.468], [0.117, 0.479, 0.398, 0.485]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 181, "edu_l1_label": "IOS"}, {"txt": "all languages have equal weight when computing the score.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 5900000, "bbox": [[0.398, 0.479, 0.859, 0.487]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 182, "edu_l1_label": "IOS"}, {"txt": "4 Results", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6000000, "bbox": [[0.117, 0.52, 0.237, 0.533]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 4, "global_sentence_id": 183, "edu_l1_label": "BOS"}, {"txt": "4.1 Overall Performance", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6100000, "bbox": [[0.117, 0.557, 0.353, 0.567]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 5, "global_sentence_id": 184, "edu_l1_label": "IOS"}, {"txt": "We benchmark a wide variety of models on KALEIDOSCOPE and present the main results in Table 2.Claude 3.5 Sonnet achieves the highest overall accuracy (62.91$\\%$), followed closely by Gemini 1.5Pro $(62.10\\%$).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6200000, "bbox": [[0.115, 0.586, 0.885, 0.686]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 185, "edu_l1_label": "IOS"}, {"txt": "GPT-4o performs notably worse $(58.32\\%$), with a high format error rate (6.52%overall, with at $10.50\\%$ for the multimodal split).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6200000, "bbox": [[0.115, 0.586, 0.885, 0.686]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 186, "edu_l1_label": "IOS"}, {"txt": "However, when considering only valid answers,GPT-4o's performance improves significantly$(+3.$78percentage points), closing the gap with other closed models and highlighting the impact of format errors (see Section 5).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6200000, "bbox": [[0.115, 0.586, 0.885, 0.686]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 187, "edu_l1_label": "IOS"}, {"txt": "Among open-weight models, Qwen2.5-VL-72B achieves the highest accuracy (52.94%), which is expected given its larger number of parameters.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6300000, "bbox": [[0.114, 0.706, 0.885, 0.787]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 188, "edu_l1_label": "IOS"}, {"txt": "In the lightweight category ($\\leq$8B parameters),Qwen2.5-VL-7B outperforms all others in both multimodal and text-only questions, with accuracy of $39.56\\%$.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6300000, "bbox": [[0.114, 0.706, 0.885, 0.787]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 189, "edu_l1_label": "IOS"}, {"txt": "Open models generally maintain low format error rates, except for Pangea, which has the highest format error rate at $13\\%$.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6300000, "bbox": [[0.114, 0.706, 0.885, 0.787]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 190, "edu_l1_label": "IOS"}, {"txt": "Table 2 also summarizes results for both the multimodal and text-only benchmark splits.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6400000, "bbox": [[0.118, 0.81, 0.824, 0.82]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 191, "edu_l1_label": "IOS"}, {"txt": "Across all models, multimodal performance is lower than text-only, with a larger drop for closed models:GPT-4o drops 21.6 accuracy points overall (10.29 points for valid answers).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6400000, "bbox": [[0.824, 0.81, 0.881, 0.819], [0.117, 0.83, 0.882, 0.836], [0.118, 0.844, 0.716, 0.854]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 192, "edu_l1_label": "IOS"}, {"txt": "Open-weight models show smaller gaps, with Molmo having the narrowest gap of only 3.69 accuracy points, though multimodal performance remains lower.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6400000, "bbox": [[0.716, 0.844, 0.881, 0.854], [0.117, 0.863, 0.88, 0.871], [0.117, 0.881, 0.426, 0.888]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 193, "edu_l1_label": "IOS"}, {"txt": "Among lightweight models, Qwen2.5-VL-7B leads on both splits, with a relatively small gap, followed by Aya-Vision-8B on text-only samples and Qwen2.5-", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6400000, "bbox": [[0.426, 0.881, 0.88, 0.888], [0.117, 0.899, 0.881, 0.905]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 194, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6500000, "bbox": [[0.494, 0.936, 0.506, 0.948]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 195, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6500000, "bbox": [[0.114, 0.085, 0.49, 0.39]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 1, "global_sentence_id": 196, "edu_l1_label": "EDU_O"}, {"txt": "(a) Accuracy as a function of modality.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6600000, "bbox": [[0.16, 0.39, 0.442, 0.401]]}]}, "tags": ["figure_caption"], "label": "figure_title", "web_segment_id": 2, "global_sentence_id": 197, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6700000, "bbox": [[0.513, 0.09, 0.886, 0.387]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 3, "global_sentence_id": 198, "edu_l1_label": "EDU_O"}, {"txt": "(b) Accuracy as a function of textual script.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6800000, "bbox": [[0.541, 0.389, 0.857, 0.402]]}]}, "tags": ["figure_caption"], "label": "figure_title", "web_segment_id": 4, "global_sentence_id": 199, "edu_l1_label": "EDU_O"}, {"txt": "Figure 2: Model Performance Analysis on KALEIDOSCOPE. (a) Accuracy (%) of models on multimodal and text-only questions, highlighting low performmance on multimodal samples. (b)Accuracy (%) by script type, revealing biases for latin scripts. Accuracy over valid responses is used to generate both figures. Identity line is added to show parity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 6900000, "bbox": [[0.115, 0.417, 0.883, 0.482]]}]}, "tags": ["figure_caption"], "label": "figure_title", "web_segment_id": 5, "global_sentence_id": 200, "edu_l1_label": "EDU_O"}, {"txt": "VL-3B in the multimodal split.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7000000, "bbox": [[0.117, 0.513, 0.369, 0.523]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 201, "edu_l1_label": "IOS"}, {"txt": "Closed models perform well on text-only questions,highlighting their strength in this modality buit also the challenges of multimodal processing.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7000000, "bbox": [[0.369, 0.513, 0.881, 0.524], [0.117, 0.532, 0.752, 0.54]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 202, "edu_l1_label": "IOS"}, {"txt": "In contrast,the smaller variation in performance across both splits by open-weight models suggests that they are less specialized, with lower overall performance but greater robustness in multimodal tasks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7000000, "bbox": [[0.752, 0.532, 0.881, 0.54], [0.116, 0.551, 0.881, 0.557], [0.117, 0.568, 0.784, 0.574]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 203, "edu_l1_label": "IOS"}, {"txt": "Lightweight models exhibit similar behavior across script types and modalities (Figure 2a), with Molmo showing the most balanced performance between Multimodal vs. text-only samples and Latin vs. non-Latin scripts.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7000000, "bbox": [[0.784, 0.568, 0.879, 0.574], [0.117, 0.585, 0.881, 0.593], [0.117, 0.6, 0.881, 0.609], [0.117, 0.619, 0.174, 0.626]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 204, "edu_l1_label": "IOS"}, {"txt": "4.2 Not All Image Types are Equal", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7100000, "bbox": [[0.117, 0.658, 0.449, 0.669]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 7, "global_sentence_id": 205, "edu_l1_label": "IOS"}, {"txt": "KALEIDOSCOPE contains eight visual information types, with accuracy varying significantly by com-plexity (Table 3).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7200000, "bbox": [[0.12, 0.689, 0.88, 0.699], [0.117, 0.71, 0.257, 0.717]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 206, "edu_l1_label": "IOS"}, {"txt": "Simpler inputs like text-rich images (Qwen2.5-VL-7B:76.3%;GPT-4o: 86.2%)and photos score higher than technical categories like Formulas and Diagrams (Qwen2.5-VL-7B:38.0%; GPT-4o: 62.9%).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7200000, "bbox": [[0.257, 0.71, 0.88, 0.718], [0.117, 0.727, 0.881, 0.733], [0.117, 0.741, 0.315, 0.751]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 207, "edu_l1_label": "IOS"}, {"txt": "Notably, Qwen2.5-VL-72B ranks second in text-rich images, surpassing both Gemini and Claude.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7200000, "bbox": [[0.315, 0.741, 0.881, 0.752], [0.117, 0.757, 0.314, 0.768]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 208, "edu_l1_label": "IOS"}, {"txt": "Larger models show specialized strengths: Gemini 1.5 Pro dominates For-mulas and Figures, GPT-4o leads in text-rich images, and Claude 3.5 Sonnet achieves the highest scores in Diagrams, Graphs, and Maps.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7200000, "bbox": [[0.314, 0.757, 0.881, 0.765], [0.121, 0.778, 0.879, 0.785], [0.117, 0.795, 0.431, 0.801]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 209, "edu_l1_label": "IOS"}, {"txt": "In contrast, Qwen2.5-VL-7B consistently outperforms all lightweight models across categories, demonstrating broader capability despite lower absolute scores.The results reveal a clear hierarchy: models handle simple visuals well but struggle with structured or symbolic data, a pattern consistent across architectures but more pronounced in smaller models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7200000, "bbox": [[0.431, 0.795, 0.883, 0.802], [0.118, 0.809, 0.882, 0.819], [0.118, 0.825, 0.881, 0.835], [0.117, 0.847, 0.883, 0.854]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 210, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7300000, "bbox": [[0.493, 0.936, 0.506, 0.949]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 211, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7300000, "bbox": [[0.115, 0.09, 0.883, 0.338]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 212, "edu_l1_label": "EDU_O"}, {"txt": "Table 3: Model Performance Breakdown by Image Type in KALEIDOSCOPE. Accuracy (%)over valid answers across image type. Bold values indicate top-performing model.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7400000, "bbox": [[0.115, 0.354, 0.882, 0.384]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 213, "edu_l1_label": "EDU_O"}, {"txt": "4.3 Resource and Script Sensitivity in Models", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7500000, "bbox": [[0.116, 0.415, 0.547, 0.426]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 3, "global_sentence_id": 214, "edu_l1_label": "IOS"}, {"txt": "Performance in KALEIDOSCOPE varies widely across all 18 languages (see Figure 3).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.117, 0.446, 0.781, 0.456]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 215, "edu_l1_label": "IOS"}, {"txt": "Models gen-erally perform well in high-resource languages (e.g., English, Spanish, German) but struggle with lower-and mid-resource ones, such as Nepali and Telugu.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.781, 0.446, 0.881, 0.456], [0.117, 0.466, 0.88, 0.472], [0.117, 0.48, 0.577, 0.49]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 216, "edu_l1_label": "IOS"}, {"txt": "This can be attributed to the limited training data for these languages, complex scripts, and the exclusive use of multimodal samples for these languages (see Table 1), which are inherently more challenging.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.577, 0.48, 0.881, 0.49], [0.117, 0.499, 0.881, 0.507], [0.117, 0.515, 0.68, 0.525]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 217, "edu_l1_label": "IOS"}, {"txt": "Lithuanian, despite being mid-resource language, stands out as the highest-performing language, with Claude 3.5 Sonnet lead-ing in overall accuracy.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.68, 0.515, 0.881, 0.525], [0.12, 0.535, 0.878, 0.541], [0.117, 0.549, 0.297, 0.558]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 218, "edu_l1_label": "IOS"}, {"txt": "This might be due the fact that all Lithuanian questions belong to College Graduation Exams, and have a major subject composition of Social Sciences and Humanities in opposition to STEM subjects, which may align well with the models' capabilities.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.297, 0.549, 0.881, 0.559], [0.117, 0.566, 0.88, 0.575], [0.117, 0.586, 0.762, 0.593]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 219, "edu_l1_label": "IOS"}, {"txt": "Closed models show similar performance within each language, except for German, where Claude excels.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.762, 0.586, 0.881, 0.592], [0.117, 0.602, 0.821, 0.61]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 220, "edu_l1_label": "IOS"}, {"txt": "In con-trast, Qwen2.5-VL-7B consistently leads all lightweight models for almost every language,and the heavier Qwen2.5-VL-72B shows the benefits of model scale.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7600000, "bbox": [[0.821, 0.602, 0.881, 0.609], [0.117, 0.618, 0.881, 0.626], [0.117, 0.634, 0.579, 0.644]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 221, "edu_l1_label": "IOS"}, {"txt": "The results show that all models are biased towards Latin script languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7700000, "bbox": [[0.118, 0.669, 0.699, 0.678]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 222, "edu_l1_label": "IOS"}, {"txt": "As shown in Figure 2b,all models are above the parity line, exhibiting consistent higher performance for Latin scripts compared to non-Latin scripts.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7700000, "bbox": [[0.699, 0.669, 0.883, 0.68], [0.117, 0.689, 0.881, 0.695], [0.117, 0.706, 0.359, 0.712]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 223, "edu_l1_label": "IOS"}, {"txt": "Full results can be found in Table 9.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7700000, "bbox": [[0.359, 0.706, 0.647, 0.712]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 224, "edu_l1_label": "IOS"}, {"txt": "4.4 STEM Questions Expose Model Deficiencies", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7800000, "bbox": [[0.117, 0.745, 0.574, 0.756]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 6, "global_sentence_id": 225, "edu_l1_label": "IOS"}, {"txt": "KALEIDOSCOPE consists of exams covering 14 subjects and domains, with Table 4 summarizing model performance on the multimodal split across subjects.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7900000, "bbox": [[0.121, 0.776, 0.881, 0.787], [0.117, 0.795, 0.592, 0.803]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 226, "edu_l1_label": "IOS"}, {"txt": "We observe that all models perform significantly better on Humanities & Social Science questions compared to other domains.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7900000, "bbox": [[0.592, 0.795, 0.874, 0.802], [0.117, 0.812, 0.841, 0.819]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 227, "edu_l1_label": "IOS"}, {"txt": "The closed models achieve high accuracy in areas like Sociology (Claude: 93.4%, GPT-4o: 93.2%), So-cial Sciences (GPT-4o: 88.1%, Gemini: 85.7%), and Language (GPT-4o: 85.8%, Claude: 85.5%).In contrast, performance in STEM subjects, including Mathematics, Physics, and Engineering, is notably lower, with most models scoring below 50%.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7900000, "bbox": [[0.841, 0.812, 0.881, 0.82], [0.117, 0.83, 0.881, 0.836], [0.117, 0.847, 0.882, 0.854], [0.117, 0.86, 0.88, 0.871], [0.117, 0.881, 0.531, 0.888]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 228, "edu_l1_label": "IOS"}, {"txt": "This suggests that while they are generally capable of recognizing visual content and retrieving surface-level knowledge, they fall short when", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 7900000, "bbox": [[0.531, 0.881, 0.881, 0.89], [0.117, 0.898, 0.88, 0.905]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 229, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8000000, "bbox": [[0.49, 0.938, 0.511, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 8, "global_sentence_id": 230, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8000000, "bbox": [[0.114, 0.085, 0.886, 0.369]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 1, "global_sentence_id": 231, "edu_l1_label": "EDU_O"}, {"txt": "Figure 3: Multimodal Accuracy by Language in KALEIDOSCOPE. Reports performance (ac-curacy %) for closed models and open-weight models on multimodal questions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8100000, "bbox": [[0.115, 0.375, 0.883, 0.405]]}]}, "tags": ["figure_caption"], "label": "figure_title", "web_segment_id": 2, "global_sentence_id": 232, "edu_l1_label": "EDU_O"}, {"txt": "it comes to performing the multi-step reasoning and problem-solving required in STEM subjects.Answering these questions often demands not just factual recall but also the ability to interpret complex diagrams, apply mathematical concepts, and reason through scientific principles - capa-bilities that current models have yet to fully master.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8200000, "bbox": [[0.117, 0.437, 0.882, 0.446], [0.118, 0.454, 0.88, 0.463], [0.117, 0.474, 0.881, 0.48], [0.117, 0.487, 0.533, 0.498]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 233, "edu_l1_label": "IOS"}, {"txt": "This highlights a key gap in their ability to bridge perception and reasoning, particularly in tasks that require deeper analytical thinking.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8200000, "bbox": [[0.533, 0.487, 0.881, 0.498], [0.117, 0.504, 0.844, 0.515]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 234, "edu_l1_label": "IOS"}, {"txt": "5 Analysis", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8300000, "bbox": [[0.117, 0.549, 0.248, 0.562]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 4, "global_sentence_id": 235, "edu_l1_label": "BOS"}, {"txt": "5.1 How Sensitive Are VLMs to Missing or Incorrect Images?", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8400000, "bbox": [[0.117, 0.585, 0.69, 0.597]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 5, "global_sentence_id": 236, "edu_l1_label": "IOS"}, {"txt": "To evaluate the dependency of multimodal questions on images, and the impact of incorrect image associations,we conducted an experiment using the multimodal split of KALEIDOSCOPE.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8500000, "bbox": [[0.118, 0.616, 0.881, 0.626], [0.117, 0.637, 0.819, 0.644]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 237, "edu_l1_label": "IOS"}, {"txt": "Follow-ing Elliott (2018); Thomason et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8500000, "bbox": [[0.819, 0.637, 0.881, 0.643], [0.117, 0.652, 0.393, 0.661]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 238, "edu_l1_label": "IOS"}, {"txt": "(2019), we created two modified versions of the dataset: (1) a 'No Image' split, where all images were removed, and (2) a 'Random Image' split,where images were randomly reassigned to questions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8500000, "bbox": [[0.393, 0.652, 0.881, 0.661], [0.119, 0.669, 0.883, 0.677], [0.117, 0.688, 0.433, 0.694]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 239, "edu_l1_label": "IOS"}, {"txt": "The aim of this experiment is to assess how much the models rely on the visual information.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8500000, "bbox": [[0.433, 0.688, 0.881, 0.695], [0.117, 0.705, 0.421, 0.711]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 240, "edu_l1_label": "IOS"}, {"txt": "We evaluate the performance of Qwen2.5-VL-7B on these modified splits, and the results are shown in Table 5.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8500000, "bbox": [[0.421, 0.705, 0.881, 0.711], [0.117, 0.722, 0.53, 0.729]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 241, "edu_l1_label": "IOS"}, {"txt": "We observe that the model performs above the random baseline (25%) across all three splits, indi-cating some ability to reason from text alone.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8600000, "bbox": [[0.119, 0.753, 0.88, 0.763], [0.117, 0.773, 0.472, 0.78]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 242, "edu_l1_label": "IOS"}, {"txt": "However, there is a drop in performance (-3.41% in Total Accuracy) when questions are presented without images, suggesting that the model does rely on visual information for accurate answers.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8600000, "bbox": [[0.472, 0.773, 0.88, 0.78], [0.118, 0.787, 0.881, 0.799], [0.117, 0.808, 0.45, 0.815]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 243, "edu_l1_label": "IOS"}, {"txt": "The performance drop is similar for both modifications;however, we observe a significantly larger format error when the model is tested with irrelevant im-ages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8600000, "bbox": [[0.45, 0.808, 0.882, 0.816], [0.117, 0.822, 0.88, 0.831], [0.117, 0.842, 0.158, 0.849]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 244, "edu_l1_label": "IOS"}, {"txt": "In several of these cases, the model actually acknowledges that the image does not correspond to the question.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8600000, "bbox": [[0.158, 0.842, 0.881, 0.849], [0.117, 0.857, 0.243, 0.866]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 245, "edu_l1_label": "IOS"}, {"txt": "In contrast, in experiments with no images, the format error rate is almost zero,", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8600000, "bbox": [[0.243, 0.857, 0.883, 0.867]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 246, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8700000, "bbox": [[0.492, 0.938, 0.508, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 8, "global_sentence_id": 247, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8700000, "bbox": [[0.187, 0.092, 0.802, 0.509]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 248, "edu_l1_label": "EDU_O"}, {"txt": "Table 4: Subject-wise Performance on KALEIDOSCOPE's Multimodal Questions. Valid accuracy ($\\%$) across examination subjects for multimodal samples, with bold highlighting top-performing models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8800000, "bbox": [[0.115, 0.52, 0.885, 0.568]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 249, "edu_l1_label": "EDU_O"}, {"txt": "indicating that the model attempts to answer even when visual inputs are missing.4", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 8900000, "bbox": [[0.117, 0.6, 0.767, 0.604]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 250, "edu_l1_label": "IOS"}, {"txt": "5.2 Scaling Model Size Improves Performance", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9000000, "bbox": [[0.117, 0.642, 0.551, 0.653]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 4, "global_sentence_id": 251, "edu_l1_label": "IOS"}, {"txt": "To analyze the impact of model size on KALEIDOSCOPE performance, we evaluated all four variants of Qwen2.5-VL.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9100000, "bbox": [[0.118, 0.672, 0.881, 0.682], [0.117, 0.692, 0.239, 0.699]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 252, "edu_l1_label": "IOS"}, {"txt": "We selected this model family for its well-distributed size range, as well as being the best performing model in the open weight model category.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9100000, "bbox": [[0.239, 0.692, 0.881, 0.701], [0.117, 0.708, 0.613, 0.717]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 253, "edu_l1_label": "IOS"}, {"txt": "We follow the same experimental setup for all model versions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9100000, "bbox": [[0.613, 0.708, 0.882, 0.716], [0.116, 0.727, 0.336, 0.734]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 254, "edu_l1_label": "IOS"}, {"txt": "Figure 4 shows the performance of Qwen2.5-VL variants on KALEIDOSCOPE.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9200000, "bbox": [[0.118, 0.758, 0.721, 0.768]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 255, "edu_l1_label": "IOS"}, {"txt": "Model size is shown in the x-axis (log-scale), while the y-axis displays accuracy for multimodal and text-only splits, and overall score.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9200000, "bbox": [[0.721, 0.758, 0.88, 0.768], [0.117, 0.776, 0.881, 0.785], [0.117, 0.795, 0.219, 0.802]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 256, "edu_l1_label": "IOS"}, {"txt": "We observe a linear relationship between the logarithm of the model size and accuracy,with larger models showing significant gains.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9200000, "bbox": [[0.219, 0.795, 0.883, 0.804], [0.118, 0.812, 0.47, 0.819]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 257, "edu_l1_label": "IOS"}, {"txt": "The largest open model model evaluated, Qwen2.5-VL-72B,still underperforms the closed models, however, these results highlight the effectiveness of scaling for open models, with clear and predictable improvements at each size tier.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9200000, "bbox": [[0.47, 0.812, 0.881, 0.819], [0.118, 0.826, 0.882, 0.836], [0.117, 0.847, 0.759, 0.854]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 258, "edu_l1_label": "IOS"}, {"txt": "${}^{4}\\mathrm {We}$observed that Qwen2.5-VL-7B tends to hallucinate when no image is present.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9300000, "bbox": [[0.115, 0.867, 0.884, 0.907]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 7, "global_sentence_id": 259, "edu_l1_label": "EDU_O"}, {"txt": "In a simple experiment using the prompt \"Describe the following image\", the model correctly describes the input image when provided.However, when no image is passed, the model hallucinates and generates a random description.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9300000, "bbox": [[0.115, 0.867, 0.884, 0.907]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 260, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9400000, "bbox": [[0.492, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 8, "global_sentence_id": 261, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9400000, "bbox": [[0.252, 0.09, 0.752, 0.186]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 262, "edu_l1_label": "EDU_O"}, {"txt": "Table 5: Image Relevance Analysis for Qwen2.5-VL-7B on KALEIDOSCOPE. Model perfor-mance across the standard multimodal, Random Image, and No-Image setups to assess the impact of visual information on question-answering accuracy.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9500000, "bbox": [[0.115, 0.202, 0.883, 0.249]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 263, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9600000, "bbox": [[0.187, 0.264, 0.808, 0.436]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 3, "global_sentence_id": 264, "edu_l1_label": "EDU_O"}, {"txt": "Figure 4: Model Size Analysis for Qwen2.5-VL Models. Performance improvement across three model sizes (3B,7B,32B, and 72B parameters) on KALEIDOSCOPE's multimodal tasks, demon-strating consistent gains from increased model capacity. Note that x-axis is shown in log-scale.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9700000, "bbox": [[0.114, 0.444, 0.885, 0.493]]}]}, "tags": ["figure_caption"], "label": "figure_title", "web_segment_id": 4, "global_sentence_id": 265, "edu_l1_label": "EDU_O"}, {"txt": "5.3 To What Extent Do Textual Augmentations Boost VLM Capabilities?", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9800000, "bbox": [[0.117, 0.525, 0.806, 0.535]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 5, "global_sentence_id": 266, "edu_l1_label": "IOS"}, {"txt": "The significant performance gap between text-only and multimodal responses raises critical ques-tions about the strengths and weaknesses of the visual processing in the tested models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9900000, "bbox": [[0.118, 0.555, 0.88, 0.564], [0.117, 0.573, 0.783, 0.582]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 267, "edu_l1_label": "IOS"}, {"txt": "In this anal-ysis, we investigate to what extent do visual processing constraints limmit multimodal capabilities,and conversely, can automatically generated textual augmentation improve model performance?", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 9900000, "bbox": [[0.783, 0.573, 0.882, 0.579], [0.116, 0.592, 0.883, 0.601], [0.117, 0.609, 0.859, 0.616]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 268, "edu_l1_label": "IOS"}, {"txt": "To explore this direction, we generate synthetic captions (using Gemini 1.5 Pro) and Optical Char-acter Recognition (OCR) text (Tesseract (Smith, 2007)) for all images in KALEIDOSCOPE, aligning with the methodology of (Das et al., 2024).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10000000, "bbox": [[0.118, 0.641, 0.882, 0.648], [0.117, 0.661, 0.881, 0.669], [0.118, 0.677, 0.455, 0.685]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 269, "edu_l1_label": "IOS"}, {"txt": "Unlike prior work that completely replaces images with text, we evaluate whether a VLM augmented with these textual inputs can boost performance.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10000000, "bbox": [[0.455, 0.677, 0.88, 0.685], [0.117, 0.693, 0.855, 0.702]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 270, "edu_l1_label": "IOS"}, {"txt": "Table 6 shows the results of augmenting visual inputs with synthetic captions and OCR text across diverse image types in KALEIDOSCOPE, measured by valid accuracy ($\\%$).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10100000, "bbox": [[0.115, 0.725, 0.885, 0.91]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 271, "edu_l1_label": "IOS"}, {"txt": "Overall, the addition of a caption and OCR text improves the performance of the selected models in 5 out of 8 image types.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10100000, "bbox": [[0.115, 0.725, 0.885, 0.91]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 272, "edu_l1_label": "IOS"}, {"txt": "Both models experienced a performance boost coordinately for Graph and Formula.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10100000, "bbox": [[0.115, 0.725, 0.885, 0.91]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 273, "edu_l1_label": "IOS"}, {"txt": "The experiment reveals that the utility of textual augmentation depends critically on image content type.While Gemini 1.5 Pro dominates overall performance, Qwen2.5-VL-7B demonstrates selective gains when provided with captions and OCR: improvements in Graph$(+0.9\\%)$,Photo$(+0.2\\%)$Formula$(+2.4\\%)$,and Text(+3.5%) suggest that textual augmentation aids interpretation of con-tent where visual elements are tightly coupled with symbolic or linguistic features (e.g., labeled axes,embedded text, or mathematical notation).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10100000, "bbox": [[0.115, 0.725, 0.885, 0.91]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 274, "edu_l1_label": "IOS"}, {"txt": "Conversely, performance declines for Diagram $(-0.1\\%)$,Map(-1.3%),and Table ($-$6.3%) with augmentation, implying that synthetic captions", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10100000, "bbox": [[0.115, 0.725, 0.885, 0.91]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 275, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10200000, "bbox": [[0.492, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 276, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10200000, "bbox": [[0.26, 0.09, 0.743, 0.284]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 277, "edu_l1_label": "EDU_O"}, {"txt": "Table 6: Accuracy on augmented multimodal inputs with image captions. Results are grouped by image type. We report Valid Accuracy $(\\%)$; the highest scores are highlighted in bold for each model. Macro averaged accuracy is reported over language for both methods.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10300000, "bbox": [[0.114, 0.297, 0.885, 0.345]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 278, "edu_l1_label": "EDU_O"}, {"txt": "may introduce noise or fail to capture structural relationships critical to these categories.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10400000, "bbox": [[0.115, 0.376, 0.885, 0.508]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 279, "edu_l1_label": "IOS"}, {"txt": "Gemini's robustness across modalities$(\\leq 2\\%$variation in most categories) suggests its stronger native visual understanding reduces reliance on supplementary text.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10400000, "bbox": [[0.115, 0.376, 0.885, 0.508]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 280, "edu_l1_label": "IOS"}, {"txt": "The results underscore that captioning ef-fectiveness is context-dependent: text augmentation benefits models most when (1) visual content inherently contains extractable text (e.g., Photo with signs, Text regions) or (2) symbolic patterns (e.g., formulas, graphs) require disambiguation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10400000, "bbox": [[0.115, 0.376, 0.885, 0.508]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 281, "edu_l1_label": "IOS"}, {"txt": "However, for structurally complex or text-sparse images (e.g., Map, Diagram), captioning may not compensate for deficiencies in spatial or relational reasoning.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10400000, "bbox": [[0.115, 0.376, 0.885, 0.508]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 282, "edu_l1_label": "IOS"}, {"txt": "Full results, including total accuracy and format error, can be found in Table 11.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10400000, "bbox": [[0.115, 0.376, 0.885, 0.508]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 283, "edu_l1_label": "IOS"}, {"txt": "5.4 Format Errors", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10500000, "bbox": [[0.117, 0.539, 0.294, 0.549]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 4, "global_sentence_id": 284, "edu_l1_label": "IOS"}, {"txt": "While our experimental setup ensures a majority of answers were extracted from model outputs,we observe occasional failures:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10600000, "bbox": [[0.117, 0.569, 0.883, 0.581], [0.117, 0.589, 0.363, 0.596]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 285, "edu_l1_label": "IOS"}, {"txt": " models struggle to follow instructions, the outputs contain for-matting errors, or models refuse toanswer (particularly for health-related or ethical questions).Figure 5 shows that unanswered questions concentrate in mid- to low-resource languages,and the distribution accumulates over non-latin scripts, likely due to tokenization challenges, insufficient language-specific training data, or visual-textual alignment difficulties.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10600000, "bbox": [[0.363, 0.589, 0.882, 0.594], [0.12, 0.607, 0.882, 0.614], [0.118, 0.622, 0.881, 0.631], [0.117, 0.638, 0.88, 0.648], [0.117, 0.655, 0.667, 0.665]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 286, "edu_l1_label": "IOS"}, {"txt": "Pangea-7B shows the high-est refusal rates, especially for Telugu (452), Hindi (130), Persian (160), and Serbian (125).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10600000, "bbox": [[0.667, 0.655, 0.881, 0.665], [0.117, 0.676, 0.831, 0.683]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 287, "edu_l1_label": "IOS"}, {"txt": "While other open models show minimal unanswered counts, indicating better format adherence.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10600000, "bbox": [[0.831, 0.676, 0.881, 0.682], [0.116, 0.692, 0.823, 0.699]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 288, "edu_l1_label": "IOS"}, {"txt": "Closed models (Claude 3.5 Sonnet, GPT-4o) display distinct behavior: their refusals concentrate on non-Latin,low-resource languages,but they also show high error rates for English questions,primarily health/medical queries due to policy constraints.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10600000, "bbox": [[0.823, 0.692, 0.881, 0.699], [0.117, 0.71, 0.881, 0.714], [0.117, 0.723, 0.881, 0.735], [0.117, 0.741, 0.507, 0.751]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 289, "edu_l1_label": "IOS"}, {"txt": "This underscores the trade-off between content moderation and benchmark performance.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10600000, "bbox": [[0.507, 0.741, 0.88, 0.751], [0.117, 0.76, 0.438, 0.768]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 290, "edu_l1_label": "IOS"}, {"txt": "6 Related Work", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10700000, "bbox": [[0.117, 0.802, 0.31, 0.816]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 6, "global_sentence_id": 291, "edu_l1_label": "BOS"}, {"txt": "6.1 General Challenges in Multilingual Evaluation of Vision-Language Models", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10800000, "bbox": [[0.117, 0.838, 0.84, 0.849]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 7, "global_sentence_id": 292, "edu_l1_label": "IOS"}, {"txt": "VLMs have demonstrated impressive performance in processing and generating text, interpreting images,and reasoning across multiple modalities.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10900000, "bbox": [[0.118, 0.869, 0.882, 0.881], [0.117, 0.887, 0.516, 0.896]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 293, "edu_l1_label": "IOS"}, {"txt": "These advances have been driven by various", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 10900000, "bbox": [[0.516, 0.887, 0.881, 0.896]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 294, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11000000, "bbox": [[0.492, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 295, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11000000, "bbox": [[0.112, 0.09, 0.891, 0.377]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 1, "global_sentence_id": 296, "edu_l1_label": "EDU_O"}, {"txt": "Figure 5: Distribution of the number of format errors for each model/language combi-nation. The languages are represented in their ISO 639 (set 1) code.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11100000, "bbox": [[0.115, 0.388, 0.883, 0.418]]}]}, "tags": ["figure_caption"], "label": "figure_title", "web_segment_id": 2, "global_sentence_id": 297, "edu_l1_label": "EDU_O"}, {"txt": "multimodal benchmarks (Li et al., 2024b;", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11100000, "bbox": [[0.117, 0.453, 0.44, 0.46]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 298, "edu_l1_label": "IOS"}, {"txt": " Vayani et al., 2024; Nayak et al., 2024; Schneider et al.,2025) that assess capabilities such as image captioning, object attribute recognition, and spatial relationship understanding.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11100000, "bbox": [[0.44, 0.453, 0.882, 0.462], [0.118, 0.468, 0.88, 0.478], [0.117, 0.487, 0.332, 0.494]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 299, "edu_l1_label": "IOS"}, {"txt": "However, most existing benchmarks prioritize high-resource languages (e.g.,English (Zang et al., 2024; Schneider et al., 2025) or Chinese (Fu et al., 2023; He et al.,2024)),resulting in significant gaps in multilingual evaluation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11100000, "bbox": [[0.332, 0.487, 0.88, 0.495], [0.118, 0.502, 0.882, 0.514], [0.117, 0.521, 0.536, 0.53]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 300, "edu_l1_label": "IOS"}, {"txt": "This focus creates disparities, particularly in evaluating performance on low-resource non-Latin languages (Hengle et al., 2024).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11100000, "bbox": [[0.536, 0.521, 0.881, 0.529], [0.117, 0.539, 0.747, 0.546]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 301, "edu_l1_label": "IOS"}, {"txt": "A common strat-egy for lower-resource languages has been to translate existing English benchmarks using tools such as ChatGPT (Lai et al., 2023), GPT-4 (Yue et al.,2025), or Google Translate (Li et al., 2023).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11100000, "bbox": [[0.747, 0.539, 0.881, 0.545], [0.117, 0.556, 0.88, 0.563], [0.117, 0.573, 0.84, 0.58]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 302, "edu_l1_label": "IOS"}, {"txt": "Such approaches, however, often fail to capture the linguistic and cultural diversity necessary for global applications (Singh et al., 2024a; Huang et al., 2025), may introduce errors or employ uncommon terms,thereby affecting the reliability of assessments, and exacerbate the problems with poverty-conscious language technology (Bird, 2022).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11100000, "bbox": [[0.84, 0.573, 0.879, 0.58], [0.117, 0.59, 0.882, 0.597], [0.117, 0.607, 0.881, 0.614], [0.117, 0.622, 0.877, 0.631], [0.117, 0.642, 0.469, 0.649]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 303, "edu_l1_label": "IOS"}, {"txt": "Furthermore, the lack of comprehensive evaluation suites for non-English languages has substantially hindered multilingual generative advancements,limiting LLMs' ability to perform equitably across diverse linguistic landscapes, a challenge par-ticularly critical for evaluating toxicity and biases in multilingual settings (Üstün et al., 2024).Addressing these limitations is essential for ensuring that VLMs perform robustly across a diverse range of real-world scenarios.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11100000, "bbox": [[0.469, 0.642, 0.881, 0.649], [0.117, 0.659, 0.883, 0.667], [0.118, 0.673, 0.881, 0.681], [0.117, 0.692, 0.882, 0.7], [0.118, 0.707, 0.881, 0.717], [0.117, 0.727, 0.344, 0.734]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 304, "edu_l1_label": "IOS"}, {"txt": "Recent efforts have attempted to address these shortcomings by incorporating culturally and lin-guistically diverse data.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11200000, "bbox": [[0.117, 0.758, 0.88, 0.768], [0.117, 0.779, 0.304, 0.785]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 305, "edu_l1_label": "IOS"}, {"txt": "Only recently have we seen some multilingual benchmarks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11200000, "bbox": [[0.304, 0.779, 0.775, 0.785]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 306, "edu_l1_label": "IOS"}, {"txt": "For example,in the reasoning space, MMLU-ProX (Xuan et al., 2025) has created a comprehensive reasoning benchmark in 13 languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11200000, "bbox": [[0.775, 0.779, 0.882, 0.788], [0.118, 0.794, 0.881, 0.804], [0.117, 0.81, 0.333, 0.819]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 307, "edu_l1_label": "IOS"}, {"txt": "Culturally-diverse Multilingual Visual Question Answering Benchmark (CVQA) (Romero et al., 2024) creates culturally relevant multiple-choice questions about images from 30 countries in 31 languages, using local languages which are then translated into English.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11200000, "bbox": [[0.333, 0.81, 0.881, 0.82], [0.117, 0.826, 0.881, 0.836], [0.117, 0.844, 0.86, 0.854]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 308, "edu_l1_label": "IOS"}, {"txt": "In contrast, KALEIDOSCOPE builds on this work by nearly doubling the number of questions while fo-cusing on 18 languages, thereby allowing for a more in-depth evaluation of each language.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11200000, "bbox": [[0.86, 0.844, 0.88, 0.854], [0.117, 0.864, 0.88, 0.871], [0.118, 0.881, 0.8, 0.888]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 309, "edu_l1_label": "IOS"}, {"txt": "Moreover,KALEIDOSCOPE focuses on multiple-choice exams covering a wide range of topics beyond cultur-", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11200000, "bbox": [[0.8, 0.881, 0.882, 0.89], [0.118, 0.895, 0.882, 0.903]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 310, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11300000, "bbox": [[0.492, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 5, "global_sentence_id": 311, "edu_l1_label": "EDU_O"}, {"txt": "ally relevant MCQA, including also STEM exams scenarios.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11300000, "bbox": [[0.117, 0.098, 0.589, 0.105]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 312, "edu_l1_label": "IOS"}, {"txt": "Additional work in evaluating VLMs includes PangeaBench, a holistic evaluation suite encompassing 14 datasets (13 pre-existing) split between multimodal and text-only tasks, covering 47 languages (Yue et al., 2025).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11300000, "bbox": [[0.589, 0.098, 0.881, 0.104], [0.117, 0.112, 0.883, 0.121], [0.117, 0.128, 0.746, 0.139]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 313, "edu_l1_label": "IOS"}, {"txt": "Similarly, Vayani et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11300000, "bbox": [[0.746, 0.128, 0.881, 0.139], [0.117, 0.148, 0.163, 0.156]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 314, "edu_l1_label": "IOS"}, {"txt": "(2024) introduce a multimodal benchmark that includes culturally diverse images paired with text across 100 languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11300000, "bbox": [[0.163, 0.148, 0.881, 0.156], [0.118, 0.165, 0.369, 0.173]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 315, "edu_l1_label": "IOS"}, {"txt": "Notably, this benchmark incorporates non-MCQA formats (e.g.,True/False and free-form answers), which is a key distinction from the MCQA format adopted in KALEIDOSCOPE.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11300000, "bbox": [[0.369, 0.165, 0.882, 0.176], [0.118, 0.18, 0.88, 0.19], [0.118, 0.197, 0.248, 0.207]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 316, "edu_l1_label": "IOS"}, {"txt": "Other benchmarks further illustrate the diversity of evaluation approaches.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11400000, "bbox": [[0.117, 0.232, 0.685, 0.242]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 317, "edu_l1_label": "IOS"}, {"txt": "For instance, MaRVL (Liu et al., 2021) assesses images in binary framework, which limits possible nuances in cultural evalu-ations.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11400000, "bbox": [[0.685, 0.232, 0.889, 0.242], [0.117, 0.252, 0.881, 0.258], [0.117, 0.269, 0.171, 0.276]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 318, "edu_l1_label": "IOS"}, {"txt": "CULTURALVQA (Nayak et al., 2024) emphasizes cultural knowledge with approximately 44.1% of its data focusing on rituals and traditions; however, it relies on open-ended questions in English, which contrasts with the MCQA and multilingual approach of KALEIDOSCOPE.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11400000, "bbox": [[0.171, 0.269, 0.881, 0.277], [0.118, 0.283, 0.881, 0.293], [0.118, 0.3, 0.827, 0.31]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 319, "edu_l1_label": "IOS"}, {"txt": "More-over, the MaXM benchmark (Changpinyo et al., 2023) addresses bias and provides multilingual,multimodal assessment across 7 languages but does not focus on cultural aspects,an area where KALEIDOSCOPE offers added value.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11400000, "bbox": [[0.827, 0.3, 0.88, 0.308], [0.117, 0.319, 0.883, 0.328], [0.119, 0.337, 0.881, 0.344], [0.117, 0.35, 0.391, 0.36]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 320, "edu_l1_label": "IOS"}, {"txt": "6.2 Exam-Style Benchmarks for Vision-Language Models", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11500000, "bbox": [[0.117, 0.393, 0.65, 0.404]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 3, "global_sentence_id": 321, "edu_l1_label": "IOS"}, {"txt": "Exam-style benchmarks have also advanced the evaluation of VLMs (see Table A.8).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.117, 0.424, 0.782, 0.434]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 322, "edu_l1_label": "IOS"}, {"txt": "Zhang et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.782, 0.424, 0.882, 0.434]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 323, "edu_l1_label": "IOS"}, {"txt": "(2023) present M3Exam, a novel benchmark sourced from real human exam questions that tests models in a multilingual, multimodal, and multilevel context using an MCQA framework.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.117, 0.441, 0.881, 0.451], [0.117, 0.462, 0.806, 0.469]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 324, "edu_l1_label": "IOS"}, {"txt": "M3Exam includes 12,317 questions in 9 languages, requiring both multilingual proficiency and cultural knowl-edge; however, only about 23% of its questions necessitate image processing.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.806, 0.462, 0.874, 0.468], [0.117, 0.477, 0.881, 0.484], [0.117, 0.496, 0.742, 0.503]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 325, "edu_l1_label": "IOS"}, {"txt": "This benchmark highlighted challenges faced by state-of-the-art models, such as GPT-4, particularly in handling low-resource and non-Latin script languages alongside complex multimodal queries.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.742, 0.496, 0.881, 0.503], [0.117, 0.51, 0.881, 0.521], [0.117, 0.527, 0.781, 0.537]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 326, "edu_l1_label": "IOS"}, {"txt": "In contrast,KALEIDOSCOPE differentiates itself by covering a larger number of languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.781, 0.527, 0.882, 0.539], [0.118, 0.544, 0.741, 0.554]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 327, "edu_l1_label": "IOS"}, {"txt": "Das et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.741, 0.544, 0.828, 0.554]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 328, "edu_l1_label": "IOS"}, {"txt": "(2024)present EXAMS-V, a multi-discipline, multimodal, multilingual exam benchmark comprising 20,932multiple-choice questions across 20 school disciplines (spanning natural sciences, social sciences,re-ligion, fine arts, and business).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.828, 0.544, 0.879, 0.555], [0.117, 0.564, 0.881, 0.571], [0.118, 0.581, 0.881, 0.586], [0.117, 0.595, 0.369, 0.605]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 329, "edu_l1_label": "IOS"}, {"txt": "EXAMS-V includes questions in 11 languages from 7 language families and incorporates four categories of multimodal features (scientific symbols,figures,graphs,and tabular data).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.369, 0.595, 0.881, 0.605], [0.118, 0.612, 0.883, 0.624], [0.117, 0.633, 0.271, 0.641]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 330, "edu_l1_label": "IOS"}, {"txt": "Despite this, only 5,086 of its questions are multimodal.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.271, 0.633, 0.738, 0.64]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 331, "edu_l1_label": "IOS"}, {"txt": "Additionally,the M5 benchmark (Schneider & Sitaram, 2024) evaluates VLMs on diverse vision-language tasks in a multilingual and multicultural context by covering 41 languages across eight datasets and five tasks, though it does not utilize an MCQA format.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.738, 0.633, 0.881, 0.64], [0.117, 0.646, 0.881, 0.657], [0.117, 0.667, 0.882, 0.674], [0.117, 0.682, 0.523, 0.691]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 332, "edu_l1_label": "IOS"}, {"txt": "In contrast, KALEIDOSCOPE stands out from these existing exam-based benchmarks by featuring a broader diversity of languages and the largest proportion of multimodal questions in an MCQA format, with 55% of its questions requiring im-age understanding.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.523, 0.682, 0.88, 0.691], [0.117, 0.699, 0.879, 0.708], [0.117, 0.718, 0.881, 0.723], [0.118, 0.735, 0.266, 0.742]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 333, "edu_l1_label": "IOS"}, {"txt": "This makes KALEIDOSCOPE, a comprehensive and challenging multimodal and multilingual testing ground for evaluating vision-language models in multilingual real-scenariOS.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11600000, "bbox": [[0.266, 0.735, 0.875, 0.742], [0.117, 0.753, 0.859, 0.759]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 334, "edu_l1_label": "IOS"}, {"txt": "6.3 Participatory Open Science Projects", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11700000, "bbox": [[0.117, 0.792, 0.498, 0.802]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 5, "global_sentence_id": 335, "edu_l1_label": "IOS"}, {"txt": "Participatory research empowers diverse communities to actively contribute to research processes,capturing linguistic subtleties and cultural nuances directly from native speakers.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11800000, "bbox": [[0.117, 0.823, 0.882, 0.835], [0.117, 0.843, 0.771, 0.85]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 336, "edu_l1_label": "IOS"}, {"txt": "Prior partic-ipatory NLP research has primarily targeted region-specific tasks such as translation, character recognition, and audio transcription.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11800000, "bbox": [[0.771, 0.843, 0.882, 0.847], [0.117, 0.857, 0.881, 0.867], [0.117, 0.877, 0.412, 0.883]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 337, "edu_l1_label": "IOS"}, {"txt": "We highlight notable initiatives here which served as our motivation and backbone framework for building KALEIDOSCOPE.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11800000, "bbox": [[0.412, 0.877, 0.881, 0.883], [0.117, 0.894, 0.632, 0.902]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 338, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 11900000, "bbox": [[0.492, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 7, "global_sentence_id": 339, "edu_l1_label": "EDU_O"}, {"txt": "In Africa, the Masakhane5 community exemplifies impactful participatory NLP by focusing on grassroots-led data collection, annotation, and model creation for African languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.117, 0.094, 0.88, 0.105], [0.118, 0.116, 0.777, 0.122]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 340, "edu_l1_label": "IOS"}, {"txt": "Nekoto et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.777, 0.116, 0.882, 0.122]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 341, "edu_l1_label": "IOS"}, {"txt": "(2020) demonstrated that communities in low-resource environments significantly contribute to NLP,even without formal training.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.117, 0.129, 0.882, 0.141], [0.117, 0.149, 0.353, 0.156]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 342, "edu_l1_label": "IOS"}, {"txt": "Subsequent efforts by Adelani et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.353, 0.149, 0.647, 0.156]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 343, "edu_l1_label": "IOS"}, {"txt": "(2023) have further advanced dataset curation and model development for underrepresented African languages using similar par-ticipatory frameworks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.647, 0.149, 0.881, 0.155], [0.117, 0.162, 0.88, 0.172], [0.117, 0.181, 0.294, 0.19]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 344, "edu_l1_label": "IOS"}, {"txt": "Similarly, the MaRVL dataset (Multicultural Reasoning over Vision and Language; Liu et al., 2021) employed native speakers from diverse linguistic backgrounds (Indone-sian,Swahili, Tamil, Turkish, and Mandarin Chinese) to contribute culturally representative im-ages, subsequently annotated by professional linguists.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.294, 0.181, 0.881, 0.189], [0.117, 0.198, 0.883, 0.206], [0.118, 0.218, 0.88, 0.224], [0.117, 0.235, 0.558, 0.242]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 345, "edu_l1_label": "IOS"}, {"txt": "Despite its cultural richness, MaRVL's modest scale (under 8,000 data points) limits broader applicability beyond evaluation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12000000, "bbox": [[0.558, 0.235, 0.883, 0.242], [0.117, 0.252, 0.786, 0.259]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 346, "edu_l1_label": "IOS"}, {"txt": "In Latin America, participatory research has also emerged and is continuously growing through the help of communities.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12100000, "bbox": [[0.117, 0.283, 0.88, 0.293], [0.117, 0.301, 0.32, 0.309]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 347, "edu_l1_label": "IOS"}, {"txt": "Recent works include Hernandez Mena & Meza Ruiz (2022),which developed eight open-access linguistic resources via structured social service programs, engaging student volunteers in transcription and segmentation tasks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12100000, "bbox": [[0.32, 0.301, 0.88, 0.31], [0.117, 0.317, 0.881, 0.328], [0.117, 0.336, 0.578, 0.344]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 348, "edu_l1_label": "IOS"}, {"txt": "Concurrently, Cañete et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12100000, "bbox": [[0.578, 0.336, 0.795, 0.344]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 349, "edu_l1_label": "IOS"}, {"txt": "(2020) and Guevara-Rukoz et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12100000, "bbox": [[0.795, 0.336, 0.879, 0.343], [0.117, 0.351, 0.283, 0.361]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 350, "edu_l1_label": "IOS"}, {"txt": "(2020) spearheaded crowd-sourced corpora addressing dialectal diversity and resource scarcity specific to Latin American Spanish.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12100000, "bbox": [[0.283, 0.351, 0.881, 0.361], [0.117, 0.371, 0.529, 0.378]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 351, "edu_l1_label": "IOS"}, {"txt": "In Southeast Asia, Project SEALD6, a collaboration between AI Singapore and Google Research,facilitated multilingual dataset collection to support regional Large Language Models (LLMs).Outputs from SEALD underpin open-source multilingual models such as SEA-LION7, Wangchan-Lion (Phatthiyaphaibun et al., 2024), and Sahabat-AI8.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12200000, "bbox": [[0.117, 0.403, 0.883, 0.414], [0.118, 0.42, 0.882, 0.43], [0.117, 0.437, 0.882, 0.444], [0.118, 0.454, 0.562, 0.463]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 352, "edu_l1_label": "IOS"}, {"txt": "Related initiatives include NusaCrowd for aggregating and standardizing Indonesian NLP datasets (Cahyawijaya et al., 2023) and the SEACrowd and SEA-VL projects aimed at comprehensive evaluation and benchmarking of LLMs across Southeast Asian languages (Cahyawijaya et al., 2025; Lovenia et al., 2024).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12200000, "bbox": [[0.562, 0.454, 0.88, 0.463], [0.118, 0.472, 0.881, 0.48], [0.117, 0.488, 0.881, 0.497], [0.117, 0.508, 0.753, 0.515]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 353, "edu_l1_label": "IOS"}, {"txt": "On a global scale, the CVQA dataset (Romero et al., 2024) was created using a participatory approach, involving native speakers and cultural experts from over 30 countries.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.117, 0.539, 0.88, 0.551], [0.117, 0.561, 0.746, 0.567]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 354, "edu_l1_label": "IOS"}, {"txt": "Annotators were selected for their fluency in local languages and cultural familiarity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.746, 0.561, 0.881, 0.567], [0.117, 0.577, 0.653, 0.583]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 355, "edu_l1_label": "IOS"}, {"txt": "Many contributors were also recognized as co-authors based on their level of involvement, reinforcing a collaborative, community-driven effort.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.653, 0.577, 0.881, 0.584], [0.117, 0.594, 0.882, 0.6], [0.117, 0.607, 0.22, 0.617]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 356, "edu_l1_label": "IOS"}, {"txt": "The Aya Initiative employed participatory methods, engaging over 3,000 contribu-tors to curate instruction datasets across 114 languages, resulting in one of the largest multilingual datasets for language model training (Singh et al., 2024b; Üstün et al., 2024).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.22, 0.607, 0.881, 0.616], [0.117, 0.626, 0.882, 0.635], [0.117, 0.641, 0.737, 0.652]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 357, "edu_l1_label": "IOS"}, {"txt": "Similarly, the IN-CLUDE benchmark (Romanou et al., 2024) leveraged participatory approaches closely aligned with our methodology.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.737, 0.641, 0.88, 0.651], [0.117, 0.658, 0.879, 0.669], [0.117, 0.679, 0.256, 0.686]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 358, "edu_l1_label": "IOS"}, {"txt": "The BigScience ROOTS corpus, developed collaboratively for the BLOOM model, exemplifies large-scale participatory data collection.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.256, 0.679, 0.875, 0.686], [0.117, 0.696, 0.569, 0.703]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 359, "edu_l1_label": "IOS"}, {"txt": "Approximately 62% of ROOTS data was crowd-sourced via global hackathons and open submissions, involving over 1,000 researchers from 60countries and more than 250 institutions, resulting in 1.6 terabytes of multilingual data (Laurençon et al., 2022).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.569, 0.696, 0.882, 0.703], [0.118, 0.713, 0.879, 0.72], [0.117, 0.731, 0.881, 0.738], [0.117, 0.748, 0.218, 0.755]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 360, "edu_l1_label": "IOS"}, {"txt": "Additionally, Uzuner et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.218, 0.748, 0.433, 0.754]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 361, "edu_l1_label": "IOS"}, {"txt": "(2010) underscored the viability of community-driven an-notation for complex, domain-specific NLP tasks like clinical text annotation, highlighting broader applicability of participatory frameworks beyond general NLP domains.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12300000, "bbox": [[0.433, 0.748, 0.881, 0.754], [0.117, 0.765, 0.881, 0.772], [0.117, 0.783, 0.674, 0.789]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 362, "edu_l1_label": "IOS"}, {"txt": "Participatory methods have also successfully extended into reinforcement learning from human", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12400000, "bbox": [[0.117, 0.812, 0.88, 0.823]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 363, "edu_l1_label": "IOS"}, {"txt": "5https://www.masakhane.io/", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12500000, "bbox": [[0.142, 0.837, 0.341, 0.849]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 364, "edu_l1_label": "EDU_O"}, {"txt": "6Southeast Asian Languages in One Network Data;", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12500000, "bbox": [[0.141, 0.853, 0.489, 0.862]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 365, "edu_l1_label": "EDU_O"}, {"txt": " https://aisingapore.org/aiproducts/southeast-asian -1anguages-in-one-network-data-seald/", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12500000, "bbox": [[0.489, 0.853, 0.881, 0.862], [0.118, 0.871, 0.406, 0.876]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 366, "edu_l1_label": "EDU_O"}, {"txt": "7https://sea-lion.ai", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12500000, "bbox": [[0.144, 0.882, 0.295, 0.89]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 367, "edu_l1_label": "EDU_O"}, {"txt": "8https://sahabat-ai.com", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12500000, "bbox": [[0.142, 0.894, 0.319, 0.903]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 368, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12600000, "bbox": [[0.492, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 10, "global_sentence_id": 369, "edu_l1_label": "EDU_O"}, {"txt": "feedback (RLHF).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12600000, "bbox": [[0.117, 0.094, 0.258, 0.105]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 370, "edu_l1_label": "IOS"}, {"txt": "For instance, the OpenAssistant project, led by LAION, utilized global crowd-sourcing to construct a multilingual corpus comprising over 161,000 messages annotated by 13,500volunteers.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12600000, "bbox": [[0.258, 0.094, 0.88, 0.104], [0.116, 0.115, 0.88, 0.121], [0.117, 0.131, 0.203, 0.138]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 371, "edu_l1_label": "IOS"}, {"txt": "This dataset facilitated robust training of dialogue-aligned language models through extensive human feedback annotations (Köpf et al., 2023).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12600000, "bbox": [[0.203, 0.131, 0.88, 0.14], [0.117, 0.148, 0.57, 0.157]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 372, "edu_l1_label": "IOS"}, {"txt": "7 Conclusion", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12700000, "bbox": [[0.117, 0.191, 0.276, 0.205]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 2, "global_sentence_id": 373, "edu_l1_label": "BOS"}, {"txt": "As generative models become increasingly multimodal and multilingual, the need for robust and culturally grounded evaluation benchmarks has never been more urgent.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12800000, "bbox": [[0.117, 0.223, 0.881, 0.234], [0.118, 0.244, 0.698, 0.251]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 374, "edu_l1_label": "IOS"}, {"txt": "In this work, we take a step toward closing this gap by introducing the largest benchmark of real-world, in-language multimodal exam questions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12800000, "bbox": [[0.698, 0.244, 0.88, 0.251], [0.117, 0.261, 0.881, 0.268], [0.12, 0.278, 0.346, 0.285]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 375, "edu_l1_label": "IOS"}, {"txt": "By grounding evaluation in authentic exam settings from around the world, our benchmark challenges models to reason about images in ways that mirror human assessment, capturing both linguistic and cultural complexity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12800000, "bbox": [[0.346, 0.278, 0.881, 0.285], [0.117, 0.294, 0.88, 0.303], [0.117, 0.313, 0.597, 0.319]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 376, "edu_l1_label": "IOS"}, {"txt": "Our findings highlight the limitations of current models in handling this intersection of skills:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12900000, "bbox": [[0.117, 0.343, 0.838, 0.354]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 377, "edu_l1_label": "IOS"}, {"txt": " mul-tilingual understanding, visual reasoning, and culturally aware problem-solving.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12900000, "bbox": [[0.838, 0.343, 0.882, 0.352], [0.118, 0.362, 0.761, 0.371]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 378, "edu_l1_label": "IOS"}, {"txt": "We hope this benchmark serves not only as a valuable tool for measuring progress but also as a call to action for developing models that are truly capable of operating across languages, cultures, and modalities.Continued investment in representative, high-quality evaluation datasets will be essential to ensure that future AI systems are equitable and globally relevant.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 12900000, "bbox": [[0.761, 0.362, 0.881, 0.371], [0.117, 0.378, 0.881, 0.388], [0.117, 0.395, 0.882, 0.405], [0.118, 0.412, 0.881, 0.422], [0.117, 0.43, 0.573, 0.439]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 379, "edu_l1_label": "IOS"}, {"txt": "Limitations", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13000000, "bbox": [[0.117, 0.474, 0.242, 0.488]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 5, "global_sentence_id": 380, "edu_l1_label": "EDU_O"}, {"txt": "While our benchmark represents an important step toward more representative multilingual multi-modal evaluations, several limitations still remain.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.117, 0.507, 0.882, 0.515], [0.117, 0.528, 0.503, 0.535]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 381, "edu_l1_label": "IOS"}, {"txt": "First, the dataset is inherently imbalanced across languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.503, 0.528, 0.881, 0.535], [0.117, 0.541, 0.201, 0.552]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 382, "edu_l1_label": "IOS"}, {"txt": "Coverage varies depending on the availability and accessibility of exam sources, with some languages significantly underrepresented.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.201, 0.541, 0.88, 0.551], [0.117, 0.563, 0.477, 0.569]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 383, "edu_l1_label": "IOS"}, {"txt": "Second, difficulty levels are not uniformly controlled.Since questions are drawn directly from real-world exams across diverse educational systems,vari-ations in exam design, curricular focus, and intended grade levels introduce potential inconsistency in task complexity across languages and modalities.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.477, 0.563, 0.882, 0.569], [0.117, 0.576, 0.88, 0.586], [0.117, 0.597, 0.88, 0.605], [0.117, 0.611, 0.515, 0.621]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 384, "edu_l1_label": "IOS"}, {"txt": "Further the chosen MCQA question format,in-herent to many exams,has issues, see Appendix B.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.515, 0.611, 0.881, 0.62], [0.117, 0.627, 0.511, 0.638]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 385, "edu_l1_label": "IOS"}, {"txt": "For instance: Exploitation of biases: Models may guess correct answers by exploiting statistical patterns or poorly designed distractors, inflating performance metrics without demonstrating genuine understanding.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.511, 0.627, 0.881, 0.637], [0.117, 0.647, 0.881, 0.656], [0.117, 0.665, 0.65, 0.671]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 386, "edu_l1_label": "IOS"}, {"txt": "Limited real-world appli-cability:Unlike open-ended queries typical in real-world applications, MCQA provides predefined options, which may not reflect natural user interactions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.65, 0.665, 0.88, 0.672], [0.118, 0.682, 0.187, 0.689], [0.197, 0.678, 0.881, 0.688], [0.117, 0.699, 0.555, 0.706]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 387, "edu_l1_label": "IOS"}, {"txt": "Choice-order sensitivity: Performance can vary based on the order of answer choices, introducing inconsistencies unrelated to model ca-pability.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.555, 0.699, 0.882, 0.706], [0.117, 0.716, 0.881, 0.722], [0.117, 0.734, 0.182, 0.741]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 388, "edu_l1_label": "IOS"}, {"txt": "Finally, while the dataset expands coverage beyond English, the overall language diversity remains limited.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.182, 0.734, 0.88, 0.742], [0.117, 0.75, 0.244, 0.757]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 389, "edu_l1_label": "IOS"}, {"txt": "Many languages, especially those spoken in low-resource regions, are still missing due to the scarcity of suitable exam material and annotators.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13100000, "bbox": [[0.244, 0.75, 0.881, 0.759], [0.117, 0.764, 0.594, 0.775]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 390, "edu_l1_label": "IOS"}, {"txt": "Acknowledgments", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13200000, "bbox": [[0.117, 0.809, 0.318, 0.822]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 7, "global_sentence_id": 391, "edu_l1_label": "EDU_O"}, {"txt": "We acknowledge the EuroHPC Joint Undertaking for awarding this project access to the EuroHPC supercomputer LEONARDO, hosted by CINECA (Italy) and the LEONARDO consortium through an EuroHPC Development Access call (ID:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13300000, "bbox": [[0.117, 0.842, 0.881, 0.852], [0.116, 0.863, 0.88, 0.87], [0.117, 0.881, 0.459, 0.886]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 392, "edu_l1_label": "EDU_O"}, {"txt": "EUHPC_D12_071).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13300000, "bbox": [[0.459, 0.881, 0.621, 0.886]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 393, "edu_l1_label": "EDU_O"}, {"txt": "This work was supported by re-search grant (VIL53122) from Villum Fonden, and by the European Union's Horizon 2020 research", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13300000, "bbox": [[0.621, 0.881, 0.881, 0.886], [0.117, 0.897, 0.88, 0.904]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 394, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.49, 0.938, 0.511, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 395, "edu_l1_label": "EDU_O"}, {"txt": "and innovation program under grant agreement No.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.117, 0.098, 0.511, 0.105]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 396, "edu_l1_label": "EDU_O"}, {"txt": "101135671 (TrustLLM).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.511, 0.098, 0.705, 0.105]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 397, "edu_l1_label": "EDU_O"}, {"txt": "EF gratefully acknowl-edges the support of the Googler Initiated Grants and the Google Award for Inclusion Research pro-grams.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.705, 0.098, 0.882, 0.102], [0.116, 0.115, 0.881, 0.121], [0.118, 0.133, 0.172, 0.139]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 398, "edu_l1_label": "EDU_O"}, {"txt": "MZ is supported by the research grant (VIL57392) from Villum Fonden.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.172, 0.133, 0.742, 0.139]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 399, "edu_l1_label": "EDU_O"}, {"txt": "JMI is supported by the National University Philippines and the UKRI Centre for Doctoral Training in Accountable,Responsible,and Transparent AI [EP/S023437/1] of the University of Bath.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13400000, "bbox": [[0.742, 0.133, 0.881, 0.138], [0.118, 0.145, 0.883, 0.157], [0.118, 0.162, 0.709, 0.173]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 400, "edu_l1_label": "EDU_O"}, {"txt": "References", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13500000, "bbox": [[0.118, 0.208, 0.237, 0.222]]}]}, "tags": ["title"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 401, "edu_l1_label": "EDU_O"}, {"txt": "Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.117, 0.241, 0.879, 0.251], [0.135, 0.258, 0.333, 0.268]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 402, "edu_l1_label": "EDU_O"}, {"txt": "The multilingual alignment prism: Aligning global and local preferences to reduce harm.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.333, 0.258, 0.88, 0.268], [0.134, 0.276, 0.261, 0.285]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 403, "edu_l1_label": "EDU_O"}, {"txt": "In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.261, 0.276, 0.76, 0.287]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 404, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.76, 0.276, 0.885, 0.287], [0.137, 0.294, 0.774, 0.303]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 405, "edu_l1_label": "EDU_O"}, {"txt": "12027-12049,Miami, Florida, USA, November 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.774, 0.294, 0.883, 0.306], [0.135, 0.31, 0.435, 0.32]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 406, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.435, 0.31, 0.774, 0.32]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 407, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2024.emnlp-main.671.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.774, 0.31, 0.88, 0.32], [0.135, 0.328, 0.343, 0.337]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 408, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2024.emnlp-main.671/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13600000, "bbox": [[0.343, 0.328, 0.811, 0.337]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 409, "edu_l1_label": "EDU_O"}, {"txt": "David Ifeoluwa Adelani, Marek Masiak, Israel Abebe Azime, Jesujoba Alabi, Atnafu Lambebo Tonja, Christine Mwase, Odunayo Ogundepo, Bonaventure F. P. Dossou, Akintunde Oladipo,Doreen Nixdorf, Chris Chinenye Emezue, Sana Al-azzawi, Blessing Sibanda, Davis David,Lolwethu Ndolela, Jonathan Mukiibi, Tunde Ajayi, Tatiana Moteu, Brian Odhiambo, Abra-ham Owodunni, Nnaemeka Obiefuna, Muhidin Mohamed, Shamsuddeen Hassan Muhammad,Teshome Mulugeta Ababu, Saheed Abdullahi Salahudeen, Mesay Gemeda Yigezu, Tajuddeen Gwadabe, Idris Abdulmumin, Mahlet Taye, Oluwabusayo Awoyomi, Iyanuoluwa Shode, Tolu-lope Adelani, Habiba Abdulganiyu, Abdul-Hakeem Omotayo, Adetola Adeeko, Abeeb Afolabi,Anuoluwapo Aremu, Olanrewaju Samuel, Clemencia Siro, Wangari Kimotho,Onyekachi Ogbu,Chinedu Mbonu, Chiamaka Chukwuneke, Samuel Fanijo, Jessica Ojo, Oyinkansola Awosan,Tadesse Kebede, Toadoum Sari Sakayo, Pamela Nyatsine, Freedmore Sidume, Oreen Yousuf,Mardiyyah Oduwole, Kanda Tshinu, Ussen Kimanuka, Thina Diko, Siyanda Nxakama, Sino-dos Nigusse, Abdulmejid Johar, Shafie Mohamed, Fuad Mire Hassan, Moges Ahmed Mehamed,Evrard Ngabire, Jules Jules, Ivan Ssenkungu, and Pontus Stenetorp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13700000, "bbox": [[0.117, 0.355, 0.881, 0.366], [0.136, 0.372, 0.883, 0.385], [0.136, 0.39, 0.883, 0.401], [0.135, 0.407, 0.88, 0.416], [0.135, 0.423, 0.883, 0.436], [0.135, 0.441, 0.88, 0.451], [0.135, 0.458, 0.88, 0.467], [0.134, 0.475, 0.883, 0.487], [0.136, 0.493, 0.883, 0.505], [0.135, 0.509, 0.883, 0.521], [0.135, 0.527, 0.883, 0.539], [0.134, 0.544, 0.881, 0.553], [0.135, 0.561, 0.883, 0.573], [0.135, 0.578, 0.694, 0.587]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 410, "edu_l1_label": "EDU_O"}, {"txt": "MasakhaNEWS: News topic classification for African languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13700000, "bbox": [[0.694, 0.578, 0.881, 0.588], [0.135, 0.596, 0.462, 0.605]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 411, "edu_l1_label": "EDU_O"}, {"txt": "In Jong C. Park, Yuki Arase, Baotian Hu, Wei Lu,Derry Wijaya, Ayu Purwarianti, and Adila Alfa Krisnadhi (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13700000, "bbox": [[0.462, 0.596, 0.883, 0.607], [0.135, 0.612, 0.652, 0.623]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 412, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 13th In-ternational Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers),pp.144-159,Nusa Dua, Bali, November 2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13700000, "bbox": [[0.652, 0.612, 0.882, 0.621], [0.136, 0.629, 0.881, 0.638], [0.135, 0.645, 0.883, 0.657], [0.135, 0.667, 0.502, 0.674]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 413, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13700000, "bbox": [[0.502, 0.667, 0.846, 0.674]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 414, "edu_l1_label": "EDU_O"}, {"txt": "doi:10.18653/v1/2023.ijcnlp-main.10.URL https://aclanthology.org/2023.ijcnlp-main.10/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13700000, "bbox": [[0.846, 0.667, 0.882, 0.674], [0.136, 0.681, 0.866, 0.691]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 415, "edu_l1_label": "EDU_O"}, {"txt": "David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Zhuang Yun Jian, Jesujoba Oluwadara Alabi,Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chia-maka Chukwuneke, Happy Buzaaba, Blessing K. Sibanda, Godson Kalipe, Jonathan Mukiibi,Salomon Kabongo KABENAMUALU, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela,Nkiruka Bridget Odu, Rooweither Mabuya, Shamsuddeen Hassan Muhammad, Salomey Osei,Sokhar Samb, Tadesse Kebede Guge, and Pontus Stenetorp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13800000, "bbox": [[0.118, 0.708, 0.881, 0.719], [0.135, 0.726, 0.882, 0.734], [0.135, 0.746, 0.883, 0.756], [0.135, 0.759, 0.883, 0.773], [0.135, 0.777, 0.883, 0.788], [0.135, 0.794, 0.623, 0.804]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 416, "edu_l1_label": "EDU_O"}, {"txt": "Irokobench: A new benchmark for african languages in the age of large language models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13800000, "bbox": [[0.623, 0.794, 0.881, 0.804], [0.135, 0.811, 0.595, 0.821]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 417, "edu_l1_label": "EDU_O"}, {"txt": "ArXiv,abs/2406.03368,2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13800000, "bbox": [[0.595, 0.811, 0.837, 0.821]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 418, "edu_l1_label": "EDU_O"}, {"txt": "URL https://api.semanticscholar.org/CorpusID:270258352.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13800000, "bbox": [[0.837, 0.811, 0.879, 0.821], [0.134, 0.829, 0.601, 0.838]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 419, "edu_l1_label": "EDU_O"}, {"txt": "Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Ak-shay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram.Mega:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13900000, "bbox": [[0.118, 0.857, 0.88, 0.866], [0.134, 0.876, 0.882, 0.884], [0.134, 0.891, 0.184, 0.902]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 420, "edu_l1_label": "EDU_O"}, {"txt": " MMultilingual evaluation of generative ai, 2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13900000, "bbox": [[0.184, 0.891, 0.537, 0.901]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 421, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2303.12528.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 13900000, "bbox": [[0.537, 0.891, 0.883, 0.902]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 422, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14000000, "bbox": [[0.49, 0.938, 0.511, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 7, "global_sentence_id": 423, "edu_l1_label": "EDU_O"}, {"txt": "Anthropic.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.117, 0.094, 0.203, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 424, "edu_l1_label": "EDU_O"}, {"txt": "The Claude 3 model family: Opus,sonnet, haiku, 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.203, 0.094, 0.646, 0.104]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 425, "edu_l1_label": "EDU_O"}, {"txt": "URL https://api.semantic scholar.org/CorpusID:268232499.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.646, 0.094, 0.881, 0.105], [0.135, 0.114, 0.417, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 426, "edu_l1_label": "EDU_O"}, {"txt": "Steven Bird.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.117, 0.14, 0.214, 0.15]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 427, "edu_l1_label": "EDU_O"}, {"txt": "Local languages, third spaces, and other high-resource scenarios.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.214, 0.14, 0.706, 0.15]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 428, "edu_l1_label": "EDU_O"}, {"txt": "In Smaranda Muresan,Preslav Nakov, and Aline Villavicencio (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.706, 0.14, 0.883, 0.152], [0.136, 0.157, 0.494, 0.168]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 429, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.494, 0.157, 0.881, 0.167], [0.136, 0.174, 0.725, 0.184]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 430, "edu_l1_label": "EDU_O"}, {"txt": "7817-7829,Dublin,Ireland, May 2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.725, 0.174, 0.883, 0.186], [0.135, 0.191, 0.284, 0.201]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 431, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.284, 0.191, 0.625, 0.201]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 432, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2022.acl-long.539.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.625, 0.191, 0.882, 0.201], [0.135, 0.209, 0.168, 0.218]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 433, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2022.acl-long.539/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14100000, "bbox": [[0.168, 0.209, 0.62, 0.219]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 434, "edu_l1_label": "EDU_O"}, {"txt": "Yuri Bizzoni, Tom S Juzek, Cristina España-Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Teich.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.117, 0.236, 0.881, 0.247], [0.135, 0.254, 0.224, 0.265]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 435, "edu_l1_label": "EDU_O"}, {"txt": "How human is machine translationese?", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.224, 0.254, 0.53, 0.264]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 436, "edu_l1_label": "EDU_O"}, {"txt": "comparing human and machine translations of text and speech.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.53, 0.254, 0.881, 0.265], [0.135, 0.274, 0.289, 0.282]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 437, "edu_l1_label": "EDU_O"}, {"txt": "In Marcello Federico, Alex Waibel, Kevin Knight, Satoshi Nakamura, HHer-mann Ney, Jan Niehues, Sebastian Stüker, Dekai Wu, Joseph Mariani, and Francois Yvon (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.289, 0.274, 0.879, 0.281], [0.135, 0.292, 0.876, 0.3]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 438, "edu_l1_label": "EDU_O"}, {"txt": "),Proceedings of the 17th International Conference on Spoken Language Translation, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.876, 0.292, 0.883, 0.3], [0.136, 0.306, 0.812, 0.316]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 439, "edu_l1_label": "EDU_O"}, {"txt": "280-290,Online, July 2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.812, 0.306, 0.883, 0.318], [0.135, 0.323, 0.279, 0.333]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 440, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.279, 0.323, 0.617, 0.333]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 441, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2020.iwslt-1.34.URL https://aclanthology.org/2020.iwslt-1.34/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14200000, "bbox": [[0.617, 0.323, 0.883, 0.333], [0.135, 0.339, 0.562, 0.35]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 442, "edu_l1_label": "EDU_O"}, {"txt": "Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14300000, "bbox": [[0.118, 0.368, 0.881, 0.378], [0.135, 0.385, 0.31, 0.395]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 443, "edu_l1_label": "EDU_O"}, {"txt": "IGLUE: A benchmark for transfer learning across modalities, tasks, and languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14300000, "bbox": [[0.31, 0.385, 0.881, 0.395], [0.134, 0.402, 0.218, 0.412]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 444, "edu_l1_label": "EDU_O"}, {"txt": "In Kamalika Chaudhuri, Stefanie Jegelka, Le Song,Csaba Szepesvari, Gang Niu,and Sivan Sabato (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14300000, "bbox": [[0.218, 0.402, 0.883, 0.414], [0.135, 0.422, 0.318, 0.431]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 445, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 39th International Conference on Machine Learning,volume 162 of Proceedings of Machine Learning Research, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14300000, "bbox": [[0.318, 0.422, 0.883, 0.431], [0.135, 0.439, 0.609, 0.446]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 446, "edu_l1_label": "EDU_O"}, {"txt": "2370-2392.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14300000, "bbox": [[0.609, 0.439, 0.698, 0.446]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 447, "edu_l1_label": "EDU_O"}, {"txt": "PMLR, 17-23 Jul 2022.URL https://proceedings.mlr.press/v162/bugliarello22a.html.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14300000, "bbox": [[0.698, 0.439, 0.883, 0.446], [0.135, 0.453, 0.682, 0.464]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 448, "edu_l1_label": "EDU_O"}, {"txt": "Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Winata, Bryan Wilie, Fajri Koto, Rah-mad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Muhammad Satrio Wicaksono, Ivan Parmonangan,Ika Alfina, Ilham Firdausi Putra, Samsul Rahmadani, Yulianti Oenang, Ali Septiandri, James Jaya, Kaustubh Dhole, Arie Suryani, Rifki Afina Putri, Dan Su, Keith Stevens, Made Nindy-atama Nityasya, Muhammad Adilazuarda, Ryan Hadiwijaya, Ryandito Diandaru, Tiezheng Yu,Vito Ghifari, Wenliang Dai, Yan Xu, Dyah Damapuspita, Haryo Wibowo, Cuk Tho, Ichwanul Karo Karo, Tirana Fatyanosa, Ziwei Ji, Graham Neubig, Timothy Baldwin, Sebastian Ruder,Pascale Fung, Herry Sujaini, Sakriani Sakti, and Ayu Purwarianti.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.117, 0.482, 0.881, 0.492], [0.135, 0.503, 0.881, 0.509], [0.135, 0.516, 0.882, 0.528], [0.135, 0.533, 0.881, 0.543], [0.135, 0.55, 0.882, 0.557], [0.135, 0.571, 0.883, 0.58], [0.135, 0.584, 0.881, 0.595], [0.136, 0.602, 0.883, 0.613], [0.135, 0.619, 0.648, 0.628]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 449, "edu_l1_label": "EDU_O"}, {"txt": "NusaCrowd: Open source ini-tiative for Indonesian NLP resources.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.648, 0.619, 0.882, 0.626], [0.135, 0.637, 0.421, 0.646]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 450, "edu_l1_label": "EDU_O"}, {"txt": "In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.421, 0.637, 0.881, 0.646], [0.136, 0.653, 0.176, 0.665]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 451, "edu_l1_label": "EDU_O"}, {"txt": "), Findings of the Association for Computational Linguistics: ACL 2023, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.176, 0.653, 0.776, 0.664]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 452, "edu_l1_label": "EDU_O"}, {"txt": "13745-13818,Toronto, Canada, July 2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.776, 0.653, 0.883, 0.664], [0.136, 0.67, 0.36, 0.681]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 453, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.36, 0.67, 0.7, 0.681]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 454, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2023.findings-acl.868.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.7, 0.67, 0.88, 0.681], [0.136, 0.691, 0.27, 0.698]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 455, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2023.findings-ac1.868/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.27, 0.691, 0.757, 0.698]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 456, "edu_l1_label": "EDU_O"}, {"txt": "Samuel Cahyawijaya, Holy Lovenia, Joel Ruben Antony Moniz, Tack Hwa Wong, Mohammad Rifqi Farhansyah, Thant Thiri Maung, Frederikus Hudi, David Anugraha, Muhammad Ravi Shulthan Habibi, Muhammad Reza Qorib, et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.117, 0.716, 0.882, 0.726], [0.135, 0.733, 0.88, 0.743], [0.136, 0.749, 0.452, 0.76]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 457, "edu_l1_label": "EDU_O"}, {"txt": "Crowdsource, crawl, or generate?", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.452, 0.749, 0.728, 0.759]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 458, "edu_l1_label": "EDU_O"}, {"txt": "creating sea-vl,a multicultural vision-language dataset for southeast asia.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.728, 0.749, 0.881, 0.76], [0.134, 0.77, 0.573, 0.777]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 459, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2503.07920,2025.URL https://arxiv.org/abs/2503.07920.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.573, 0.77, 0.883, 0.777], [0.135, 0.783, 0.478, 0.794]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 460, "edu_l1_label": "EDU_O"}, {"txt": "Isaac Caswell, Theresa Breiner, Daan van Esch, and Ankur Bapna.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.118, 0.812, 0.663, 0.823]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 461, "edu_l1_label": "EDU_O"}, {"txt": "Language ID in the wild:Unexpected challenges on the path to a thousand-language web text corpus.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.663, 0.812, 0.88, 0.822], [0.141, 0.83, 0.749, 0.84]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 462, "edu_l1_label": "EDU_O"}, {"txt": "In Donia Scott,Nuria Bel, and Chengqing Zong (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.749, 0.83, 0.883, 0.842], [0.135, 0.846, 0.426, 0.858]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 463, "edu_l1_label": "EDU_O"}, {"txt": "), PProceedings of the 28th International Conference on Com-putational Linguistics, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.426, 0.846, 0.882, 0.855], [0.135, 0.868, 0.341, 0.874]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 464, "edu_l1_label": "EDU_O"}, {"txt": "6588-6608, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.341, 0.868, 0.882, 0.874], [0.135, 0.881, 0.475, 0.891]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 465, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2020.coling-main.579.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.475, 0.881, 0.832, 0.891]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 466, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2020.coling-main.579/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14400000, "bbox": [[0.832, 0.881, 0.88, 0.891], [0.134, 0.9, 0.565, 0.908]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 467, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14500000, "bbox": [[0.489, 0.936, 0.511, 0.95]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 8, "global_sentence_id": 468, "edu_l1_label": "EDU_O"}, {"txt": "José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14600000, "bbox": [[0.118, 0.094, 0.833, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 469, "edu_l1_label": "EDU_O"}, {"txt": "Span-ish pre-trained bert model and evaluation data.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14600000, "bbox": [[0.833, 0.094, 0.88, 0.104], [0.135, 0.113, 0.506, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 470, "edu_l1_label": "EDU_O"}, {"txt": "In PML4DC at ICLR 2020, 2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14600000, "bbox": [[0.506, 0.113, 0.774, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 471, "edu_l1_label": "EDU_O"}, {"txt": "Beer Changpinyo, Linting Xue, Michal Yarom, Ashish Thapliyal, Idan Szpektor, Julien Amelot,Xi Chen, and Radu Soricut.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14700000, "bbox": [[0.117, 0.14, 0.883, 0.152], [0.135, 0.157, 0.353, 0.167]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 472, "edu_l1_label": "EDU_O"}, {"txt": "Maxm: Towards multilingual visual question answering.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14700000, "bbox": [[0.353, 0.157, 0.79, 0.167]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 473, "edu_l1_label": "EDU_O"}, {"txt": "In Findings of ACL: EMNLP, 2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14700000, "bbox": [[0.79, 0.157, 0.882, 0.169], [0.136, 0.177, 0.323, 0.184]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 474, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2209.05401.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14700000, "bbox": [[0.323, 0.177, 0.672, 0.184]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 475, "edu_l1_label": "EDU_O"}, {"txt": "Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu,Zhuyun Dai, Mirac Suzgun,Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel-lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao,Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin,Adam Roberts,Denny Zhou, Quoc V. Le, and Jason Wei.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.117, 0.202, 0.881, 0.212], [0.136, 0.219, 0.883, 0.232], [0.135, 0.237, 0.882, 0.244], [0.134, 0.254, 0.883, 0.266], [0.135, 0.271, 0.883, 0.284], [0.135, 0.289, 0.581, 0.299]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 476, "edu_l1_label": "EDU_O"}, {"txt": "Scaling instruction-finetuned language models,2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.581, 0.289, 0.881, 0.299], [0.135, 0.309, 0.243, 0.316]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 477, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2210.11416.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.243, 0.309, 0.592, 0.316]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 478, "edu_l1_label": "EDU_O"}, {"txt": "Cohere-For-AI-Team.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.117, 0.334, 0.285, 0.344]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 479, "edu_l1_label": "EDU_O"}, {"txt": "Aya vision: Expanding the worlds ai can see, March 2025.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.285, 0.334, 0.767, 0.344]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 480, "edu_l1_label": "EDU_O"}, {"txt": "URL https://cohere.com/blog/aya-vision.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.767, 0.334, 0.879, 0.344], [0.135, 0.351, 0.4, 0.36]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 481, "edu_l1_label": "EDU_O"}, {"txt": "John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker.RLHF can speak many languages:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.118, 0.379, 0.882, 0.39], [0.136, 0.396, 0.404, 0.407]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 482, "edu_l1_label": "EDU_O"}, {"txt": " Unlocking multilingual preference optimization for LLMs.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.404, 0.396, 0.861, 0.407]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 483, "edu_l1_label": "EDU_O"}, {"txt": "In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.861, 0.396, 0.881, 0.407], [0.135, 0.414, 0.6, 0.426]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 484, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.6, 0.414, 0.882, 0.424], [0.136, 0.434, 0.598, 0.441]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 485, "edu_l1_label": "EDU_O"}, {"txt": "13134-13156, Miami, Florida, USA,November 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.598, 0.434, 0.883, 0.442], [0.135, 0.447, 0.262, 0.458]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 486, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.262, 0.447, 0.6, 0.458]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 487, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2024.emnlp-main.729.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.6, 0.447, 0.883, 0.458], [0.135, 0.466, 0.169, 0.475]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 488, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2024.emnlp-main.729/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14800000, "bbox": [[0.169, 0.466, 0.639, 0.475]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 489, "edu_l1_label": "EDU_O"}, {"txt": "Rocktim Jyoti Das, Simeon Emilov Hristov, Haonan Li, Dimitar Iliyanov Dimitrov, Ivan Koychev,and Preslav Nakov.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.117, 0.494, 0.882, 0.505], [0.135, 0.514, 0.288, 0.521]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 490, "edu_l1_label": "EDU_O"}, {"txt": "EXAMS-V: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.288, 0.514, 0.881, 0.52], [0.134, 0.531, 0.406, 0.538]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 491, "edu_l1_label": "EDU_O"}, {"txt": "In Lun-Wei Ku, Andre Martins, and Vivek Srikumar(eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.406, 0.531, 0.875, 0.54]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 492, "edu_l1_label": "EDU_O"}, {"txt": "),Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.875, 0.531, 0.883, 0.54], [0.136, 0.544, 0.882, 0.554], [0.136, 0.562, 0.294, 0.572]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 493, "edu_l1_label": "EDU_O"}, {"txt": "7768-7791,Bangkok, Thailand, August 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.294, 0.562, 0.645, 0.572]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 494, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.645, 0.562, 0.882, 0.572], [0.135, 0.578, 0.226, 0.588]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 495, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2024.acl-long.420.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.226, 0.578, 0.511, 0.589]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 496, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2024.ac1-1ong.420/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.511, 0.578, 0.88, 0.588], [0.135, 0.599, 0.213, 0.605]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 497, "edu_l1_label": "EDU_O"}, {"txt": "Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mo-hammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson,Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lam-bert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat,Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi,Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.124, 0.624, 0.881, 0.634], [0.135, 0.641, 0.883, 0.654], [0.136, 0.658, 0.881, 0.669], [0.135, 0.676, 0.88, 0.685], [0.134, 0.693, 0.881, 0.703], [0.135, 0.71, 0.882, 0.72], [0.135, 0.727, 0.881, 0.737], [0.135, 0.744, 0.883, 0.757], [0.135, 0.761, 0.569, 0.771]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 498, "edu_l1_label": "EDU_O"}, {"txt": "Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.569, 0.761, 0.881, 0.771], [0.135, 0.782, 0.564, 0.789]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 499, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2409.17146,2024.URL https://arxiv.org/abs/2409.17146.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.564, 0.782, 0.883, 0.789], [0.135, 0.795, 0.479, 0.805]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 500, "edu_l1_label": "EDU_O"}, {"txt": "Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld,Margaret Mitchell, and Matt Gardner.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.118, 0.823, 0.883, 0.835], [0.135, 0.841, 0.444, 0.851]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 501, "edu_l1_label": "EDU_O"}, {"txt": "Documenting large webtext corpora: A case study on the colossal clean crawled corpus.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.444, 0.841, 0.88, 0.851], [0.135, 0.859, 0.396, 0.867]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 502, "edu_l1_label": "EDU_O"}, {"txt": "In Marie-Francine Moens, Xuanjing Huang,Lucia Specia,and Scott Wen-tau Yih (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.396, 0.859, 0.881, 0.867], [0.135, 0.875, 0.331, 0.887]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 503, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.331, 0.875, 0.883, 0.884], [0.136, 0.893, 0.335, 0.903]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 504, "edu_l1_label": "EDU_O"}, {"txt": "1286-1305, Online and Punta Cana, Dominican Republic, November", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 14900000, "bbox": [[0.335, 0.893, 0.881, 0.903]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 505, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.489, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 506, "edu_l1_label": "EDU_O"}, {"txt": "2021.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.135, 0.095, 0.178, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 507, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.178, 0.095, 0.52, 0.104]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 508, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2021.emnlp-main.98.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.52, 0.095, 0.836, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 509, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2021.emnlp-main.98/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.836, 0.095, 0.878, 0.104], [0.134, 0.113, 0.547, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 510, "edu_l1_label": "EDU_O"}, {"txt": "Mengnan Du,Fengxiang He, Na Zou, Dacheng Tao, and Xia Hu.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.124, 0.14, 0.611, 0.15]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 511, "edu_l1_label": "EDU_O"}, {"txt": "Shortcut learning of large language models in natural language understanding.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.611, 0.14, 0.882, 0.15], [0.135, 0.16, 0.459, 0.169]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 512, "edu_l1_label": "EDU_O"}, {"txt": "Commun.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.459, 0.16, 0.545, 0.167]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 513, "edu_l1_label": "EDU_O"}, {"txt": "ACM, 67(1):110-120, December 2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.545, 0.16, 0.838, 0.167]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 514, "edu_l1_label": "EDU_O"}, {"txt": "ISSN 0001-0782. doi: 10.1145/3596490.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.838, 0.16, 0.88, 0.167], [0.134, 0.174, 0.397, 0.184]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 515, "edu_l1_label": "EDU_O"}, {"txt": "URL https://doi.org/10.1145/3596490.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.397, 0.174, 0.737, 0.183]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 516, "edu_l1_label": "EDU_O"}, {"txt": "Desmond Elliott.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.117, 0.202, 0.252, 0.213]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 517, "edu_l1_label": "EDU_O"}, {"txt": "Adversarial evaluation of multimodal machine translation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.252, 0.202, 0.71, 0.212]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 518, "edu_l1_label": "EDU_O"}, {"txt": "In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.71, 0.202, 0.881, 0.213], [0.135, 0.22, 0.56, 0.232]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 519, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.56, 0.22, 0.88, 0.23], [0.136, 0.237, 0.57, 0.247]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 520, "edu_l1_label": "EDU_O"}, {"txt": "2974-2978, Brussels, Belgium, October-November 2018.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.57, 0.237, 0.882, 0.244], [0.134, 0.254, 0.263, 0.265]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 521, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.263, 0.254, 0.604, 0.265]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 522, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/D18-1329.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.604, 0.254, 0.838, 0.265]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 523, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/D18-1329/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.838, 0.254, 0.88, 0.265], [0.134, 0.273, 0.453, 0.282]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 524, "edu_l1_label": "EDU_O"}, {"txt": "Desmond Elliott, Stella Frank, Khalil Sima'an, and Lucia Specia.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.117, 0.3, 0.627, 0.31]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 525, "edu_l1_label": "EDU_O"}, {"txt": "Multi30K: Multilingual English-German image descriptions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.627, 0.3, 0.881, 0.309], [0.135, 0.317, 0.355, 0.326]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 526, "edu_l1_label": "EDU_O"}, {"txt": "In Anya Belz, Erkut Erdem, Krystian Mikolajczyk,and Katerina Pastra (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.355, 0.317, 0.881, 0.327], [0.136, 0.334, 0.232, 0.345]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 527, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 5th Workshop on Vision and Language, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.232, 0.334, 0.73, 0.344]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 528, "edu_l1_label": "EDU_O"}, {"txt": "70-74,Berlin,Ger-many, August 2016.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.73, 0.334, 0.88, 0.343], [0.135, 0.354, 0.295, 0.361]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 529, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.295, 0.354, 0.637, 0.361]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 530, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/W16-3210.URL https://aclanthology.org/W16-3210/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15000000, "bbox": [[0.637, 0.354, 0.882, 0.361], [0.136, 0.367, 0.498, 0.378]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 531, "edu_l1_label": "EDU_O"}, {"txt": "Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Man-deep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch,Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.117, 0.396, 0.881, 0.406], [0.139, 0.413, 0.883, 0.425], [0.135, 0.431, 0.819, 0.441]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 532, "edu_l1_label": "EDU_O"}, {"txt": "Beyond english-centric multilingual machine translation, 2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.819, 0.431, 0.878, 0.44], [0.135, 0.451, 0.557, 0.457]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 533, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2010.11125.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.557, 0.451, 0.881, 0.458], [0.135, 0.466, 0.159, 0.475]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 534, "edu_l1_label": "EDU_O"}, {"txt": "Stella Frank, Desmond Elliott, and Lucia Specia.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.117, 0.494, 0.497, 0.504]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 535, "edu_l1_label": "EDU_O"}, {"txt": "Assessing multilingual multimodal image descrip-tion: Studies of native speaker preferences and translator choices.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.497, 0.494, 0.881, 0.501], [0.134, 0.512, 0.641, 0.521]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 536, "edu_l1_label": "EDU_O"}, {"txt": "Natural Language Engineering,24(3):393-413,2018.doi: 10.1017/S1351324918000074.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.641, 0.512, 0.883, 0.523], [0.135, 0.528, 0.561, 0.538]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 537, "edu_l1_label": "EDU_O"}, {"txt": "Chaoyou Fu,Peixian Chen, Yunhang SShen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng,Ke Li,Xing Sun, et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.117, 0.556, 0.881, 0.566], [0.135, 0.573, 0.381, 0.583]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 538, "edu_l1_label": "EDU_O"}, {"txt": "Mme: A comprehensive evaluation benchmark for multimodal large language models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.381, 0.573, 0.882, 0.583], [0.134, 0.59, 0.314, 0.6]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 539, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2306.13394,2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.314, 0.59, 0.623, 0.6]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 540, "edu_l1_label": "EDU_O"}, {"txt": "Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Pa-pangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay,Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson,Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, Di Jin, Dimitra Gkatzia, Dragomir Radev, Elizabeth Clark, Esin Durmus, Faisal Ladhak, Filip Ginter, Genta Indra Winata, Hen-drik Strobelt, Hiroaki Hayashi, Jekaterina Novikova, Jenna Kanerva, Jenny Chim, Jiawei Zhou,Jordan Clive, Joshua Maynez, João Sedoc, Juraj Juraska, Kaustubh Dhole, Khyathi Raghavi Chandu, Laura Perez Beltrachini, Leonardo F. R. Ribeiro, Lewis Tunstall, Li Zhang,Mahim Pushkarna, Mathias Creutz, Michael White, Mihir Sanjay Kale, Moussa Kamal Eddine, Nico Daheim, Nishant Subramani, Ondrej Dusek, Paul Pu Liang, Pawan Sasanka Ammanamanchi,Qi Zhu, Ratish Puduppully, Reno Kriz, Rifat Shahriyar, Ronald Cardenas, Saad Mahamood,Salomey Osei, Samuel Cahyawijaya, Sanja Stajner, Sebastien Montella, Shailza Jolly, Simon Mille, Tahmid Hasan, Tianhao Shen, Tosin Adewumi, Vikas Raunak, Vipul Raheja, Vitaly Niko-laev, Vivian Tsai, Yacine Jernite,Ying Xu, Yisi Sang, Yixin Liu, and Yufang Hou.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.117, 0.619, 0.881, 0.628], [0.135, 0.639, 0.881, 0.645], [0.135, 0.652, 0.883, 0.665], [0.135, 0.669, 0.881, 0.68], [0.135, 0.688, 0.882, 0.694], [0.135, 0.704, 0.883, 0.717], [0.135, 0.721, 0.881, 0.731], [0.135, 0.739, 0.88, 0.749], [0.135, 0.756, 0.88, 0.765], [0.136, 0.773, 0.883, 0.784], [0.135, 0.79, 0.883, 0.802], [0.135, 0.807, 0.881, 0.817], [0.134, 0.824, 0.882, 0.831], [0.135, 0.841, 0.807, 0.851]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 541, "edu_l1_label": "EDU_O"}, {"txt": "GEMv2:Multilingual NLG benchmarking in a single line of code.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.807, 0.841, 0.882, 0.851], [0.135, 0.858, 0.604, 0.868]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 542, "edu_l1_label": "EDU_O"}, {"txt": "In Wanxiang Che and Ekaterina Shutova (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.604, 0.858, 0.881, 0.867], [0.135, 0.875, 0.249, 0.886]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 543, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 2022 Conference on Empirical Methods in Natural Lan-guage Processing: Systemn Demonstrations, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.249, 0.875, 0.881, 0.885], [0.135, 0.896, 0.515, 0.903]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 544, "edu_l1_label": "EDU_O"}, {"txt": "266-281, Abu Dhabi, UAE, December 2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15100000, "bbox": [[0.515, 0.896, 0.883, 0.903]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 545, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15200000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 546, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15200000, "bbox": [[0.135, 0.095, 0.479, 0.104]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 547, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2022.emnlp-demos.27.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15200000, "bbox": [[0.479, 0.095, 0.833, 0.104]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 548, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2022.emnlp-demos.27/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15200000, "bbox": [[0.833, 0.095, 0.878, 0.104], [0.134, 0.113, 0.556, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 549, "edu_l1_label": "EDU_O"}, {"txt": "Gregor Geigle, Florian Schneider, Carolin Holtermann, Chris Biemann, Radu Timofte,Anne Lauscher, and Goran Glavaš.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15300000, "bbox": [[0.117, 0.138, 0.881, 0.148], [0.135, 0.155, 0.36, 0.165]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 550, "edu_l1_label": "EDU_O"}, {"txt": "Centurio: On drivers of multilingual ability of large vision-language model, 2025.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15300000, "bbox": [[0.36, 0.155, 0.881, 0.166], [0.139, 0.176, 0.234, 0.182]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 551, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2501.05122.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15300000, "bbox": [[0.234, 0.176, 0.585, 0.182]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 552, "edu_l1_label": "EDU_O"}, {"txt": "Gemini Team Google, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.117, 0.199, 0.88, 0.21], [0.135, 0.216, 0.599, 0.226]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 553, "edu_l1_label": "EDU_O"}, {"txt": "GGemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.599, 0.216, 0.88, 0.226], [0.135, 0.237, 0.577, 0.244]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 554, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2403.05530.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.577, 0.237, 0.881, 0.243], [0.135, 0.252, 0.178, 0.26]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 555, "edu_l1_label": "EDU_O"}, {"txt": "Adriana Guevara-Rukoz, Isin Demirsahin, Fei He, Shan-Hui Cathy Chu, Supheakmungkol Sarin,Knot Pipatsrisawat, Alexander Gutkin, Alena Butryna, and Oddur Kjartansson.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.118, 0.277, 0.883, 0.289], [0.137, 0.294, 0.761, 0.304]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 556, "edu_l1_label": "EDU_O"}, {"txt": "Crowdsourcing Latin American Spanish for low-resource text-to-speech.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.761, 0.294, 0.881, 0.306], [0.135, 0.311, 0.573, 0.321]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 557, "edu_l1_label": "EDU_O"}, {"txt": "In Nicoletta Calzolari, Frédéric Béchet,Philippe Blache, Khalid Choukri,Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isa-hara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.573, 0.311, 0.883, 0.324], [0.136, 0.328, 0.882, 0.335], [0.135, 0.346, 0.881, 0.356], [0.135, 0.363, 0.252, 0.374]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 558, "edu_l1_label": "EDU_O"}, {"txt": "),Proceedings of the Twelfth Language Resources and Evaluation Conference,pp.6504-6513,Marseille, France, May 2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.252, 0.363, 0.883, 0.373], [0.135, 0.381, 0.453, 0.39]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 559, "edu_l1_label": "EDU_O"}, {"txt": "European Language Resources Association.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.453, 0.381, 0.796, 0.39]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 560, "edu_l1_label": "EDU_O"}, {"txt": "ISBN 979-10-95546-34-4.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.796, 0.381, 0.882, 0.387], [0.136, 0.397, 0.249, 0.407]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 561, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2020.1rec-1.801/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.249, 0.397, 0.682, 0.407]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 562, "edu_l1_label": "EDU_O"}, {"txt": "Zheqi He,Xinya Wu, Pengfei Zhou,Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu,and Hua Huang.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.117, 0.423, 0.881, 0.433], [0.141, 0.441, 0.195, 0.451]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 563, "edu_l1_label": "EDU_O"}, {"txt": "Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.195, 0.441, 0.88, 0.451], [0.134, 0.461, 0.216, 0.468]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 564, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2401.14011,2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15400000, "bbox": [[0.216, 0.461, 0.525, 0.468]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 565, "edu_l1_label": "EDU_O"}, {"txt": "Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.117, 0.484, 0.879, 0.494], [0.135, 0.502, 0.225, 0.512]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 566, "edu_l1_label": "EDU_O"}, {"txt": "Measuring massive multitask language understanding.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.225, 0.502, 0.658, 0.512]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 567, "edu_l1_label": "EDU_O"}, {"txt": "In International Conference on Learning Representations, 2021.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.658, 0.502, 0.882, 0.511], [0.136, 0.522, 0.41, 0.53]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 568, "edu_l1_label": "EDU_O"}, {"txt": "URL https://openreview.net/forum?id=d7KBjmI3GmQ.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.41, 0.522, 0.86, 0.528]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 569, "edu_l1_label": "EDU_O"}, {"txt": "Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.117, 0.545, 0.685, 0.555]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 570, "edu_l1_label": "EDU_O"}, {"txt": "Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models, 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.685, 0.545, 0.881, 0.555], [0.135, 0.562, 0.84, 0.573]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 571, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2408.10151.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.84, 0.562, 0.882, 0.573], [0.134, 0.58, 0.435, 0.59]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 572, "edu_l1_label": "EDU_O"}, {"txt": "Carlos Daniel Hernandez Mena and Ivan Vladimir Meza Ruiz.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.117, 0.605, 0.601, 0.616]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 573, "edu_l1_label": "EDU_O"}, {"txt": "Creating Mexican Spanish language resources through the social service program.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.601, 0.605, 0.881, 0.616], [0.134, 0.626, 0.489, 0.633]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 574, "edu_l1_label": "EDU_O"}, {"txt": "In Chris Callison-Burch, Christopher Cieri, James Fiumara, and Mark Liberman (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.489, 0.626, 0.881, 0.633], [0.135, 0.64, 0.408, 0.652]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 575, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the2nd Workshop on Novel Incentives in Data Collection from People: models, implementations,challenges and results within LREC 2022, pp.20-24,Marseille, France, June 2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.408, 0.64, 0.882, 0.65], [0.136, 0.657, 0.882, 0.667], [0.135, 0.676, 0.425, 0.685]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 576, "edu_l1_label": "EDU_O"}, {"txt": "European Language Resources Association.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.425, 0.676, 0.771, 0.685]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 577, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2022.nidcp-1.4/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15500000, "bbox": [[0.771, 0.676, 0.88, 0.685], [0.135, 0.692, 0.453, 0.703]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 578, "edu_l1_label": "EDU_O"}, {"txt": "Kaiyu Huang,Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang,Weijian Yi,Yulong Mao, Jinchen Liu, Yuzhuang Xu, Jinan Xu, Jian-Yun Nie, and Yang Liu.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.117, 0.718, 0.881, 0.73], [0.135, 0.735, 0.73, 0.746]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 579, "edu_l1_label": "EDU_O"}, {"txt": "A survey on large language models with multilingualism: Recent advances and new frontiers, 2025.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.73, 0.735, 0.881, 0.746], [0.134, 0.752, 0.773, 0.762]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 580, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org,", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.773, 0.752, 0.879, 0.763], [0.135, 0.77, 0.243, 0.78]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 581, "edu_l1_label": "EDU_O"}, {"txt": "Pratik Joshi, Christain Barnes, Sebastin Santy, Simran Khanuja, Sanket Shah, Anirudh Srini-vasan,Satwik Bhattamishra, Sunayana Sitaram, Monojit Choudhury, and Kalika Bali.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.118, 0.796, 0.881, 0.804], [0.135, 0.817, 0.814, 0.824]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 582, "edu_l1_label": "EDU_O"}, {"txt": "Unsung challenges of building and deploying language technologies for low resource language commu-nities.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.814, 0.817, 0.881, 0.825], [0.134, 0.831, 0.88, 0.84], [0.134, 0.85, 0.184, 0.858]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 583, "edu_l1_label": "EDU_O"}, {"txt": "In Dipti Misra Sharma and Pushpak Bhattacharya (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.184, 0.85, 0.652, 0.859]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 584, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 16th In-ternational Conference on Natural Language Processing, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.652, 0.85, 0.88, 0.858], [0.136, 0.866, 0.609, 0.875]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 585, "edu_l1_label": "EDU_O"}, {"txt": "211-219, International Institute of Information Technology, Hyderabad, India, December 2019.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.609, 0.866, 0.882, 0.874], [0.135, 0.882, 0.615, 0.891]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 586, "edu_l1_label": "EDU_O"}, {"txt": "NLP Association of India.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.615, 0.882, 0.833, 0.892]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 587, "edu_l1_label": "EDU_O"}, {"txt": "URL", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15600000, "bbox": [[0.833, 0.882, 0.879, 0.892]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 588, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.489, 0.938, 0.511, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 11, "global_sentence_id": 589, "edu_l1_label": "EDU_O"}, {"txt": "Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.118, 0.094, 0.771, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 590, "edu_l1_label": "EDU_O"}, {"txt": "The state and fate of linguistic diversity and inclusion in the NLP world.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.771, 0.094, 0.879, 0.104], [0.135, 0.112, 0.586, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 591, "edu_l1_label": "EDU_O"}, {"txt": "In Dan Jurafsky, Joyce Chai,Natalie Schluter, and Joel Tetreault (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.586, 0.112, 0.881, 0.122], [0.135, 0.129, 0.406, 0.14]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 592, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.406, 0.129, 0.881, 0.139], [0.134, 0.145, 0.404, 0.156]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 593, "edu_l1_label": "EDU_O"}, {"txt": "6282-6293, Online, July 2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.404, 0.145, 0.641, 0.156]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 594, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.641, 0.145, 0.882, 0.156], [0.135, 0.162, 0.226, 0.172]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 595, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2020.acl-main.560.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.226, 0.162, 0.528, 0.172]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 596, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2020.ac 1-main.560/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15700000, "bbox": [[0.528, 0.162, 0.881, 0.172], [0.135, 0.181, 0.242, 0.19]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 597, "edu_l1_label": "EDU_O"}, {"txt": "Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam,Keith Stevens,Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexan-der Mattick.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.117, 0.208, 0.882, 0.22], [0.136, 0.225, 0.88, 0.235], [0.135, 0.241, 0.882, 0.249], [0.135, 0.259, 0.235, 0.269]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 598, "edu_l1_label": "EDU_O"}, {"txt": "Openassistant conversations - democratizing large language model alignment.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.235, 0.259, 0.858, 0.269]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 599, "edu_l1_label": "EDU_O"}, {"txt": "In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.858, 0.259, 0.881, 0.269], [0.135, 0.276, 0.772, 0.287]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 600, "edu_l1_label": "EDU_O"}, {"txt": "),Advances in Neural Information Processing Systems, volume 36, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.772, 0.276, 0.882, 0.286], [0.136, 0.293, 0.576, 0.303]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 601, "edu_l1_label": "EDU_O"}, {"txt": "47669-47681.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.576, 0.293, 0.686, 0.303]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 602, "edu_l1_label": "EDU_O"}, {"txt": "Curran Associates,Inc.,2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.686, 0.293, 0.883, 0.305], [0.135, 0.311, 0.178, 0.319]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 603, "edu_l1_label": "EDU_O"}, {"txt": "URL https://proceedings.neurips.cc/paper_files/paper/2023/file/949f0f8f32267d297c2d4e3ee10a2e7e-Paper-Datasets_and_Benchmarks.pdf.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.178, 0.311, 0.881, 0.32], [0.135, 0.329, 0.648, 0.338]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 604, "edu_l1_label": "EDU_O"}, {"txt": "Moshe Koppel and Noam Ordan.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.117, 0.355, 0.377, 0.365]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 605, "edu_l1_label": "EDU_O"}, {"txt": "Translationese and its dialects.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.377, 0.355, 0.624, 0.365]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 606, "edu_l1_label": "EDU_O"}, {"txt": "In Dekang Lin, Yuji Matsumoto,and Rada Mihalcea (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.624, 0.355, 0.883, 0.366], [0.135, 0.375, 0.331, 0.384]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 607, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 49th Annual Meeting of the Association for Compu-tational Linguistics: Human Language Technologies, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.331, 0.375, 0.879, 0.381], [0.136, 0.39, 0.565, 0.399]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 608, "edu_l1_label": "EDU_O"}, {"txt": "1318-1326, Portland, Oregon, USA, June 2011.Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.565, 0.39, 0.881, 0.399], [0.135, 0.407, 0.515, 0.416]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 609, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/P11-1132/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.515, 0.407, 0.883, 0.416]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 610, "edu_l1_label": "EDU_O"}, {"txt": "Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh,Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan,Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Pa-padimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji,Andre Niyongabo Rubungo,Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad,Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F.P.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.116, 0.433, 0.885, 0.652]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 611, "edu_l1_label": "EDU_O"}, {"txt": "Dossou,Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Balli, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya,Duygu Ataman,Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi.Quality at a glance: An audit of web-crawled multilingual datasets.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.116, 0.433, 0.885, 0.652]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 612, "edu_l1_label": "EDU_O"}, {"txt": "Transactions of the As-sociation for Computational Linguistics, 10:50-72, 2022. doi: 10.1162/tacl-$2$ 00447.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.116, 0.433, 0.885, 0.652]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 613, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2022.tacl-1.4/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.116, 0.433, 0.885, 0.652]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 614, "edu_l1_label": "EDU_O"}, {"txt": "Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.117, 0.666, 0.88, 0.676], [0.136, 0.683, 0.199, 0.693]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 615, "edu_l1_label": "EDU_O"}, {"txt": "Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.199, 0.683, 0.88, 0.693], [0.134, 0.701, 0.377, 0.711]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 616, "edu_l1_label": "EDU_O"}, {"txt": "In Yansong Feng and Els Lefever (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.377, 0.701, 0.687, 0.712]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 617, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 2023Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pp.318-327,Singapore, December 2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.687, 0.701, 0.883, 0.711], [0.136, 0.717, 0.882, 0.728], [0.135, 0.735, 0.42, 0.745]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 618, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.42, 0.735, 0.757, 0.745]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 619, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v 1/2023.emnlp-demo.28.URL https://aclanthology.org/2023.emnlp-demo.28/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15800000, "bbox": [[0.757, 0.735, 0.881, 0.745], [0.136, 0.752, 0.779, 0.762]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 620, "edu_l1_label": "EDU_O"}, {"txt": "Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral,Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen,Jörg Frohberg, Mario Saško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont,Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen,Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tris-tan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh ChienVu,Itziar Gonzalez-Dios,Aitor Soroa,Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani,Long", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 15900000, "bbox": [[0.117, 0.778, 0.882, 0.791], [0.135, 0.796, 0.883, 0.809], [0.135, 0.814, 0.881, 0.824], [0.135, 0.831, 0.882, 0.843], [0.135, 0.848, 0.882, 0.855], [0.135, 0.866, 0.882, 0.874], [0.135, 0.882, 0.583, 0.891], [0.589, 0.882, 0.883, 0.893], [0.138, 0.899, 0.881, 0.91]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 621, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.489, 0.938, 0.511, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 7, "global_sentence_id": 622, "edu_l1_label": "EDU_O"}, {"txt": "Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.136, 0.095, 0.88, 0.105], [0.145, 0.111, 0.581, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 623, "edu_l1_label": "EDU_O"}, {"txt": "The bigscience roots corpus: A 1.6tb composite multilingual dataset.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.581, 0.111, 0.88, 0.122], [0.134, 0.132, 0.38, 0.139]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 624, "edu_l1_label": "EDU_O"}, {"txt": "In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.38, 0.132, 0.88, 0.138], [0.135, 0.146, 0.231, 0.158]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 625, "edu_l1_label": "EDU_O"}, {"txt": "), Advances in Neural Information Processing Systems, volume 35, pp.31809-31826.Curran Associates, Inc., 2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.231, 0.146, 0.883, 0.156], [0.135, 0.163, 0.371, 0.172]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 626, "edu_l1_label": "EDU_O"}, {"txt": "URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ce9e92e3de2372a4b93353eb7f3dc0bd-Paper-Datasets_and_Benchmarks.pdf.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.371, 0.163, 0.881, 0.172], [0.135, 0.181, 0.833, 0.19]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 627, "edu_l1_label": "EDU_O"}, {"txt": "Bohao Li,Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan.Seed-bench:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.117, 0.208, 0.882, 0.218], [0.135, 0.225, 0.225, 0.235]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 628, "edu_l1_label": "EDU_O"}, {"txt": " Benchmarking multimodal large language models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.225, 0.225, 0.628, 0.235]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 629, "edu_l1_label": "EDU_O"}, {"txt": "In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.628, 0.225, 0.882, 0.235], [0.136, 0.246, 0.601, 0.253]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 630, "edu_l1_label": "EDU_O"}, {"txt": "13299-13308, 2024a.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.601, 0.246, 0.768, 0.253]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 631, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.1109/CVPR52733.2024.01263.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.768, 0.246, 0.88, 0.254], [0.135, 0.259, 0.327, 0.269]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 632, "edu_l1_label": "EDU_O"}, {"txt": "Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.117, 0.287, 0.779, 0.297]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 633, "edu_l1_label": "EDU_O"}, {"txt": "Bactrian-x:Multilingual replicable instruction-followving models with low-rank adaptation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.779, 0.287, 0.881, 0.297], [0.135, 0.304, 0.758, 0.314]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 634, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2305.15011,2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.758, 0.304, 0.879, 0.314], [0.136, 0.324, 0.324, 0.331]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 635, "edu_l1_label": "EDU_O"}, {"txt": "Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi,Chaoyou Fu, et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.117, 0.349, 0.882, 0.361], [0.135, 0.366, 0.279, 0.376]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 636, "edu_l1_label": "EDU_O"}, {"txt": "A survey on benchmarks of multimodal large language models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.279, 0.366, 0.767, 0.376]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 637, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2408.08632,2024b.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.767, 0.366, 0.879, 0.376], [0.136, 0.386, 0.331, 0.393]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 638, "edu_l1_label": "EDU_O"}, {"txt": "Wenyan Li, Crystina Zhang, Jiaang Li, Qiwei Peng, Raphael Tang, Li Zhou, Weijia Zhang, Guimin Hu, Yifei Yuan, Anders Søgaard, Daniel Hershcovich, and Desmond Elliott.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.117, 0.411, 0.881, 0.421], [0.135, 0.429, 0.73, 0.439]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 639, "edu_l1_label": "EDU_O"}, {"txt": "FoodieQA: A mul-timodal dataset for fine-grained understanding of Chinese food culture.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.73, 0.429, 0.882, 0.436], [0.135, 0.447, 0.703, 0.456]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 640, "edu_l1_label": "EDU_O"}, {"txt": "In Yaser Al-Onaizan,Mohit Bansal, and Yun-Nung Chen (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.703, 0.447, 0.882, 0.456], [0.135, 0.463, 0.451, 0.475]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 641, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 2024 Conference on Empirical Meth-ods in Natural Language Processing, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.451, 0.463, 0.88, 0.472], [0.136, 0.483, 0.454, 0.49]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 642, "edu_l1_label": "EDU_O"}, {"txt": "19077-19095,Miami, Florida, USA, November 2024c.Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.454, 0.483, 0.883, 0.49], [0.135, 0.497, 0.476, 0.507]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 643, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2024.emnlp-main.1063.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.476, 0.497, 0.834, 0.507]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 644, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2024.emnlp-main.1063/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.834, 0.497, 0.879, 0.506], [0.133, 0.515, 0.566, 0.524]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 645, "edu_l1_label": "EDU_O"}, {"txt": "Chin-Yew Lin.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.117, 0.541, 0.23, 0.552]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 646, "edu_l1_label": "EDU_O"}, {"txt": "ROUGE: A package for automatic evaluation of summaries.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.23, 0.541, 0.699, 0.552]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 647, "edu_l1_label": "EDU_O"}, {"txt": "In Text Summarization Branches Out, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.699, 0.541, 0.882, 0.552], [0.137, 0.559, 0.274, 0.569]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 648, "edu_l1_label": "EDU_O"}, {"txt": "74-81,Barcelona, Spain, July 2004.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.274, 0.559, 0.551, 0.568]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 649, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.URL https://aclanthology.org/W04-1013/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.551, 0.559, 0.882, 0.569], [0.135, 0.575, 0.498, 0.585]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 650, "edu_l1_label": "EDU_O"}, {"txt": "Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.118, 0.604, 0.881, 0.613], [0.145, 0.621, 0.194, 0.631]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 651, "edu_l1_label": "EDU_O"}, {"txt": "Visually grounded reasoning across languages and cultures.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.194, 0.621, 0.67, 0.631]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 652, "edu_l1_label": "EDU_O"}, {"txt": "In Marie-Francine Moens,Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.67, 0.621, 0.882, 0.633], [0.134, 0.638, 0.603, 0.65]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 653, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.603, 0.638, 0.883, 0.648], [0.136, 0.659, 0.586, 0.665]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 654, "edu_l1_label": "EDU_O"}, {"txt": "10467-10485, Online and Punta Cana,Dominican Republic, November 2021.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.586, 0.659, 0.883, 0.668], [0.135, 0.672, 0.431, 0.682]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 655, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.431, 0.672, 0.771, 0.682]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 656, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2021.emnlp-main.818.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.771, 0.672, 0.88, 0.682], [0.135, 0.69, 0.343, 0.699]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 657, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2021.emnlp-main.818/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.343, 0.69, 0.811, 0.699]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 658, "edu_l1_label": "EDU_O"}, {"txt": "Shayne Longpre,Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, Xinyi Wu, Enrico Shippole, Kurt Bollacker, Tongshuang Wu, Luis Villa, Alex Pentland, and Sara Hooker.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.116, 0.717, 0.874, 0.727], [0.135, 0.735, 0.88, 0.744], [0.135, 0.751, 0.815, 0.761]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 659, "edu_l1_label": "EDU_O"}, {"txt": "A large-scale audit of dataset licensing and attribution in ai.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.815, 0.751, 0.882, 0.759], [0.135, 0.771, 0.543, 0.778]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 660, "edu_l1_label": "EDU_O"}, {"txt": "Nature Machine Intelligence, 6:975-987,082024. doi: 10.1038/s42256-024-00878-8.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.543, 0.771, 0.881, 0.778], [0.135, 0.785, 0.444, 0.795]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 661, "edu_l1_label": "EDU_O"}, {"txt": "Shayne Longpre, Nikhil Singh, Manuel Cherep, Kushagra Tiwary, Joanna Materzynska, William Brannon, Robert Mahari, Naana Obeng-Marnu, Manan Dey, Mohammed Hamdy, Nayan Saxena,Ahmad Mustafa Anis, Emad A. Alghamdi, Vu Minh Chien, Da Yin, Kun Qian, Yizhi Li, Minnie Liang, An Dinh, Shrestha Mohanty, Deividas Mataciunas, Tobin South, Jianguo Zhang, Ariel N.Lee, Campbell S. Lund, Christopher Klamm, Damien Sileo, Diganta Misra, Enrico Shippole,Kevin Klyman, Lester JV Miranda, Niklas Muennighoff, Seonghyeon Ye, Seungone Kim, Vipul", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16000000, "bbox": [[0.117, 0.814, 0.881, 0.824], [0.14, 0.831, 0.882, 0.843], [0.136, 0.848, 0.881, 0.857], [0.135, 0.864, 0.883, 0.875], [0.135, 0.882, 0.883, 0.893], [0.138, 0.899, 0.881, 0.909]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 662, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.489, 0.938, 0.511, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 10, "global_sentence_id": 663, "edu_l1_label": "EDU_O"}, {"txt": "Gupta, Vivek Sharma, Xuhui Zhou, Caiming Xiong,Luis Villa, Stella Biderman, Alex Pentland,Sara Hooker, and Jad Kabbara.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.135, 0.095, 0.883, 0.107], [0.135, 0.112, 0.382, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 664, "edu_l1_label": "EDU_O"}, {"txt": "Bridging the data provenance gap across text, speech and video,2025.URL https://arxiv.org/abs/2412.17847.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.382, 0.112, 0.883, 0.124], [0.135, 0.129, 0.527, 0.139]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 665, "edu_l1_label": "EDU_O"}, {"txt": "Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James Validad Miranda, Jen-nifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial,Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Jann Railey Montalan, Ryan Ignatius Hadiwijaya, Joanito Agili Lopo, William Nixon,Börje F. Karlsson, James Jaya, Ryandito Diandaru, Yuze Gao, Patrick Amadeus Irawan, Bin Wang, Jan Christian Blaise Cruz, Chenxi Whitehouse, Ivan Halim Parmonangan, Maria Khelli,Wenyu Zhang,Lucky Susanto,Reynard Adha Ryanda, Sonny Lazuardi Hermawan, Dan John Velasco, Muhammad Dehan Al Kautsar, Willy Fitra Hendria, Yasmin Moslem, Noah Flynn,Muhammad Farid Adilazuarda, Haochen Li, Johanes Lee, R. Damanhuri, Shuo Sun, Muham-mad Reza Qorib, Amirbek Djanibekov, Wei Qi Leong, Quyet V. Do, Niklas Muennighoff, Tan-rada Pansuwan, Ilham Firdausi Putra, Yan Xu, Tai Ngee Chia, Ayu Purwarianti, Sebastian Ruder, William Chandra Tjhi, Peerat Limkonchotiwat, Alham Fikri Aji, Sedrick Keh, Genta In-dra Winata, Ruochen Zhang, Fajri Koto, Zheng Xin Yong, and Samuel Cahyawijaya.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.118, 0.157, 0.88, 0.167], [0.134, 0.177, 0.883, 0.186], [0.135, 0.191, 0.881, 0.201], [0.141, 0.208, 0.883, 0.221], [0.135, 0.226, 0.881, 0.235], [0.136, 0.243, 0.883, 0.255], [0.135, 0.261, 0.881, 0.27], [0.135, 0.277, 0.883, 0.289], [0.135, 0.294, 0.88, 0.304], [0.134, 0.314, 0.88, 0.32], [0.134, 0.331, 0.88, 0.338], [0.135, 0.345, 0.881, 0.355], [0.14, 0.362, 0.784, 0.372]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 666, "edu_l1_label": "EDU_O"}, {"txt": "SEACrowd:A multilingual multimodal data hub and benchmark suite for Southeast Asian languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.784, 0.362, 0.882, 0.372], [0.135, 0.38, 0.815, 0.389]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 667, "edu_l1_label": "EDU_O"}, {"txt": "In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.815, 0.38, 0.881, 0.39], [0.136, 0.397, 0.565, 0.408]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 668, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.565, 0.397, 0.882, 0.407], [0.137, 0.414, 0.56, 0.424]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 669, "edu_l1_label": "EDU_O"}, {"txt": "5155-5203, Miami, Florida, USA, Novem-ber 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.56, 0.414, 0.88, 0.424], [0.135, 0.431, 0.21, 0.441]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 670, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.21, 0.431, 0.552, 0.441]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 671, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2024.emnlp-main.296.URL https://aclanthology.org/2024.emnlp-main.296/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.552, 0.431, 0.883, 0.441], [0.135, 0.448, 0.599, 0.458]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 672, "edu_l1_label": "EDU_O"}, {"txt": "Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu,Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord,Peter Clark, and Ashwin Kalyan.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.118, 0.477, 0.882, 0.487], [0.134, 0.494, 0.394, 0.503]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 673, "edu_l1_label": "EDU_O"}, {"txt": "Learn to explain: Multimodal reasoning via thought chains for science question answering.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.394, 0.494, 0.881, 0.504], [0.134, 0.514, 0.352, 0.521]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 674, "edu_l1_label": "EDU_O"}, {"txt": "In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.352, 0.514, 0.88, 0.521], [0.135, 0.528, 0.234, 0.539]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 675, "edu_l1_label": "EDU_O"}, {"txt": "), Advances in Neural Information Processing Systems, volume 35, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.234, 0.528, 0.79, 0.539]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 676, "edu_l1_label": "EDU_O"}, {"txt": "2507-2521.Curran Associates, Inc., 2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.79, 0.528, 0.883, 0.538], [0.135, 0.545, 0.369, 0.555]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 677, "edu_l1_label": "EDU_O"}, {"txt": "URL https://proceedings.neurips.cc/paper_files/paper/2022/file/11332b6b6cf4485b84afadb1352d3a9a-Paper-Conference.pdf.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.369, 0.545, 0.881, 0.555], [0.135, 0.563, 0.712, 0.571]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 678, "edu_l1_label": "EDU_O"}, {"txt": "Pan Lu,Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng,Kai-Wei Chang, Michel Galley, and Jianfeng Gao.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.117, 0.59, 0.882, 0.602], [0.136, 0.607, 0.517, 0.617]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 679, "edu_l1_label": "EDU_O"}, {"txt": "Mathvista: Evaluating mathematical reasoning of foundation models in vvisual contexts.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.517, 0.607, 0.881, 0.619], [0.134, 0.628, 0.447, 0.635]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 680, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2310.02255,2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.447, 0.628, 0.757, 0.635]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 681, "edu_l1_label": "EDU_O"}, {"txt": "Alexandra Luccioni and Joseph Viviano.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.117, 0.653, 0.44, 0.664]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 682, "edu_l1_label": "EDU_O"}, {"txt": "What's in the box?", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.44, 0.653, 0.601, 0.663]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 683, "edu_l1_label": "EDU_O"}, {"txt": "an analysis of undesirable content in the Common Crawl corpus.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.601, 0.653, 0.88, 0.664], [0.134, 0.671, 0.381, 0.68]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 684, "edu_l1_label": "EDU_O"}, {"txt": "In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.381, 0.671, 0.882, 0.681], [0.135, 0.687, 0.176, 0.699]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 685, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.176, 0.687, 0.882, 0.697], [0.136, 0.708, 0.883, 0.714], [0.136, 0.721, 0.236, 0.732]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 686, "edu_l1_label": "EDU_O"}, {"txt": "182-189, Online, August 2021.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.236, 0.721, 0.49, 0.732]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 687, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.49, 0.721, 0.841, 0.732]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 688, "edu_l1_label": "EDU_O"}, {"txt": "doi:10.18653/v1/2021.acl-short.24.URL https://aclanthology.org/2021.acl-short.24/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.841, 0.721, 0.882, 0.732], [0.136, 0.739, 0.827, 0.749]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 689, "edu_l1_label": "EDU_O"}, {"txt": "Li Lucy,Suchin Gururangan, Luca Soldaini, Emma Strubell, DDavid Bamman, Lauren Klein, and Jesse Dodge.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.124, 0.767, 0.881, 0.777], [0.135, 0.784, 0.233, 0.794]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 690, "edu_l1_label": "EDU_O"}, {"txt": "AboutMe: Using self-descriptions in webpages to document the effects of English pre-training data filters.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.233, 0.784, 0.88, 0.793], [0.134, 0.802, 0.288, 0.811]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 691, "edu_l1_label": "EDU_O"}, {"txt": "In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.288, 0.802, 0.737, 0.813]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 692, "edu_l1_label": "EDU_O"}, {"txt": "),Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.737, 0.802, 0.882, 0.811], [0.136, 0.819, 0.883, 0.83], [0.135, 0.84, 0.161, 0.846]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 693, "edu_l1_label": "EDU_O"}, {"txt": "7393-7420,Bangkok, Thailand, August 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.161, 0.84, 0.515, 0.845]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 694, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.515, 0.84, 0.85, 0.846]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 695, "edu_l1_label": "EDU_O"}, {"txt": "doi:10.18653/v1/2024.acl-long.400.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.85, 0.84, 0.882, 0.846], [0.135, 0.852, 0.377, 0.862]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 696, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2024.acl-long.400/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.377, 0.852, 0.828, 0.862]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 697, "edu_l1_label": "EDU_O"}, {"txt": "Muhammad Maaz, Hanoona Abdul Rasheed, Abdelrahman M. Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Timothy Baldwin, Michael Felsberg, and Fahad Shahbaz", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16100000, "bbox": [[0.117, 0.881, 0.874, 0.891], [0.135, 0.898, 0.88, 0.908]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 698, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 8, "global_sentence_id": 699, "edu_l1_label": "EDU_O"}, {"txt": "Khan.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.144, 0.094, 0.185, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 700, "edu_l1_label": "EDU_O"}, {"txt": "Palo: A polyglot large multimodal model for 5b people.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.185, 0.094, 0.642, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 701, "edu_l1_label": "EDU_O"}, {"txt": "ArXiv, abs/2402.14818, 2024.URL https://api.semanticscholar.org/CorpusID:267782854.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.642, 0.094, 0.883, 0.105], [0.135, 0.111, 0.644, 0.121]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 702, "edu_l1_label": "EDU_O"}, {"txt": "Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hen-dricks, Karolina Stanczak, and Aishwarya Agrawal.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.117, 0.14, 0.881, 0.148], [0.135, 0.157, 0.545, 0.167]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 703, "edu_l1_label": "EDU_O"}, {"txt": "Benchmarking vision language models for cultural understanding.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.545, 0.157, 0.88, 0.167], [0.134, 0.177, 0.322, 0.184]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 704, "edu_l1_label": "EDU_O"}, {"txt": "In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.322, 0.177, 0.833, 0.186]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 705, "edu_l1_label": "EDU_O"}, {"txt": "), Pro-ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.833, 0.177, 0.88, 0.183], [0.136, 0.195, 0.835, 0.201]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 706, "edu_l1_label": "EDU_O"}, {"txt": "5769-5790,Miami, Florida, USA, November 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.835, 0.195, 0.882, 0.198], [0.135, 0.209, 0.495, 0.218]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 707, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.495, 0.209, 0.844, 0.218]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 708, "edu_l1_label": "EDU_O"}, {"txt": "doi:10.18653/v1/2024.emnlp-main.329.URL https://aclanthology.org/2024.emnlp-main.329/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.844, 0.209, 0.882, 0.218], [0.136, 0.227, 0.879, 0.235]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 709, "edu_l1_label": "EDU_O"}, {"txt": "Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Taiwo Fagbohungbe,Solomon Oluwole Akinola, Shamsuddeen Muhammad, Salomon Kabongo Kabenamualu, Sa-lomey Osei, Freshia Sackey, Rubungo Andre Niyongabo,Ricky Macharm, Perez Ogayo, Ore-vaoghene Ahia, Musie Meressa Berhe, Mofetoluwa Adeyemi, Masabata Mokgesi-Selinga, Lawrence Okegbemi,Laura Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji,Kathleen Siminyu,Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Abbott, Iroro Orife, Ignatius Ezeani,Idris Abdulkadir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko,Murhabazi Espoir, Elan van Biljon, Daniel Whitenack, Christopher Onyefuluchi, Chris Chinenye Emezue, Bonaventure F. P. Dossou, Blessing Sibanda, Blessing Bassey, Ayodele Olabiyi, Arshath Ramkilowan, Alp Öktem, Adewale Akinfaderin, and Abdallah Bashir.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.117, 0.254, 0.882, 0.266], [0.135, 0.271, 0.88, 0.281], [0.134, 0.289, 0.882, 0.296], [0.134, 0.308, 0.881, 0.315], [0.135, 0.323, 0.883, 0.335], [0.135, 0.339, 0.883, 0.351], [0.135, 0.357, 0.883, 0.369], [0.135, 0.374, 0.881, 0.383], [0.136, 0.391, 0.88, 0.401], [0.135, 0.408, 0.679, 0.417]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 710, "edu_l1_label": "EDU_O"}, {"txt": "Participatory research for low-resourced machine translation: A case study in African languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.679, 0.408, 0.881, 0.418], [0.134, 0.426, 0.674, 0.436]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 711, "edu_l1_label": "EDU_O"}, {"txt": "In Trevor Cohn, Yulan He,and Yang Liu (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.674, 0.426, 0.883, 0.437], [0.135, 0.446, 0.291, 0.454]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 712, "edu_l1_label": "EDU_O"}, {"txt": "), Findings of the Association for Computational Linguistics: EMNLP 2020,pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.291, 0.446, 0.883, 0.455], [0.135, 0.463, 0.16, 0.469]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 713, "edu_l1_label": "EDU_O"}, {"txt": "2144-2160, Online, November 2020.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.16, 0.463, 0.44, 0.469]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 714, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.44, 0.463, 0.775, 0.469]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 715, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2020.findings-emnlp.195.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.775, 0.463, 0.88, 0.47], [0.135, 0.478, 0.366, 0.487]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 716, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2020.findings-emnlp.195/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.366, 0.478, 0.873, 0.485]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 717, "edu_l1_label": "EDU_O"}, {"txt": "OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman,DDiogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian,Jeff Belgum, Irwan Bello, Jake Berdine,Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks,Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey,Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen,Ruby Chen, Jason Chen,Mark Chen,Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecof-fet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman,Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun GGogi-neni,Gabriel Goh,Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene,Joshua Gross, Shixiang Shane Gu,Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris,Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain,Shawn Jain, Joanne Jang,Angela Jiang,Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto,Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Ni-tish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, An-drew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe,Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin,Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju,Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer,", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16200000, "bbox": [[0.117, 0.505, 0.881, 0.515], [0.136, 0.522, 0.881, 0.532], [0.136, 0.539, 0.883, 0.551], [0.135, 0.556, 0.88, 0.568], [0.136, 0.573, 0.883, 0.585], [0.135, 0.59, 0.883, 0.603], [0.135, 0.607, 0.881, 0.619], [0.135, 0.625, 0.88, 0.635], [0.135, 0.642, 0.88, 0.652], [0.136, 0.659, 0.882, 0.666], [0.135, 0.676, 0.883, 0.687], [0.135, 0.693, 0.882, 0.701], [0.135, 0.713, 0.881, 0.72], [0.135, 0.727, 0.883, 0.739], [0.136, 0.744, 0.881, 0.754], [0.141, 0.761, 0.883, 0.773], [0.135, 0.778, 0.883, 0.791], [0.136, 0.797, 0.882, 0.803], [0.135, 0.814, 0.882, 0.823], [0.136, 0.83, 0.881, 0.84], [0.135, 0.847, 0.882, 0.859], [0.136, 0.864, 0.881, 0.876], [0.135, 0.881, 0.883, 0.893], [0.136, 0.898, 0.882, 0.91]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 718, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 5, "global_sentence_id": 719, "edu_l1_label": "EDU_O"}, {"txt": "Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko,Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk,David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo,Hyeonwoo Noh,Long Ouyang,Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ash-ley Pantuliano, Giambattista Parascandolo, Joel Parish,Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Hen-rique Ponde de Oliveira Pinto, Michael, PPokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Pow-ell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry,Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh,Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin,Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Pet-roski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson,Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek,Juan Felipe Cerón Uribe, Andrea Vallone,Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright,Justin Jay Wang, Alvin WVang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Weli-hinda,Peter Welinder, Jiayi Weng, Lilian Weng, Matt WViethoff,Dave Willner, Clemens Winter,Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao,Tao Xu,Sarah Yoo,Kevin Yu, Qiming Yuan, Wojciech Zaremba,Rowan Zellers,Chong Zhang,Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph.Gpt-4 technical report, 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.136, 0.095, 0.881, 0.104], [0.135, 0.112, 0.881, 0.122], [0.146, 0.129, 0.883, 0.14], [0.135, 0.146, 0.882, 0.158], [0.135, 0.162, 0.88, 0.172], [0.134, 0.18, 0.882, 0.19], [0.134, 0.198, 0.881, 0.207], [0.134, 0.217, 0.882, 0.222], [0.135, 0.234, 0.881, 0.242], [0.141, 0.249, 0.882, 0.259], [0.136, 0.266, 0.881, 0.276], [0.135, 0.283, 0.881, 0.293], [0.135, 0.3, 0.883, 0.311], [0.134, 0.317, 0.88, 0.326], [0.135, 0.337, 0.883, 0.347], [0.135, 0.351, 0.883, 0.362], [0.135, 0.368, 0.883, 0.379], [0.135, 0.385, 0.882, 0.393], [0.134, 0.403, 0.882, 0.414], [0.135, 0.419, 0.883, 0.432], [0.136, 0.437, 0.883, 0.448], [0.135, 0.454, 0.883, 0.464], [0.135, 0.47, 0.361, 0.481]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 720, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2303.08774.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.361, 0.47, 0.711, 0.481]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 721, "edu_l1_label": "EDU_O"}, {"txt": "Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.118, 0.499, 0.635, 0.509]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 722, "edu_l1_label": "EDU_O"}, {"txt": "Bleu: a method for automatic evaluation of machine translation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.635, 0.499, 0.881, 0.509], [0.134, 0.52, 0.408, 0.527]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 723, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.408, 0.52, 0.88, 0.527], [0.134, 0.533, 0.478, 0.543]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 724, "edu_l1_label": "EDU_O"}, {"txt": "311-318, USA, 2002.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.478, 0.533, 0.642, 0.543]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 725, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.642, 0.533, 0.882, 0.544], [0.135, 0.549, 0.226, 0.56]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 726, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.3115/1073083.1073135.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.226, 0.549, 0.471, 0.56]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 727, "edu_l1_label": "EDU_O"}, {"txt": "URL https://doi.org/10.3115/1073083.1073135.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.471, 0.549, 0.883, 0.56]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 728, "edu_l1_label": "EDU_O"}, {"txt": "Wannaphong Phatthiyaphaibun, Surapon Nonesung, Patomporn Payoungkhamdee, Peerat Limkon-chotiwat, Can Udomcharoenchaikit, Jitkapat Sawatphol, Chompakorn Chaksangchaichot, Ekapol Chuangsuwanich, and Sarana Nutanong.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.117, 0.578, 0.88, 0.588], [0.134, 0.598, 0.881, 0.606], [0.135, 0.614, 0.455, 0.625]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 729, "edu_l1_label": "EDU_O"}, {"txt": "Wangchanlion and wangchanx mrc eval, 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.455, 0.614, 0.837, 0.624]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 730, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2403.16127.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.837, 0.614, 0.88, 0.624], [0.134, 0.632, 0.435, 0.641]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 731, "edu_l1_label": "EDU_O"}, {"txt": "Luiza Pozzobon, Patrick Lewis, Sara Hooker, and Beyza Ermis.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.117, 0.658, 0.61, 0.669]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 732, "edu_l1_label": "EDU_O"}, {"txt": "From one to many: Expanding the scope of toxicity mitigation in language models, 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.61, 0.658, 0.881, 0.669], [0.134, 0.679, 0.554, 0.686]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 733, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2403.03893.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.554, 0.679, 0.881, 0.686], [0.135, 0.694, 0.159, 0.703]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 734, "edu_l1_label": "EDU_O"}, {"txt": "Chen Qiu, Dan Oneatǎ, Emanuele Bugliarello, Stella Frank, and Desmond Elliott.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.117, 0.721, 0.778, 0.732]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 735, "edu_l1_label": "EDU_O"}, {"txt": "Multilingual multimodal learning with machine translated text.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.778, 0.721, 0.882, 0.731], [0.138, 0.742, 0.539, 0.749]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 736, "edu_l1_label": "EDU_O"}, {"txt": "In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.539, 0.742, 0.881, 0.748], [0.136, 0.754, 0.266, 0.768]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 737, "edu_l1_label": "EDU_O"}, {"txt": "),Findings of the Association for Computational Linguistics: EMNLP 2022,pp.4178-4193,Abu Dhabi, United Arab Emirates, December 2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.266, 0.754, 0.883, 0.766], [0.136, 0.773, 0.639, 0.783]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 738, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.639, 0.773, 0.881, 0.783], [0.135, 0.789, 0.226, 0.799]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 739, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2022.findings-emnlp.308.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.226, 0.789, 0.571, 0.799]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 740, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2022.findings-emnlp.308/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.571, 0.789, 0.881, 0.799], [0.134, 0.807, 0.335, 0.817]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 741, "edu_l1_label": "EDU_O"}, {"txt": "Qwen-Team.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.117, 0.836, 0.218, 0.846]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 742, "edu_l1_label": "EDU_O"}, {"txt": "Qwen2.5-vl, January 2025.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.218, 0.836, 0.432, 0.846]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 743, "edu_l1_label": "EDU_O"}, {"txt": "URL https://qwenlm.github.io/blog/qwen2.5-v1/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.432, 0.836, 0.862, 0.846]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 744, "edu_l1_label": "EDU_O"}, {"txt": "Rita Ramos, Emanuele Bugliarello, Bruno Martins, and Desmond Elliott.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.118, 0.864, 0.708, 0.874]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 745, "edu_l1_label": "EDU_O"}, {"txt": "PAELLA: Parameter-efficient lightweight language-agnostic captioning model.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.708, 0.864, 0.881, 0.874], [0.135, 0.884, 0.586, 0.891]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 746, "edu_l1_label": "EDU_O"}, {"txt": "In Kevin Duh, Helena Gomez,and Steven Bethard (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.586, 0.884, 0.88, 0.891], [0.135, 0.898, 0.301, 0.911]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 747, "edu_l1_label": "EDU_O"}, {"txt": "), Findings of the Association for Computational Linguistics: NAACL 2024,", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16300000, "bbox": [[0.301, 0.898, 0.882, 0.91]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 748, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16400000, "bbox": [[0.489, 0.938, 0.511, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 8, "global_sentence_id": 749, "edu_l1_label": "EDU_O"}, {"txt": "pp.3549-3564,Mexico City, Mexico, June 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16400000, "bbox": [[0.141, 0.098, 0.511, 0.104]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 750, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16400000, "bbox": [[0.511, 0.098, 0.848, 0.104]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 751, "edu_l1_label": "EDU_O"}, {"txt": "doi:10.18653/v1/2024.findings-naacl.225.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16400000, "bbox": [[0.848, 0.098, 0.882, 0.104], [0.135, 0.112, 0.427, 0.121]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 752, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2024.findings-naacl .225/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16400000, "bbox": [[0.427, 0.112, 0.881, 0.122], [0.136, 0.136, 0.186, 0.14]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 753, "edu_l1_label": "EDU_O"}, {"txt": "Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shiv-alika Singh, Rishabh Maheshwary, Micol Altomare, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Viraat Aryabumi, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sharad Duwal, Daniil Dzenhaliou,Daniel Fernando Erazo Florez, Fabian Farestam, Joseph Marvin Imperial, Shayekh Bin Islam,Perttu Isotalo, Maral Jabbarishiviari, Börje F. Karlsson, Eldar Khalilov,Christopher Klamm,Fajri Koto, Dominik Krzemiński, Gabriel Adriano de Melo, Syrielle Montariol, Yiyang Nan, Joel Niklaus, Jekaterina Novikova, Johan Samir Obando Ceron, Debjit Paul, Esther Ploeger, Jebish Purbey, Swati Rajwal, Selvan Sunitha Ravi, Sara Rydell, Roshan Santhosh, Drishti Sharma, Mar-jana Prifti Skenduli, Arshia Soltani Moakhar, Bardia Soltani Moakhar, Ran Tamir, Ayush Kumar Tarun, Azmine Toushik Wasi, Thenuka Ovin Weerasinghe, Serhan Yilmaz, Mike Zhang, Imanol Schlag, Marzieh Fadaee, Sara Hooker, and Antoine Bosselut.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16500000, "bbox": [[0.117, 0.156, 0.881, 0.164], [0.135, 0.177, 0.88, 0.184], [0.136, 0.191, 0.88, 0.202], [0.135, 0.208, 0.883, 0.22], [0.136, 0.225, 0.883, 0.237], [0.135, 0.242, 0.883, 0.254], [0.136, 0.259, 0.882, 0.269], [0.135, 0.275, 0.88, 0.286], [0.136, 0.293, 0.882, 0.3], [0.134, 0.311, 0.881, 0.32], [0.136, 0.328, 0.881, 0.337], [0.135, 0.345, 0.618, 0.355]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 754, "edu_l1_label": "EDU_O"}, {"txt": "Include: Evaluating multilingual language understanding with regional knowledge.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16500000, "bbox": [[0.618, 0.345, 0.882, 0.354], [0.134, 0.361, 0.524, 0.372]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 755, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2411.19799,2024.URL https://arxiv.org/abs/2411.19799.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16500000, "bbox": [[0.524, 0.361, 0.88, 0.372], [0.134, 0.38, 0.435, 0.389]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 756, "edu_l1_label": "EDU_O"}, {"txt": "David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani,David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Gan-zorig Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok Bi-adglign Ademtew, Hernán Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan,Laura Alonso Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fer-nando D'Haro, Marcelo Viridiano, Marcos Estecha-Garitagoitia, Maria Camila Buitrago Cabrera,Mario Rodríguez-Cantelar, Mélanie Jouitteau, Mihail Mihaylov, Mohamed Fazli Mohamed Imam,Muhammad Farid Adilazuarda, Munkhjargal Gochoo, Munkh-Erdene Otgonbold, Naome Etori,Olivier Niyomugisha, Paula Mónica Silva, Pranjal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang,Ryandito Diandaru, Samuel Cahyawijaya, Santiago Góngora,Soyeong Jeong,Sukannya Purkayastha, Tatsuki Kuribayashi, Thanmay Jayakumar, Tiago Timponi Torrent, Toqeer Ehsan,Vladimir Araujo, Yova Kementchedjhieva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, Thamar Solorio, and Alham Fikri Aji.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16600000, "bbox": [[0.117, 0.405, 0.881, 0.416], [0.136, 0.423, 0.881, 0.433], [0.135, 0.44, 0.883, 0.451], [0.135, 0.457, 0.881, 0.467], [0.135, 0.478, 0.882, 0.482], [0.135, 0.495, 0.881, 0.501], [0.135, 0.509, 0.883, 0.521], [0.135, 0.525, 0.881, 0.535], [0.135, 0.546, 0.882, 0.555], [0.135, 0.559, 0.883, 0.572], [0.135, 0.577, 0.883, 0.59], [0.135, 0.594, 0.88, 0.604], [0.135, 0.61, 0.881, 0.621], [0.136, 0.629, 0.883, 0.64], [0.135, 0.646, 0.88, 0.656], [0.134, 0.662, 0.734, 0.673]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 757, "edu_l1_label": "EDU_O"}, {"txt": "Cvqa: Culturally-diverse multilingual visual question answering benchmark.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16600000, "bbox": [[0.734, 0.662, 0.881, 0.672], [0.135, 0.681, 0.588, 0.69]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 758, "edu_l1_label": "EDU_O"}, {"txt": "NeurIPS 2024, 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16600000, "bbox": [[0.588, 0.681, 0.757, 0.69]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 759, "edu_l1_label": "EDU_O"}, {"txt": "Florian Schneider and Sunayana Sitaram.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16700000, "bbox": [[0.118, 0.707, 0.444, 0.717]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 760, "edu_l1_label": "EDU_O"}, {"txt": "M5-a diverse benchmark to assess the performance of large multimodal models across multilingual and multicultural vision-language tasks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16700000, "bbox": [[0.444, 0.707, 0.883, 0.717], [0.134, 0.724, 0.807, 0.734]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 761, "edu_l1_label": "EDU_O"}, {"txt": "In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16700000, "bbox": [[0.807, 0.724, 0.88, 0.734], [0.135, 0.742, 0.569, 0.754]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 762, "edu_l1_label": "EDU_O"}, {"txt": "), Findings of the Association for Com-putational Linguistics: EMNLP 2024, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16700000, "bbox": [[0.569, 0.742, 0.884, 0.749], [0.135, 0.763, 0.473, 0.769]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 763, "edu_l1_label": "EDU_O"}, {"txt": "4309-4345, Miami, Florida, USA, November 2024.Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16700000, "bbox": [[0.473, 0.763, 0.883, 0.769], [0.136, 0.776, 0.475, 0.786]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 764, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2024.findings-emnlp.250.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16700000, "bbox": [[0.475, 0.776, 0.836, 0.786]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 765, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2024.findings-emnlp.250/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16700000, "bbox": [[0.836, 0.776, 0.88, 0.786], [0.134, 0.794, 0.592, 0.803]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 766, "edu_l1_label": "EDU_O"}, {"txt": "Florian Schneider, Carolin Holtermann, Chris Biemann, and Anne Lauscher.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16800000, "bbox": [[0.117, 0.82, 0.721, 0.83]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 767, "edu_l1_label": "EDU_O"}, {"txt": "Gimmick - globally inclusive multimodal multitask cultural knowledge benchmarking, 2025.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16800000, "bbox": [[0.721, 0.82, 0.881, 0.832], [0.134, 0.838, 0.702, 0.847]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 768, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2502.13766.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16800000, "bbox": [[0.702, 0.838, 0.878, 0.847], [0.135, 0.857, 0.306, 0.864]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 769, "edu_l1_label": "EDU_O"}, {"txt": "Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I Adelani, Jian Gang Ngui, DDaniel Vila-Suero, Peerat Limkonchotiwat,Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 16900000, "bbox": [[0.117, 0.882, 0.882, 0.892], [0.135, 0.899, 0.882, 0.908]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 770, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17000000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 7, "global_sentence_id": 771, "edu_l1_label": "EDU_O"}, {"txt": "Global mmlu:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17000000, "bbox": [[0.135, 0.094, 0.242, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 772, "edu_l1_label": "EDU_O"}, {"txt": " Understanding and addressing cultural and linguistic biases in multilingual evalu-ation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17000000, "bbox": [[0.242, 0.094, 0.881, 0.104], [0.135, 0.115, 0.179, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 773, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2412.03304,2024a.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17000000, "bbox": [[0.179, 0.115, 0.5, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 774, "edu_l1_label": "EDU_O"}, {"txt": "Shivalika Singh, Freddie Vargus, Daniel D'souza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko,Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O'Mahony,Mike Zhang,Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Moura, Dominik Krzemiński,Hakimeh Fadaei, Irem Ergun, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Chien,Sebastian Ruder, Surya Guthikonda, Emad Alghamdi, Sebastian Gehrmann, Niklas Muennighoff,Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.117, 0.14, 0.881, 0.15], [0.136, 0.157, 0.88, 0.167], [0.141, 0.174, 0.88, 0.184], [0.135, 0.191, 0.882, 0.202], [0.136, 0.208, 0.883, 0.221], [0.135, 0.226, 0.75, 0.235]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 775, "edu_l1_label": "EDU_O"}, {"txt": "Aya dataset: An open-access collection for multilingual instruction tuning.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.75, 0.226, 0.881, 0.236], [0.135, 0.246, 0.587, 0.253]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 776, "edu_l1_label": "EDU_O"}, {"txt": "In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.587, 0.246, 0.878, 0.253], [0.135, 0.26, 0.309, 0.271]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 777, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 62nd Annual Meeting of the Association for Com-putational Linguistics (Volume 1: Long Papers), pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.309, 0.26, 0.879, 0.27], [0.135, 0.28, 0.55, 0.287]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 778, "edu_l1_label": "EDU_O"}, {"txt": "11521-11567,Bangkok, Thailand, August 2024b.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.55, 0.28, 0.88, 0.287], [0.135, 0.294, 0.187, 0.304]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 779, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.187, 0.294, 0.533, 0.304]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 780, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2024.acl-long.620.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.533, 0.294, 0.836, 0.304]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 781, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2024.acl-long.620/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17100000, "bbox": [[0.836, 0.294, 0.878, 0.304], [0.134, 0.313, 0.537, 0.321]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 782, "edu_l1_label": "EDU_O"}, {"txt": "Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I. Adelani, Jian Gang Ngui, Daniel Vila-Suero,Peerat Limkonchotiwat,Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng,Shayne Longpre, Wei-Yin Ko, Sebastian Ruder, Madeline Smith, Antoine Bosselut, Alice Oh,Andre F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee,Beyza Ermis, and Sara Hooker.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17200000, "bbox": [[0.117, 0.34, 0.882, 0.35], [0.135, 0.357, 0.88, 0.366], [0.135, 0.373, 0.883, 0.385], [0.136, 0.391, 0.88, 0.401], [0.135, 0.407, 0.335, 0.417]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 783, "edu_l1_label": "EDU_O"}, {"txt": "Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation, 2025.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17200000, "bbox": [[0.335, 0.407, 0.881, 0.418], [0.134, 0.426, 0.439, 0.435]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 784, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2412.03304.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17200000, "bbox": [[0.439, 0.426, 0.788, 0.436]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 785, "edu_l1_label": "EDU_O"}, {"txt": "Ray Smith.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17200000, "bbox": [[0.118, 0.454, 0.207, 0.464]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 786, "edu_l1_label": "EDU_O"}, {"txt": "An overview of the Tesseract OCR engine.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17200000, "bbox": [[0.207, 0.454, 0.537, 0.464]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 787, "edu_l1_label": "EDU_O"}, {"txt": "In Proc.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17200000, "bbox": [[0.537, 0.454, 0.602, 0.464]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 788, "edu_l1_label": "EDU_O"}, {"txt": "Ninth Int.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17200000, "bbox": [[0.602, 0.454, 0.687, 0.464]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 789, "edu_l1_label": "EDU_O"}, {"txt": "Conference on Document Analysis and Recognition (ICDAR), pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17200000, "bbox": [[0.687, 0.454, 0.883, 0.464], [0.136, 0.47, 0.445, 0.481]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 790, "edu_l1_label": "EDU_O"}, {"txt": "629-633, 2007.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17200000, "bbox": [[0.445, 0.47, 0.562, 0.482]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 791, "edu_l1_label": "EDU_O"}, {"txt": "Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17300000, "bbox": [[0.117, 0.499, 0.642, 0.51]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 792, "edu_l1_label": "EDU_O"}, {"txt": "Crossmodal-3600: A massively multilingual multimodal evaluation dataset.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17300000, "bbox": [[0.642, 0.499, 0.881, 0.511], [0.134, 0.52, 0.471, 0.526]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 793, "edu_l1_label": "EDU_O"}, {"txt": "In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17300000, "bbox": [[0.471, 0.52, 0.881, 0.528], [0.135, 0.533, 0.177, 0.545]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 794, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17300000, "bbox": [[0.177, 0.533, 0.883, 0.545], [0.135, 0.554, 0.158, 0.561]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 795, "edu_l1_label": "EDU_O"}, {"txt": "715-729, Abu Dhabi, United Arab Emirates, December 2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17300000, "bbox": [[0.158, 0.554, 0.641, 0.561]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 796, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17300000, "bbox": [[0.641, 0.554, 0.881, 0.56], [0.135, 0.567, 0.225, 0.578]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 797, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2022.emnlp-main.45.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17300000, "bbox": [[0.225, 0.567, 0.536, 0.577]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 798, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2022.em nlp-main.45/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17300000, "bbox": [[0.536, 0.567, 0.881, 0.578], [0.134, 0.587, 0.252, 0.595]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 799, "edu_l1_label": "EDU_O"}, {"txt": "Jesse Thomason, Daniel Gordon, and Yonatan Bisk.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17400000, "bbox": [[0.118, 0.612, 0.526, 0.623]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 800, "edu_l1_label": "EDU_O"}, {"txt": "Shifting the baseline: Single modality perfor-mance on visual navigation & QA.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17400000, "bbox": [[0.526, 0.612, 0.882, 0.621], [0.135, 0.633, 0.413, 0.64]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 801, "edu_l1_label": "EDU_O"}, {"txt": "In Jill Burstein, Christy Doran, and Thamar Solorio (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17400000, "bbox": [[0.413, 0.633, 0.875, 0.642]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 802, "edu_l1_label": "EDU_O"}, {"txt": "),Proceedings of the 2019 Conference of the North American Chapter of the Association for Com-putational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),pp.1977-1983,Minneapolis,Minnesota, June 2019.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17400000, "bbox": [[0.875, 0.633, 0.883, 0.641], [0.136, 0.646, 0.882, 0.655], [0.135, 0.669, 0.883, 0.675], [0.136, 0.681, 0.507, 0.692]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 803, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17400000, "bbox": [[0.507, 0.681, 0.847, 0.692]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 804, "edu_l1_label": "EDU_O"}, {"txt": "doi:10.18653/v1/N19-1197.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17400000, "bbox": [[0.847, 0.681, 0.882, 0.692], [0.136, 0.699, 0.321, 0.71]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 805, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/N19-1197/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17400000, "bbox": [[0.321, 0.699, 0.686, 0.708]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 806, "edu_l1_label": "EDU_O"}, {"txt": "Ahmet Ustün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D'souza,Gbemileke Onilude,Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre,Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer,and Sara Hooker.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17500000, "bbox": [[0.118, 0.727, 0.882, 0.739], [0.135, 0.744, 0.881, 0.754], [0.135, 0.761, 0.761, 0.771]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 807, "edu_l1_label": "EDU_O"}, {"txt": "Aya model: An instruction finetuned open-access multilingual language model.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17500000, "bbox": [[0.761, 0.761, 0.881, 0.771], [0.134, 0.779, 0.627, 0.789]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 808, "edu_l1_label": "EDU_O"}, {"txt": "In Lun-Wei Ku, Andre Martins,and Vivek Srikumar (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17500000, "bbox": [[0.627, 0.779, 0.883, 0.791], [0.135, 0.799, 0.338, 0.807]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 809, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 62nd Annual Meeting of the Association for Com-putational Linguistics (Volume 1:Long Papers), pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17500000, "bbox": [[0.338, 0.799, 0.882, 0.802], [0.135, 0.816, 0.55, 0.823]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 810, "edu_l1_label": "EDU_O"}, {"txt": "15894-15939, Bangkok,Thailand, August 2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17500000, "bbox": [[0.55, 0.816, 0.88, 0.822], [0.135, 0.829, 0.179, 0.84]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 811, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17500000, "bbox": [[0.179, 0.829, 0.524, 0.84]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 812, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2024.acl-long.845.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17500000, "bbox": [[0.524, 0.829, 0.836, 0.84]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 813, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/2024.acl-long.845/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17500000, "bbox": [[0.836, 0.829, 0.88, 0.84], [0.134, 0.848, 0.537, 0.857]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 814, "edu_l1_label": "EDU_O"}, {"txt": "Özlem Uzuner, Imre Solti, Fei Xia, and Eithon Cadag.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17600000, "bbox": [[0.117, 0.872, 0.563, 0.885]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 815, "edu_l1_label": "EDU_O"}, {"txt": "Community annotation experiment for ground truth generation for the i2b2 medication challenge.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17600000, "bbox": [[0.563, 0.872, 0.881, 0.885], [0.135, 0.896, 0.609, 0.903]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 816, "edu_l1_label": "EDU_O"}, {"txt": "Journal of the American Medical", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17600000, "bbox": [[0.609, 0.896, 0.884, 0.902]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 817, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17700000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 818, "edu_l1_label": "EDU_O"}, {"txt": "Informatics Association, 17(5):", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17700000, "bbox": [[0.136, 0.094, 0.384, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 819, "edu_l1_label": "EDU_O"}, {"txt": "519-523, 09 2010.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17700000, "bbox": [[0.384, 0.094, 0.518, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 820, "edu_l1_label": "EDU_O"}, {"txt": "ISSN 1067-5027. doi: 10.1136/jamia.2010.004200.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17700000, "bbox": [[0.518, 0.094, 0.88, 0.104], [0.135, 0.112, 0.168, 0.121]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 821, "edu_l1_label": "EDU_O"}, {"txt": "URL https://doi.org/10.1136/jamia.2010.004200.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17700000, "bbox": [[0.168, 0.112, 0.6, 0.121]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 822, "edu_l1_label": "EDU_O"}, {"txt": "Emiel van Miltenburg, Desmond Elliott, and Piek Vossen.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17800000, "bbox": [[0.118, 0.139, 0.556, 0.149]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 823, "edu_l1_label": "EDU_O"}, {"txt": "Cross-linguistic differences and similarities in image descriptions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17800000, "bbox": [[0.556, 0.139, 0.881, 0.148], [0.134, 0.157, 0.306, 0.165]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 824, "edu_l1_label": "EDU_O"}, {"txt": "In Jose M. Alonso, Alberto Bugarín, and Ehud Reiter (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17800000, "bbox": [[0.306, 0.157, 0.78, 0.167]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 825, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 10th International Conference on Natural Language Generation, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17800000, "bbox": [[0.78, 0.157, 0.883, 0.166], [0.136, 0.176, 0.729, 0.183]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 826, "edu_l1_label": "EDU_O"}, {"txt": "21-30, Santiago de Compostela, Spain, September 2017.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17800000, "bbox": [[0.729, 0.176, 0.881, 0.183], [0.135, 0.189, 0.419, 0.2]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 827, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17800000, "bbox": [[0.419, 0.189, 0.756, 0.2]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 828, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v 1/W17-3503.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17800000, "bbox": [[0.756, 0.189, 0.881, 0.199], [0.136, 0.207, 0.24, 0.217]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 829, "edu_l1_label": "EDU_O"}, {"txt": "URL https://aclanthology.org/W17-3503/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17800000, "bbox": [[0.24, 0.207, 0.606, 0.217]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 830, "edu_l1_label": "EDU_O"}, {"txt": "Eva Vanmassenhove, Dimitar Shterionov, and Matthew Gwilliam.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17900000, "bbox": [[0.118, 0.234, 0.621, 0.244]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 831, "edu_l1_label": "EDU_O"}, {"txt": "Machine translationese: Effects of algorithmic bias on linguistic complexity in machine translation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17900000, "bbox": [[0.621, 0.234, 0.883, 0.244], [0.135, 0.254, 0.623, 0.262]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 832, "edu_l1_label": "EDU_O"}, {"txt": "In Paola Merlo, Jorg Tiedemann,and Reut Tsarfaty (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17900000, "bbox": [[0.623, 0.254, 0.883, 0.264], [0.135, 0.271, 0.333, 0.279]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 833, "edu_l1_label": "EDU_O"}, {"txt": "), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17900000, "bbox": [[0.333, 0.271, 0.881, 0.278], [0.136, 0.285, 0.632, 0.295]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 834, "edu_l1_label": "EDU_O"}, {"txt": "2203-2213, Online, April 2021.Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17900000, "bbox": [[0.632, 0.285, 0.882, 0.295], [0.135, 0.302, 0.47, 0.312]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 835, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.18653/v1/2021.eacl-main.188.URL https://aclanthology.org/2021.eacl-main.188/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 17900000, "bbox": [[0.47, 0.302, 0.879, 0.312], [0.135, 0.319, 0.49, 0.33]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 836, "edu_l1_label": "EDU_O"}, {"txt": "Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadglign Ademtew, Yahya Hmaiti, Amandeep Kumar, Kartik Kuckreja,Mykola Maslych,Wafa Al Ghallabi, Mihail Mihaylov, Chao Qin, Abdelrahman M. Shaker, Mike Zhang, Mahardika Krisna Ihsani, Amiel Esplana, Monil Gokani, Shachar Mirkin, Harsh Singh,Ashay Srivastava, Endre Hamerlik, Fathinah Asma Izzati, Fadillah Adamsyah Maani, Sebastian Cavada, Jenny Chim, Rohit Gupta, Sanjay Manjunath, Kamila Zhumakhanova, Feno Herini-aina Rabevohitra, Azril Hafizi Amirudin, Muhammad Ridzuan, Daniya Najiha Abdul Kareem,Ketan More,Kunyang Li, Pramesh Shakya, Muhammad Saad, Amirpouya Ghasemaghaei, Amir-bek Djanibekov, Dilshod Azizov, Branislava Jankovic, Naman Bhatia, Alvaro Cabrera, Johan Obando-Ceron, Olympiah Otieno, Fabian Farestam, Muztoba Rabbani, Sanoojan Baliah, San-tosh Sanjeev, Abduragim Shtanchaev, Maheen Fatima, Thao Nguyen, Amrin Kareem,Toluwani Aremu, Nathan Xavier, Amit Bhatkal, Hawau Olamide Toyin, Aman Chadha,Hisham Cholakkal,Rao Muhammad Anwer, Michael Felsberg, Jorma Laaksonen, Thamar Solorio, Monojit Choud-hury, Ivan Laptev, Mubarak Shah, Salman Khan, and Fahad Shahbaz Khan.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18000000, "bbox": [[0.118, 0.346, 0.881, 0.356], [0.135, 0.364, 0.883, 0.375], [0.135, 0.381, 0.88, 0.39], [0.135, 0.398, 0.883, 0.409], [0.136, 0.415, 0.88, 0.425], [0.135, 0.432, 0.882, 0.439], [0.135, 0.451, 0.883, 0.46], [0.138, 0.466, 0.882, 0.473], [0.135, 0.484, 0.88, 0.493], [0.135, 0.501, 0.881, 0.51], [0.135, 0.52, 0.88, 0.528], [0.135, 0.535, 0.883, 0.547], [0.136, 0.552, 0.881, 0.561], [0.134, 0.569, 0.763, 0.578]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 837, "edu_l1_label": "EDU_O"}, {"txt": "All languages matter: Evaluating Imms on culturally diverse 100 languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18000000, "bbox": [[0.763, 0.569, 0.881, 0.578], [0.135, 0.589, 0.638, 0.596]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 838, "edu_l1_label": "EDU_O"}, {"txt": "ArXiv, abs/2411.16508,2024.URL https://api.semanticscholar.org/CorpusID:274234962.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18000000, "bbox": [[0.638, 0.589, 0.883, 0.596], [0.135, 0.602, 0.645, 0.613]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 839, "edu_l1_label": "EDU_O"}, {"txt": "Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18100000, "bbox": [[0.118, 0.631, 0.629, 0.64]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 840, "edu_l1_label": "EDU_O"}, {"txt": "Cider: Consensus-based image description evaluation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18100000, "bbox": [[0.629, 0.631, 0.881, 0.64], [0.135, 0.646, 0.314, 0.657]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 841, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the IEEE conference on computer vision and pattern recognition,pp.4566-4575,2015.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18100000, "bbox": [[0.314, 0.646, 0.881, 0.657], [0.136, 0.668, 0.391, 0.674]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 842, "edu_l1_label": "EDU_O"}, {"txt": "Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18200000, "bbox": [[0.118, 0.691, 0.789, 0.701]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 843, "edu_l1_label": "EDU_O"}, {"txt": "Measuring multimodal mathematical reasoning with math-vision dataset,2024a.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18200000, "bbox": [[0.789, 0.691, 0.881, 0.703], [0.135, 0.711, 0.672, 0.718]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 844, "edu_l1_label": "EDU_O"}, {"txt": "Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo,Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max KU, Kai Wang, Alex Zhuang, Rongqi Fan,Xiang Yue, and Wenhu Chen.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18300000, "bbox": [[0.118, 0.735, 0.881, 0.747], [0.135, 0.753, 0.881, 0.765], [0.135, 0.77, 0.417, 0.78]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 845, "edu_l1_label": "EDU_O"}, {"txt": "Mmlu-pro: A more robust and challenging multi-task lan-guage understanding benchmark.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18300000, "bbox": [[0.417, 0.77, 0.881, 0.779], [0.135, 0.79, 0.395, 0.796]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 846, "edu_l1_label": "EDU_O"}, {"txt": "In A. Globerson, L. Mackey, D. Belgrave,A.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18300000, "bbox": [[0.395, 0.79, 0.752, 0.796]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 847, "edu_l1_label": "EDU_O"}, {"txt": "Fan, U. Paquet,J.Tomczak,and C. Zhang (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18300000, "bbox": [[0.752, 0.79, 0.882, 0.798], [0.135, 0.803, 0.382, 0.815]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 848, "edu_l1_label": "EDU_O"}, {"txt": "), Advances in Neural Information Processing Systems, volume 37,pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18300000, "bbox": [[0.382, 0.803, 0.883, 0.815], [0.135, 0.824, 0.158, 0.833]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 849, "edu_l1_label": "EDU_O"}, {"txt": "95266-95290.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18300000, "bbox": [[0.158, 0.824, 0.272, 0.831]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 850, "edu_l1_label": "EDU_O"}, {"txt": "Curran Associates, Inc., 2024b.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18300000, "bbox": [[0.272, 0.824, 0.517, 0.831]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 851, "edu_l1_label": "EDU_O"}, {"txt": "URL https://proceedings.neurips.cc/pap er_files/paper/2024/file/ad236edc564f3e3156e1b2feafb99a24-Paper-Datasets_and_Be nchmarks_Track.pdf.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18300000, "bbox": [[0.517, 0.824, 0.881, 0.833], [0.135, 0.841, 0.881, 0.848], [0.134, 0.857, 0.307, 0.866]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 852, "edu_l1_label": "EDU_O"}, {"txt": "Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi,Quoc V. Le, and Dennyy Zhou.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18400000, "bbox": [[0.118, 0.881, 0.883, 0.893], [0.135, 0.899, 0.382, 0.909]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 853, "edu_l1_label": "EDU_O"}, {"txt": "Chain-of-thought prompting elicits reasoning in large language", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18400000, "bbox": [[0.382, 0.899, 0.881, 0.91]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 854, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18500000, "bbox": [[0.489, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 9, "global_sentence_id": 855, "edu_l1_label": "EDU_O"}, {"txt": "models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18500000, "bbox": [[0.135, 0.098, 0.197, 0.105]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 856, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18500000, "bbox": [[0.197, 0.098, 0.882, 0.107], [0.136, 0.112, 0.503, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 857, "edu_l1_label": "EDU_O"}, {"txt": "Curran Associates Inc. ISBN 9781713871088.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18500000, "bbox": [[0.503, 0.112, 0.86, 0.122]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 858, "edu_l1_label": "EDU_O"}, {"txt": "Peng Xu,Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18600000, "bbox": [[0.117, 0.14, 0.88, 0.15], [0.136, 0.157, 0.392, 0.167]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 859, "edu_l1_label": "EDU_O"}, {"txt": "Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18600000, "bbox": [[0.392, 0.157, 0.881, 0.167], [0.135, 0.177, 0.321, 0.184]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 860, "edu_l1_label": "EDU_O"}, {"txt": "IEEE Transactions on Pattern Analysis and Machine Intelligence,47(3):1877-1893,2025. doi: 10.1109/TPAMI.2024.3507000.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18600000, "bbox": [[0.321, 0.177, 0.883, 0.184], [0.136, 0.191, 0.551, 0.201]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 2, "global_sentence_id": 861, "edu_l1_label": "EDU_O"}, {"txt": "Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Yun Xing, Junjue Wang,Huitao Li,Xin Li,Kunyu Yu, et al.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18700000, "bbox": [[0.117, 0.22, 0.88, 0.23], [0.134, 0.236, 0.354, 0.247]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 862, "edu_l1_label": "EDU_O"}, {"txt": "Mmlu-prox: A multilingual benchmark for advanced large language model evaluation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18700000, "bbox": [[0.354, 0.236, 0.881, 0.247], [0.134, 0.257, 0.275, 0.265]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 863, "edu_l1_label": "EDU_O"}, {"txt": "arXiv preprint arXiv:2503.10497,2025.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18700000, "bbox": [[0.275, 0.257, 0.585, 0.265]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 3, "global_sentence_id": 864, "edu_l1_label": "EDU_O"}, {"txt": "Xiang Yue,Yuansheng Ni, Tianyu Zheng,Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang,Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu,Ruibin Yuan, Renliang Sun, Ming Yin,Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18800000, "bbox": [[0.116, 0.282, 0.881, 0.293], [0.136, 0.301, 0.881, 0.311], [0.135, 0.316, 0.881, 0.327], [0.135, 0.334, 0.184, 0.344]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 865, "edu_l1_label": "EDU_O"}, {"txt": "Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18800000, "bbox": [[0.184, 0.334, 0.881, 0.344], [0.135, 0.351, 0.252, 0.361]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 866, "edu_l1_label": "EDU_O"}, {"txt": "In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18800000, "bbox": [[0.252, 0.351, 0.881, 0.361], [0.136, 0.368, 0.239, 0.378]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 867, "edu_l1_label": "EDU_O"}, {"txt": "9556-9567,2024a.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18800000, "bbox": [[0.239, 0.368, 0.384, 0.378]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 868, "edu_l1_label": "EDU_O"}, {"txt": "doi: 10.1109/CVPR52733.2024.00913.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18800000, "bbox": [[0.384, 0.368, 0.684, 0.378]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 4, "global_sentence_id": 869, "edu_l1_label": "EDU_O"}, {"txt": "Xiang Yue,Tianyu Zheng,Yuansheng Ni,Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun,Botao Yu,Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18900000, "bbox": [[0.116, 0.396, 0.882, 0.409], [0.135, 0.414, 0.722, 0.424]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 870, "edu_l1_label": "EDU_O"}, {"txt": "Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark,2024b.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18900000, "bbox": [[0.722, 0.414, 0.881, 0.424], [0.134, 0.434, 0.678, 0.441]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 871, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.or g/abs/2409.02813.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 18900000, "bbox": [[0.678, 0.434, 0.881, 0.441], [0.135, 0.451, 0.288, 0.457]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 5, "global_sentence_id": 872, "edu_l1_label": "EDU_O"}, {"txt": "Xiang Yue, Yueqi Song, Akari Asai, Seungone Kim, Jean de Dieu Nyandwi, Simran Khanuja, Anjali Kantharuban, Lintang Sutawika, Sathyanarayanan Ramamoorthy, and Graham Neubig.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19000000, "bbox": [[0.117, 0.475, 0.882, 0.486], [0.14, 0.493, 0.817, 0.503]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 873, "edu_l1_label": "EDU_O"}, {"txt": "Pangea:A fully open multilingual multimodal llm for 39 languages, 2025.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19000000, "bbox": [[0.817, 0.493, 0.881, 0.504], [0.136, 0.511, 0.645, 0.521]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 874, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/ab s/2410.16153.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19000000, "bbox": [[0.645, 0.511, 0.881, 0.52], [0.135, 0.53, 0.252, 0.538]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 6, "global_sentence_id": 875, "edu_l1_label": "EDU_O"}, {"txt": "Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Y. Zou.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19100000, "bbox": [[0.125, 0.556, 0.829, 0.566]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 876, "edu_l1_label": "EDU_O"}, {"txt": "When and why vision-language models behave like bags-of-words, and what to do about it?", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19100000, "bbox": [[0.829, 0.556, 0.88, 0.566], [0.135, 0.576, 0.812, 0.583]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 877, "edu_l1_label": "EDU_O"}, {"txt": "ArXiv,abs/2210.01936,2022.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19100000, "bbox": [[0.812, 0.576, 0.883, 0.585], [0.135, 0.593, 0.31, 0.6]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 878, "edu_l1_label": "EDU_O"}, {"txt": "URL https://api.semanticscholar.org/CorpusID:252734947.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19100000, "bbox": [[0.31, 0.593, 0.824, 0.6]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 7, "global_sentence_id": 879, "edu_l1_label": "EDU_O"}, {"txt": "Yuhang Zang,Wei Li, Jun Han, Kaiyang Zhou, and Chen Change Loy.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19200000, "bbox": [[0.117, 0.618, 0.663, 0.628]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 880, "edu_l1_label": "EDU_O"}, {"txt": "Contextual object detection with multimodal large language models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19200000, "bbox": [[0.663, 0.618, 0.881, 0.629], [0.135, 0.638, 0.456, 0.646]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 881, "edu_l1_label": "EDU_O"}, {"txt": "International Journal of Computer Vision, pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19200000, "bbox": [[0.456, 0.638, 0.838, 0.646]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 882, "edu_l1_label": "EDU_O"}, {"txt": "1-19,2024.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19200000, "bbox": [[0.838, 0.638, 0.883, 0.647], [0.135, 0.653, 0.176, 0.663]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 8, "global_sentence_id": 883, "edu_l1_label": "EDU_O"}, {"txt": "Mike Zhang and Antonio Toral.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19300000, "bbox": [[0.115, 0.681, 0.883, 0.797]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 884, "edu_l1_label": "EDU_O"}, {"txt": "The effect of translationese in machine translation test sets.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19300000, "bbox": [[0.115, 0.681, 0.883, 0.797]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 885, "edu_l1_label": "EDU_O"}, {"txt": "In Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Had-dow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Christof Monz,Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19300000, "bbox": [[0.115, 0.681, 0.883, 0.797]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 886, "edu_l1_label": "EDU_O"}, {"txt": "),Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research$Pa-$pers),pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19300000, "bbox": [[0.115, 0.681, 0.883, 0.797]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 887, "edu_l1_label": "EDU_O"}, {"txt": "73-81,Florence,Italy, August 2019.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19300000, "bbox": [[0.115, 0.681, 0.883, 0.797]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 888, "edu_l1_label": "EDU_O"}, {"txt": "Association for Computational Linguistics.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19300000, "bbox": [[0.115, 0.681, 0.883, 0.797]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 889, "edu_l1_label": "EDU_O"}, {"txt": "doi:10.18653/v1/W19-5208.URL https://aclanthology.org/W19-5208/.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19300000, "bbox": [[0.115, 0.681, 0.883, 0.797]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 9, "global_sentence_id": 890, "edu_l1_label": "EDU_O"}, {"txt": "Wenxuan Zhang, Sharifah Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing.M3exam:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19400000, "bbox": [[0.118, 0.812, 0.882, 0.823], [0.145, 0.83, 0.207, 0.84]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 891, "edu_l1_label": "EDU_O"}, {"txt": " A multilingual, multimodal, multilevel benchmark for examining large language mod-els.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19400000, "bbox": [[0.207, 0.83, 0.88, 0.839], [0.134, 0.849, 0.164, 0.857]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 892, "edu_l1_label": "EDU_O"}, {"txt": "In Advances in Neural Information Processing Systems 36: Annual Conference on Neu-ral Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, 2023.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19400000, "bbox": [[0.164, 0.849, 0.879, 0.856], [0.136, 0.867, 0.835, 0.874]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 893, "edu_l1_label": "EDU_O"}, {"txt": "URL https://arxiv.org/abs/2306.05179.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19400000, "bbox": [[0.835, 0.867, 0.88, 0.874], [0.134, 0.881, 0.435, 0.891]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 10, "global_sentence_id": 894, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19500000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 11, "global_sentence_id": 895, "edu_l1_label": "EDU_O"}, {"txt": "Mingyang Zhou,Luowei Zhou,Shuohang Wang, Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu.Uc2:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19600000, "bbox": [[0.13, 0.095, 0.882, 0.105], [0.141, 0.112, 0.171, 0.121]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 896, "edu_l1_label": "EDU_O"}, {"txt": " Universal cross-lingual cross-modal vision-and-language pre-training.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19600000, "bbox": [[0.171, 0.112, 0.714, 0.121]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 897, "edu_l1_label": "EDU_O"}, {"txt": "In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR),pp.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19600000, "bbox": [[0.714, 0.112, 0.882, 0.122], [0.136, 0.128, 0.677, 0.139], [0.689, 0.129, 0.792, 0.14]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 898, "edu_l1_label": "EDU_O"}, {"txt": "4155-4165,June 2021.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19600000, "bbox": [[0.792, 0.129, 0.882, 0.141], [0.135, 0.146, 0.218, 0.157]]}]}, "tags": ["text"], "label": "reference", "web_segment_id": 1, "global_sentence_id": 899, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19700000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 2, "global_sentence_id": 900, "edu_l1_label": "EDU_O"}, {"txt": "A Data Collection Details", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19800000, "bbox": [[0.117, 0.093, 0.423, 0.106]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 1, "global_sentence_id": 901, "edu_l1_label": "EDU_O"}, {"txt": "A.1 Dataset Fields", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 19900000, "bbox": [[0.117, 0.129, 0.301, 0.14]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 2, "global_sentence_id": 902, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20000000, "bbox": [[0.116, 0.153, 0.897, 0.649]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 3, "global_sentence_id": 903, "edu_l1_label": "EDU_O"}, {"txt": "Table 7: Structured dataset fields with descriptions used in data collection protocol.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20100000, "bbox": [[0.171, 0.665, 0.828, 0.679]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 4, "global_sentence_id": 904, "edu_l1_label": "EDU_O"}, {"txt": "A.2 Categories of Visual Elements", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20200000, "bbox": [[0.117, 0.706, 0.444, 0.717]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 5, "global_sentence_id": 905, "edu_l1_label": "EDU_O"}, {"txt": "We group the visual elements into eight primary categories in KALEIDOSCOPE.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20300000, "bbox": [[0.118, 0.736, 0.746, 0.747]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 906, "edu_l1_label": "EDU_O"}, {"txt": "If an image falls into multiple categories, we assign the most representative based on the image's content.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20300000, "bbox": [[0.746, 0.736, 0.883, 0.747], [0.117, 0.754, 0.806, 0.765]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 907, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20400000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 7, "global_sentence_id": 908, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20400000, "bbox": [[0.115, 0.082, 0.873, 0.746]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 909, "edu_l1_label": "EDU_O"}, {"txt": "Continued on next page", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20400000, "bbox": [[0.728, 0.746, 0.875, 0.754]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 2, "global_sentence_id": 910, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20400000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 911, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20400000, "bbox": [[0.119, 0.085, 0.884, 0.756]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 912, "edu_l1_label": "EDU_O"}, {"txt": "Table 8: Types of visual elements or images in the KALEIDOSCOPE benchmark. The correct answer is highlighted in Bold Green. Some samples are reformatted for better presentation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20500000, "bbox": [[0.11, 0.766, 0.881, 0.795]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 913, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20600000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 914, "edu_l1_label": "EDU_O"}, {"txt": "A.3 Complete Results", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20700000, "bbox": [[0.117, 0.093, 0.33, 0.104]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 1, "global_sentence_id": 915, "edu_l1_label": "EDU_O"}, {"txt": "We report full multimodal performances in table 9 for each language and in table 10 for each subject.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20800000, "bbox": [[0.118, 0.124, 0.879, 0.134], [0.117, 0.145, 0.18, 0.152]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 916, "edu_l1_label": "EDU_O"}, {"txt": "Each table reports, for each model and category; Total Accuracy %: the accuracy over all samples, Valid Accuracy %: the accuracy over successfully extracted answers and Format Error % (FE): the proportion of unextracted answers.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20800000, "bbox": [[0.18, 0.145, 0.881, 0.152], [0.117, 0.163, 0.88, 0.169], [0.118, 0.176, 0.551, 0.186]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 917, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 20900000, "bbox": [[0.116, 0.205, 0.885, 0.485]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 3, "global_sentence_id": 918, "edu_l1_label": "EDU_O"}, {"txt": "Table 9: Total Accuracy %, Valid Accuracy % and Format Error % (FE) grouped by Language in KALEIDOSCOPE for multimodal samples.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21000000, "bbox": [[0.116, 0.5, 0.883, 0.53]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 4, "global_sentence_id": 919, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21100000, "bbox": [[0.49, 0.938, 0.509, 0.948]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 5, "global_sentence_id": 920, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21100000, "bbox": [[0.116, 0.089, 0.884, 0.398]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 921, "edu_l1_label": "EDU_O"}, {"txt": "Table 10: Total Accuracy %, Valid Accuracy % and Format Error % (FE) grouped by Subject in KALEIDOSCOPE for multimodal samples.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21200000, "bbox": [[0.115, 0.414, 0.885, 0.444]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 922, "edu_l1_label": "EDU_O"}, {"txt": "We report full multimodal performances in table 9 for each langumage and in table 10 for each subject.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21300000, "bbox": [[0.117, 0.485, 0.88, 0.496], [0.117, 0.506, 0.181, 0.513]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 923, "edu_l1_label": "EDU_O"}, {"txt": "Each table reports, for each model and category; Total Accuracy %: the accuracy over all samples,Valid Accuracy %: the accuracy over successfully extracted answers and Format Error % (FE): the proportion of unextracted answers.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21300000, "bbox": [[0.181, 0.506, 0.881, 0.513], [0.117, 0.523, 0.881, 0.53], [0.125, 0.538, 0.551, 0.547]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 924, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21400000, "bbox": [[0.222, 0.568, 0.778, 0.759]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 4, "global_sentence_id": 925, "edu_l1_label": "EDU_O"}, {"txt": "Table 11: Total Accuracy %, Valid Accuracy % and Format Error % (FE) grouped by Image Type in KALEIDOSCOPE for captioning+ OCR experiment.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21500000, "bbox": [[0.115, 0.774, 0.885, 0.804]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 5, "global_sentence_id": 926, "edu_l1_label": "EDU_O"}, {"txt": "A.4 License", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21600000, "bbox": [[0.116, 0.846, 0.236, 0.857]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 6, "global_sentence_id": 927, "edu_l1_label": "EDU_O"}, {"txt": "To ensure ethical data usage, we prioritize sources that permit redistribution and academic use.During data collection, we filter out content from sources with restrictive licensing policies.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21700000, "bbox": [[0.118, 0.876, 0.882, 0.886], [0.117, 0.894, 0.833, 0.904]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 928, "edu_l1_label": "EDU_O"}, {"txt": "Addi-", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21700000, "bbox": [[0.833, 0.894, 0.88, 0.903]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 929, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21800000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 8, "global_sentence_id": 930, "edu_l1_label": "EDU_O"}, {"txt": "tionally, our dataset does not include personally identifiable information, and all collected exams are either publicly available or obtained under appropriate agreements.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21800000, "bbox": [[0.117, 0.096, 0.881, 0.105], [0.117, 0.115, 0.669, 0.121]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 931, "edu_l1_label": "EDU_O"}, {"txt": "A.5 Prompts", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 21900000, "bbox": [[0.116, 0.154, 0.248, 0.165]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 2, "global_sentence_id": 932, "edu_l1_label": "EDU_O"}, {"txt": "The prompts that we used to perform all experiments were designed to ensure consistency across languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22000000, "bbox": [[0.118, 0.185, 0.881, 0.194], [0.117, 0.202, 0.2, 0.212]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 933, "edu_l1_label": "EDU_O"}, {"txt": "Examples are shown both in English and Spanish as an overview.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22000000, "bbox": [[0.2, 0.202, 0.718, 0.212]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 934, "edu_l1_label": "EDU_O"}, {"txt": "Below is a summary of the key components.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22000000, "bbox": [[0.718, 0.202, 0.88, 0.214], [0.117, 0.222, 0.297, 0.229]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 935, "edu_l1_label": "EDU_O"}, {"txt": "A.5.1 System Message", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22100000, "bbox": [[0.117, 0.262, 0.315, 0.273]]}]}, "tags": ["title"], "label": "title3", "web_segment_id": 4, "global_sentence_id": 936, "edu_l1_label": "EDU_O"}, {"txt": "A system message sets the context for the model, instructing it to act as an expert in solving multiple-choice questions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22200000, "bbox": [[0.117, 0.291, 0.88, 0.301], [0.117, 0.311, 0.254, 0.319]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 937, "edu_l1_label": "EDU_O"}, {"txt": "For zero-shot CoT prompting, the message is provided in all the evaluation languages to support language-specific evaluation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22200000, "bbox": [[0.254, 0.311, 0.881, 0.319], [0.117, 0.326, 0.506, 0.336]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 938, "edu_l1_label": "EDU_O"}, {"txt": "·Zero-shot CoT:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22300000, "bbox": [[0.143, 0.372, 0.298, 0.38]]}]}, "tags": ["title"], "label": "content", "web_segment_id": 6, "global_sentence_id": 939, "edu_l1_label": "EDU_O"}, {"txt": "- English:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22400000, "bbox": [[0.181, 0.406, 0.262, 0.411]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 940, "edu_l1_label": "EDU_O"}, {"txt": " You are an expert at solving multiple-choice questions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22400000, "bbox": [[0.262, 0.406, 0.779, 0.412]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 941, "edu_l1_label": "EDU_O"}, {"txt": "Carefully analyze the question, think step by step, and provide your FINAL answer between the tags <ANSWER>X </ANSWER>, where X is ONLY the correct choice.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22400000, "bbox": [[0.779, 0.406, 0.881, 0.413], [0.201, 0.421, 0.929, 0.427], [0.201, 0.437, 0.356, 0.444], [0.366, 0.436, 0.807, 0.445]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 942, "edu_l1_label": "EDU_O"}, {"txt": "Do not write any additional text between the tags.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22400000, "bbox": [[0.807, 0.436, 0.881, 0.445], [0.201, 0.455, 0.596, 0.463]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 943, "edu_l1_label": "EDU_O"}, {"txt": "- Spanish:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22500000, "bbox": [[0.182, 0.479, 0.269, 0.484]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 944, "edu_l1_label": "EDU_O"}, {"txt": " Eres un experto en resolver preguntas de opción múltiple.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22500000, "bbox": [[0.269, 0.479, 0.797, 0.484]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 945, "edu_l1_label": "EDU_O"}, {"txt": "Analiza cuidadosamente la pregunta, piensa paso a paso y proporciona tu respuesta FINAL entre las etiquetas <ANSWER> X </ANSWER>, donde X es ÚNICAMENTE la opción correcta.No escribas ningún texto adicional entre las etiquetas.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22500000, "bbox": [[0.797, 0.479, 0.881, 0.483], [0.201, 0.494, 0.927, 0.501], [0.201, 0.511, 0.964, 0.518], [0.2, 0.525, 0.706, 0.536]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 946, "edu_l1_label": "EDU_O"}, {"txt": "·Direct answer:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22600000, "bbox": [[0.143, 0.558, 0.291, 0.566]]}]}, "tags": ["title"], "label": "content", "web_segment_id": 9, "global_sentence_id": 947, "edu_l1_label": "EDU_O"}, {"txt": "You are a helpful assistant who answers multiple-choice questions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22700000, "bbox": [[0.161, 0.579, 0.767, 0.589]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 948, "edu_l1_label": "EDU_O"}, {"txt": "For each question,output your final answer in JSON format with the following structure: {\"choice\":\"The correct option (e.g.,A,B,C,or D)\"}.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22700000, "bbox": [[0.767, 0.579, 0.953, 0.59], [0.162, 0.598, 0.905, 0.605], [0.163, 0.613, 0.556, 0.623], [0.56, 0.619, 0.565, 0.623]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 949, "edu_l1_label": "EDU_O"}, {"txt": "ONLY output this format exactly.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22700000, "bbox": [[0.565, 0.619, 0.879, 0.624]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 950, "edu_l1_label": "EDU_O"}, {"txt": "Do not include any additional text or explanations outside the JSON structure.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22700000, "bbox": [[0.879, 0.619, 0.917, 0.622], [0.161, 0.633, 0.852, 0.639]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 951, "edu_l1_label": "EDU_O"}, {"txt": "A.5.2 Keywords", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22800000, "bbox": [[0.117, 0.673, 0.258, 0.682]]}]}, "tags": ["title"], "label": "title3", "web_segment_id": 11, "global_sentence_id": 952, "edu_l1_label": "EDU_O"}, {"txt": "Language-specific keywords are used to structure the prompts consistently across languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22900000, "bbox": [[0.115, 0.702, 0.883, 0.749]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 953, "edu_l1_label": "EDU_O"}, {"txt": "These include terms for$\"Q$uestion, ${}^{\\prime \\prime }\\mathrm {O}$ptions,\",\"and$\"Ar$nswer\" to be included when generating the prompt.For example:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 22900000, "bbox": [[0.115, 0.702, 0.883, 0.749]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 954, "edu_l1_label": "EDU_O"}, {"txt": "·English:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23000000, "bbox": [[0.143, 0.784, 0.223, 0.791]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 955, "edu_l1_label": "EDU_O"}, {"txt": " {\"question\": \"Question\", \"options\":  \"Options\", \"answer\": \"Answer\"}", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23000000, "bbox": [[0.223, 0.784, 0.561, 0.791], [0.583, 0.782, 0.865, 0.792]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 956, "edu_l1_label": "EDU_O"}, {"txt": "·Spanish:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23100000, "bbox": [[0.143, 0.811, 0.227, 0.818]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 957, "edu_l1_label": "EDU_O"}, {"txt": " {\"question\": \"Pregunta\", \"options\":\"Opciones\",\"answer\": \"Respuesta\"}", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23100000, "bbox": [[0.227, 0.811, 0.565, 0.818], [0.585, 0.809, 0.905, 0.819]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 958, "edu_l1_label": "EDU_O"}, {"txt": "A.5.3 Prompt Examples", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23200000, "bbox": [[0.117, 0.852, 0.325, 0.862]]}]}, "tags": ["title"], "label": "title3", "web_segment_id": 15, "global_sentence_id": 959, "edu_l1_label": "EDU_O"}, {"txt": "System messages A.5.1 and Keywords A.5.2 are used to systematically craft the prompt for a model in a specific language.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23300000, "bbox": [[0.117, 0.882, 0.881, 0.892], [0.117, 0.9, 0.289, 0.908]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 16, "global_sentence_id": 960, "edu_l1_label": "EDU_O"}, {"txt": "We show examples of both a closed and an open model in Table 12.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23300000, "bbox": [[0.289, 0.9, 0.822, 0.908]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 16, "global_sentence_id": 961, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23400000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 17, "global_sentence_id": 962, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23400000, "bbox": [[0.143, 0.094, 0.874, 0.737]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 963, "edu_l1_label": "EDU_O"}, {"txt": "Respuesta:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23400000, "bbox": [[0.509, 0.744, 0.586, 0.753]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 2, "global_sentence_id": 964, "edu_l1_label": "EDU_O"}, {"txt": "Table 12: Prompt examples in KALEIDOSCOPE. Multimodal prompt samples with interleaved image are shown for an open model and a closed model.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23500000, "bbox": [[0.115, 0.77, 0.885, 0.801]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 3, "global_sentence_id": 965, "edu_l1_label": "EDU_O"}, {"txt": "A.6 Captioning & OCR", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23600000, "bbox": [[0.117, 0.842, 0.345, 0.854]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 4, "global_sentence_id": 966, "edu_l1_label": "EDU_O"}, {"txt": "We instantiated Gemini 1.5 Pro with the following instruction to generate synthetic captions from the images in KALEIDOSCOPE.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23700000, "bbox": [[0.117, 0.873, 0.874, 0.883], [0.117, 0.892, 0.358, 0.901]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 967, "edu_l1_label": "EDU_O"}, {"txt": "Prompts with image augmentations are shown in Table 13.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23700000, "bbox": [[0.358, 0.892, 0.823, 0.901]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 968, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23800000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 6, "global_sentence_id": 969, "edu_l1_label": "EDU_O"}, {"txt": "Open Model", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 23900000, "bbox": [[0.146, 0.098, 0.257, 0.108]]}]}, "tags": ["title"], "label": "title1", "web_segment_id": 1, "global_sentence_id": 970, "edu_l1_label": "EDU_O"}, {"txt": "SYSTEM:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24000000, "bbox": [[0.145, 0.124, 0.198, 0.132]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 971, "edu_l1_label": "EDU_O"}, {"txt": "You are a helpful assistant who answers multiple-choice questions.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24100000, "bbox": [[0.145, 0.138, 0.445, 0.146], [0.145, 0.154, 0.347, 0.159]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 972, "edu_l1_label": "EDU_O"}, {"txt": "For each question, output your final answer in JSON format with the following structure:\"choice\":\"The correct option (e.g.,A,B, C, or D)\".", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24100000, "bbox": [[0.347, 0.154, 0.422, 0.16], [0.145, 0.168, 0.43, 0.174], [0.146, 0.181, 0.459, 0.188], [0.145, 0.193, 0.437, 0.203], [0.145, 0.207, 0.248, 0.214]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 973, "edu_l1_label": "EDU_O"}, {"txt": "ONLY output this format exactly.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24100000, "bbox": [[0.248, 0.207, 0.436, 0.215], [0.145, 0.223, 0.206, 0.23]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 974, "edu_l1_label": "EDU_O"}, {"txt": "Do not include any additional text or explanations outside the JSON structure.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24100000, "bbox": [[0.206, 0.223, 0.484, 0.229], [0.145, 0.236, 0.475, 0.242]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 975, "edu_l1_label": "EDU_O"}, {"txt": "USER:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24200000, "bbox": [[0.145, 0.263, 0.182, 0.27]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 4, "global_sentence_id": 976, "edu_l1_label": "EDU_O"}, {"txt": "#include<stdio.h>", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24300000, "bbox": [[0.146, 0.275, 0.236, 0.28]]}]}, "tags": ["text"], "label": "code", "web_segment_id": 5, "global_sentence_id": 977, "edu_l1_label": "EDU_O"}, {"txt": "int main(int argc, char *argv[]){", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24300000, "bbox": [[0.147, 0.289, 0.322, 0.296]]}]}, "tags": ["text"], "label": "code", "web_segment_id": 6, "global_sentence_id": 978, "edu_l1_label": "EDU_O"}, {"txt": "$$chara='P';$$", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24300000, "bbox": [[0.172, 0.302, 0.243, 0.311]]}]}, "tags": ["equation"], "label": "code", "web_segment_id": 7, "global_sentence_id": 979, "edu_l1_label": "EDU_O"}, {"txt": "$$charb='x';$$", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24300000, "bbox": [[0.172, 0.316, 0.243, 0.326]]}]}, "tags": ["equation"], "label": "code", "web_segment_id": 8, "global_sentence_id": 980, "edu_l1_label": "EDU_O"}, {"txt": "$$\\mathrm {c}=(\\mathrm {a}\\&\\mathrm {~b})+{}^{\\prime }\\star ^{\\prime }$$", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24300000, "bbox": [[0.172, 0.331, 0.296, 0.341]]}]}, "tags": ["equation"], "label": "code", "web_segment_id": 9, "global_sentence_id": 981, "edu_l1_label": "EDU_O"}, {"txt": "$$chard=(a\\vert b)-'-';$$", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24300000, "bbox": [[0.172, 0.346, 0.296, 0.355]]}]}, "tags": ["equation"], "label": "code", "web_segment_id": 10, "global_sentence_id": 982, "edu_l1_label": "EDU_O"}, {"txt": "$$char\\quad e=(a^b)+'+';$$", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24300000, "bbox": [[0.172, 0.36, 0.296, 0.37]]}]}, "tags": ["equation"], "label": "code", "web_segment_id": 11, "global_sentence_id": 983, "edu_l1_label": "EDU_O"}, {"txt": "$printf(\"\\%c$ $8c$tc\\n\",c,d,e);", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24300000, "bbox": [[0.172, 0.374, 0.333, 0.384]]}]}, "tags": ["text"], "label": "code", "web_segment_id": 12, "global_sentence_id": 984, "edu_l1_label": "EDU_O"}, {"txt": "return 0;", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24300000, "bbox": [[0.173, 0.391, 0.22, 0.397]]}]}, "tags": ["text"], "label": "code", "web_segment_id": 13, "global_sentence_id": 985, "edu_l1_label": "EDU_O"}, {"txt": "1", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24400000, "bbox": [[0.146, 0.405, 0.15, 0.417]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 14, "global_sentence_id": 986, "edu_l1_label": "EDU_O"}, {"txt": "ASCII encoding for relevant characters is given below", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24400000, "bbox": [[0.147, 0.42, 0.338, 0.426]]}]}, "tags": ["text"], "label": "code", "web_segment_id": 15, "global_sentence_id": 987, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24500000, "bbox": [[0.302, 0.432, 0.4, 0.464]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 16, "global_sentence_id": 988, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24500000, "bbox": [[0.185, 0.434, 0.286, 0.464]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 17, "global_sentence_id": 989, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24500000, "bbox": [[0.252, 0.475, 0.313, 0.504]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 18, "global_sentence_id": 990, "edu_l1_label": "EDU_O"}, {"txt": "Caption:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24600000, "bbox": [[0.143, 0.54, 0.494, 0.881]]}]}, "tags": ["text"], "label": "table_title", "web_segment_id": 19, "global_sentence_id": 991, "edu_l1_label": "EDU_O"}, {"txt": " The code initializes character variables 'a' to 'P' and 'b' to ${}^{\\prime }\\mathrm {X}^{\\prime }$.It then calculates 'c','d',and'e'using bitwise operations (&,1,~)and character addition with $,*'$ ,'-',and $'+'$,respectively.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24600000, "bbox": [[0.143, 0.54, 0.494, 0.881]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 19, "global_sentence_id": 992, "edu_l1_label": "EDU_O"}, {"txt": "The 'printf' function outputs the characters c, d, and e. Below the code,three tables display ASCII values: one for uppercase letters 'A' to 'Z'(65 to 90),another for lowercase letters 'a' to 'z'(97 to 122), and a third for symbols '*',${}^{\\prime }+{}^{\\prime }$,and'-'(42,43,and 45,respectively).Ellipses (...) within the tables indicate omitted values between the shown characters.OCR: ##include<stdio.$h>$ \\nint main(int argc,\\n \\nchar $a='P';nchar$ $b='x'$;\\nchar $c=$(a &\\nchar$d=(a$ I\\nchar e $=$ (a\\u 201c\\n \\nprintf $(\\backslash \"sc$\\%\\nreturn 0;\\n \\n $\\backslash \\}\\backslash n$\\nchar *argv[]) \\{\\n \\nby $+$ te;\\nb)$-'-$se\\\\n\\\",c,d,e);\\n \\nASCII encoding for relevant characters is given below\\n \\n 42| 43) 45\\n \\n Question: What is printed by the following ANSI C program?", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24600000, "bbox": [[0.143, 0.54, 0.494, 0.881]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 19, "global_sentence_id": 993, "edu_l1_label": "EDU_O"}, {"txt": "Options: A.)", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24600000, "bbox": [[0.143, 0.54, 0.494, 0.881]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 19, "global_sentence_id": 994, "edu_l1_label": "EDU_O"}, {"txt": "z K s B.)", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24600000, "bbox": [[0.143, 0.54, 0.494, 0.881]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 19, "global_sentence_id": 995, "edu_l1_label": "EDU_O"}, {"txt": "122 7583 C.) $*-+D.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24600000, "bbox": [[0.143, 0.54, 0.494, 0.881]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 19, "global_sentence_id": 996, "edu_l1_label": "EDU_O"}, {"txt": ")$Px+", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24600000, "bbox": [[0.143, 0.54, 0.494, 0.881]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 19, "global_sentence_id": 997, "edu_l1_label": "EDU_O"}, {"txt": "Answer:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24700000, "bbox": [[0.146, 0.886, 0.198, 0.895]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 20, "global_sentence_id": 998, "edu_l1_label": "EDU_O"}, {"txt": "Closed Model", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24800000, "bbox": [[0.51, 0.098, 0.632, 0.108]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 21, "global_sentence_id": 999, "edu_l1_label": "EDU_O"}, {"txt": "SYSTEM:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24900000, "bbox": [[0.509, 0.223, 0.562, 0.231]]}]}, "tags": ["title"], "label": "O", "web_segment_id": 22, "global_sentence_id": 1000, "edu_l1_label": "EDU_O"}, {"txt": "Eres un experto en resolver preguntas de opción múltiple.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24900000, "bbox": [[0.509, 0.237, 0.816, 0.245], [0.51, 0.253, 0.635, 0.258]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 23, "global_sentence_id": 1001, "edu_l1_label": "EDU_O"}, {"txt": "Analiza cuidadosamente la pregunta, piensa paso a paso y proporciona tu respuesta FINAL entre las etiquetas <ANSWER> X </ANSWER>, donde X es ÚNICAMENTE la opción correcta.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24900000, "bbox": [[0.635, 0.253, 0.84, 0.258], [0.509, 0.267, 0.833, 0.273], [0.509, 0.28, 0.801, 0.287], [0.509, 0.294, 0.84, 0.301], [0.509, 0.307, 0.655, 0.314]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 23, "global_sentence_id": 1002, "edu_l1_label": "EDU_O"}, {"txt": "No escribas ningún texto adicional entre las etiquetas.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 24900000, "bbox": [[0.655, 0.307, 0.809, 0.315], [0.509, 0.321, 0.786, 0.328]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 23, "global_sentence_id": 1003, "edu_l1_label": "EDU_O"}, {"txt": "USER:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25000000, "bbox": [[0.508, 0.348, 0.547, 0.355]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 24, "global_sentence_id": 1004, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25100000, "bbox": [[0.522, 0.364, 0.67, 0.448]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 25, "global_sentence_id": 1005, "edu_l1_label": "EDU_O"}, {"txt": "Caption:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25200000, "bbox": [[0.509, 0.467, 0.57, 0.474]]}]}, "tags": ["text"], "label": "table_title", "web_segment_id": 26, "global_sentence_id": 1006, "edu_l1_label": "EDU_O"}, {"txt": " This table presents data on total weight, measured in grams, across four months.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25200000, "bbox": [[0.57, 0.467, 0.838, 0.474], [0.509, 0.482, 0.8, 0.488], [0.515, 0.496, 0.562, 0.502]]}]}, "tags": ["text"], "label": "table_title", "web_segment_id": 26, "global_sentence_id": 1007, "edu_l1_label": "EDU_O"}, {"txt": "The table consists of two columns:Mes(Month) and Peso total(Total Weight).Month 1 shows a weight of 1,500 grams, Month 2 shows 2,600 grams, Month 3 shows 3,700grams, and Month 4 shows 4,800 grams.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25200000, "bbox": [[0.562, 0.496, 0.838, 0.502], [0.509, 0.508, 0.816, 0.516], [0.509, 0.522, 0.846, 0.529], [0.509, 0.535, 0.816, 0.544], [0.509, 0.551, 0.795, 0.557]]}]}, "tags": ["text"], "label": "table_title", "web_segment_id": 26, "global_sentence_id": 1008, "edu_l1_label": "EDU_O"}, {"txt": "The table is a simple grid format with plain black text on a white background.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25200000, "bbox": [[0.795, 0.551, 0.831, 0.556], [0.509, 0.564, 0.816, 0.571], [0.508, 0.577, 0.761, 0.585]]}]}, "tags": ["text"], "label": "table_title", "web_segment_id": 26, "global_sentence_id": 1009, "edu_l1_label": "EDU_O"}, {"txt": "OCR:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25200000, "bbox": [[0.509, 0.591, 0.539, 0.598]]}]}, "tags": ["text"], "label": "title1", "web_segment_id": 27, "global_sentence_id": 1010, "edu_l1_label": "EDU_O"}, {"txt": " Wes | Pesototal\\n 1 [1500 gramos\\n 2I_2600 grmos\\n 3 | 3700 gemoe\\na \\n \\n \\u 201cZ00 grams\\n \\n", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25200000, "bbox": [[0.539, 0.591, 0.824, 0.598], [0.511, 0.604, 0.824, 0.613], [0.51, 0.619, 0.647, 0.626]]}]}, "tags": ["text"], "label": "title1", "web_segment_id": 27, "global_sentence_id": 1011, "edu_l1_label": "EDU_O"}, {"txt": "Pregunta:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25300000, "bbox": [[0.508, 0.633, 0.578, 0.64]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 28, "global_sentence_id": 1012, "edu_l1_label": "BOS"}, {"txt": " Un perro cachorro tenía un peso de 1.500 gramos al mes de nacido.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25300000, "bbox": [[0.578, 0.633, 0.831, 0.64], [0.509, 0.647, 0.762, 0.655]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 28, "global_sentence_id": 1013, "edu_l1_label": "IOS"}, {"txt": "En la tabla se muestra el peso del cachorro en los primeros cuatro meses.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25300000, "bbox": [[0.762, 0.647, 0.817, 0.655], [0.509, 0.661, 0.817, 0.669], [0.509, 0.674, 0.707, 0.682]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 28, "global_sentence_id": 1014, "edu_l1_label": "IOS"}, {"txt": "De acuerdo con la tabla,Cuál es el cambio del peso del cachorro entre un mes y el mes siguiente?Opciones:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25300000, "bbox": [[0.707, 0.674, 0.829, 0.681], [0.509, 0.688, 0.824, 0.696], [0.509, 0.704, 0.824, 0.71], [0.509, 0.716, 0.577, 0.722]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 28, "global_sentence_id": 1015, "edu_l1_label": "IOS"}, {"txt": "A.)", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25400000, "bbox": [[0.509, 0.729, 0.53, 0.737]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 29, "global_sentence_id": 1016, "edu_l1_label": "IOS"}, {"txt": "Disminuyo 1.500 gramos", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25400000, "bbox": [[0.53, 0.729, 0.708, 0.737]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 29, "global_sentence_id": 1017, "edu_l1_label": "IOS"}, {"txt": "B.)", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25500000, "bbox": [[0.509, 0.743, 0.531, 0.752]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 30, "global_sentence_id": 1018, "edu_l1_label": "IOS"}, {"txt": "Disminuyo 2.600 gramos", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25500000, "bbox": [[0.531, 0.743, 0.708, 0.751]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 30, "global_sentence_id": 1019, "edu_l1_label": "IOS"}, {"txt": "C.) Aumentó 1.100 gramos", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25600000, "bbox": [[0.509, 0.757, 0.693, 0.765]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 31, "global_sentence_id": 1020, "edu_l1_label": "IOS"}, {"txt": "D.) Aumentó 3.300 gramos", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25700000, "bbox": [[0.508, 0.771, 0.693, 0.778]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 32, "global_sentence_id": 1021, "edu_l1_label": "IOS"}, {"txt": "Respuesta:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25800000, "bbox": [[0.509, 0.785, 0.585, 0.792]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 33, "global_sentence_id": 1022, "edu_l1_label": "EDU_O"}, {"txt": "Table 13: Caption$+$OCR prompt examples in KALEIDOSCOPE. Prompts are shown for open and closed models in English and Spanish. Caption and OCR additions are highlighted in green.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 25900000, "bbox": [[0.115, 0.912, 0.883, 0.946]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 34, "global_sentence_id": 1023, "edu_l1_label": "EDU_O"}, {"txt": "Gemini 1.5 Pro's prompt for captioning:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26000000, "bbox": [[0.117, 0.094, 0.476, 0.106]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 1024, "edu_l1_label": "IOS"}, {"txt": "$**$Instruction:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26100000, "bbox": [[0.115, 0.141, 0.266, 0.15]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 1025, "edu_l1_label": "EDU_O"}, {"txt": "**", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26100000, "bbox": [[0.115, 0.141, 0.266, 0.15]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 1026, "edu_l1_label": "EDU_O"}, {"txt": "You are an expert image captioner.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26200000, "bbox": [[0.117, 0.158, 0.43, 0.167]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 1027, "edu_l1_label": "EDU_O"}, {"txt": "Generate highly detailed, precise,", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26200000, "bbox": [[0.43, 0.158, 0.752, 0.169]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 1028, "edu_l1_label": "EDU_O"}, {"txt": "and academically relevant textual descriptions of images sourced from exam questions,ensuring all critical visual elements are captured for accurate problem-solving.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26200000, "bbox": [[0.117, 0.178, 0.899, 0.186], [0.118, 0.194, 0.853, 0.203]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 1029, "edu_l1_label": "EDU_O"}, {"txt": "**Guidelines:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26300000, "bbox": [[0.118, 0.229, 0.236, 0.236]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 1030, "edu_l1_label": "EDU_O"}, {"txt": "**", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26300000, "bbox": [[0.236, 0.229, 0.254, 0.234]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 1031, "edu_l1_label": "EDU_O"}, {"txt": "Exam-Specific Analysis:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26400000, "bbox": [[0.117, 0.261, 0.327, 0.27]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 1032, "edu_l1_label": "EDU_O"}, {"txt": "- Primary Elements:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26500000, "bbox": [[0.118, 0.299, 0.291, 0.304]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1033, "edu_l1_label": "EDU_O"}, {"txt": " Identify and describe key components (e.g., diagrams, charts,graphs,labels, symbols, annotations) and their exact attributes (e.g., numerical values, units,directional arrows, text annotations).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26500000, "bbox": [[0.291, 0.299, 0.935, 0.306], [0.117, 0.312, 0.926, 0.323], [0.117, 0.329, 0.464, 0.339]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1034, "edu_l1_label": "EDU_O"}, {"txt": "- Secondary Details:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26600000, "bbox": [[0.118, 0.368, 0.299, 0.372]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 1035, "edu_l1_label": "EDU_O"}, {"txt": " Note stylistic features (e.g.,\"black-and-white schematic,\"\"color-coded bars in a graph\"), spatial relationships (e.g., \"force vectors pointing northwest\"), and contextual clues (e.g., axes labels, legends, scales).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26600000, "bbox": [[0.299, 0.368, 0.586, 0.375], [0.598, 0.363, 0.853, 0.368], [0.118, 0.381, 0.807, 0.39], [0.117, 0.4, 0.854, 0.408]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 8, "global_sentence_id": 1036, "edu_l1_label": "EDU_O"}, {"txt": "- Textual Elements:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26700000, "bbox": [[0.118, 0.436, 0.29, 0.441]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 1037, "edu_l1_label": "EDU_O"}, {"txt": " Explicitly transcribe all visible text (e.g., labels like \"Mitochondria,\" numbers like \"5V,\" titles like \"Figure 2: Velocity vs. Time\").", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26700000, "bbox": [[0.29, 0.436, 0.826, 0.441], [0.118, 0.449, 0.834, 0.459]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 9, "global_sentence_id": 1038, "edu_l1_label": "EDU_O"}, {"txt": "Academic Precision:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26800000, "bbox": [[0.117, 0.483, 0.291, 0.492]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 10, "global_sentence_id": 1039, "edu_l1_label": "EDU_O"}, {"txt": "- Technical Focus:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26900000, "bbox": [[0.115, 0.514, 0.92, 0.563]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 1040, "edu_l1_label": "EDU_O"}, {"txt": " Prioritize details critical to exam questions (e.g., \"aright triangle with hypotenuse labeled$c=10$m,\"\"abar graph comparing GDP of 5 countries,with Japan's bar shaded blue at 4.3 trillion\").", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 26900000, "bbox": [[0.115, 0.514, 0.92, 0.563]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 11, "global_sentence_id": 1041, "edu_l1_label": "EDU_O"}, {"txt": "- Diagrams/Charts:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27000000, "bbox": [[0.118, 0.591, 0.281, 0.595]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 1042, "edu_l1_label": "EDU_O"}, {"txt": " Specify type (e.g.,\"pie chart,\" \"circuit diagram\")and components (e.g.,\"resistor symbol connected to a battery\").", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27000000, "bbox": [[0.281, 0.591, 0.466, 0.598], [0.477, 0.586, 0.76, 0.596], [0.771, 0.588, 0.9, 0.595], [0.118, 0.602, 0.565, 0.613]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 12, "global_sentence_id": 1043, "edu_l1_label": "EDU_O"}, {"txt": "- Scientific Relevance:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27100000, "bbox": [[0.116, 0.634, 0.865, 0.668]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 1044, "edu_l1_label": "IOS"}, {"txt": " Highlight measurements,units, symbols(e.g., $\"T=25^{\\circ }C$,$\\text {\"a}$pulley system with frictionless ropes\").", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27100000, "bbox": [[0.116, 0.634, 0.865, 0.668]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 13, "global_sentence_id": 1045, "edu_l1_label": "IOS"}, {"txt": "Structure & Clarity:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27200000, "bbox": [[0.117, 0.688, 0.299, 0.698]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 14, "global_sentence_id": 1046, "edu_l1_label": "EDU_O"}, {"txt": "- Begin with the image's purpose (e.g., \"A biology diagram of a plant cell\")", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27300000, "bbox": [[0.118, 0.727, 0.815, 0.732]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 15, "global_sentence_id": 1047, "edu_l1_label": "IOS"}, {"txt": "followed by a systematic breakdown (left-to-right, top-to-bottom, or by functional layers).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27300000, "bbox": [[0.117, 0.739, 0.954, 0.749]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 16, "global_sentence_id": 1048, "edu_l1_label": "IOS"}, {"txt": "- Use neutral, objective language.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27400000, "bbox": [[0.118, 0.779, 0.43, 0.784]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 17, "global_sentence_id": 1049, "edu_l1_label": "IOS"}, {"txt": "Avoid assumptions unless implied by context (e.g.,", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27400000, "bbox": [[0.43, 0.779, 0.899, 0.785]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 17, "global_sentence_id": 1050, "edu_l1_label": "IOS"}, {"txt": "$\"a$ downward arrow labeled $9.8\\mathrm {\\sim m}/\\mathrm {s}^{2}$ likely representing gravitational acceleration\").", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27400000, "bbox": [[0.116, 0.788, 0.891, 0.805]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 18, "global_sentence_id": 1051, "edu_l1_label": "EDU_O"}, {"txt": "**Output Format:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27500000, "bbox": [[0.118, 0.826, 0.263, 0.835]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 19, "global_sentence_id": 1052, "edu_l1_label": "EDU_O"}, {"txt": "**", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27500000, "bbox": [[0.263, 0.826, 0.282, 0.832]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 19, "global_sentence_id": 1053, "edu_l1_label": "EDU_O"}, {"txt": "- Single paragraph (4-6 sentences).", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27600000, "bbox": [[0.118, 0.864, 0.439, 0.869]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 20, "global_sentence_id": 1054, "edu_l1_label": "IOS"}, {"txt": "- Example:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27700000, "bbox": [[0.118, 0.898, 0.208, 0.903]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 21, "global_sentence_id": 1055, "edu_l1_label": "IOS"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27800000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 22, "global_sentence_id": 1056, "edu_l1_label": "EDU_O"}, {"txt": "$\"A$ physics diagram depicts two blocks on a frictionless inclined plane:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27900000, "bbox": [[0.113, 0.094, 0.91, 0.176]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 1057, "edu_l1_label": "IOS"}, {"txt": " Block A ($(5kg)$is connected via a rope to Block B ($(3kg)$  over a pulley.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27900000, "bbox": [[0.113, 0.094, 0.91, 0.176]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 1058, "edu_l1_label": "IOS"}, {"txt": "Angle theta$=0^{\\circ }$,with vectors labeled F_normal and F_gravity.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27900000, "bbox": [[0.113, 0.094, 0.91, 0.176]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 1059, "edu_l1_label": "IOS"}, {"txt": "A scale beside the plane shows time$\\mathrm {t}=0\\mathrm {\\sim s}$to t$=5s$.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27900000, "bbox": [[0.113, 0.094, 0.91, 0.176]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 1060, "edu_l1_label": "IOS"}, {"txt": "Text at the bottom reads: 'Calculate tension in the rope.'", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27900000, "bbox": [[0.113, 0.094, 0.91, 0.176]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 1061, "edu_l1_label": "IOS"}, {"txt": "The image is monochrome, with dashed lines indicating motion direction.\"", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 27900000, "bbox": [[0.113, 0.094, 0.91, 0.176]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 1062, "edu_l1_label": "IOS"}, {"txt": "Constraints:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28000000, "bbox": [[0.117, 0.198, 0.225, 0.208]]}]}, "tags": ["title"], "label": "content", "web_segment_id": 2, "global_sentence_id": 1063, "edu_l1_label": "IOS"}, {"txt": "- Avoid Omissions:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28100000, "bbox": [[0.118, 0.237, 0.281, 0.242]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 1064, "edu_l1_label": "IOS"}, {"txt": " Ensure no labels, numbers, or symbols are overlooked, even if small or peripheral.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28100000, "bbox": [[0.281, 0.237, 0.853, 0.242], [0.118, 0.252, 0.301, 0.258]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 1065, "edu_l1_label": "IOS"}, {"txt": "- Neutral Tone:", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28200000, "bbox": [[0.115, 0.282, 0.894, 0.328]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 1066, "edu_l1_label": "IOS"}, {"txt": " Exclude subjective interpretations (e.g., \"messy handwriting\"or \"complex diagram\"') unless style is exam-relevant (e.g.,$\"a$ hand-drawn sketch with annotations\").", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28200000, "bbox": [[0.115, 0.282, 0.894, 0.328]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 4, "global_sentence_id": 1067, "edu_l1_label": "IOS"}, {"txt": "A.7 Open-Weight Models CoT Results", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28300000, "bbox": [[0.116, 0.362, 0.485, 0.372]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 5, "global_sentence_id": 1068, "edu_l1_label": "IOS"}, {"txt": "To benchmark the models, we initially designed a$CoT$ prompt that instructed the models to think step-by-step and then provide the correct answer, marking the choice with the tags <ANSWER></ANSWER>.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28400000, "bbox": [[0.115, 0.391, 0.883, 0.459]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 1069, "edu_l1_label": "IOS"}, {"txt": "However, in preliminary experiments, we found this instruction too complex for mid-to small-sized models (32B-3B), which struggled to follow it consistently.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28400000, "bbox": [[0.115, 0.391, 0.883, 0.459]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 6, "global_sentence_id": 1070, "edu_l1_label": "IOS"}, {"txt": "In Table 14, we compare results using$CoT$versus the direct English-language prompt adopted in our final evaluation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28500000, "bbox": [[0.114, 0.477, 0.885, 0.593]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1071, "edu_l1_label": "IOS"}, {"txt": "The error rate was considerably higher for most models, even after cleaning and extracting answers with regex matching their typical output formats.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28500000, "bbox": [[0.114, 0.477, 0.885, 0.593]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1072, "edu_l1_label": "IOS"}, {"txt": "Two exceptions were Pangea and Molmo, which showed lower error rates with theCoT prompt.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28500000, "bbox": [[0.114, 0.477, 0.885, 0.593]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1073, "edu_l1_label": "IOS"}, {"txt": "However, this was because both ignored the reasoning instruction and simply output the selected option, making extraction easier.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28500000, "bbox": [[0.114, 0.477, 0.885, 0.593]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1074, "edu_l1_label": "IOS"}, {"txt": "Prompt choice significantly impacted performance: the direct English prompt improved results across all models except Pangea,which remained unchanged.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28500000, "bbox": [[0.114, 0.477, 0.885, 0.593]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1075, "edu_l1_label": "IOS"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28600000, "bbox": [[0.157, 0.614, 0.842, 0.811]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 8, "global_sentence_id": 1076, "edu_l1_label": "EDU_O"}, {"txt": "Table 14: Comparison of CoT and direct English prompting on KALEIDOSCOPE for small models.. Reported values are macro-averaged accuracy $\\%$) across all languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28700000, "bbox": [[0.115, 0.828, 0.883, 0.859]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 9, "global_sentence_id": 1077, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28800000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 10, "global_sentence_id": 1078, "edu_l1_label": "EDU_O"}, {"txt": "A.8 Comparison with Other Benchmarks", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 28900000, "bbox": [[0.117, 0.093, 0.503, 0.104]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 1, "global_sentence_id": 1079, "edu_l1_label": "IOS"}, {"txt": "Table A.8 offers a concise comparison of key multimodal benchmarks.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29000000, "bbox": [[0.118, 0.124, 0.661, 0.134]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 1080, "edu_l1_label": "IOS"}, {"txt": "MMMU (Yue et al.,2024a),SEED-Bench (Li et al., 2024a), and MME (Fu et al., 2023) are single-language datasets focused mainly on image-text pairs, with SEED-Bench also incorporating video-text.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29000000, "bbox": [[0.661, 0.124, 0.882, 0.137], [0.117, 0.141, 0.88, 0.151], [0.117, 0.162, 0.738, 0.169]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 1081, "edu_l1_label": "IOS"}, {"txt": "MME is notably smaller and only partially human-annotated, using mostly true/false formats.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29000000, "bbox": [[0.738, 0.162, 0.881, 0.171], [0.117, 0.18, 0.713, 0.186]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 1082, "edu_l1_label": "IOS"}, {"txt": "In contrast, M3Exam (Zhang et al., 2023), EXAMS-V (Das et al., 2024), and M5 (Schneider & Sitaram, 2024) introduce multilingualism-M5 being the most extensive with 41 languages-though much of its content is not multiple-choice and lacks verified annotations.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29000000, "bbox": [[0.713, 0.18, 0.874, 0.186], [0.117, 0.193, 0.881, 0.203], [0.117, 0.213, 0.883, 0.221], [0.117, 0.23, 0.507, 0.237]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 2, "global_sentence_id": 1083, "edu_l1_label": "IOS"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29100000, "bbox": [[0.125, 0.256, 0.886, 0.468]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 3, "global_sentence_id": 1084, "edu_l1_label": "EDU_O"}, {"txt": "Table 15: Comparison of Multimodal Benchmarks.${}^{\\dagger }\\mathrm {All}$1 but 40 questions are in English that measure machine translation capability from Chinese to English.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29200000, "bbox": [[0.115, 0.48, 0.883, 0.513]]}]}, "tags": ["table_caption"], "label": "table_title", "web_segment_id": 4, "global_sentence_id": 1085, "edu_l1_label": "EDU_O"}, {"txt": "KALEIDOSCOPE stands out by offering a balanced composition of 20,911 samples across 18 lan-guages, with a strong focus on multimodal reasoning (11,459 Image-Textsamples),comprehensive human annotation, and a consistent multiple-choice setup.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29300000, "bbox": [[0.118, 0.535, 0.881, 0.545], [0.118, 0.556, 0.881, 0.562], [0.118, 0.569, 0.591, 0.58]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 1086, "edu_l1_label": "IOS"}, {"txt": "Compared to existing benchmarks,KALEIDOSCOPE is more linguistically diverse than M3Exam and EXAMS-V, includes more multi-modal samples than M5, and ensures higher quality through expert-verified annotations, making it a robust and equuitable benchmark for evaluating multilingual multimodal models.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29300000, "bbox": [[0.591, 0.569, 0.883, 0.581], [0.121, 0.586, 0.882, 0.594], [0.117, 0.607, 0.88, 0.614], [0.117, 0.624, 0.754, 0.631]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 5, "global_sentence_id": 1087, "edu_l1_label": "IOS"}, {"txt": "B Evaluation Metrics and the MCQA Framework", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29400000, "bbox": [[0.118, 0.665, 0.685, 0.679]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 6, "global_sentence_id": 1088, "edu_l1_label": "EDU_O"}, {"txt": "Traditional evaluation metrics for VLMs, such as exact match accuracy, BLEU (Papineni et al.,2002), ROUGE (Lin, 2004), and CIDEr (Vedantam et al., 2015), rely on surface-level n-gram com-parisons that often penalize semantically equivalent answers phrased differently from reference texts.In contrast, the multiple-choice question answering (MCQA) framework (Hendrycks et al., 2021;", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29500000, "bbox": [[0.118, 0.7, 0.882, 0.713], [0.118, 0.718, 0.88, 0.727], [0.118, 0.738, 0.882, 0.744], [0.118, 0.751, 0.882, 0.765]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1089, "edu_l1_label": "EDU_O"}, {"txt": "Romero et al., 2024; Lu et al., 2022; Yue et al., 2024a) offers a more humman-like evaluation paradigm by providing predefined answer options.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29500000, "bbox": [[0.118, 0.769, 0.874, 0.779], [0.117, 0.785, 0.431, 0.796]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1090, "edu_l1_label": "EDU_O"}, {"txt": "This reduces ambiguity in scoring and facilitates the cre-ation of evaluation datasets that capture both domain knowledge and linguistic/cultural nuances across languages.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29500000, "bbox": [[0.431, 0.785, 0.881, 0.795], [0.117, 0.806, 0.881, 0.812], [0.117, 0.823, 0.255, 0.83]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1091, "edu_l1_label": "EDU_O"}, {"txt": "Although concerns regarding oversaturation and reliance on superficial cues in MCQA exist (Du et al., 2023; Yuksekgonul et al., 2022), these can be mitigated by extending the an-swer option space and applying rigorous filtering strategies (WNang et al., 2024b; Yue et al., 2024a).Our primary challenge lies in the scarcityy of questions that are both multimodal and culturally agnostic.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29500000, "bbox": [[0.255, 0.823, 0.881, 0.83], [0.118, 0.838, 0.881, 0.847], [0.117, 0.858, 0.882, 0.865], [0.117, 0.871, 0.881, 0.883], [0.117, 0.893, 0.188, 0.899]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1092, "edu_l1_label": "EDU_O"}, {"txt": "As demonstrated by results from KALEIDOSCOPE and related studies (Maaz et al., 2024;", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29500000, "bbox": [[0.188, 0.893, 0.882, 0.902]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 7, "global_sentence_id": 1093, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29600000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 8, "global_sentence_id": 1094, "edu_l1_label": "EDU_O"}, {"txt": "Nayak et al., 2024), oversaturation is not a prevalent issue.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29600000, "bbox": [[0.118, 0.095, 0.578, 0.105]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 1095, "edu_l1_label": "EDU_O"}, {"txt": "Consequently, bridging this evaluation gap is of key importance.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29600000, "bbox": [[0.578, 0.095, 0.88, 0.105], [0.117, 0.115, 0.313, 0.121]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 1096, "edu_l1_label": "EDU_O"}, {"txt": "To ensure high data quality, source data in KALEIDOSCOPE are manually verified by qualified processors in accordance with established criteria (2.2), maintaining a clear distinction between verified and unverified data.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29600000, "bbox": [[0.313, 0.115, 0.881, 0.123], [0.117, 0.132, 0.881, 0.138], [0.117, 0.145, 0.491, 0.156]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 1, "global_sentence_id": 1097, "edu_l1_label": "EDU_O"}, {"txt": "B.1 Selected Dataset Samples", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29700000, "bbox": [[0.118, 0.188, 0.408, 0.199]]}]}, "tags": ["title"], "label": "title2", "web_segment_id": 2, "global_sentence_id": 1098, "edu_l1_label": "EDU_O"}, {"txt": "The subsequent table presents one sample from each dataset, including the question, the associated image,the provided answers, and the correct answer highlighted in green.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29800000, "bbox": [[0.118, 0.219, 0.881, 0.229], [0.117, 0.237, 0.689, 0.246]]}]}, "tags": ["text"], "label": "content", "web_segment_id": 3, "global_sentence_id": 1099, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 4, "global_sentence_id": 1100, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.144, 0.094, 0.861, 0.848]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 1101, "edu_l1_label": "EDU_O"}, {"txt": "Continued on next page", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.709, 0.86, 0.854, 0.868]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 2, "global_sentence_id": 1102, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 1103, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.138, 0.093, 0.859, 0.791]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 1104, "edu_l1_label": "EDU_O"}, {"txt": "Continued on next page", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.708, 0.802, 0.854, 0.811]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 2, "global_sentence_id": 1105, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 1106, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.141, 0.081, 0.872, 0.844]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 1107, "edu_l1_label": "EDU_O"}, {"txt": "Continued on next page", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.709, 0.85, 0.854, 0.858]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 2, "global_sentence_id": 1108, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 1109, "edu_l1_label": "EDU_O"}, {"txt": "figure", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.154, 0.098, 0.849, 0.372]]}]}, "tags": ["figure"], "label": "figure", "web_segment_id": 1, "global_sentence_id": 1110, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.139, 0.079, 0.87, 0.818]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 2, "global_sentence_id": 1111, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 1112, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.136, 0.087, 0.875, 0.79]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 1113, "edu_l1_label": "EDU_O"}, {"txt": "Continued on next page", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.708, 0.793, 0.854, 0.801]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 2, "global_sentence_id": 1114, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.49, 0.938, 0.509, 0.946]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 1115, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.144, 0.084, 0.859, 0.79]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 1116, "edu_l1_label": "EDU_O"}, {"txt": "Continued on next page", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.709, 0.795, 0.854, 0.804]]}]}, "tags": ["text"], "label": "O", "web_segment_id": 2, "global_sentence_id": 1117, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.49, 0.938, 0.508, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 1118, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.141, 0.09, 0.86, 0.823]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 1119, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.708, 0.828, 0.855, 0.838]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 2, "global_sentence_id": 1120, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 1121, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.144, 0.09, 0.878, 0.851]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 1122, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.708, 0.855, 0.855, 0.865]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 2, "global_sentence_id": 1123, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 1124, "edu_l1_label": "EDU_O"}, {"txt": "table", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 29900000, "bbox": [[0.144, 0.091, 0.886, 0.838]]}]}, "tags": ["table"], "label": "table", "web_segment_id": 1, "global_sentence_id": 1125, "edu_l1_label": "EDU_O"}, {"txt": "Table 16: Samples from various exams in the KALEIDOSCOPE benchmark. The correct answer is highlighted in Bold Green. Some samples are reformmatted for better presentation.", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 30000000, "bbox": [[0.111, 0.836, 0.881, 0.868]]}]}, "tags": ["figure_caption"], "label": "table_title", "web_segment_id": 2, "global_sentence_id": 1126, "edu_l1_label": "EDU_O"}, {"txt": "footer", "language": "english", "position": {"pdf_position": [{"page_number": 0, "shape": [1650, 1275], "position_id": 30100000, "bbox": [[0.49, 0.938, 0.509, 0.947]]}]}, "tags": ["footer"], "label": "O", "web_segment_id": 3, "global_sentence_id": 1127, "edu_l1_label": "EDU_O"}], "type": "PDF"}
# Kaleidoscope: In-language Exams forMassively Multilingual Vision Evaluation
## 1 Introduction
## 2 The KALEIDOSCOPE Benchmark
### 2.1 Global Collaboration
### 2.2 Data Pipeline
#### Collection:
#### Processing:
#### Quality Assessment:
### 2.3 Data Statistics
## 3 Experimental Setup
### 3.1 Models
### 3.2 Evaluation Setup
### 3.3 Evaluation Metrics
## 4 Results
### 4.1 Overall Performance
### 4.2 Not All Image Types are Equal
### 4.3 Resource and Script Sensitivity in Models
### 4.4 STEM Questions Expose Model Deficiencies
## 5 Analysis
### 5.1 How Sensitive Are VLMs to Missing or Incorrect Images?
### 5.2 Scaling Model Size Improves Performance
### 5.3 To What Extent Do Textual Augmentations Boost VLM Capabilities?
### 5.4 Format Errors
## 6 Related Work
### 6.1 General Challenges in Multilingual Evaluation of Vision-Language Models
### 6.2 Exam-Style Benchmarks for Vision-Language Models
### 6.3 Participatory Open Science Projects
## 7 Conclusion
