{"entry_id": "fe896cc5-6a7c-43af-b7a0-2321f468c1af", "infos": [{"txt": "LLM-Powered Sorting with TrueSkill", "language": "english", "position": {"atoms": [{"position_id": 0, "txt": "LLM-Powered Sorting with TrueSkill", "x": "//title"}]}, "tags": [], "label": "O", "web_segment_id": 0, "global_sentence_id": 0, "edu_l1_label": "EDU_O"}, {"txt": "LLM-Powered Sorting with TrueSkill", "language": "english", "position": {"atoms": [{"position_id": 318, "txt": "LLM-Powered Sorting with TrueSkill", "x": "/html/body/main/div/div/article/div[1]/h1"}]}, "tags": ["h1"], "label": "article_title", "web_segment_id": 1, "global_sentence_id": 1, "edu_l1_label": "BOT"}, {"txt": "Thariq Shihipar -  11 February 2025 ", "language": "english", "position": {"atoms": [{"position_id": 320, "txt": "Thariq Shihipar - ", "x": "/html/body/main/div/div/article/div[1]/div/p"}, {"position_id": 322, "txt": " 11 February 2025 ", "x": "/html/body/main/div/div/article/div[1]/div/p/time"}]}, "tags": [], "label": "author", "web_segment_id": 2, "global_sentence_id": 2, "edu_l1_label": "EDU_O"}, {"txt": "Large Language Models (LLMs) are remarkably good at understanding and comparing concepts, but getting them to consistently sort large amounts of data is still quite difficult.", "language": "english", "position": {"atoms": [{"position_id": 324, "txt": "Large Language Models (LLMs) are remarkably good at understanding and comparing concepts, but getting them to consistently sort large amounts of data is still quite difficult.", "x": "/html/body/main/div/div/article/div[2]/div/p[2]"}]}, "tags": [], "label": "content", "web_segment_id": 3, "global_sentence_id": 3, "edu_l1_label": "BOS"}, {"txt": "In this post, I share a technique that combines the semantic understanding of LLMs with the mathematical rigor of ELO algorithms like TrueSkill to create robust, scalable semantic sorting.", "language": "english", "position": {"atoms": [{"position_id": 326, "txt": "In this post, I share a technique that combines the semantic understanding of LLMs with the mathematical rigor of ELO algorithms like TrueSkill to create robust, scalable semantic sorting.", "x": "/html/body/main/div/div/article/div[2]/div/p[5]"}]}, "tags": [], "label": "content", "web_segment_id": 4, "global_sentence_id": 4, "edu_l1_label": "IOS"}, {"txt": "The Challenge with LLM Sorting", "language": "english", "position": {"atoms": [{"position_id": 328, "txt": "The Challenge with LLM Sorting", "x": "/html/body/main/div/div/article/div[2]/h2[1]"}]}, "tags": ["h2"], "label": "title1", "web_segment_id": 5, "global_sentence_id": 5, "edu_l1_label": "BOS"}, {"txt": "Let’s say you’d like a LLM to create an ordered list of items, based on a prompt.", "language": "english", "position": {"atoms": [{"position_id": 330, "txt": "Let’s say you’d like a LLM to create an ordered list of items, based on a prompt.", "x": "/html/body/main/div/div/article/div[2]/p[1]"}]}, "tags": [], "label": "content", "web_segment_id": 6, "global_sentence_id": 6, "edu_l1_label": "IOS"}, {"txt": " For example, you might sort job candidates, feature requests, movie recommendations, or customer feedback.", "language": "english", "position": {"atoms": [{"position_id": 331, "txt": " For example, you might sort job candidates, feature requests, movie recommendations, or customer feedback.", "x": "/html/body/main/div/div/article/div[2]/p[1]"}]}, "tags": [], "label": "content", "web_segment_id": 6, "global_sentence_id": 7, "edu_l1_label": "IOS"}, {"txt": "When you ask an LLM to sort a large list of items by itself you run into several problems:", "language": "english", "position": {"atoms": [{"position_id": 333, "txt": "When you ask an LLM to sort a large list of items by itself you run into several problems:", "x": "/html/body/main/div/div/article/div[2]/p[2]"}]}, "tags": [], "label": "content", "web_segment_id": 7, "global_sentence_id": 8, "edu_l1_label": "IOS"}, {"txt": "Context window limitations - You can only feed so many items at once", "language": "english", "position": {"atoms": [{"position_id": 335, "txt": "Context window limitations", "x": "/html/body/main/div/div/article/div[2]/ol[1]/li[1]/strong"}, {"position_id": 337, "txt": " - You can only feed so many items at once", "x": "/html/body/main/div/div/article/div[2]/ol[1]/li[1]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 8, "global_sentence_id": 9, "edu_l1_label": "IOS"}, {"txt": "Consistency issues - The model might rank A > B and B > C, but then rank C > A another time", "language": "english", "position": {"atoms": [{"position_id": 339, "txt": "Consistency issues", "x": "/html/body/main/div/div/article/div[2]/ol[1]/li[2]/strong"}, {"position_id": 341, "txt": " - The model might rank A > B and B > C, but then rank C > A another time", "x": "/html/body/main/div/div/article/div[2]/ol[1]/li[2]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 9, "global_sentence_id": 10, "edu_l1_label": "IOS"}, {"txt": "Quality degradation - As the list grows longer, the quality of sorting tends to decrease as attention degrades", "language": "english", "position": {"atoms": [{"position_id": 343, "txt": "Quality degradation", "x": "/html/body/main/div/div/article/div[2]/ol[1]/li[3]/strong"}, {"position_id": 345, "txt": " - As the list grows longer, the quality of sorting tends to decrease as attention degrades", "x": "/html/body/main/div/div/article/div[2]/ol[1]/li[3]/strong"}]}, "tags": ["strong", "li"], "label": "content", "web_segment_id": 10, "global_sentence_id": 11, "edu_l1_label": "IOS"}, {"txt": "Token limits - Getting back large sorted lists can bump up against output token limits", "language": "english", "position": {"atoms": [{"position_id": 347, "txt": "Token limits", "x": "/html/body/main/div/div/article/div[2]/ol[1]/li[4]/strong"}, {"position_id": 349, "txt": " - Getting back large sorted lists can bump up against output token limits", "x": "/html/body/main/div/div/article/div[2]/ol[1]/li[4]/strong"}]}, "tags": ["strong", "li"], "label": "content", "web_segment_id": 11, "global_sentence_id": 12, "edu_l1_label": "IOS"}, {"txt": "Enter TrueSkill", "language": "english", "position": {"atoms": [{"position_id": 351, "txt": "Enter TrueSkill", "x": "/html/body/main/div/div/article/div[2]/h2[2]"}]}, "tags": ["h2"], "label": "title1", "web_segment_id": 12, "global_sentence_id": 13, "edu_l1_label": "BOS"}, {"txt": "TrueSkill (or any ELO algorithm), originally developed by Microsoft for Xbox Live matchmaking, provides an elegant solution to these challenges.", "language": "english", "position": {"atoms": [{"position_id": 353, "txt": "TrueSkill (or any ELO algorithm), originally developed by Microsoft for Xbox Live matchmaking, provides an elegant solution to these challenges.", "x": "/html/body/main/div/div/article/div[2]/p[3]"}]}, "tags": [], "label": "content", "web_segment_id": 13, "global_sentence_id": 14, "edu_l1_label": "IOS"}, {"txt": " Trueskill allowed Microsoft to compare two players who had never played before, using the history of their games to estimate their skill level.", "language": "english", "position": {"atoms": [{"position_id": 354, "txt": " Trueskill allowed Microsoft to compare two players who had never played before, using the history of their games to estimate their skill level.", "x": "/html/body/main/div/div/article/div[2]/p[3]"}]}, "tags": [], "label": "content", "web_segment_id": 13, "global_sentence_id": 15, "edu_l1_label": "IOS"}, {"txt": "In this context, we can think of each item as a player, and the “games” between them are run by the LLM sorting them in a small batch.", "language": "english", "position": {"atoms": [{"position_id": 356, "txt": "In this context, we can think of each item as a player, and the “games” between them are run by the LLM sorting them in a small batch.", "x": "/html/body/main/div/div/article/div[2]/p[4]"}]}, "tags": [], "label": "content", "web_segment_id": 14, "global_sentence_id": 16, "edu_l1_label": "IOS"}, {"txt": "So, instead of asking a LLM to sort everything at once, we can:", "language": "english", "position": {"atoms": [{"position_id": 358, "txt": "So, instead of asking a LLM to sort everything at once, we can:", "x": "/html/body/main/div/div/article/div[2]/p[5]"}]}, "tags": [], "label": "content", "web_segment_id": 15, "global_sentence_id": 17, "edu_l1_label": "IOS"}, {"txt": "Break the data into small, manageable batches", "language": "english", "position": {"atoms": [{"position_id": 360, "txt": "Break the data into small, manageable batches", "x": "/html/body/main/div/div/article/div[2]/ul[1]/li[1]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 16, "global_sentence_id": 18, "edu_l1_label": "IOS"}, {"txt": "Have the LLM order these smaller batches (play a “game” between them)", "language": "english", "position": {"atoms": [{"position_id": 362, "txt": "Have the LLM order these smaller batches ", "x": "/html/body/main/div/div/article/div[2]/ul[1]/li[2]"}, {"position_id": 364, "txt": "(play a “game” between them)", "x": "/html/body/main/div/div/article/div[2]/ul[1]/li[2]/em"}]}, "tags": ["li", "em"], "label": "content", "web_segment_id": 17, "global_sentence_id": 19, "edu_l1_label": "IOS"}, {"txt": "Use TrueSkill to update the ratings of the items based on the results of the games", "language": "english", "position": {"atoms": [{"position_id": 366, "txt": "Use TrueSkill to update the ratings of the items based on the results of the games", "x": "/html/body/main/div/div/article/div[2]/ul[1]/li[3]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 18, "global_sentence_id": 20, "edu_l1_label": "IOS"}, {"txt": "In the end, we will have a global ranking of the items, and a confidence interval for each item’s rating, informed by the LLMs knowledge.", "language": "english", "position": {"atoms": [{"position_id": 368, "txt": "In the end, we will have a global ranking of the items, and a confidence interval for each item’s rating, informed by the LLMs knowledge.", "x": "/html/body/main/div/div/article/div[2]/p[6]"}]}, "tags": [], "label": "content", "web_segment_id": 19, "global_sentence_id": 21, "edu_l1_label": "IOS"}, {"txt": "This scales to very large datasets, and new items can be added incrementally without re-sorting everything.", "language": "english", "position": {"atoms": [{"position_id": 370, "txt": "This scales to very large datasets, and new items can be added incrementally without re-sorting everything.", "x": "/html/body/main/div/div/article/div[2]/p[7]"}]}, "tags": [], "label": "content", "web_segment_id": 20, "global_sentence_id": 22, "edu_l1_label": "IOS"}, {"txt": "Implementation", "language": "english", "position": {"atoms": [{"position_id": 372, "txt": "Implementation", "x": "/html/body/main/div/div/article/div[2]/h2[3]"}]}, "tags": ["h2"], "label": "title1", "web_segment_id": 21, "global_sentence_id": 23, "edu_l1_label": "BOS"}, {"txt": "Here’s how it works in practice:", "language": "english", "position": {"atoms": [{"position_id": 374, "txt": "Here’s how it works in practice:", "x": "/html/body/main/div/div/article/div[2]/p[8]"}]}, "tags": [], "label": "content", "web_segment_id": 22, "global_sentence_id": 24, "edu_l1_label": "IOS"}, {"txt": "Initialize a TrueSkill rating for each item in your dataset", "language": "english", "position": {"atoms": [{"position_id": 376, "txt": "Initialize a TrueSkill rating for each item in your dataset", "x": "/html/body/main/div/div/article/div[2]/ol[2]/li[1]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 23, "global_sentence_id": 25, "edu_l1_label": "IOS"}, {"txt": "Repeatedly: ", "language": "english", "position": {"atoms": [{"position_id": 378, "txt": "Repeatedly: ", "x": "/html/body/main/div/div/article/div[2]/ol[2]/li[2]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 24, "global_sentence_id": 26, "edu_l1_label": "IOS"}, {"txt": "Sample a small batch of items (e.g., 10)", "language": "english", "position": {"atoms": [{"position_id": 380, "txt": "Sample a small batch of items (e.g., 10)", "x": "/html/body/main/div/div/article/div[2]/ol[2]/li[2]/ul/li[1]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 25, "global_sentence_id": 27, "edu_l1_label": "IOS"}, {"txt": "Ask the LLM to sort them", "language": "english", "position": {"atoms": [{"position_id": 382, "txt": "Ask the LLM to sort them", "x": "/html/body/main/div/div/article/div[2]/ol[2]/li[2]/ul/li[2]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 26, "global_sentence_id": 28, "edu_l1_label": "IOS"}, {"txt": "Update TrueSkill ratings based on these results", "language": "english", "position": {"atoms": [{"position_id": 384, "txt": "Update TrueSkill ratings based on these results", "x": "/html/body/main/div/div/article/div[2]/ol[2]/li[2]/ul/li[3]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 27, "global_sentence_id": 29, "edu_l1_label": "IOS"}, {"txt": "After sufficient iterations, sort by the conservative rating estimate (μ - 3σ)", "language": "english", "position": {"atoms": [{"position_id": 386, "txt": "After sufficient iterations, sort by the conservative rating estimate (μ - 3σ)", "x": "/html/body/main/div/div/article/div[2]/ol[2]/li[3]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 28, "global_sentence_id": 30, "edu_l1_label": "IOS"}, {"txt": "Here’s a simplified version of the core loop:", "language": "english", "position": {"atoms": [{"position_id": 388, "txt": "Here’s a simplified version of the core loop:", "x": "/html/body/main/div/div/article/div[2]/p[9]"}]}, "tags": [], "label": "content", "web_segment_id": 29, "global_sentence_id": 31, "edu_l1_label": "IOS"}, {"txt": "import trueskillfrom typing import Dict, List# Initialize ratings for all itemsratings:", "language": "english", "position": {"atoms": [{"position_id": 390, "txt": "import", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[1]/span[1]"}, {"position_id": 392, "txt": " trueskill", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[1]/span[2]"}, {"position_id": 394, "txt": "from", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[2]/span[1]"}, {"position_id": 396, "txt": " typing ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[2]/span[2]"}, {"position_id": 398, "txt": "import", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[2]/span[3]"}, {"position_id": 400, "txt": " Dict, List", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[2]/span[4]"}, {"position_id": 402, "txt": "# Initialize ratings for all items", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[4]/span"}, {"position_id": 404, "txt": "ratings:", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[5]/span[1]"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 32, "edu_l1_label": "EDU_O"}, {"txt": " Dict[int, trueskill.Rating] = {    item_id:", "language": "english", "position": {"atoms": [{"position_id": 405, "txt": " Dict[", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[5]/span[1]"}, {"position_id": 407, "txt": "int", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[5]/span[2]"}, {"position_id": 409, "txt": ", trueskill.Rating] ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[5]/span[3]"}, {"position_id": 411, "txt": "=", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[5]/span[4]"}, {"position_id": 413, "txt": " {", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[5]/span[5]"}, {"position_id": 415, "txt": "    item_id:", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[6]/span[1]"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 33, "edu_l1_label": "EDU_O"}, {"txt": " trueskill.Rating() for item_id in all_items}for match in range(num_matches):", "language": "english", "position": {"atoms": [{"position_id": 416, "txt": " trueskill.Rating() ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[6]/span[1]"}, {"position_id": 418, "txt": "for", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[6]/span[2]"}, {"position_id": 420, "txt": " item_id ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[6]/span[3]"}, {"position_id": 422, "txt": "in", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[6]/span[4]"}, {"position_id": 424, "txt": " all_items", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[6]/span[5]"}, {"position_id": 426, "txt": "}", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[7]/span"}, {"position_id": 428, "txt": "for", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[9]/span[1]"}, {"position_id": 430, "txt": " match ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[9]/span[2]"}, {"position_id": 432, "txt": "in", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[9]/span[3]"}, {"position_id": 434, "txt": " range", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[9]/span[4]"}, {"position_id": 436, "txt": "(num_matches):", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[9]/span[5]"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 34, "edu_l1_label": "EDU_O"}, {"txt": "    # Sample batch of items    batch = random.sample(all_items, batch_size)    # Get sorted order from LLM    sorted_batch = llm_sort(batch)    # Update TrueSkill ratings    teams = [[ratings[item]] for item in sorted_batch]    new_ratings = trueskill.rate(teams)    # Update stored ratings    for i, item in enumerate(sorted_batch):", "language": "english", "position": {"atoms": [{"position_id": 438, "txt": "    # Sample batch of items", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[10]/span"}, {"position_id": 440, "txt": "    batch ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[11]/span[1]"}, {"position_id": 442, "txt": "=", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[11]/span[2]"}, {"position_id": 444, "txt": " random.sample(all_items, batch_size)", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[11]/span[3]"}, {"position_id": 446, "txt": "    # Get sorted order from LLM", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[13]/span"}, {"position_id": 448, "txt": "    sorted_batch ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[14]/span[1]"}, {"position_id": 450, "txt": "=", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[14]/span[2]"}, {"position_id": 452, "txt": " llm_sort(batch)", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[14]/span[3]"}, {"position_id": 454, "txt": "    # Update TrueSkill ratings", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[16]/span"}, {"position_id": 456, "txt": "    teams ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[17]/span[1]"}, {"position_id": 458, "txt": "=", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[17]/span[2]"}, {"position_id": 460, "txt": " [[ratings[item]] ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[17]/span[3]"}, {"position_id": 462, "txt": "for", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[17]/span[4]"}, {"position_id": 464, "txt": " item ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[17]/span[5]"}, {"position_id": 466, "txt": "in", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[17]/span[6]"}, {"position_id": 468, "txt": " sorted_batch]", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[17]/span[7]"}, {"position_id": 470, "txt": "    new_ratings ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[18]/span[1]"}, {"position_id": 472, "txt": "=", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[18]/span[2]"}, {"position_id": 474, "txt": " trueskill.rate(teams)", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[18]/span[3]"}, {"position_id": 476, "txt": "    # Update stored ratings", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[20]/span"}, {"position_id": 478, "txt": "    for", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[21]/span[1]"}, {"position_id": 480, "txt": " i, item ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[21]/span[2]"}, {"position_id": 482, "txt": "in", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[21]/span[3]"}, {"position_id": 484, "txt": " enumerate", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[21]/span[4]"}, {"position_id": 486, "txt": "(sorted_batch):", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[21]/span[5]"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 35, "edu_l1_label": "EDU_O"}, {"txt": "        ratings[item] = new_ratings[i][0]", "language": "english", "position": {"atoms": [{"position_id": 488, "txt": "        ratings[item] ", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[22]/span[1]"}, {"position_id": 490, "txt": "=", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[22]/span[2]"}, {"position_id": 492, "txt": " new_ratings[i][", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[22]/span[3]"}, {"position_id": 494, "txt": "0", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[22]/span[4]"}, {"position_id": 496, "txt": "]", "x": "/html/body/main/div/div/article/div[2]/pre[1]/code/span[22]/span[5]"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 36, "edu_l1_label": "EDU_O"}, {"txt": "LLM Sorting Prompt", "language": "english", "position": {"atoms": [{"position_id": 498, "txt": "LLM Sorting Prompt", "x": "/html/body/main/div/div/article/div[2]/h3[1]"}]}, "tags": ["h3"], "label": "title2", "web_segment_id": 31, "global_sentence_id": 37, "edu_l1_label": "IOS"}, {"txt": "The key to making this work well is crafting good prompts for the LLM sorting step.", "language": "english", "position": {"atoms": [{"position_id": 500, "txt": "The key to making this work well is crafting good prompts for the LLM sorting step.", "x": "/html/body/main/div/div/article/div[2]/p[10]"}]}, "tags": [], "label": "content", "web_segment_id": 32, "global_sentence_id": 38, "edu_l1_label": "IOS"}, {"txt": " Here’s an example prompt template, but you should customize it to your use case.", "language": "english", "position": {"atoms": [{"position_id": 501, "txt": " Here’s an example prompt template, but you should customize it to your use case.", "x": "/html/body/main/div/div/article/div[2]/p[10]"}]}, "tags": [], "label": "content", "web_segment_id": 32, "global_sentence_id": 39, "edu_l1_label": "IOS"}, {"txt": "prompt = f\"\"\"You are an expert at analyzing {domain}.Please sort these items by {criteria}.Consider:", "language": "english", "position": {"atoms": [{"position_id": 503, "txt": "prompt ", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[1]/span[1]"}, {"position_id": 505, "txt": "=", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[1]/span[2]"}, {"position_id": 507, "txt": " f", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[1]/span[3]"}, {"position_id": 509, "txt": "\"\"\"You are an expert at analyzing ", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[1]/span[4]"}, {"position_id": 511, "txt": "{", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[1]/span[5]"}, {"position_id": 513, "txt": "domain", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[1]/span[6]"}, {"position_id": 515, "txt": "}", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[1]/span[7]"}, {"position_id": 517, "txt": ".", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[1]/span[8]"}, {"position_id": 519, "txt": "Please sort these items by ", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[2]/span[1]"}, {"position_id": 521, "txt": "{", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[2]/span[2]"}, {"position_id": 523, "txt": "criteria", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[2]/span[3]"}, {"position_id": 525, "txt": "}", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[2]/span[4]"}, {"position_id": 527, "txt": ".", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[2]/span[5]"}, {"position_id": 529, "txt": "Consider:", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[3]/span"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 40, "edu_l1_label": "EDU_O"}, {"txt": "- {consideration_1}- {consideration_2}- {consideration_3}Items to sort:", "language": "english", "position": {"atoms": [{"position_id": 531, "txt": "- ", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[4]/span[1]"}, {"position_id": 533, "txt": "{", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[4]/span[2]"}, {"position_id": 535, "txt": "consideration_1", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[4]/span[3]"}, {"position_id": 537, "txt": "}", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[4]/span[4]"}, {"position_id": 539, "txt": "- ", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[5]/span[1]"}, {"position_id": 541, "txt": "{", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[5]/span[2]"}, {"position_id": 543, "txt": "consideration_2", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[5]/span[3]"}, {"position_id": 545, "txt": "}", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[5]/span[4]"}, {"position_id": 547, "txt": "- ", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[6]/span[1]"}, {"position_id": 549, "txt": "{", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[6]/span[2]"}, {"position_id": 551, "txt": "consideration_3", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[6]/span[3]"}, {"position_id": 553, "txt": "}", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[6]/span[4]"}, {"position_id": 555, "txt": "Items to sort:", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[8]/span"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 41, "edu_l1_label": "EDU_O"}, {"txt": "{\"\\n\".join(f'{item[\"id\"]}. \"", "language": "english", "position": {"atoms": [{"position_id": 557, "txt": "{", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[1]"}, {"position_id": 559, "txt": "\"", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[2]"}, {"position_id": 561, "txt": "\\n", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[3]"}, {"position_id": 563, "txt": "\"", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[4]"}, {"position_id": 565, "txt": ".join(", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[5]"}, {"position_id": 567, "txt": "f", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[6]"}, {"position_id": 569, "txt": "'", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[7]"}, {"position_id": 571, "txt": "{", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[8]"}, {"position_id": 573, "txt": "item[", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[9]"}, {"position_id": 575, "txt": "\"id\"", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[10]"}, {"position_id": 577, "txt": "]", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[11]"}, {"position_id": 579, "txt": "}", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[12]"}, {"position_id": 581, "txt": ". \"", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[13]"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 42, "edu_l1_label": "EDU_O"}, {"txt": "{item[\"text\"]}\"' for item in items)}Return ONLY a comma-separated list of ids sorted in order,with no other text or explanation.", "language": "english", "position": {"atoms": [{"position_id": 583, "txt": "{", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[14]"}, {"position_id": 585, "txt": "item[", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[15]"}, {"position_id": 587, "txt": "\"text\"", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[16]"}, {"position_id": 589, "txt": "]", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[17]"}, {"position_id": 591, "txt": "}", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[18]"}, {"position_id": 593, "txt": "\"'", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[19]"}, {"position_id": 595, "txt": " for", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[20]"}, {"position_id": 597, "txt": " item ", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[21]"}, {"position_id": 599, "txt": "in", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[22]"}, {"position_id": 601, "txt": " items)", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[23]"}, {"position_id": 603, "txt": "}", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[9]/span[24]"}, {"position_id": 605, "txt": "Return ONLY a comma-separated list of ids sorted in order,", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[11]/span"}, {"position_id": 607, "txt": "with no other text or explanation.", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[12]/span"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 43, "edu_l1_label": "EDU_O"}, {"txt": " For example: \"", "language": "english", "position": {"atoms": [{"position_id": 608, "txt": " For example: \"", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[12]/span"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 44, "edu_l1_label": "EDU_O"}, {"txt": "123,456,789\"\"\"\"", "language": "english", "position": {"atoms": [{"position_id": 609, "txt": "123,456,789\"", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[12]/span"}, {"position_id": 611, "txt": "\"\"\"", "x": "/html/body/main/div/div/article/div[2]/pre[2]/code/span[13]/span"}]}, "tags": [], "label": "code", "web_segment_id": 30, "global_sentence_id": 45, "edu_l1_label": "EDU_O"}, {"txt": "By requesting a specific output format and keeping the task focused, we get more reliable results from the LLM.", "language": "english", "position": {"atoms": [{"position_id": 613, "txt": "By requesting a specific output format and keeping the task focused, we get more reliable results from the LLM.", "x": "/html/body/main/div/div/article/div[2]/p[11]"}]}, "tags": [], "label": "content", "web_segment_id": 33, "global_sentence_id": 46, "edu_l1_label": "IOS"}, {"txt": "An Example", "language": "english", "position": {"atoms": [{"position_id": 615, "txt": "An Example", "x": "/html/body/main/div/div/article/div[2]/h2[4]"}]}, "tags": ["h2"], "label": "title1", "web_segment_id": 34, "global_sentence_id": 47, "edu_l1_label": "BOS"}, {"txt": "Let’s walk through a concrete example.", "language": "english", "position": {"atoms": [{"position_id": 617, "txt": "Let’s walk through a concrete example.", "x": "/html/body/main/div/div/article/div[2]/p[12]"}]}, "tags": [], "label": "content", "web_segment_id": 35, "global_sentence_id": 48, "edu_l1_label": "IOS"}, {"txt": "Let’s say you have over 10,000 pieces of customer feedback you’ve collected over the past year.", "language": "english", "position": {"atoms": [{"position_id": 619, "txt": "Let’s say you have over 10,000 pieces of customer feedback you’ve collected over the past year.", "x": "/html/body/main/div/div/article/div[2]/p[13]"}]}, "tags": [], "label": "content", "web_segment_id": 36, "global_sentence_id": 49, "edu_l1_label": "IOS"}, {"txt": " And you want to organize it by “actionability”.", "language": "english", "position": {"atoms": [{"position_id": 620, "txt": " And you want to organize it by “actionability”.", "x": "/html/body/main/div/div/article/div[2]/p[13]"}]}, "tags": [], "label": "content", "web_segment_id": 36, "global_sentence_id": 50, "edu_l1_label": "IOS"}, {"txt": "You can write a prompt for actionability that you like, getting the LLM to combine factors like technical feasibility, potential impact, and clarity of the request.", "language": "english", "position": {"atoms": [{"position_id": 622, "txt": "You can write a prompt for actionability that you like, getting the LLM to combine factors like technical feasibility, potential impact, and clarity of the request.", "x": "/html/body/main/div/div/article/div[2]/p[14]"}]}, "tags": [], "label": "content", "web_segment_id": 37, "global_sentence_id": 51, "edu_l1_label": "IOS"}, {"txt": " Maybe even your codebase or product roadmap.", "language": "english", "position": {"atoms": [{"position_id": 623, "txt": " Maybe even your codebase or product roadmap.", "x": "/html/body/main/div/div/article/div[2]/p[14]"}]}, "tags": [], "label": "content", "web_segment_id": 37, "global_sentence_id": 52, "edu_l1_label": "IOS"}, {"txt": "The feedback might look like this:", "language": "english", "position": {"atoms": [{"position_id": 625, "txt": "The feedback might look like this:", "x": "/html/body/main/div/div/article/div[2]/p[15]"}]}, "tags": [], "label": "content", "web_segment_id": 38, "global_sentence_id": 53, "edu_l1_label": "IOS"}, {"txt": "“The blue button is hard to see\"", "language": "english", "position": {"atoms": [{"position_id": 627, "txt": "“The blue button is hard to see\"", "x": "/html/body/main/div/div/article/div[2]/ul[2]/li[1]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 39, "global_sentence_id": 54, "edu_l1_label": "IOS"}, {"txt": "\"Would be nice if it worked more like Excel\"", "language": "english", "position": {"atoms": [{"position_id": 629, "txt": "\"Would be nice if it worked more like Excel\"", "x": "/html/body/main/div/div/article/div[2]/ul[2]/li[2]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 40, "global_sentence_id": 55, "edu_l1_label": "IOS"}, {"txt": "\"Sometimes crashes when I have too many tabs open\"", "language": "english", "position": {"atoms": [{"position_id": 631, "txt": "\"Sometimes crashes when I have too many tabs open\"", "x": "/html/body/main/div/div/article/div[2]/ul[2]/li[3]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 41, "global_sentence_id": 56, "edu_l1_label": "IOS"}, {"txt": "\"Love the product but wish it was faster\"", "language": "english", "position": {"atoms": [{"position_id": 633, "txt": "\"Love the product but wish it was faster\"", "x": "/html/body/main/div/div/article/div[2]/ul[2]/li[4]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 42, "global_sentence_id": 57, "edu_l1_label": "IOS"}, {"txt": "\"The API needs better documentation”", "language": "english", "position": {"atoms": [{"position_id": 635, "txt": "\"The API needs better documentation”", "x": "/html/body/main/div/div/article/div[2]/ul[2]/li[5]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 43, "global_sentence_id": 58, "edu_l1_label": "IOS"}, {"txt": "Traditional sorting methods struggle here because:", "language": "english", "position": {"atoms": [{"position_id": 637, "txt": "Traditional sorting methods struggle here because:", "x": "/html/body/main/div/div/article/div[2]/p[16]"}]}, "tags": [], "label": "content", "web_segment_id": 44, "global_sentence_id": 59, "edu_l1_label": "IOS"}, {"txt": "Simple keyword matchin or embedding misses the meaning", "language": "english", "position": {"atoms": [{"position_id": 639, "txt": "Simple keyword matchin or embedding misses the meaning", "x": "/html/body/main/div/div/article/div[2]/ul[3]/li[1]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 45, "global_sentence_id": 60, "edu_l1_label": "IOS"}, {"txt": "Manual sorting of 10,000 items is overwhelming", "language": "english", "position": {"atoms": [{"position_id": 641, "txt": "Manual sorting of 10,000 items is overwhelming", "x": "/html/body/main/div/div/article/div[2]/ul[3]/li[2]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 46, "global_sentence_id": 61, "edu_l1_label": "IOS"}, {"txt": "Each piece of feedback might be actionable for different reasons", "language": "english", "position": {"atoms": [{"position_id": 643, "txt": "Each piece of feedback might be actionable for different reasons", "x": "/html/body/main/div/div/article/div[2]/ul[3]/li[3]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 47, "global_sentence_id": 62, "edu_l1_label": "IOS"}, {"txt": "LLMs can’t handle 10,000 items at once", "language": "english", "position": {"atoms": [{"position_id": 645, "txt": "LLMs can’t handle 10,000 items at once", "x": "/html/body/main/div/div/article/div[2]/ul[3]/li[4]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 48, "global_sentence_id": 63, "edu_l1_label": "IOS"}, {"txt": "The system will then break these into smaller batches, and ask the LLM to sort them, perhaps like so:", "language": "english", "position": {"atoms": [{"position_id": 647, "txt": "The system will then break these into smaller batches, and ask the LLM to sort them, perhaps like so:", "x": "/html/body/main/div/div/article/div[2]/p[17]"}]}, "tags": [], "label": "content", "web_segment_id": 49, "global_sentence_id": 64, "edu_l1_label": "IOS"}, {"txt": "1. \"Save button sometimes doesn't work\"2. \"Would be cool to have dark mode\"3. \"The UX feels clunky\"4. \"Need better keyboard shortcuts\"5. \"Can't find where to change my password\"", "language": "english", "position": {"atoms": [{"position_id": 649, "txt": "1. \"Save button sometimes doesn't work\"", "x": "/html/body/main/div/div/article/div[2]/pre[3]/code/span[1]/span"}, {"position_id": 651, "txt": "2. \"Would be cool to have dark mode\"", "x": "/html/body/main/div/div/article/div[2]/pre[3]/code/span[2]/span"}, {"position_id": 653, "txt": "3. \"The UX feels clunky\"", "x": "/html/body/main/div/div/article/div[2]/pre[3]/code/span[3]/span"}, {"position_id": 655, "txt": "4. \"Need better keyboard shortcuts\"", "x": "/html/body/main/div/div/article/div[2]/pre[3]/code/span[4]/span"}, {"position_id": 657, "txt": "5. \"Can't find where to change my password\"", "x": "/html/body/main/div/div/article/div[2]/pre[3]/code/span[5]/span"}]}, "tags": [], "label": "content", "web_segment_id": 30, "global_sentence_id": 65, "edu_l1_label": "IOS"}, {"txt": "When are you “done”?", "language": "english", "position": {"atoms": [{"position_id": 659, "txt": "When are you “done”?", "x": "/html/body/main/div/div/article/div[2]/h2[5]"}]}, "tags": ["h2"], "label": "title1", "web_segment_id": 50, "global_sentence_id": 66, "edu_l1_label": "BOS"}, {"txt": "The number of matches needed for to converge on a result scales with dataset size and distribution of results.", "language": "english", "position": {"atoms": [{"position_id": 661, "txt": "The number of matches needed for to converge on a result scales with dataset size and distribution of results.", "x": "/html/body/main/div/div/article/div[2]/p[18]"}]}, "tags": [], "label": "content", "web_segment_id": 51, "global_sentence_id": 67, "edu_l1_label": "IOS"}, {"txt": " Luckily TrueSkill is very robust to the number of matches played, and LLMs are quite stable in their predictions.", "language": "english", "position": {"atoms": [{"position_id": 662, "txt": " Luckily TrueSkill is very robust to the number of matches played, and LLMs are quite stable in their predictions.", "x": "/html/body/main/div/div/article/div[2]/p[18]"}]}, "tags": [], "label": "content", "web_segment_id": 51, "global_sentence_id": 68, "edu_l1_label": "IOS"}, {"txt": "I’ve found that even 1-2 matches per item can be enough to get a good enough result for most use cases.", "language": "english", "position": {"atoms": [{"position_id": 664, "txt": "I’ve found that even 1-2 matches per item can be enough to get a good enough result for most use cases.", "x": "/html/body/main/div/div/article/div[2]/p[19]"}]}, "tags": [], "label": "content", "web_segment_id": 52, "global_sentence_id": 69, "edu_l1_label": "IOS"}, {"txt": " For a list of 10,000 items and a 10 item batch size, this could mean as little as 500 matches.", "language": "english", "position": {"atoms": [{"position_id": 665, "txt": " For a list of 10,000 items and a 10 item batch size, this could mean as little as 500 matches.", "x": "/html/body/main/div/div/article/div[2]/p[19]"}]}, "tags": [], "label": "content", "web_segment_id": 52, "global_sentence_id": 70, "edu_l1_label": "IOS"}, {"txt": "If you want a more quantitative measure of how good your results are, TrueSkill gives each item a rating, which is a combination of its skill level and its uncertainty.", "language": "english", "position": {"atoms": [{"position_id": 667, "txt": "If you want a more quantitative measure of how good your results are, TrueSkill gives each item a rating, which is a combination of its skill level and its uncertainty.", "x": "/html/body/main/div/div/article/div[2]/p[20]"}]}, "tags": [], "label": "content", "web_segment_id": 53, "global_sentence_id": 71, "edu_l1_label": "IOS"}, {"txt": "μ (mu):", "language": "english", "position": {"atoms": [{"position_id": 669, "txt": "μ (mu):", "x": "/html/body/main/div/div/article/div[2]/ul[4]/li[1]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 54, "global_sentence_id": 72, "edu_l1_label": "IOS"}, {"txt": " The estimated skill rating", "language": "english", "position": {"atoms": [{"position_id": 670, "txt": " The estimated skill rating", "x": "/html/body/main/div/div/article/div[2]/ul[4]/li[1]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 54, "global_sentence_id": 73, "edu_l1_label": "IOS"}, {"txt": "σ (sigma):", "language": "english", "position": {"atoms": [{"position_id": 672, "txt": "σ (sigma):", "x": "/html/body/main/div/div/article/div[2]/ul[4]/li[2]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 55, "global_sentence_id": 74, "edu_l1_label": "IOS"}, {"txt": " The uncertainty in that rating", "language": "english", "position": {"atoms": [{"position_id": 673, "txt": " The uncertainty in that rating", "x": "/html/body/main/div/div/article/div[2]/ul[4]/li[2]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 55, "global_sentence_id": 75, "edu_l1_label": "IOS"}, {"txt": "Sigma is usually initialized to 8.333, which is a good starting point, you can measure the confidence of your results by looking at how much the sigma values change.", "language": "english", "position": {"atoms": [{"position_id": 675, "txt": "Sigma is usually initialized to 8.333, which is a good starting point, you can measure the confidence of your results by looking at how much the sigma values change.", "x": "/html/body/main/div/div/article/div[2]/p[21]"}]}, "tags": [], "label": "content", "web_segment_id": 56, "global_sentence_id": 76, "edu_l1_label": "IOS"}, {"txt": "And of course course, remember the golden rule of Machine Learning look at your data, to make sure the results are what you expect.", "language": "english", "position": {"atoms": [{"position_id": 677, "txt": "And of course course, remember the golden rule of Machine Learning ", "x": "/html/body/main/div/div/article/div[2]/p[22]"}, {"position_id": 679, "txt": "look at your data", "x": "/html/body/main/div/div/article/div[2]/p[22]/strong"}, {"position_id": 681, "txt": ", to make sure the results are what you expect.", "x": "/html/body/main/div/div/article/div[2]/p[22]/strong"}]}, "tags": ["strong"], "label": "content", "web_segment_id": 57, "global_sentence_id": 77, "edu_l1_label": "IOS"}, {"txt": "Alternatives", "language": "english", "position": {"atoms": [{"position_id": 683, "txt": "Alternatives", "x": "/html/body/main/div/div/article/div[2]/h2[6]"}]}, "tags": ["h2"], "label": "title1", "web_segment_id": 58, "global_sentence_id": 78, "edu_l1_label": "BOS"}, {"txt": "Individual Scoring", "language": "english", "position": {"atoms": [{"position_id": 685, "txt": "Individual Scoring", "x": "/html/body/main/div/div/article/div[2]/h3[2]"}]}, "tags": ["h3"], "label": "title2", "web_segment_id": 59, "global_sentence_id": 79, "edu_l1_label": "IOS"}, {"txt": "A common alternative to sorting is to score each item individually, and then sort by score.", "language": "english", "position": {"atoms": [{"position_id": 687, "txt": "A common alternative to sorting is to score each item individually, and then sort by score.", "x": "/html/body/main/div/div/article/div[2]/p[23]"}]}, "tags": [], "label": "content", "web_segment_id": 60, "global_sentence_id": 80, "edu_l1_label": "IOS"}, {"txt": "This is a good approach if you have a clear scoring function, but models struggle to score things consistently.", "language": "english", "position": {"atoms": [{"position_id": 689, "txt": "This is a good approach if you have a clear scoring function, but models struggle to score things consistently.", "x": "/html/body/main/div/div/article/div[2]/p[24]"}]}, "tags": [], "label": "content", "web_segment_id": 61, "global_sentence_id": 81, "edu_l1_label": "IOS"}, {"txt": "If you score on a range of 1-10, you can expect many ties and clumps around common scores like 7. If you score on a range of 1-100, you should expect more randomness in the scores.", "language": "english", "position": {"atoms": [{"position_id": 691, "txt": "If you score on a range of 1-10, you can expect many ties and clumps around common scores like 7. If you score on a range of 1-100, you should expect more randomness in the scores.", "x": "/html/body/main/div/div/article/div[2]/p[25]"}]}, "tags": [], "label": "content", "web_segment_id": 62, "global_sentence_id": 82, "edu_l1_label": "IOS"}, {"txt": "But more importantly, it’s easier for LLMs (and humans!) to make relative judgments (“A is more interesting than B”) than absolute ones (“A is 73/100 interesting”)", "language": "english", "position": {"atoms": [{"position_id": 693, "txt": "But more importantly, it’s easier for LLMs (and humans!) to make relative judgments (“A is more interesting than B”) than absolute ones (“A is 73/100 interesting”)", "x": "/html/body/main/div/div/article/div[2]/p[26]"}]}, "tags": [], "label": "content", "web_segment_id": 63, "global_sentence_id": 83, "edu_l1_label": "IOS"}, {"txt": "Embedding-based Sorting", "language": "english", "position": {"atoms": [{"position_id": 695, "txt": "Embedding-based Sorting", "x": "/html/body/main/div/div/article/div[2]/h3[3]"}]}, "tags": ["h3"], "label": "title2", "web_segment_id": 64, "global_sentence_id": 84, "edu_l1_label": "IOS"}, {"txt": "Given the cheapness of embeddings, embedding-based sorting is often deployed as well.", "language": "english", "position": {"atoms": [{"position_id": 697, "txt": "Given the cheapness of embeddings, embedding-based sorting is often deployed as well.", "x": "/html/body/main/div/div/article/div[2]/p[27]"}]}, "tags": [], "label": "content", "web_segment_id": 65, "global_sentence_id": 85, "edu_l1_label": "IOS"}, {"txt": "The basic process is:", "language": "english", "position": {"atoms": [{"position_id": 699, "txt": "The basic process is:", "x": "/html/body/main/div/div/article/div[2]/p[28]"}]}, "tags": [], "label": "content", "web_segment_id": 66, "global_sentence_id": 86, "edu_l1_label": "IOS"}, {"txt": "Get embeddings for each item", "language": "english", "position": {"atoms": [{"position_id": 701, "txt": "Get embeddings for each item", "x": "/html/body/main/div/div/article/div[2]/ol[3]/li[1]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 67, "global_sentence_id": 87, "edu_l1_label": "IOS"}, {"txt": "Choose a reference point (e.g., embedding of “very interesting”)", "language": "english", "position": {"atoms": [{"position_id": 703, "txt": "Choose a reference point (e.g., embedding of “very interesting”)", "x": "/html/body/main/div/div/article/div[2]/ol[3]/li[2]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 68, "global_sentence_id": 88, "edu_l1_label": "IOS"}, {"txt": "Sort items by cosine similarity to this reference point", "language": "english", "position": {"atoms": [{"position_id": 705, "txt": "Sort items by cosine similarity to this reference point", "x": "/html/body/main/div/div/article/div[2]/ol[3]/li[3]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 69, "global_sentence_id": 89, "edu_l1_label": "IOS"}, {"txt": "While cheap, embedding-based sorting loses a lot of the nuance of the sorting criteria and sometimes be affected by the semantic space of the embedding model (e.g. the color blue might be closer to the concept of usability than the color red)", "language": "english", "position": {"atoms": [{"position_id": 707, "txt": "While cheap, embedding-based sorting loses a lot of the nuance of the sorting criteria and sometimes be affected by the semantic space of the embedding model (e.g. the color blue might be closer to the concept of usability than the color red)", "x": "/html/body/main/div/div/article/div[2]/p[29]"}]}, "tags": [], "label": "content", "web_segment_id": 70, "global_sentence_id": 90, "edu_l1_label": "IOS"}, {"txt": "That said, embeddings can be useful in conjunction with TrueSkill:", "language": "english", "position": {"atoms": [{"position_id": 709, "txt": "That said, embeddings can be useful in conjunction with TrueSkill:", "x": "/html/body/main/div/div/article/div[2]/p[30]"}]}, "tags": [], "label": "content", "web_segment_id": 71, "global_sentence_id": 91, "edu_l1_label": "IOS"}, {"txt": "Use embeddings for initial rough sorting", "language": "english", "position": {"atoms": [{"position_id": 711, "txt": "Use embeddings for initial rough sorting", "x": "/html/body/main/div/div/article/div[2]/ol[4]/li[1]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 72, "global_sentence_id": 92, "edu_l1_label": "IOS"}, {"txt": "Use embedding similarity to choose batches for comparison", "language": "english", "position": {"atoms": [{"position_id": 713, "txt": "Use embedding similarity to choose batches for comparison", "x": "/html/body/main/div/div/article/div[2]/ol[4]/li[2]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 73, "global_sentence_id": 93, "edu_l1_label": "IOS"}, {"txt": "Combine embedding and TrueSkill scores for hybrid ranking", "language": "english", "position": {"atoms": [{"position_id": 715, "txt": "Combine embedding and TrueSkill scores for hybrid ranking", "x": "/html/body/main/div/div/article/div[2]/ol[4]/li[3]"}]}, "tags": ["li"], "label": "content", "web_segment_id": 74, "global_sentence_id": 94, "edu_l1_label": "IOS"}, {"txt": "Optimization", "language": "english", "position": {"atoms": [{"position_id": 717, "txt": "Optimization", "x": "/html/body/main/div/div/article/div[2]/h2[7]"}]}, "tags": ["h2"], "label": "title1", "web_segment_id": 75, "global_sentence_id": 95, "edu_l1_label": "BOS"}, {"txt": "Smart Batch Selection", "language": "english", "position": {"atoms": [{"position_id": 719, "txt": "Smart Batch Selection", "x": "/html/body/main/div/div/article/div[2]/h3[4]"}]}, "tags": ["h3"], "label": "title2", "web_segment_id": 76, "global_sentence_id": 96, "edu_l1_label": "IOS"}, {"txt": "Rather than selecting batches randomly, we can be strategic about which items to compare.", "language": "english", "position": {"atoms": [{"position_id": 721, "txt": "Rather than selecting batches randomly, we can be strategic about which items to compare.", "x": "/html/body/main/div/div/article/div[2]/p[31]"}]}, "tags": [], "label": "content", "web_segment_id": 77, "global_sentence_id": 97, "edu_l1_label": "IOS"}, {"txt": " Some effective strategies include:", "language": "english", "position": {"atoms": [{"position_id": 722, "txt": " Some effective strategies include:", "x": "/html/body/main/div/div/article/div[2]/p[31]"}]}, "tags": [], "label": "content", "web_segment_id": 77, "global_sentence_id": 98, "edu_l1_label": "IOS"}, {"txt": "Rating-based grouping:", "language": "english", "position": {"atoms": [{"position_id": 724, "txt": "Rating-based grouping", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[1]/strong"}, {"position_id": 726, "txt": ":", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[1]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 78, "global_sentence_id": 99, "edu_l1_label": "IOS"}, {"txt": " Compare items with similar ratings to refine close distinctions", "language": "english", "position": {"atoms": [{"position_id": 727, "txt": " Compare items with similar ratings to refine close distinctions", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[1]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 78, "global_sentence_id": 100, "edu_l1_label": "IOS"}, {"txt": "High uncertainty pairs:", "language": "english", "position": {"atoms": [{"position_id": 729, "txt": "High uncertainty pairs", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[2]/strong"}, {"position_id": 731, "txt": ":", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[2]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 79, "global_sentence_id": 101, "edu_l1_label": "IOS"}, {"txt": " Prioritize comparisons between items with high sigma values", "language": "english", "position": {"atoms": [{"position_id": 732, "txt": " Prioritize comparisons between items with high sigma values", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[2]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 79, "global_sentence_id": 102, "edu_l1_label": "IOS"}, {"txt": "Bridge comparisons:", "language": "english", "position": {"atoms": [{"position_id": 734, "txt": "Bridge comparisons", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[3]/strong"}, {"position_id": 736, "txt": ":", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[3]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 80, "global_sentence_id": 103, "edu_l1_label": "IOS"}, {"txt": " Occasionally compare items from different rating tiers to verify tier boundaries", "language": "english", "position": {"atoms": [{"position_id": 737, "txt": " Occasionally compare items from different rating tiers to verify tier boundaries", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[3]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 80, "global_sentence_id": 104, "edu_l1_label": "IOS"}, {"txt": "Novel pair maximization:", "language": "english", "position": {"atoms": [{"position_id": 739, "txt": "Novel pair maximization", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[4]/strong"}, {"position_id": 741, "txt": ":", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[4]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 81, "global_sentence_id": 105, "edu_l1_label": "IOS"}, {"txt": " Prioritize pairs that haven’t been compared before", "language": "english", "position": {"atoms": [{"position_id": 742, "txt": " Prioritize pairs that haven’t been compared before", "x": "/html/body/main/div/div/article/div[2]/ol[5]/li[4]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 81, "global_sentence_id": 106, "edu_l1_label": "IOS"}, {"txt": "Using Confidence Intervals", "language": "english", "position": {"atoms": [{"position_id": 744, "txt": "Using Confidence Intervals", "x": "/html/body/main/div/div/article/div[2]/h3[5]"}]}, "tags": ["h3"], "label": "title2", "web_segment_id": 82, "global_sentence_id": 107, "edu_l1_label": "IOS"}, {"txt": "The sigma value provides a built-in confidence measure.", "language": "english", "position": {"atoms": [{"position_id": 746, "txt": "The sigma value provides a built-in confidence measure.", "x": "/html/body/main/div/div/article/div[2]/p[32]"}]}, "tags": [], "label": "content", "web_segment_id": 83, "global_sentence_id": 108, "edu_l1_label": "IOS"}, {"txt": " We can use this to:", "language": "english", "position": {"atoms": [{"position_id": 747, "txt": " We can use this to:", "x": "/html/body/main/div/div/article/div[2]/p[32]"}]}, "tags": [], "label": "content", "web_segment_id": 83, "global_sentence_id": 109, "edu_l1_label": "IOS"}, {"txt": "Guide batch selection:", "language": "english", "position": {"atoms": [{"position_id": 749, "txt": "Guide batch selection", "x": "/html/body/main/div/div/article/div[2]/ol[6]/li[1]/strong"}, {"position_id": 751, "txt": ":", "x": "/html/body/main/div/div/article/div[2]/ol[6]/li[1]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 84, "global_sentence_id": 110, "edu_l1_label": "IOS"}, {"txt": " Prioritize items with high sigma values", "language": "english", "position": {"atoms": [{"position_id": 752, "txt": " Prioritize items with high sigma values", "x": "/html/body/main/div/div/article/div[2]/ol[6]/li[1]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 84, "global_sentence_id": 111, "edu_l1_label": "IOS"}, {"txt": "Detect convergence:", "language": "english", "position": {"atoms": [{"position_id": 754, "txt": "Detect convergence", "x": "/html/body/main/div/div/article/div[2]/ol[6]/li[2]/strong"}, {"position_id": 756, "txt": ":", "x": "/html/body/main/div/div/article/div[2]/ol[6]/li[2]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 85, "global_sentence_id": 112, "edu_l1_label": "IOS"}, {"txt": " Monitor when sigma values stabilize", "language": "english", "position": {"atoms": [{"position_id": 757, "txt": " Monitor when sigma values stabilize", "x": "/html/body/main/div/div/article/div[2]/ol[6]/li[2]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 85, "global_sentence_id": 113, "edu_l1_label": "IOS"}, {"txt": "Balance exploration/exploitation:", "language": "english", "position": {"atoms": [{"position_id": 759, "txt": "Balance exploration/exploitation", "x": "/html/body/main/div/div/article/div[2]/ol[6]/li[3]/strong"}, {"position_id": 761, "txt": ":", "x": "/html/body/main/div/div/article/div[2]/ol[6]/li[3]/strong"}]}, "tags": ["li", "strong"], "label": "content", "web_segment_id": 86, "global_sentence_id": 114, "edu_l1_label": "IOS"}, {"txt": " Mix high and low sigma items in batches", "language": "english", "position": {"atoms": [{"position_id": 762, "txt": " Mix high and low sigma items in batches", "x": "/html/body/main/div/div/article/div[2]/ol[6]/li[3]/strong"}]}, "tags": ["strong", "li"], "label": "content", "web_segment_id": 86, "global_sentence_id": 115, "edu_l1_label": "IOS"}, {"txt": "Conclusion", "language": "english", "position": {"atoms": [{"position_id": 764, "txt": "Conclusion", "x": "/html/body/main/div/div/article/div[2]/h2[8]"}]}, "tags": ["h2"], "label": "title1", "web_segment_id": 87, "global_sentence_id": 116, "edu_l1_label": "BOS"}, {"txt": "I’ve run into this problem a few times now, and recommended this solution to a few friends, so I hope it’s useful for you as well.", "language": "english", "position": {"atoms": [{"position_id": 766, "txt": "I’ve run into this problem a few times now, and recommended this solution to a few friends, so I hope it’s useful for you as well.", "x": "/html/body/main/div/div/article/div[2]/p[33]"}]}, "tags": [], "label": "content", "web_segment_id": 88, "global_sentence_id": 117, "edu_l1_label": "IOS"}, {"txt": " Let me know if it is!", "language": "english", "position": {"atoms": [{"position_id": 767, "txt": " Let me know if it is!", "x": "/html/body/main/div/div/article/div[2]/p[33]"}]}, "tags": [], "label": "content", "web_segment_id": 88, "global_sentence_id": 118, "edu_l1_label": "IOS"}, {"txt": "If you’re running into this problem at a more complicated scale, send me an email (trq212 AT gmail DOT com) and I’ll see if I can help.", "language": "english", "position": {"atoms": [{"position_id": 769, "txt": "If you’re running into this problem at a more complicated scale, send me an email (trq212 AT gmail DOT com) and I’ll see if I can help.", "x": "/html/body/main/div/div/article/div[2]/p[34]"}]}, "tags": [], "label": "content", "web_segment_id": 89, "global_sentence_id": 119, "edu_l1_label": "IOS"}], "type": "WEB"}
# LLM-Powered Sorting with TrueSkill
## The Challenge with LLM Sorting
## Enter TrueSkill
## Implementation
### LLM Sorting Prompt
## An Example
## When are you “done”?
## Alternatives
### Individual Scoring
### Embedding-based Sorting
## Optimization
### Smart Batch Selection
### Using Confidence Intervals
## Conclusion