README.md

---
datasets:
- multimolecule/rfam
library_name: multimolecule
license: agpl-3.0
mask_token: <mask>
pipeline_tag: fill-mask
tags:
- Biology
- RNA
- ncRNA
- rna
widget:
- example_title: microRNA 21
  mask_index: 11
  mask_index_1based: 12
  masked_char: A
  output:
  - label: U
    score: 0.314087
  - label: K
    score: 0.260464
  - label: G
    score: 0.215995
  - label: '-'
    score: 0.041525
  - label: D
    score: 0.034758
  pipeline_tag: fill-mask
  sequence_type: ncRNA
  task: fill-mask
  text: UAGCUUAUCAG<mask>CUGAUGUUGA
- example_title: microRNA 146a
  mask_index: 10
  mask_index_1based: 11
  masked_char: A
  output:
  - label: '-'
    score: 0.327472
  - label: U
    score: 0.31148
  - label: K
    score: 0.095658
  - label: W
    score: 0.035357
  - label: D
    score: 0.03324
  pipeline_tag: fill-mask
  sequence_type: ncRNA
  task: fill-mask
  text: UGAGAACUGA<mask>UUCCAUGGGUU
- example_title: microRNA 155
  mask_index: 15
  mask_index_1based: 16
  masked_char: A
  output:
  - label: U
    score: 0.70388
  - label: '-'
    score: 0.124142
  - label: K
    score: 0.042144
  - label: W
    score: 0.034779
  - label: Y
    score: 0.018222
  pipeline_tag: fill-mask
  sequence_type: ncRNA
  task: fill-mask
  text: UUAAUGCUAAUCGUG<mask>UAGGGGUU
- example_title: RNA component of mitochondrial RNA processing endoribonuclease
  mask_index: 11
  mask_index_1based: 12
  masked_char: A
  output:
  - label: G
    score: 0.127718
  - label: K
    score: 0.109365
  - label: U
    score: 0.093649
  - label: D
    score: 0.077012
  - label: R
    score: 0.069837
  pipeline_tag: fill-mask
  sequence_type: ncRNA
  task: fill-mask
  text: GGUUCGUGCUG<mask>AGGCCUGUAUCCUAGGCUACACACUGAGGACUCUGUUCCUCCCCUUUCCGCCUAGGGGAAAGUCCCCGGACCUCGGGCAGAGAGUGCCACGUGCAUACGCACGUAGACAUUCCCCGCUUCCCACUCCAAAGUCCGCCAAGAAGCGUAUCCCGCUGAGCGGCGUGGCGCGGGGGCGUCAUCCGUCAGCUCCCUCUAGUUACGCAGGCAGUGCGUGUCCGCGCACCAACCACACGGGGCUCAUUCUCAGCGCGGCUGUAAAAAAAAA
- example_title: 7SK small nuclear RNA
  mask_index: 13
  mask_index_1based: 14
  masked_char: A
  output:
  - label: G
    score: 0.100307
  - label: K
    score: 0.100124
  - label: U
    score: 0.099941
  - label: D
    score: 0.07411
  - label: B
    score: 0.069722
  pipeline_tag: fill-mask
  sequence_type: ncRNA
  task: fill-mask
  text: GGAUGUGAGGGCG<mask>UCUGGCUGCGACAUCUGUCACCCCAUUGAUCGCCAGGGUUGAUUCGGCUGAUCUGGCUGGCUAGGCGGGUGUCCCCUUCCUCCCUCACCGCUCCAUGUGCGUCCCUCCCGAAGCUGCGCGCUCGGUCGAAGAGGACGACCAUCCCCGAUAGAGGAGGACCGGUCUUCGGUCAAGGGUAUACGAGUAGCUGCGCUCCCCUGCUAGAACCUCCAAACAAGCUCUCAAGGUCCAUUUGUAGGAGAACGUAGGGUAGUCAAGCUUCCAAGACUCCAGACACAUCCAAAUGAGGCGCUGCAUGUGGCAGUCUGCCUUUCUUUU
- example_title: telomerase RNA component
  mask_index: 23
  mask_index_1based: 24
  masked_char: A
  output:
  - label: U
    score: 0.286223
  - label: Y
    score: 0.108965
  - label: K
    score: 0.104227
  - label: B
    score: 0.076667
  - label: W
    score: 0.05469
  pipeline_tag: fill-mask
  sequence_type: ncRNA
  task: fill-mask
  text: GGGUUGCGGAGGGUGGGCCUGGG<mask>GGGGUGGUGGCCAUUUUUUGUCUAACCCUAACUGAGAAGGGCGUAGGCGCCGUGCUUUUGCUCCCCGCGCGCUGUUUUUCUCGCUGACUUUCAGCGGGCGGAAAAGCCUCGGCCUGCCGCCUUCCACCGUUCAUUCUAGAGCAAACAAAAAAUGUCAGCUGCUGGCCCGUUCGCCCCUCCCGGGGACCUGCGGCGGGUCGCCUGCCCAGCCCCCGAACCCCGCCUGGAGGCCGCGGUCGGCCCGGGGCUUCUCCGGAGGCACCCACUGCCACCGCGAAGAGUUGGGCUCUGUCAGCCGCGGGUCUCUCGGGGGCGAGGGCGAGGUUCAGGCCUUUCAGGCCGCAGGAAGAGGAACGGAGCGAGUCCCCGCGCGCGGCGCGAUUCCCUGAGCUGUGGGACGUGCACCCAGGACUCGGCUCACACAUGC
- example_title: vault RNA 2-1
  mask_index: 12
  mask_index_1based: 13
  masked_char: A
  output:
  - label: G
    score: 0.126666
  - label: R
    score: 0.119311
  - label: A
    score: 0.112384
  - label: D
    score: 0.109336
  - label: K
    score: 0.107844
  pipeline_tag: fill-mask
  sequence_type: ncRNA
  task: fill-mask
  text: CGGGUCGGAGUU<mask>GCUCAAGCGGUUACCUCCUCAUGCCGGACUUUCUAUCUGUCCAUCUCUGUGCUGGGGUUCGAGACCCGCGGGUGCUUACUGACCCUUUUAUGCAA
- example_title: brain cytoplasmic RNA 1
  mask_index: 18
  mask_index_1based: 19
  masked_char: A
  output:
  - label: U
    score: 0.24415
  - label: K
    score: 0.136772
  - label: G
    score: 0.076619
  - label: D
    score: 0.073373
  - label: W
    score: 0.071802
  pipeline_tag: fill-mask
  sequence_type: ncRNA
  task: fill-mask
  text: GGCCGGGCGCGGUGGCUC<mask>CGCCUGUAAUCCCAGCUCUCAGGGAGGCUAAGAGGCGGGAGGAUAGCUUGAGCCCAGGAGUUCGAGACCUGCCUGGGCAAUAUAGCGAGACCCCGUUCUCCAGAAAAAGGAAAAAAAAAAACAAAAGACAAAAAAAAAAUAAGCGUAACUUCCCUCAAAGCAACAACCCCCCCCCCCCUUU
- example_title: HIV-1 TAR-WT
  mask_index: 13
  mask_index_1based: 14
  masked_char: A
  output:
  - label: U
    score: 0.42819
  - label: Y
    score: 0.143348
  - label: K
    score: 0.079119
  - label: B
    score: 0.066973
  - label: C
    score: 0.047989
  pipeline_tag: fill-mask
  sequence_type: ncRNA
  task: fill-mask
  text: GGUCUCUCUGGUU<mask>GACCAGAUCUGAGCCUGGGAGCUCUCUGGCUAACUAGGGAACC
- example_title: prion protein (Kanno blood group)
  mask_index: 21
  mask_index_1based: 22
  masked_char: A
  output:
  - label: U
    score: 0.36797
  - label: K
    score: 0.127267
  - label: Y
    score: 0.114058
  - label: B
    score: 0.083041
  - label: G
    score: 0.044017
  pipeline_tag: fill-mask
  sequence_type: mRNA
  task: fill-mask
  text: AUGGCGAACCUUGGCUGCUGG<mask>UGCUGGUUCUCUUUGUGGCCACAUGGAGUGACCUGGGCCUCUGC
- example_title: interleukin 10
  mask_index: 11
  mask_index_1based: 12
  masked_char: A
  output:
  - label: G
    score: 0.194995
  - label: K
    score: 0.158988
  - label: U
    score: 0.129629
  - label: D
    score: 0.093929
  - label: R
    score: 0.079955
  pipeline_tag: fill-mask
  sequence_type: mRNA
  task: fill-mask
  text: AUGCACAGCUC<mask>GCACUGCUCUGUUGCCUGGUCCUCCUGACUGGGGUGAGGGCC
- example_title: Zaire ebolavirus
  mask_index: 11
  mask_index_1based: 12
  masked_char: A
  output:
  - label: U
    score: 0.089905
  - label: W
    score: 0.084702
  - label: K
    score: 0.083046
  - label: D
    score: 0.081949
  - label: A
    score: 0.079799
  pipeline_tag: fill-mask
  sequence_type: mRNA
  task: fill-mask
  text: AAUGUUCAAAC<mask>CUUUGUGAAGCUCUGUUAGCUGAUGGUCUUGCUAAAGCAUUUCCUAGCAAUAUGAUGGUAGUCACAGAGCGUGAGCAAAAAGAAAGCUUAUUGCAUCAAGCAUCAUGGCACCACACAAGUGAUGAUUUUGGUGAGCAUGCCACAGUUAGAGGGAGUAGCUUUGUAACUGAUUUAGAGAAAUACAAUCUUGCAUUUAGAUAUGAGUUUACAGCACCUUUUAUAGAAUAUUGUAACCGUUGCUAUGGUGUUAAGAAUGUUUUUAAUUGGAUGCAUUAUACAAUCCCACAGUGUUAU
- example_title: SARS coronavirus
  mask_index: 14
  mask_index_1based: 15
  masked_char: A
  output:
  - label: G
    score: 0.095108
  - label: R
    score: 0.083014
  - label: K
    score: 0.077897
  - label: D
    score: 0.07604
  - label: A
    score: 0.072459
  pipeline_tag: fill-mask
  sequence_type: mRNA
  task: fill-mask
  text: AUGUUUAUUUUCUU<mask>UUAUUUCUUACUCUCACUAGUGGUAGUGACCUUGACCGGUGCACCACUUUUGAUGAUGUUCAAGCUCCUAAUUACACUCAACAUACUUCAUCUAUGAGGGGGGUUUACUAUCCUGAUGAAAUUUUUAGAUCAGACACUCUUUAUUUAACUCAGGAUUUAUUUCUUCCAUUUUAUUCUAAUGUUACAGGGUUUCAUACUAUUAAUCAUACGUUUGACAACCCUGUCAUACCUUUUAAGGAUGGUAUUUAUUUUGCUGCCACAGAGAAAUCAAAUGUUGUCCGUGGUUGGGUUUUUGGUUCUACCAUGAACAACAAGUCACAGUCGGUGAUUAUUAUUAACAAUUCUACUAAUGUUGUUAUACGAGCAUGUAACUUUGAAUUGUGUGACAACCCUUUCUUUGCUGUUUCUAAACCCAUGGGUACACAGACACAUACUAUGAUAUUCGAUAAUGCAUUUAAAUGCACUUUCGAGUACAUAUCU
- example_title: insulin
  mask_index: 12
  mask_index_1based: 13
  masked_char: A
  output:
  - label: G
    score: 0.161098
  - label: R
    score: 0.101944
  - label: S
    score: 0.076365
  - label: V
    score: 0.072189
  - label: K
    score: 0.070794
  pipeline_tag: fill-mask
  sequence_type: mRNA
  task: fill-mask
  text: AUGGCCCUGUGG<mask>UGCGCCUCCUGCCCCUGCUGGCGCUGCUGGCCCUCUGGGGACCUGACCCAGCCGCAGCCUUUGUGAACCAACACCUGUGCGGCUCACACCUGGUGGAAGCUCUCUACCUAGUGUGCGGGGAACGAGGCUUCUUCUACACACCCAAGACCCGCCGGGAGGCAGAGGACCUGCAGGUGGGGCAGGUGGAGCUGGGCGGGGGCCCUGGUGCAGGCAGCCUGCAGCCCUUGGCCCUGGAGGGGUCCCUGCAGAAGCGUGGCAUUGUGGAACAAUGCUGUACCAGCAUCUGCUCCCUCUACCAGCUGGAGAACUACUGCAACUAG
- example_title: cyclin dependent kinase inhibitor 2A
  mask_index: 18
  mask_index_1based: 19
  masked_char: A
  output:
  - label: G
    score: 0.102352
  - label: S
    score: 0.088292
  - label: C
    score: 0.076164
  - label: B
    score: 0.072571
  - label: K
    score: 0.070839
  pipeline_tag: fill-mask
  sequence_type: mRNA
  task: fill-mask
  text: AUGGAGCCGGCGGCGGGG<mask>GCAGCAUGGAGCCUUCGGCUGACUGGCUGGCCACGGCCGCGGCCCGGGGUCGGGUAGAGGAGGUGCGGGCGCUGCUGGAGGCGGGGGCGCUGCCCAACGCACCGAAUAGUUACGGUCGGAGGCCGAUCCAGGUCAUGAUGAUGGGCAGCGCCCGAGUGGCGGAGCUGCUGCUGCUCCACGGCGCGGAGCCCAACUGCGCCGACCCCGCCACUCUCACCCGACCCGUGCACGACGCUGCCCGGGAGGGCUUCCUGGACACGCUGGUGGUGCUGCACCGGGCCGGGGCGCGGCUGGACGUGCGCGAUGCCUGGGGCCGUCUGCCCGUGGACCUGGCUGAGGAGCUGGGCCAUCGCGAUGUCGCACGGUACCUGCGCGCGGCUGCGGGGGGCACCAGAGGCAGUAACCAUGCCCGCAUAGAUGCCGCGGAAGGUCCCUCAGACAUCCCCGAUUGA
- example_title: human papillomavirus type 16 E6
  mask_index: 10
  mask_index_1based: 11
  masked_char: A
  output:
  - label: G
    score: 0.089809
  - label: K
    score: 0.086284
  - label: U
    score: 0.082897
  - label: D
    score: 0.071567
  - label: B
    score: 0.067529
  pipeline_tag: fill-mask
  sequence_type: mRNA
  task: fill-mask
  text: AUGCACCAAA<mask>GAGAACUGCAAUGUUUCAGGACCCACAGGAGCGACCCAGAAAGUUACCACAGUUAUGCACAGAGCUGCAAACAACUAUACAUGAUAUAAUAUUAGAAUGUGUGUACUGCAAGCAACAGUUACUGCGACGUGAGGUAUAUGACUUUGCUUUUCGGGAUUUAUGCAUAGUAUAUAGAGAUGGGAAUCCAUAUGCUGUAUGUGAUAAAUGUUUAAAGUUUUAUUCUAAAAUUAGUGAGUAUAGACAUUAUUGUUAUAGUUUGUAUGGAACAACAUUAGAACAGCAAUACAACAAACCGUUGUGUGAUUUGUUAAUUAGGUGUAUUAACUGUCAAAAGCCACUGUGUCCUGAAGAAAAGCAAAGACAUCUGGACAAAAAGCAAAGAUUCCAUAAUAUAAGGGGUCGGUGGACCGGUCGAUGUAUGUCUUGUUGCAGAUCAUCAAGAACACGUAGAGAAACCCAGCUGUAA
- example_title: NRAS proto-oncogene
  mask_index: 36
  mask_index_1based: 37
  masked_char: A
  output:
  - label: U
    score: 0.635724
  - label: K
    score: 0.093051
  - label: Y
    score: 0.063058
  - label: B
    score: 0.037834
  - label: W
    score: 0.032982
  pipeline_tag: fill-mask
  sequence_type: 5' UTR
  task: fill-mask
  text: GGGGCCGGAAGUGCCGCUCCUUGGUGGGGGCUGUUC<mask>UGGCGGUUCCGGGGUCUCCAACAUUUUUCCCGGCUGUGGUCCUAAAUCUGUCCAAAGCAGAGGCAGUGGAGCUUGAGGUUCUUGCUGGUGUGAA
- example_title: amyloid beta precursor protein
  mask_index: 15
  mask_index_1based: 16
  masked_char: A
  output:
  - label: G
    score: 0.138262
  - label: S
    score: 0.086267
  - label: R
    score: 0.078015
  - label: K
    score: 0.077845
  - label: V
    score: 0.068936
  pipeline_tag: fill-mask
  sequence_type: 5' UTR
  task: fill-mask
  text: GUCAGUUUCCUCGGC<mask>GCGGUAGGCGAGAGCACGCGGAGGAGCGUGCGCGGGGGCCCCGGGAGACGGCGGCGGUGGCGGCGCGGGCAGAGCAAGGACGCGGCGGAUCCCACUCGCACAGCAGCGCACUCGGUGCCCCGCGCAGGGUCGCG
- example_title: RUNX family transcription factor 1
  mask_index: 15
  mask_index_1based: 16
  masked_char: A
  output:
  - label: G
    score: 0.174696
  - label: K
    score: 0.150702
  - label: U
    score: 0.130003
  - label: D
    score: 0.084631
  - label: R
    score: 0.068284
  pipeline_tag: fill-mask
  sequence_type: 5' UTR
  task: fill-mask
  text: ACUUCUUUGGGCCUC<mask>UAAACAACCACAGAACCACAAGUUGGGUAGCCUGGCAGUGUCAGAAGUCUGAACCCAGCAUAGUGGUCAGCAGGCAGGACGAAUCACACUGAAUGCAAACCACAGGGUUUCGCAGCGUGGUAAAAGAAAUCAUUGAGUCCCCCGCCUUCAGAAGAGGGUGCAUUUUCAGGAGGAAGCG
- example_title: fragile X messenger ribonucleoprotein 1
  mask_index: 15
  mask_index_1based: 16
  masked_char: A
  output:
  - label: G
    score: 0.162588
  - label: S
    score: 0.109068
  - label: K
    score: 0.08251
  - label: B
    score: 0.07927
  - label: C
    score: 0.073166
  pipeline_tag: fill-mask
  sequence_type: 5' UTR
  task: fill-mask
  text: CUCAGUCAGGCGCUC<mask>GCUCCGUUUCGGUUUCACUUCCGGUGGAGGGCCGCCUCUGAGCGGGCGGCGGGCCGACGGCGAGCGCGGGCGGCGGCGGUGACGGAGGCGCCGCUGCCAGGGGGCGUGCGGCAGCGCGGCGGCGGCGGCGGCGGCGGCGGCGGCGGAGGCGGCGGCGGCGGCGGCGGCGGCGGCGGCUGGGCCUCGAGCGCCCGCAGCCCACCUCUCGGGGGCGGGCUCCCGGCGCUAGCAGGGCUGAAGAGAAG
- example_title: MYC proto-oncogene
  mask_index: 10
  mask_index_1based: 11
  masked_char: A
  output:
  - label: G
    score: 0.148459
  - label: K
    score: 0.089077
  - label: S
    score: 0.083355
  - label: R
    score: 0.072752
  - label: B
    score: 0.071878
  pipeline_tag: fill-mask
  sequence_type: 5' UTR
  task: fill-mask
  text: AACUCGCUGU<mask>GUAAUUCCAGCGAGAGGCAGAGGGAGCGAGCGGGCGGCCGGCUAGGGUGGAAGAGCCGGGCGAGCAGAGCUGCGCUGCGGGCGUCCUGGGAAGGGAGAUCCGGAGCGAAUAGGGGGCUUCGCCUCUGGCCCAGCCCUCCCGCUGAUCCCCCAGCCAGCGGUCCGCAACCCUUGCCGCAUCCACGAAACUUUGCCCAUAGCAGCGGGCGGGCACUUUGCACUGGAACUUACAACACCCGAGCAAGGACGCGACUCUCCCGACGCGGGGAGGCUAUUCUGCCCAUUUGGGGACACUUCCCCGCCGCUGCCAGGACCCGCUUCUCUGAAAGGCUCUCCUUGCAGCUGCUUAGACG
- example_title: activating transcription factor 4
  mask_index: 20
  mask_index_1based: 21
  masked_char: A
  output:
  - label: U
    score: 0.23485
  - label: K
    score: 0.133435
  - label: G
    score: 0.075814
  - label: B
    score: 0.068826
  - label: D
    score: 0.068213
  pipeline_tag: fill-mask
  sequence_type: 5' UTR
  task: fill-mask
  text: CAUUUCUACUUUGCCCGCCC<mask>CAGAUGUAGUUUUCUCUGCGCGUGUGCGUUUUCCCUCCUCCCCGCCCUCAGGGUCCACGGCCACCAUGGCGUAUUAGGGGCAGCAGUGCCUGCGGCAGCAUUGGCCUUUGCAGCGGCGGCAGCAGCACCAGGCUCUGCAGCGGCAACCCCCAGCGGCUUAAGCCAUGGCGCUUCUCACGGCAUUCAGCAGCAGCGUUGCUGUAACCGACAAAGACACCUUCGAAUUAAGCACAUUCCUCGAUUCCAGCAAAGCACCGCAAC
- example_title: Human GPI protein p137
  mask_index: 11
  mask_index_1based: 12
  masked_char: A
  output:
  - label: A
    score: 0.128298
  - label: R
    score: 0.123536
  - label: G
    score: 0.11895
  - label: D
    score: 0.084011
  - label: W
    score: 0.070603
  pipeline_tag: fill-mask
  sequence_type: 3' UTR
  task: fill-mask
  text: UUUUUAAAAGG<mask>AAAGAUACCAAAUGCCUGCUGCUACCACCCUUUUCAAUUGCUAUGUUUUGAAAGGCACCAGUAUGUGUUUUAGAUUGAUUUAAAUGUUUCAUUUAAAUCACGGACAGUAGUUUCAGUUCUGAUGGUAUAAGCAAAACAAAUAAAACGUUUAUAAAAGUUGUAUCUUGAAACACUGGUGUUCAACAGCUAGCAGCUUAUGUGAUUCACCCCAUGCCACGUUAGUGUCACAAAUUUUAUGGUUUAUCUCCAGCAACAUUUCUCUAGUACUUGCACUUAUUAUCUGAAUUC
- example_title: nucleophosmin 1
  mask_index: 11
  mask_index_1based: 12
  masked_char: A
  output:
  - label: G
    score: 0.099704
  - label: R
    score: 0.097548
  - label: A
    score: 0.095438
  - label: D
    score: 0.080819
  - label: K
    score: 0.074372
  pipeline_tag: fill-mask
  sequence_type: 3' UTR
  task: fill-mask
  text: GAAAAUAGUUU<mask>AACAAUUUGUUAAAAAAUUUUCCGUCUUAUUUCAUUUCUGUAACAGUUGAUAUCUGGCUGUCCUUUUUAUAAUGCAGAGUGAGAACUUUCCCUACCGUGUUUGAUAAAUGUUGUCCAGGUUCUAUUGCCAAGAAUGUGUUGUCCAAAAUGCCUGUUUAGUUUUUAAAGAUGGAACUCCACCCUUUGCUUGGUUUUAAGUAUGUAUGGAAUGUUAUGAUAGGACAUAGUAGUAGCGGUGGUCAGACAUGGAAAUGGUGGGGAGACAAAAAUAUACAUGUGAAAUAAAACUCAGUAUUUUAAUAAAGUAGCACGGUUUCUAUUGA
- example_title: superoxide dismutase 1
  mask_index: 12
  mask_index_1based: 13
  masked_char: A
  output:
  - label: A
    score: 0.101106
  - label: R
    score: 0.095146
  - label: G
    score: 0.089537
  - label: D
    score: 0.083374
  - label: W
    score: 0.080453
  pipeline_tag: fill-mask
  sequence_type: 3' UTR
  task: fill-mask
  text: ACAUUCCCUUGG<mask>UGUAGUCUGAGGCCCCUUAACUCAUCUGUUAUCCUGCUAGCUGUAGAAAUGUAUCCUGAUAAACAUUAAACACUGUAAUCUUAAAAGUGUAAUUGUGUGACUUUUUCAGAGUUGCUUUAAAGUACCUGUAGUGAGAAACUGAUUUAUGAUCACUUGGAAGAUUUGUAUAGUUUUAUAAAACUCAGUUAAAAUGUCUGUUUCAAUGACCUGUAUUUUGCCAGACUUAAAUCACAGAUGGGUAUUAAACUUGUCAGAAUUUCUUUGUCAUUCAAGCCUGUGAAUAAAAACCCUGUAUGGCACUUAUUAUGAGGCUAUUAAAAGAAUCCAAAUUCAAACUAAA
- example_title: hemoglobin subunit alpha 2
  mask_index: 13
  mask_index_1based: 14
  masked_char: A
  output:
  - label: G
    score: 0.170329
  - label: K
    score: 0.107836
  - label: R
    score: 0.08528
  - label: D
    score: 0.079185
  - label: U
    score: 0.068272
  pipeline_tag: fill-mask
  sequence_type: 3' UTR
  task: fill-mask
  text: CUGGAGCCUCGGU<mask>GCCGUUCCUCCUGCCCGCUGGGCCUCCCAACGGGCCCUCCUCCCCUCCUUGCACCGGCCCUUCCUGGUCUUUGAAUAAAGUCUGAGUGGGCAGCA
- example_title: BRAF proto-oncogene
  mask_index: 12
  mask_index_1based: 13
  masked_char: A
  output:
  - label: A
    score: 0.112735
  - label: R
    score: 0.094389
  - label: W
    score: 0.079391
  - label: D
    score: 0.07927
  - label: G
    score: 0.079028
  pipeline_tag: fill-mask
  sequence_type: 3' UTR
  task: fill-mask
  text: AACAAAUGAGUG<mask>GAGAGUUCAGGAGAGUAGCAACAAAAGGAAAAUAAAUGAACAUAUGUUUGCUUAUAUGUUAAAUUGAAUAAAAUACUCUCUUUUUUUUUAAGGUGAACCAAAGAACACUUGUGUGGUUAAAGACUAGAUAUAAUUUUUCCCCAAACUAAAAUUUAUACUUAACAUUGGAUUUUUAACAUCCAAGGGUUAAAAUACAUAGACAUUGCUAAAAAUUGGCAGAGCCUCUUCUAGAGGCUUUACUUUCUGUUCCGGGUUUGUAUCAUUCACUUGGUUAUUUUAAGUAGUAAACUUCAGUUUCUCAUGCAACUUUUGUUGCCAGCUAUCACAUGUCCACUAGGGACUCCAGAAGAAGACCCUACCUAUGCCUGUGUUUGCAGGUGAGAAGUUGGCAGUCGGUUAGCCUGGG
- example_title: H3 clustered histone 1
  mask_index: 17
  mask_index_1based: 18
  masked_char: A
  output:
  - label: G
    score: 0.304871
  - label: K
    score: 0.199453
  - label: U
    score: 0.130486
  - label: D
    score: 0.079415
  - label: R
    score: 0.061954
  pipeline_tag: fill-mask
  sequence_type: 3' UTR
  task: fill-mask
  text: UUACUGUGGUCUCUCUG<mask>CGGUCCAAGCAAAGGCUCUUUUCAGAGCCACCACCUUUUC
---

# RNA-MSM

Pre-trained model on non-coding RNA (ncRNA) with multi (homologous) sequence alignment using a masked language modeling (MLM) objective.

## Disclaimer

This is an UNOFFICIAL implementation of the [Multiple sequence alignment-based RNA language model and its application to structural inference](https://doi.org/10.1093/nar/gkad1031) by Yikun Zhang, Mei Lang, Jiuhong Jiang, Zhiqiang Gao, et al.

The OFFICIAL repository of RNA-MSM is at [yikunpku/RNA-MSM](https://github.com/yikunpku/RNA-MSM).

> [!CAUTION]
> The MultiMolecule team is aware of a potential risk in reproducing the results of RNA-MSM.
>
> The original implementation of RNA-MSM does not append `<eos>` token to the end of the input sequence in consistent to MSA Transformer.
> This should not affect the performance of the model in most cases, but it can lead to unexpected behavior in some cases.
>
> Please set `eos_token=None` in the tokenizer and set `eos_token_id=None` in the model configuration if you want the exact behavior of the original implementation.
>
> See more at [issue #10](https://github.com/yikunpku/RNA-MSM/issues/10)

> [!TIP]
> The MultiMolecule team has confirmed that the provided model and checkpoints are producing the same intermediate representations as the original implementation.

**The team releasing RNA-MSM did not write this model card for this model so this model card has been written by the MultiMolecule team.**

## Model Details

RNA-MSM is a [bert](https://huggingface.co/google-bert/bert-base-uncased)-style model.
RNA-MSM follows the [MSA](https://doi.org/10.1101/2021.02.12.430858) architecture, where it uses a column-wise attention and a row-wise attention to reduce the computational complexity over conventional self-attention.

RNA-MSM is pre-trained on a large corpus of non-coding RNA sequences with multiple sequence alignment (MSA) in a self-supervised fashion.
This means that the model was trained on the raw nucleotides of RNA sequences only, with an automatic process to generate inputs and labels from those texts.
Please refer to the [Training Details](#training-details) section for more information on the training process.

### Model Specification

| Num Layers | Hidden Size | Num Heads | Intermediate Size | Num Parameters (M) | FLOPs (G) | MACs (G) | Max Num Tokens |
| ---------- | ----------- | --------- | ----------------- | ------------------ | --------- | -------- | -------------- |
| 10         | 768         | 12        | 3072              | 95.92              | 92.84     | 46.31    | 1024           |

### Links

- **Code**: [multimolecule.rnamsm](https://github.com/DLS5-Omics/multimolecule/tree/master/multimolecule/models/rnamsm)
- **Weights**: [multimolecule/rnamsm](https://huggingface.co/multimolecule/rnamsm)
- **Data**: [multimolecule/rfam](https://huggingface.co/datasets/multimolecule/rfam)
- **Paper**: [Multiple sequence alignment-based RNA language model and its application to structural inference](https://doi.org/10.1093/nar/gkad1031)
- **Developed by**: Yikun Zhang, Mei Lang, Jiuhong Jiang, Zhiqiang Gao, Fan Xu, Thomas Litfin, Ke Chen, Jaswinder Singh, Xiansong Huang, Guoli Song, Yonghong Tian, Jian Zhan, Jie Chen, Yaoqi Zhou
- **Model type**: [BERT](https://huggingface.co/google-bert/bert-base-uncased) - [MSA](https://doi.org/10.1101/2021.02.12.430858)
- **Original Repository**: [yikunpku/RNA-MSM](https://github.com/yikunpku/RNA-MSM)

## Usage

The model file depends on the [`multimolecule`](https://multimolecule.danling.org) library. You can install it using pip:

```bash
pip install multimolecule
```

### Direct Use

#### Masked Language Modeling

You can use this model directly with a pipeline for masked language modeling:

```python
import multimolecule  # you must import multimolecule to register models
from transformers import pipeline

predictor = pipeline("fill-mask", model="multimolecule/rnamsm")
output = predictor("gguc<mask>cucugguuagaccagaucugagccu")
```

#### RNA Secondary Structure Prediction

You can use this model directly with a pipeline for secondary structure prediction:

```python
import multimolecule  # you must import multimolecule to register models
from transformers import pipeline

predictor = pipeline("rna-secondary-structure", model="multimolecule/rnamsm")
output = predictor("ggucucucugguuagaccagaucugagccu")
```

### Downstream Use

#### Extract Features

Here is how to use this model to get the features of a given sequence in PyTorch:

```python
from multimolecule import RnaTokenizer, RnaMsmModel


tokenizer = RnaTokenizer.from_pretrained("multimolecule/rnamsm")
model = RnaMsmModel.from_pretrained("multimolecule/rnamsm")

text = "UAGCUUAUCAGACUGAUGUUG"
input = tokenizer(text, return_tensors="pt")

output = model(**input)
```

#### Sequence Classification / Regression

> [!NOTE]
> This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for sequence classification or regression.

Here is how to use this model as backbone to fine-tune for a sequence-level task in PyTorch:

```python
import torch
from multimolecule import RnaTokenizer, RnaMsmForSequencePrediction


tokenizer = RnaTokenizer.from_pretrained("multimolecule/rnamsm")
model = RnaMsmForSequencePrediction.from_pretrained("multimolecule/rnamsm")

text = "UAGCUUAUCAGACUGAUGUUG"
input = tokenizer(text, return_tensors="pt")
label = torch.tensor([1])

output = model(**input, labels=label)
```

#### Token Classification / Regression

> [!NOTE]
> This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for token classification or regression.

Here is how to use this model as backbone to fine-tune for a nucleotide-level task in PyTorch:

```python
import torch
from multimolecule import RnaTokenizer, RnaMsmForTokenPrediction


tokenizer = RnaTokenizer.from_pretrained("multimolecule/rnamsm")
model = RnaMsmForNucleotidPrediction.from_pretrained("multimolecule/rnamsm")

text = "UAGCUUAUCAGACUGAUGUUG"
input = tokenizer(text, return_tensors="pt")
label = torch.randint(2, (len(text), ))

output = model(**input, labels=label)
```

#### Contact Classification / Regression

> [!NOTE]
> This model is not fine-tuned for any specific task. You will need to fine-tune the model on a downstream task to use it for contact classification or regression.

Here is how to use this model as backbone to fine-tune for a contact-level task in PyTorch:

```python
import torch
from multimolecule import RnaTokenizer, RnaMsmForContactPrediction


tokenizer = RnaTokenizer.from_pretrained("multimolecule/rnamsm")
model = RnaMsmForContactPrediction.from_pretrained("multimolecule/rnamsm")

text = "UAGCUUAUCAGACUGAUGUUG"
input = tokenizer(text, return_tensors="pt")
label = torch.randint(2, (len(text), len(text)))

output = model(**input, labels=label)
```

## Training Details

RNA-MSM used Masked Language Modeling (MLM) as the pre-training objective: taking a sequence, the model randomly masks 15% of the tokens in the input then runs the entire masked sentence through the model and has to predict the masked tokens. This is comparable to the Cloze task in language modeling.

### Training Data

The RNA-MSM model was pre-trained on [Rfam](https://rfam.org).
The Rfam database is a collection of RNA sequence families of structural RNAs including non-coding RNA genes as well as cis-regulatory elements.
RNA-MSM used Rfam 14.7 which contains 4,069 RNA families.

To avoid potential overfitting in structural inference, RNA-MSM excluded families with experimentally determined structures, such as ribosomal RNAs, transfer RNAs, and small nuclear RNAs. The final dataset contains 3,932 RNA families. The median value for the number of MSA sequences for these families by RNAcmap3 is 2,184.

To increase the number of homologous sequences, RNA-MSM used an automatic pipeline, RNAcmap3, for homolog search and sequence alignment. RNAcmap3 is a pipeline that combines the BLAST-N, INFERNAL, Easel, RNAfold and evolutionary coupling tools to generate homologous sequences.

RNA-MSM preprocessed all tokens by replacing "T"s with "U"s and substituting "R", "Y", "K", "M", "S", "W", "B", "D", "H", "V", "N" with "X".

Note that [`RnaTokenizer`][multimolecule.RnaTokenizer] will convert "T"s to "U"s for you, you may disable this behaviour by passing `replace_T_with_U=False`. `RnaTokenizer` does not perform other substitutions.

### Training Procedure

#### Preprocessing

RNA-MSM used masked language modeling (MLM) as the pre-training objective. The masking procedure is similar to the one used in BERT:

- Mask rate: 15%
- Replacement: `<mask>` for 80% of masked tokens
- Replacement: random token for 10% of masked tokens
- Replacement: unchanged token for 10% of masked tokens

#### Pre-training

The model was trained on 8 NVIDIA V100 GPUs with 32GiB memories.

- Batch Size: 1
- Epochs: 300
- Optimizer: Adam
- Learning rate: 3e-4
- Learning rate warm-up: 16,000 steps
- Weight decay: 3e-4
- Dropout: 0.1

## Citation

```bibtex
@article{zhang2023multiple,
    author = {Zhang, Yikun and Lang, Mei and Jiang, Jiuhong and Gao, Zhiqiang and Xu, Fan and Litfin, Thomas and Chen, Ke and Singh, Jaswinder and Huang, Xiansong and Song, Guoli and Tian, Yonghong and Zhan, Jian and Chen, Jie and Zhou, Yaoqi},
    title = "{Multiple sequence alignment-based RNA language model and its application to structural inference}",
    journal = {Nucleic Acids Research},
    volume = {52},
    number = {1},
    pages = {e3-e3},
    year = {2023},
    month = {11},
    abstract = "{Compared with proteins, DNA and RNA are more difficult languages to interpret because four-letter coded DNA/RNA sequences have less information content than 20-letter coded protein sequences. While BERT (Bidirectional Encoder Representations from Transformers)-like language models have been developed for RNA, they are ineffective at capturing the evolutionary information from homologous sequences because unlike proteins, RNA sequences are less conserved. Here, we have developed an unsupervised multiple sequence alignment-based RNA language model (RNA-MSM) by utilizing homologous sequences from an automatic pipeline, RNAcmap, as it can provide significantly more homologous sequences than manually annotated Rfam. We demonstrate that the resulting unsupervised, two-dimensional attention maps and one-dimensional embeddings from RNA-MSM contain structural information. In fact, they can be directly mapped with high accuracy to 2D base pairing probabilities and 1D solvent accessibilities, respectively. Further fine-tuning led to significantly improved performance on these two downstream tasks compared with existing state-of-the-art techniques including SPOT-RNA2 and RNAsnap2. By comparison, RNA-FM, a BERT-based RNA language model, performs worse than one-hot encoding with its embedding in base pair and solvent-accessible surface area prediction. We anticipate that the pre-trained RNA-MSM model can be fine-tuned on many other tasks related to RNA structure and function.}",
    doi = {10.1093/nar/gkad1031},
    url = {https://doi.org/10.1093/nar/gkad1031},
    eprint = {https://academic.oup.com/nar/article-pdf/52/1/e3/55443207/gkad1031.pdf},
}
```

> [!NOTE]
> The artifacts distributed in this repository are part of the MultiMolecule project.
> If MultiMolecule supports your research, please cite the MultiMolecule project as follows:

```bibtex
@software{chen_2024_12638419,
  author    = {Chen, Zhiyuan and Zhu, Sophia Y.},
  title     = {MultiMolecule},
  doi       = {10.5281/zenodo.12638419},
  publisher = {Zenodo},
  url       = {https://doi.org/10.5281/zenodo.12638419},
  year      = 2024,
  month     = may,
  day       = 4
}
```

## Contact

Please use GitHub issues of [MultiMolecule](https://github.com/DLS5-Omics/multimolecule/issues) for any questions or comments on the model card.

Please contact the authors of the [RNA-MSM paper](https://doi.org/10.1093/nar/gkad1031) for questions or comments on the paper/model.

## License

This model implementation is licensed under the [GNU Affero General Public License](license.md).

For additional terms and clarifications, please refer to our [License FAQ](license-faq.md).

```spdx
SPDX-License-Identifier: AGPL-3.0-or-later
```