Kaestli, D. Kotlinski, U. Langenegger, M. Missiroli690000-0002-1780-1344, L.
Noehte69, T. Rohe ETH Zurich - Institute for Particle Physics and Astrophysics
(IPA), Zurich, Switzerland K. Androsov680000-0003-2694-6542, M.
Backhaus0000-0002-5888-2304, P. Berger, A. Calandri0000-0001-7774-0099, A. De
Cosa, G. Dissertori0000-0002-4549-2569, M. Dittmar, M. Donegà, C.
Dorfer0000-0002-2163-442X, F. Eble, K. Gedia, F. Glessgen, T.A. Gómez
Espinosa0000-0002-9443-7769, C. Grab0000-0002-6182-3380, D. Hits, W.
Lustermann, A.-M. Lyon, R.A. Manzoni0000-0002-7584-5038, L.
Marchese0000-0001-6627-8716, C. Martin Perez, M.T. Meinhard, F. Nessi-Tedaldi,
J. Niedziela0000-0002-9514-0799, F. Pauss, V. Perovic, S.
Pigazzini0000-0002-8046-4344, M.G. Ratti0000-0003-1777-7855, M. Reichmann, C.
Reissel, T. Reitenspiess, B. Ristic0000-0002-8610-1130, D. Ruini, D.A. Sanz
Becerra0000-0002-6610-4019, V. Stampf, J. Steggemann680000-0003-4420-5510, R.
Wallny0000-0001-8038-1613 Universität Zürich, Zurich, Switzerland C.
Amsler700000-0002-7695-501X, P. Bärtschi, C. Botta0000-0002-8072-795X, D.
Brzhechko, M.F. Canelli0000-0001-6361-2117, K. Cormier, A. De
Wit0000-0002-5291-1661, R. Del Burgo, J.K. Heikkilä0000-0002-0538-1469, M.
Huwiler, W. Jin, A. Jofrehei0000-0002-8992-5426, B.
Kilminster0000-0002-6657-0407, S. Leontsinis0000-0002-7561-6091, S.P. Liechti,
A. Macchiolo0000-0003-0199-6957, P. Meiring, V.M. Mikuni0000-0002-1579-2421,
U. Molinatti, I. Neutelings, A. Reimers, P. Robmann, S. Sanchez
Cruz0000-0002-9991-195X, K. Schweiger0000-0002-5846-3919, M. Senger, Y.
Takahashi0000-0001-5184-2265 National Central University, Chung-Li, Taiwan C.
Adloff71, C.M. Kuo, W. Lin, A. Roy0000-0002-5622-4260, T.
Sarkar390000-0003-0582-4167, S.S. Yu National Taiwan University (NTU), Taipei,
Taiwan L. Ceard, Y. Chao, K.F. Chen0000-0003-1304-3782, P.H.
Chen0000-0002-0468-8805, P.s. Chen, H. Cheng0000-0001-6456-7178, W.-S.
Hou0000-0002-4260-5118, Y.y. Li, R.-S. Lu, E. Paganis0000-0002-1950-8993, A.
Psallidas, A. Steen, H.y. Wu, E. Yazgan0000-0001-5732-7950, P.r. Yu
Chulalongkorn University, Faculty of Science, Department of Physics, Bangkok,
Thailand B. Asavapibhop0000-0003-1892-7130, C.
Asawatangtrakuldee0000-0003-2234-7219, N. Srimanobhas0000-0003-3563-2959
Çukurova University, Physics Department, Science and Art Faculty, Adana,
Turkey F. Boran0000-0002-3611-390X, S. Damarseckin72, Z.S.
Demiroglu0000-0001-7977-7127, F. Dolek0000-0001-7092-5517, I.
Dumanoglu730000-0002-0039-5503, E. Eskut, Y. Guler740000-0001-7598-5252, E.
Gurpinar Guler740000-0002-6172-0285, C. Isik, O. Kara, A. Kayis Topaksu, U.
Kiminsu0000-0001-6940-7800, G. Onengut, K. Ozdemir75, A. Polatoz, A.E.
Simsek0000-0002-9074-2256, B. Tali76, U.G. Tok0000-0002-3039-021X, S.
Turkcapar, I.S. Zorbakir0000-0002-5962-2221 Middle East Technical University,
Physics Department, Ankara, Turkey G. Karapinar, K.
Ocalan770000-0002-8419-1400, M. Yalvac780000-0003-4915-9162 Bogazici
University, Istanbul, Turkey B. Akgun, I.O. Atakisi0000-0002-9231-7464, E.
Gulmez0000-0002-6353-518X, M. Kaya790000-0003-2890-4493, O. Kaya80, Ö.
Özçelik, S. Tekten81, E.A. Yetkin820000-0002-9007-8260 Istanbul Technical
University, Istanbul, Turkey A. Cakir0000-0002-8627-7689, K.
Cankocak730000-0002-3829-3481, Y. Komurcu, S. Sen830000-0001-7325-1087
Istanbul University, Istanbul, Turkey S. Cerci76, I. Hos84, B. Isildak85, B.
Kaynak, S. Ozkorucuklu, H. Sert0000-0003-0716-6727, D. Sunar
Cerci760000-0002-5412-4688, C. Zorbilmez Institute for Scintillation Materials
of National Academy of Science of Ukraine, Kharkov, Ukraine B. Grynyov
National Scientific Center, Kharkov Institute of Physics and Technology,
Kharkov, Ukraine L. Levchuk0000-0001-5889-7410 University of Bristol, Bristol,
United Kingdom D. Anthony, E. Bhal0000-0003-4494-628X, S. Bologna, J.J.
Brooke0000-0002-6078-3348, A. Bundock0000-0002-2916-6456, E.
Clement0000-0003-3412-4004, D. Cussans0000-0001-8192-0826, H.
Flacher0000-0002-5371-941X, J. Goldstein0000-0003-1591-6014, G.P. Heath, H.F.
Heath0000-0001-6576-9740, L. Kreczko0000-0003-2341-8330, B.
Krikler0000-0001-9712-0030, S. Paramesvaran, S. Seif El Nasr-Storey, V.J.
Smith, N. Stylianou860000-0002-0113-6829, K. Walkingshaw Pass, R. White
Rutherford Appleton Laboratory, Didcot, United Kingdom K.W. Bell, A.
Belyaev870000-0002-1733-4408, C. Brew0000-0001-6595-8365, R.M. Brown, D.J.A.
Cockerill, C. Cooke, K.V. Ellis, K. Harder, S. Harper, M.-L. Holmberg88, J.
Linacre0000-0001-7555-652X, K. Manolopoulos, D.M. Newbold0000-0002-9015-9634,
E. Olaiya, D. Petyt, T. Reis0000-0003-3703-6624, T. Schuh, C.H. Shepherd-
Themistocleous, I.R. Tomalin, T. Williams0000-0002-8724-4678 Imperial College,
London, United Kingdom R. Bainbridge0000-0001-9157-4832, P.
Bloch0000-0001-6716-979X, S. Bonomally, J. Borg0000-0002-7716-7621, S. Breeze,
O. Buchmuller, V. Cepaitis0000-0002-4809-4056, G.S.
Chahal890000-0003-0320-4407, D. Colling, P. Dauncey0000-0001-6839-9466, G.
Davies0000-0001-8668-5001, M. Della Negra0000-0001-6497-8081, S. Fayer, G.
Fedi0000-0001-9101-2573, G. Hall0000-0002-6299-8385, M.H. Hassanshahi, G.
Iles, J. Langford, L. Lyons, A.-M. Magnan, S. Malik, A.
Martelli0000-0003-3530-2255, D.G. Monk, J. Nash900000-0003-0607-6519, M.
Pesaresi, B.C. Radburn-Smith, D.M. Raymond, A. Richards, A. Rose, E.
Scott0000-0003-0352-6836, C. Seez, A. Shtipliyski, A.
Tapper0000-0003-4543-864X, K. Uchida, T. Virdee210000-0001-7429-2198, M.
Vojinovic0000-0001-8665-2808, N. Wardle0000-0003-1344-3356, S.N.
Webb0000-0003-4749-8814, D. Winterbottom Brunel University, Uxbridge, United
Kingdom K. Coldham, J.E. Cole0000-0001-5638-7599, A. Khan, P.
Kyberd0000-0002-7353-7090, I.D. Reid0000-0002-9235-779X, L. Teodorescu, S.
Zahid0000-0003-2123-3607 Baylor University, Waco, Texas, USA S.
Abdullin0000-0003-4885-6935, A. Brinkerhoff0000-0002-4853-0401, B.
Caraway0000-0002-6088-2020, J. Dittmann0000-0002-1911-3158, K.
Hatakeyama0000-0002-6012-2451, A.R. Kanuganti, B. McMaster0000-0002-4494-0446,
N. Pastika, M. Saunders0000-0003-1572-9075, S. Sawant, C. Sutantawibul, J.
Wilson0000-0002-5672-7394 Catholic University of America, Washington, DC, USA
R. Bartek0000-0002-1686-2882, A. Dominguez0000-0002-7420-5493, R.
Uniyal0000-0001-7345-6293, A.M. Vargas Hernandez The University of Alabama,
Tuscaloosa, Alabama, USA A. Buccilli0000-0001-6240-8931, S.I.
Cooper0000-0002-4618-0313, D. Di Croce0000-0002-1122-7919, S.V.
Gleyzer0000-0002-6222-8102, C. Henderson0000-0002-6986-9404, C.U.
Perez0000-0002-6861-2674, P. Rumerio910000-0002-1702-5541, C.
West0000-0003-4460-2241 Boston University, Boston, Massachusetts, USA A.
Akpinar0000-0001-7510-6617, A. Albert0000-0003-2369-9507, D.
Arcaro0000-0001-9457-8302, C. Cosby0000-0003-0352-6561, Z.
Demiragli0000-0001-8521-737X, E. Fontanesi, D. Gastler, S.
May0000-0002-6351-6122, J. Rohlf0000-0001-6423-9799, K.
Salyer0000-0002-6957-1077, D. Sperka, D. Spitzbart0000-0003-2025-2742, I.
Suarez0000-0002-5374-6995, A. Tsatsos, S. Yuan, D. Zou Brown University,
Providence, Rhode Island, USA G. Benelli0000-0003-4461-8905, B.
Burkle0000-0003-1645-822X, X. Coubez22, D. Cutts0000-0003-1041-7099, M.
Hadley0000-0002-7068-4327, U. Heintz0000-0002-7590-3058, J.M.
Hogan920000-0002-8604-3452, T. Kwon, G. Landsberg0000-0002-4184-9380, K.T.
Lau0000-0003-1371-8575, D. Li, M. Lukasik, J. Luo0000-0002-4108-8681, M.
Narain, N. Pervan, S. Sagir930000-0002-2614-5860, F. Simpson, E.
Usai0000-0001-9323-2107, W.Y. Wong, X. Yan0000-0002-6426-0560, D.
Yu0000-0001-5921-5231, W. Zhang University of California, Davis, Davis,
California, USA J. Bonilla0000-0002-6982-6121, C. Brainerd0000-0002-9552-1006,
R. Breedon, M. Calderon De La Barca Sanchez, M. Chertok0000-0002-2729-6273, J.
Conway0000-0003-2719-5779, P.T. Cox, R. Erbacher, G. Haza, F.
Jensen0000-0003-3769-9081, O. Kukral, R. Lander, M.
Mulhearn0000-0003-1145-6436, D. Pellett, B. Regnery0000-0003-1539-923X, D.
Taylor0000-0002-4274-3983, Y. Yao0000-0002-5990-4245, F.
Zhang0000-0002-6158-2468 University of California, Los Angeles, California,
USA M. Bachtis0000-0003-3110-0701, R. Cousins0000-0002-5963-0467, A.
Datta0000-0003-2695-7719, D. Hamilton, J. Hauser0000-0002-9781-4873, M.
Ignatenko, M.A. Iqbal, T. Lam, W.A. Nash, S. Regnard0000-0002-9818-6725, D.
Saltzberg0000-0003-0658-9146, B. Stone, V. Valuev0000-0002-0783-6703
University of California, Riverside, Riverside, California, USA Y. Chen, R.
Clare0000-0003-3293-5305, J.W. Gary0000-0003-0175-5731, M. Gordon, G.
Hanson0000-0002-7273-4009, G. Karapostoli0000-0002-4280-2541, O.R.
Long0000-0002-2180-7634, N. Manganelli, W. Si0000-0002-5879-6326, S. Wimpenny,
Y. Zhang University of California, San Diego, La Jolla, California, USA J.G.
Branson, P. Chang0000-0002-2095-6320, S. Cittolin, S.
Cooperstein0000-0003-0262-3132, N. Deelen0000-0003-4010-7155, D.
Diaz0000-0001-6834-1176, J. Duarte0000-0002-5076-7096, R.
Gerosa0000-0001-8359-3734, L. Giannini0000-0002-5621-7706, J. Guiang, R.
Kansal0000-0003-2445-1060, V. Krutelyov0000-0002-1386-0232, R. Lee, J.
Letts0000-0002-0156-1251, M. Masciovecchio0000-0002-8200-9425, F. Mokhtar, M.
Pieri0000-0003-3303-6301, B.V. Sathia Narayanan0000-0003-2076-5126, V.
Sharma0000-0003-1736-8795, M. Tadel, F. Würthwein0000-0001-5912-6124, Y.
Xiang0000-0003-4112-7457, A. Yagil0000-0002-6108-4004 University of
California, Santa Barbara - Department of Physics, Santa Barbara, California,
USA N. Amin, C. Campagnari0000-0002-8978-8177, M. Citron0000-0001-6250-8465,
G. Collura0000-0002-4160-1844, A. Dorsett, V. Dutta0000-0001-5958-829X, J.
Incandela0000-0001-9850-2030, M. Kilpatrick0000-0002-2602-0566, J.
Kim0000-0002-2072-6082, B. Marsh, H. Mei, M. Oshiro, M.
Quinnan0000-0003-2902-5597, J. Richman, U. Sarica0000-0002-1557-4424, F.
Setti, J. Sheplock, P. Siddireddy, D. Stuart, S. Wang0000-0001-7887-1728
California Institute of Technology, Pasadena, California, USA A.
Bornheim0000-0002-0128-0871, O. Cerri, I. Dutta0000-0003-0953-4503, J.M.
Lawhorn0000-0002-8597-9259, N. Lu0000-0002-2631-6770, J. Mao, H.B.
Newman0000-0003-0964-1480, T.Q. Nguyen0000-0003-3954-5131, M.
Spiropulu0000-0001-8172-7081, J.R. Vlimant0000-0002-9705-101X, C.
Wang0000-0002-0117-7196, S. Xie0000-0003-2509-5731, Z.
Zhang0000-0002-1630-0986, R.Y. Zhu0000-0003-3091-7461 Carnegie Mellon
University, Pittsburgh, Pennsylvania, USA J. Alison0000-0003-0843-1641, S.
An0000-0002-9740-1622, M.B. Andrews, P. Bryant0000-0001-8145-6322, T.
Ferguson0000-0001-5822-3731, A. Harilal, C. Liu, T.
Mudholkar0000-0002-9352-8140, M. Paulini0000-0002-6714-5787, A. Sanchez, W.
Terrill University of Colorado Boulder, Boulder, Colorado, USA J.P.
Cumalat0000-0002-6032-5857, W.T. Ford0000-0001-8703-6943, A. Hassani, G.
Karathanasis, E. MacDonald, R. Patel, A. Perloff0000-0001-5230-0396, C.
Savard, N. Schonbeck, K. Stenson0000-0003-4888-205X, K.A.
Ulmer0000-0001-6875-9177, S.R. Wagner0000-0002-9269-5772, N. Zipper Cornell
University, Ithaca, New York, USA J. Alexander0000-0002-2046-342X, S. Bright-
Thonney0000-0003-1889-7824, X. Chen0000-0002-8157-1328, Y.
Cheng0000-0002-2602-935X, D.J. Cranshaw0000-0002-7498-2129, S. Hogan, J.
Monroy0000-0002-7394-4710, J.R. Patterson0000-0002-3815-3649, D.
Quach0000-0002-1622-0134, J. Reichert0000-0003-2110-8021, M.
Reid0000-0001-7706-1416, A. Ryd, W. Sun0000-0003-0649-5086, J.
Thom0000-0002-4870-8468, P. Wittich0000-0002-7401-2181, R.
Zou0000-0002-0542-1264 Fermi National Accelerator Laboratory, Batavia,
Illinois, USA M. Albrow0000-0001-7329-4925, M. Alyari0000-0001-9268-3360, G.
Apollinari, A. Apresyan0000-0002-6186-0130, A. Apyan0000-0002-9418-6656,
L.A.T. Bauerdick0000-0002-7170-9012, D. Berry0000-0002-5383-8320, J.
Berryhill0000-0002-8124-3033, P.C. Bhat, K. Burkett0000-0002-2284-4744, J.N.
Butler, A. Canepa, G.B. Cerati0000-0003-3548-0262, H.W.K.
Cheung0000-0001-6389-9357, F. Chlebana, K.F. Di Petrillo0000-0001-8001-4602,
J. Dickinson0000-0001-5450-5328, V.D. Elvira0000-0003-4446-4395, Y. Feng, J.
Freeman, Z. Gecse, L. Gray, D. Green, S. Grünendahl0000-0002-4857-0294, O.
Gutsche0000-0002-8015-9622, R.M. Harris0000-0003-1461-3425, R. Heller, T.C.
Herwig0000-0002-4280-6382, J. Hirschauer0000-0002-8244-0805, B.
Jayatilaka0000-0001-7912-5612, S. Jindariani, M. Johnson, U. Joshi, T.
Klijnsma0000-0003-1675-6040, B. Klima0000-0002-3691-7625, K.H.M. Kwok, S.
Lammel0000-0003-0027-635X, D. Lincoln0000-0002-0599-7407, R. Lipton, T. Liu,
C. Madrid, K. Maeshima, C. Mantilla0000-0002-0177-5903, D. Mason, P.
McBride0000-0001-6159-7750, P. Merkel, S. Mrenna0000-0001-8731-160X, S.
Nahn0000-0002-8949-0178, J. Ngadiuba0000-0002-0055-2935, V. Papadimitriou, K.
Pedro0000-0003-2260-9151, C. Pena620000-0002-4500-7930, F.
Ravera0000-0003-3632-0287, A. Reinsvold Hall940000-0003-1653-8553, L.
Ristori0000-0003-1950-2492, E. Sexton-Kennedy0000-0001-9171-1980, N.
Smith0000-0002-0324-3054, A. Soha0000-0002-5968-1192, L. Spiegel, S.
Stoynev0000-0003-4563-7702, J. Strait0000-0002-7233-8348, L.
Taylor0000-0002-6584-2538, S. Tkaczyk, N.V. Tran0000-0002-8440-6854, L.
Uplegger0000-0002-9202-803X, E.W. Vaandering0000-0003-3207-6950, H.A.
Weber0000-0002-5074-0539 University of Florida, Gainesville, Florida, USA P.
Avery, D. Bourilkov0000-0003-0260-4935, L. Cadamuro0000-0001-8789-610X, V.
Cherepanov, R.D. Field, D. Guerrero, B.M. Joshi0000-0002-4723-0968, M. Kim, E.
Koenig, J. Konigsberg0000-0001-6850-8765, A. Korytov, K.H. Lo, K.
Matchev0000-0003-4182-9096, N. Menendez0000-0002-3295-3194, G.
Mitselmakher0000-0001-5745-3658, A. Muthirakalayil Madhu, N. Rawal, D.
Rosenzweig, S. Rosenzweig, K. Shi0000-0002-2475-0055, J.
Wang0000-0003-3879-4873, Z. Wu0000-0003-2165-9501, E.
Yigitbasi0000-0002-9595-2623, X. Zuo Florida State University, Tallahassee,
Florida, USA T. Adams0000-0001-8049-5143, A. Askew0000-0002-7172-1396, R.
Habibullah0000-0002-3161-8300, V. Hagopian, K.F. Johnson, R. Khurana, T.
Kolberg0000-0002-0211-6109, G. Martinez, H. Prosper0000-0002-4077-2713, C.
Schiber, O. Viazlo0000-0002-2957-0301, R. Yohay0000-0002-0124-9065, J. Zhang
Florida Institute of Technology, Melbourne, Florida, USA M.M.
Baarmand0000-0002-9792-8619, S. Butalla, T. Elkafrawy950000-0001-9930-6445, M.
Hohlmann0000-0003-4578-9319, R. Kumar Verma0000-0002-8264-156X, D.
Noonan0000-0002-3932-3769, M. Rahmani, F. Yumiceva0000-0003-2436-5074
University of Illinois at Chicago (UIC), Chicago, Illinois, USA M.R. Adams, H.
Becerril Gonzalez0000-0001-5387-712X, R. Cavanaugh0000-0001-7169-3420, S.
Dittmer, O. Evdokimov0000-0002-1250-8931, C.E. Gerber0000-0002-8116-9021, D.J.
Hofman0000-0002-2449-3845, A.H. Merrit, C. Mills0000-0001-8035-4818, G.
Oh0000-0003-0744-1063, T. Roy, S. Rudrabhatla, M.B. Tonjes0000-0002-2617-9315,
N. Varelas0000-0002-9397-5514, J. Viinikainen0000-0003-2530-4265, X. Wang, Z.
Ye0000-0001-6091-6772 The University of Iowa, Iowa City, Iowa, USA M.
Alhusseini0000-0002-9239-470X, K. Dilsiz960000-0003-0138-3368, L. Emediato,
R.P. Gandrajula0000-0001-9053-3182, O.K. Köseyan0000-0001-9040-3468, J.-P.
Merlo, A. Mestvirishvili97, J. Nachtman, H. Ogul980000-0002-5121-2893, Y.
Onel0000-0002-8141-7769, A. Penzo, C. Snyder, E. Tiras990000-0002-5628-7464
Johns Hopkins University, Baltimore, Maryland, USA O.
Amram0000-0002-3765-3123, B. Blumenfeld0000-0003-1150-1735, L.
Corcodilos0000-0001-6751-3108, J. Davis, A.V. Gritsan0000-0002-3545-7970, S.
Kyriacou, P. Maksimovic0000-0002-2358-2168, J. Roskes0000-0001-8761-0490, M.
Swartz, T.Á. Vámi0000-0002-0959-9211 The University of Kansas, Lawrence,
Kansas, USA A. Abreu, J. Anguiano, C. Baldenegro Barrera0000-0002-6033-8885,
P. Baringer0000-0002-3691-8388, A. Bean0000-0001-5967-8674, Z. Flowers, T.
Isidori, S. Khalil0000-0001-8630-8046, J. King, G.
Krintiras0000-0002-0380-7577, A. Kropivnitskaya0000-0002-8751-6178, M.
Lazarovits, C. Le Mahieu, C. Lindsey, J. Marquez, N.
Minafra0000-0003-4002-1888, M. Murray0000-0001-7219-4818, M. Nickel, C.
Rogan0000-0002-4166-4503, C. Royon, R. Salvatico0000-0002-2751-0567, S.
Sanders, E. Schmitz, C. Smith0000-0003-0505-0528, Q. Wang0000-0003-3804-3244,
Z. Warner, J. Williams0000-0002-9810-7097, G. Wilson0000-0003-0917-4763 Kansas
State University, Manhattan, Kansas, USA S. Duric, A.
Ivanov0000-0002-9270-5643, K. Kaadze0000-0003-0571-163X, D. Kim, Y.
Maravin0000-0002-9449-0666, T. Mitchell, A. Modak, K. Nam Lawrence Livermore
National Laboratory, Livermore, California, USA F. Rebassoo, D. Wright
University of Maryland, College Park, Maryland, USA E. Adams, A. Baden, O.
Baron, A. Belloni0000-0002-1727-656X, S.C. Eno0000-0003-4282-2515, N.J.
Hadley0000-0002-1209-6471, S. Jabeen0000-0002-0155-7383, R.G. Kellogg, T.
Koeth, Y. Lai, S. Lascio, A.C. Mignerey, S. Nabili, C.
Palmer0000-0003-0510-141X, M. Seidel0000-0003-3550-6151, A.
Skuja0000-0002-7312-6339, L. Wang, K. Wong0000-0002-9698-1354 Massachusetts
Institute of Technology, Cambridge, Massachusetts, USA D. Abercrombie, G.
Andreassi, R. Bi, W. Busza0000-0002-3831-9071, I.A. Cali, Y.
Chen0000-0003-2582-6469, M. D’Alfonso0000-0002-7409-7904, J. Eysermans, C.
Freer0000-0002-7967-4635, G. Gomez Ceballos, M. Goncharov, P. Harris, M. Hu,
M. Klute0000-0002-0869-5631, D. Kovalskyi0000-0002-6923-293X, J. Krupa, Y.-J.
Lee0000-0003-2593-7767, C. Mironov0000-0002-8599-2437, C.
Paus0000-0002-6047-4211, D. Rankin0000-0001-8411-9620, C.
Roland0000-0002-7312-5854, G. Roland, Z. Shi0000-0001-5498-8825, G.S.F.
Stephans0000-0003-3106-4894, J. Wang, Z. Wang0000-0002-3074-3767, B.
Wyslouch0000-0003-3681-0649 University of Minnesota, Minneapolis, Minnesota,
USA R.M. Chatterjee, A. Evans0000-0002-7427-1079, J. Hiltbrand, Sh.
Jain0000-0003-1770-5309, M. Krohn, Y. Kubota, J. Mans0000-0003-2840-1087, M.
Revering, R. Rusack0000-0002-7633-749X, R. Saradhy, N.
Schroeder0000-0002-8336-6141, N. Strobbe0000-0001-8835-8282, M.A. Wadud
University of Nebraska-Lincoln, Lincoln, Nebraska, USA K.
Bloom0000-0002-4272-8900, M. Bryson, S. Chauhan0000-0002-6544-5794, D.R.
Claes, C. Fangmeier, L. Finco0000-0002-2630-5465, F. Golf0000-0003-3567-9351,
C. Joo, I. Kravchenko0000-0003-0068-0395, I. Reed, J.E. Siado, G.R.
Snow${}^{\textrm{\textdagger}}$, W. Tabb, A. Wightman, F. Yan, A.G.
Zecchinelli State University of New York at Buffalo, Buffalo, New York, USA G.
Agarwal0000-0002-2593-5297, H. Bandyopadhyay0000-0001-9726-4915, L.
Hay0000-0002-7086-7641, I. Iashvili0000-0003-1948-5901, A. Kharchilava, C.
McLean0000-0002-7450-4805, D. Nguyen, J. Pekkanen0000-0002-6681-7668, S.
Rappoccio0000-0002-5449-2560, A. Williams0000-0003-4055-6532 Northeastern
University, Boston, Massachusetts, USA G. Alverson0000-0001-6651-1178, E.
Barberis, Y. Haddad0000-0003-4916-7752, Y. Han, A. Hortiangtham, A. Krishna,
J. Li0000-0001-5245-2074, J. Lidrych0000-0003-1439-0196, G. Madigan, B.
Marzocchi0000-0001-6687-6214, D.M. Morse0000-0003-3163-2169, V. Nguyen, T.
Orimoto0000-0002-8388-3341, A. Parker, L. Skinnari0000-0002-2019-6755, A.
Tishelman-Charny, T. Wamorkar, B. Wang0000-0003-0796-2475, A. Wisecarver, D.
Wood0000-0002-6477-801X Northwestern University, Evanston, Illinois, USA S.
Bhattacharya0000-0002-0526-6161, J. Bueghly, Z. Chen0000-0003-4521-6086, A.
Gilbert0000-0001-7560-5790, T. Gunter0000-0002-7444-5622, K.A. Hahn, Y. Liu,
N. Odell, M.H. Schmitt0000-0003-0814-3578, M. Velasco University of Notre
Dame, Notre Dame, Indiana, USA R. Band0000-0003-4873-0523, R. Bucci, M.
Cremonesi, A. Das0000-0001-9115-9698, N. Dev0000-0003-2792-0491, R.
Goldouzian0000-0002-0295-249X, M. Hildreth, K. Hurtado
Anampa0000-0002-9779-3566, C. Jessop0000-0002-6885-3611, K.
Lannon0000-0002-9706-0098, J. Lawrence, N. Loukas0000-0003-0049-6918, D.
Lutton, J. Mariano, N. Marinelli, I. Mcalister, T.
McCauley0000-0001-6589-8286, C. Mcgrady, K. Mohrman, C. Moore, Y. Musienko55,
R. Ruchti, A. Townsend, M. Wayne, M. Zarucki0000-0003-1510-5772, L. Zygala The
Ohio State University, Columbus, Ohio, USA B. Bylsma, L.S.
Durkin0000-0002-0477-1051, B. Francis0000-0002-1414-6583, C.
Hill0000-0003-0059-0779, M. Nunez Ornelas0000-0003-2663-7379, K. Wei, B.L.
Winer, B.R. Yates0000-0001-7366-1318 Princeton University, Princeton, New
Jersey, USA F.M. Addesa0000-0003-0484-5804, B. Bonham0000-0002-2982-7621, P.
Das0000-0002-9770-1377, G. Dezoort, P. Elmer0000-0001-6830-3356, A.
Frankenthal0000-0002-2583-5982, B. Greenberg0000-0002-4922-1934, N. Haubrich,
S. Higginbotham, A. Kalogeropoulos0000-0003-3444-0314, G. Kopp, S.
Kwan0000-0002-5308-7707, D. Lange, D. Marlow0000-0002-6395-1079, K.
Mei0000-0003-2057-2025, I. Ojalvo, J. Olsen0000-0002-9361-5762, D.
Stickland0000-0003-4702-8820, C. Tully0000-0001-6771-2174 University of Puerto
Rico, Mayaguez, Puerto Rico, USA S. Malik0000-0002-6356-2655, S. Norberg
Purdue University, West Lafayette, Indiana, USA A.S. Bakshi, V.E.
Barnes0000-0001-6939-3445, R. Chawla0000-0003-4802-6819, S.
Das0000-0001-6701-9265, L. Gutay, M. Jones0000-0002-9951-4583, A.W.
Jung0000-0003-3068-3212, D. Kondratyev0000-0002-7874-2480, A.M. Koshy, M. Liu,
G. Negro, N. Neumeister0000-0003-2356-1700, G. Paspalaki, S.
Piperov0000-0002-9266-7819, A. Purohit, J.F. Schulte0000-0003-4421-680X, M.
Stojanovic17, J. Thieman0000-0001-7684-6588, F. Wang0000-0002-8313-0809, R.
Xiao0000-0001-7292-8527, W. Xie0000-0003-1430-9191 Purdue University
Northwest, Hammond, Indiana, USA J. Dolen0000-0003-1141-3823, N. Parashar Rice
University, Houston, Texas, USA D. Acosta0000-0001-5367-1738, A.
Baty0000-0001-5310-3466, T. Carnahan, M. Decaro, S.
Dildick0000-0003-0554-4755, K.M. Ecklund0000-0002-6976-4637, S. Freed, P.
Gardner, F.J.M. Geurts0000-0003-2856-9090, A. Kumar0000-0002-5180-6595, W. Li,
B.P. Padley0000-0002-3572-5701, R. Redjimi, J. Rotter, W.
Shi0000-0002-8102-9002, A.G. Stahl Leiton0000-0002-5397-252X, S.
Yang0000-0002-2075-8631, L. Zhang100, Y. Zhang0000-0002-6812-761X University
of Rochester, Rochester, New York, USA A. Bodek0000-0003-0409-0341, P. de
Barbaro, R. Demina0000-0002-7852-167X, J.L. Dulemba0000-0002-9842-7015, C.
Fallon, T. Ferbel0000-0002-6733-131X, M. Galanti, A. Garcia-
Bellido0000-0002-1407-1972, O. Hindrichs0000-0001-7640-5264, A.
Khukhunaishvili, E. Ranken, R. Taus, G.P. Van Onsem0000-0002-1664-2337
Rutgers, The State University of New Jersey, Piscataway, New Jersey, USA B.
Chiarito, J.P. Chou0000-0001-6315-905X, A. Gandrakota0000-0003-4860-3233, Y.
Gershtein0000-0002-4871-5449, E. Halkiadakis0000-0002-3584-7856, A. Hart, M.
Heindl0000-0002-2831-463X, O. Karacheban250000-0002-2785-3762, I. Laflotte, A.
Lath0000-0003-0228-9760, R. Montalvo, K. Nash, M. Osherson, S.
Salur0000-0002-4995-9285, S. Schnetzer, S. Somalwar0000-0002-8856-7401, R.
Stone, S.A. Thayil0000-0002-1469-0335, S. Thomas, H. Wang0000-0002-3027-0752
University of Tennessee, Knoxville, Tennessee, USA H. Acharya, A.G.
Delannoy0000-0003-1252-6213, S. Fiorendi0000-0003-3273-9419, S.
Spanier0000-0002-8438-3197 Texas A&M University, College Station, Texas, USA
O. Bouhali1010000-0001-7139-7322, M. Dalchenko0000-0002-0137-136X, A.
Delgado0000-0003-3453-7204, R. Eusebi, J. Gilmore, T. Huang, T. Kamon102, H.
Kim0000-0003-4986-1728, S. Luo0000-0003-3122-4245, S. Malhotra, R. Mueller, D.
Overton, D. Rathjens0000-0002-8420-1488, A. Safonov0000-0001-9497-5471 Texas
Tech University, Lubbock, Texas, USA N. Akchurin, J. Damgov, V. Hegde, S.
Kunori, K. Lamichhane, S.W. Lee0000-0002-3388-8339, T. Mengke, S.
Muthumuni0000-0003-0432-6895, T. Peltola0000-0002-4732-4008, I. Volobouev, Z.
Wang, A. Whitbeck Vanderbilt University, Nashville, Tennessee, USA E.
Appelt0000-0003-3389-4584, S. Greene, A. Gurrola0000-0002-2793-4052, W. Johns,
A. Melo, K. Padeken0000-0001-7251-9125, F. Romeo0000-0002-1297-6065, P.
Sheldon0000-0003-1550-5223, S. Tuo, J. Velkovska0000-0003-1423-5241 University
of Virginia, Charlottesville, Virginia, USA M.W. Arenton0000-0002-6188-1011,
B. Cardwell, B. Cox0000-0003-3752-4759, G. Cummings0000-0002-8045-7806, J.
Hakala0000-0001-9586-3316, R. Hirosky0000-0003-0304-6330, M.
Joyce0000-0003-1112-5880, A. Ledovskoy0000-0003-4861-0943, A. Li, C.
Neu0000-0003-3644-8627, C.E. Perez Lara0000-0003-0199-8864, B.
Tannenwald0000-0002-5570-8095, S. White0000-0002-6181-4935 Wayne State
University, Detroit, Michigan, USA N. Poudyal0000-0003-4278-3464 University of
Wisconsin - Madison, Madison, WI, Wisconsin, USA S. Banerjee, K.
Black0000-0001-7320-5080, T. Bose0000-0001-8026-5380, S.
Dasu0000-0001-5993-9045, I. De Bruyn0000-0003-1704-4360, P.
Everaerts0000-0003-3848-324X, C. Galloni, H. He, M.
Herndon0000-0003-3043-1090, A. Herve, U. Hussain, A. Lanaro, A. Loeliger, R.
Loveless, J. Madhusudanan Sreekala0000-0003-2590-763X, A. Mallampalli, A.
Mohammadi, D. Pinna, A. Savin, V. Shang, V. Sharma0000-0003-1287-1471, W.H.
Smith0000-0003-3195-0909, D. Teague, S. Trembath-Reichert, W.
Vetens0000-0003-1058-1163 †: Deceased
1: Also at TU Wien, Wien, Austria
2: Also at Institute of Basic and Applied Sciences, Faculty of Engineering,
Arab Academy for Science, Technology and Maritime Transport, Alexandria, Egypt
3: Also at Université Libre de Bruxelles, Bruxelles, Belgium
4: Also at Universidade Estadual de Campinas, Campinas, Brazil
5: Also at Federal University of Rio Grande do Sul, Porto Alegre, Brazil
6: Also at The University of the State of Amazonas, Manaus, Brazil
7: Also at University of Chinese Academy of Sciences, Beijing, China
8: Also at Department of Physics, Tsinghua University, Beijing, China
9: Also at UFMS, Nova Andradina, Brazil
10: Also at Nanjing Normal University Department of Physics, Nanjing, China
11: Now at The University of Iowa, Iowa City, Iowa, USA
12: Also at Institute for Theoretical and Experimental Physics named by A.I.
Alikhanov of NRC ‘Kurchatov Institute’, Moscow, Russia
13: Also at Joint Institute for Nuclear Research, Dubna, Russia
14: Also at Cairo University, Cairo, Egypt
15: Also at Helwan University, Cairo, Egypt
16: Now at Zewail City of Science and Technology, Zewail, Egypt
17: Also at Purdue University, West Lafayette, Indiana, USA
18: Also at Université de Haute Alsace, Mulhouse, France
19: Also at Tbilisi State University, Tbilisi, Georgia
20: Also at Erzincan Binali Yildirim University, Erzincan, Turkey
21: Also at CERN, European Organization for Nuclear Research, Geneva,
Switzerland
22: Also at RWTH Aachen University, III. Physikalisches Institut A, Aachen,
Germany
23: Also at University of Hamburg, Hamburg, Germany
24: Also at Isfahan University of Technology, Isfahan, Iran
25: Also at Brandenburg University of Technology, Cottbus, Germany
26: Also at Forschungszentrum Jülich, Juelich, Germany
27: Also at Physics Department, Faculty of Science, Assiut University, Assiut,
Egypt
28: Also at Karoly Robert Campus, MATE Institute of Technology, Gyongyos,
Hungary
29: Also at Institute of Physics, University of Debrecen, Debrecen, Hungary
30: Also at Institute of Nuclear Research ATOMKI, Debrecen, Hungary
31: Now at Universitatea Babes-Bolyai - Facultatea de Fizica, Cluj-Napoca,
Romania
32: Also at MTA-ELTE Lendület CMS Particle and Nuclear Physics Group, Eötvös
Loránd University, Budapest, Hungary
33: Also at Wigner Research Centre for Physics, Budapest, Hungary
34: Also at IIT Bhubaneswar, Bhubaneswar, India
35: Also at Institute of Physics, Bhubaneswar, India
36: Also at Punjab Agricultural University, Ludhiana, India
37: Also at Shoolini University, Solan, India
38: Also at University of Hyderabad, Hyderabad, India
39: Also at University of Visva-Bharati, Santiniketan, India
40: Also at Indian Institute of Science (IISc), Bangalore, India
41: Also at Indian Institute of Technology (IIT), Mumbai, India
42: Also at Deutsches Elektronen-Synchrotron, Hamburg, Germany
43: Now at Department of Physics, Isfahan University of Technology, Isfahan,
Iran
44: Also at Department of Electrical and Computer Engineering, Isfahan
University of Technology, Isfahan, Iran
45: Also at Department of Physics, University of Science and Technology of
Mazandaran, Behshahr, Iran
46: Now at INFN Sezione di Bari, Università di Bari, Politecnico di Bari,
Bari, Italy
47: Also at Italian National Agency for New Technologies, Energy and
Sustainable Economic Development, Bologna, Italy
48: Also at Centro Siciliano di Fisica Nucleare e di Struttura Della Materia,
Catania, Italy
49: Also at Scuola Superiore Meridionale, Università di Napoli Federico II,
Napoli, Italy
50: Also at Università di Napoli ’Federico II’, Napoli, Italy
51: Also at Consiglio Nazionale delle Ricerche - Istituto Officina dei
Materiali, Perugia, Italy
52: Also at Riga Technical University, Riga, Latvia
53: Also at Consejo Nacional de Ciencia y Tecnología, Mexico City, Mexico
54: Also at IRFU, CEA, Université Paris-Saclay, Gif-sur-Yvette, France
55: Also at Institute for Nuclear Research, Moscow, Russia
56: Now at National Research Nuclear University ’Moscow Engineering Physics
Institute’ (MEPhI), Moscow, Russia
57: Also at Institute of Nuclear Physics of the Uzbekistan Academy of
Sciences, Tashkent, Uzbekistan
58: Also at St. Petersburg Polytechnic University, St. Petersburg, Russia
59: Also at University of Florida, Gainesville, Florida, USA
60: Also at Imperial College, London, United Kingdom
61: Also at P.N. Lebedev Physical Institute, Moscow, Russia
62: Also at California Institute of Technology, Pasadena, California, USA
63: Also at Budker Institute of Nuclear Physics, Novosibirsk, Russia
64: Also at Faculty of Physics, University of Belgrade, Belgrade, Serbia
65: Also at Trincomalee Campus, Eastern University, Sri Lanka, Nilaveli, Sri
Lanka
66: Also at INFN Sezione di Pavia, Università di Pavia, Pavia, Italy
67: Also at National and Kapodistrian University of Athens, Athens, Greece
68: Also at Ecole Polytechnique Fédérale Lausanne, Lausanne, Switzerland
69: Also at Universität Zürich, Zurich, Switzerland
70: Also at Stefan Meyer Institute for Subatomic Physics, Vienna, Austria
71: Also at Laboratoire d’Annecy-le-Vieux de Physique des Particules,
IN2P3-CNRS, Annecy-le-Vieux, France
72: Also at Şırnak University, Sirnak, Turkey
73: Also at Near East University, Research Center of Experimental Health
Science, Nicosia, Turkey
74: Also at Konya Technical University, Konya, Turkey
75: Also at Piri Reis University, Istanbul, Turkey
76: Also at Adiyaman University, Adiyaman, Turkey
77: Also at Necmettin Erbakan University, Konya, Turkey
78: Also at Bozok Universitetesi Rektörlügü, Yozgat, Turkey
79: Also at Marmara University, Istanbul, Turkey
80: Also at Milli Savunma University, Istanbul, Turkey
81: Also at Kafkas University, Kars, Turkey
82: Also at Istanbul Bilgi University, Istanbul, Turkey
83: Also at Hacettepe University, Ankara, Turkey
84: Also at Istanbul University - Cerrahpasa, Faculty of Engineering,
Istanbul, Turkey
85: Also at Ozyegin University, Istanbul, Turkey
86: Also at Vrije Universiteit Brussel, Brussel, Belgium
87: Also at School of Physics and Astronomy, University of Southampton,
Southampton, United Kingdom
88: Also at Rutherford Appleton Laboratory, Didcot, United Kingdom
89: Also at IPPP Durham University, Durham, United Kingdom
90: Also at Monash University, Faculty of Science, Clayton, Australia
91: Also at Università di Torino, Torino, Italy
92: Also at Bethel University, St. Paul, Minneapolis, USA
93: Also at Karamanoğlu Mehmetbey University, Karaman, Turkey
94: Also at United States Naval Academy, Annapolis, Maryland, USA
95: Also at Ain Shams University, Cairo, Egypt
96: Also at Bingol University, Bingol, Turkey
97: Also at Georgian Technical University, Tbilisi, Georgia
98: Also at Sinop University, Sinop, Turkey
99: Also at Erciyes University, Kayseri, Turkey
100: Also at Institute of Modern Physics and Key Laboratory of Nuclear Physics
and Ion-beam Application (MOE) - Fudan University, Shanghai, China
101: Also at Texas A&M University at Qatar, Doha, Qatar
102: Also at Kyungpook National University, Daegu, Korea


# Traveling Words: A Geometric Interpretation of Transformers

Raul Molina
<EMAIL_ADDRESS>
(September 2023)

##  1 Abstract

Transformers have significantly advanced the field of natural language
processing, but comprehending their internal mechanisms remains a challenge.
In this paper, we introduce a novel geometric perspective that elucidates the
inner mechanisms of transformer operations. Our primary contribution is
illustrating how layer normalization confines the latent features to a hyper-
sphere, subsequently enabling attention to mold the semantic representation of
words on this surface. This geometric viewpoint seamlessly connects
established properties such as iterative refinement and contextual embeddings.
We validate our insights by probing a pre-trained 124M parameter GPT-2 model.
Our findings reveal clear query-key attention patterns in early layers and
build upon prior observations regarding the subject-specific nature of
attention heads at deeper layers. Harnessing these geometric insights, we
present an intuitive understanding of transformers, depicting them as
processes that model the trajectory of word particles along the hyper-sphere.

##  2 Introduction

The transformer architecture [Vaswani et al., 2017] has sparked a significant
shift in Artificial Intelligence (AI). It is the central component behind some
of the most advanced conversational AI systems [Brown et al., 2020, Thoppilan
et al., 2022, Bai et al., 2022], and has been established as state-of-the-art
for Natural Language Processing (NLP), Computer Vision (CV) and Robotics
applications, and many others [OpenAI, 2023, Google, 2023, Chen et al., 2023,
Zong et al., 2022, Driess et al., 2023].

Recent work on the interpretability of the transformer architecture has
focused on analyzing weights in relation to the word embedding space used in
its input and output layers Dar et al. [2022], Elhage et al. [2021], Geva et
al. [2022], Brody et al. [2023], Windsor [2022], Millidge and Black [2022].
Elhage et al. [2021] introduces “Transformer Circuits”, a theoretical
framework that decomposes the transformer computation into two main
components: a residual stream that carries information from input to output
layers and attention/feed-forward updates that modify the information flowing
in the residual stream. A key development from their work is the grouping of
the $W_{Q}$$W_{K}^{T}$ and $W_{O}$$W_{V}^{T}$ matrices from the attention
mechanism, representing low-rank approximations of virtual matrices $W_{QK}$
and $W_{OV}$, respectively. These virtual matrices define interactions between
different words in the input sequence $X$ within a canonical feature space $E$
given by the word embedding matrix $W_{E}$. The resulting values from these
interactions are used to update the information carried throughout the
residual stream. Geva et al. [2022] further decompose the operations within
the Transformer, demonstrating that the updates from the feed-forward module
can be decomposed into a linear combination of sub-updates given by the weight
matrix of the feed-forward module’s second layer $W_{2}$. The matrix $W_{2}$
also interacts within the canonical space $E$ and allows the authors to
measure the impact of each sub-update on the model’s final prediction using
the matrix $W_{E}$ as a probe. Dar et al. [2022] incorporate these ideas to
show that it is not only possible to interpret the outcomes of each
Transformer operation in relation to the canonical space $E$ but also the
weights themselves, enabling them to do zero-shot model stitching by
“translating” between the canonical spaces of different language models.
Finally, Millidge and Black [2022] note that analysis on the singular vectors
of the $W_{OV}$ matrices provides better practical results when compared to
analysis of its row and column weights.

A complimentary perspective to the line of work on Transformer Circuits comes
from the geometric interpretation of layer normalization [Ba et al., 2016] by
Brody et al. [2023]. The authors prove that layer normalization is equivalent
to projecting features onto the hyperplane defined by the
$\overrightarrow{\mathbbm{1}}$ vector and then scaling the projection by
$\sqrt{d}$. They show that these properties are crucial for the attention
mechanism to either attend to all keys equally or to avoid the problem of
having “unselectable” keys (relevant keys within the convex hull of a set of
non-relevant keys). Windsor [2022] provides further evidence of the
representational power of layer normalization, visualizing the highly non-
linear behavior that arises from this operation. The authors demonstrate that,
when used as an activation function within a neural network, layer
normalization is capable of solving complex classification tasks.

In this work, we connect these ideas under a single interpretation: word
particles traveling around the surface of a hyper-sphere, completing a journey
that goes from a previous word to the next and transforming their meaning
throughout this process. An illustrated summary of this interpretation is
given in Figure 1.

Figure 1: Overview of the proposed geometric interpretation of Transformers.
In it, the phrase “Traveling Words” is to be completed by a given transformer
model. The input token “Traveling ” is embedded as a word particle using an
embedding matrix $W_{E}$ and projected onto a hyper-sphere using layer
normalization. Each subsequent layer in the transformer determines the path
that the particle will follow along the surface of the hyper-sphere,
culminating on the region closest to the next token: “Words”.

##  3 Transformers as a Composition of Geometric Primitives

In this section, we analyze each of the transformer’s components from a
geometric perspective, leveraging the interpretation of one component to
analyze the next. We begin with the layer normalization function, for which we
demonstrate that it constrains $d$-dimensional input features to lie within
the surface of a $(d-1)$ dimensional hyper-sphere. Then we consider the role
of the $W_{QK}$ matrix in terms of geometric transformations on said hyper-
sphere, and $W_{VO}$ as a key-value mapping from the hyper-sphere back to
$\mathbb{R}^{d}$. Additionally, we review the key-value interpretation of the
feed-forward module proposed by Geva et al. [2021]. Finally, we discuss the
role of the embedding matrix $W_{E}$ on the transformer’s output
probabilities.

###  3.1 Layer Normalization

In its original formulation [Ba et al., 2016], layer normalization is
introduced in terms of the mean $\mu$ and standard deviation $\sigma$ of an
input feature vector $X\in\mathbb{R}^{d}$:

$\text{LayerNorm}(X)=\frac{X-\mu}{\sigma}$ (1)

Where both the mean and standard deviation are taken along the feature
dimension $d$ such that:

$\displaystyle\mu$ $\displaystyle=\frac{1}{d}\sum_{i}^{d}x_{i}$
$\displaystyle\sigma$
$\displaystyle=\sqrt{\frac{1}{d}\sum_{i}^{d}(x_{i}-\mu)^{2}}$

Brody et al. [2023] note that the numerator in Equation 1 is itself an
operation between $X$ and the vector $\boldsymbol{\mu}$ defined as:

$\boldsymbol{\mu}=[\mu,\mu,\dots,\mu]\in\mathbb{R}^{d}$

The resulting vector $(X-\boldsymbol{\mu})$ is shown to be orthogonal to the
$\overrightarrow{\mathbbm{1}}$ vector, and thus layer normalization can be
interpreted as a projection of $X$ onto the hyperplane defined by the normal
vector $\overrightarrow{\mathbbm{1}}$. Brody et al. [2023] also show that the
division by $\sigma$ acts as a scaling factor that modifies the norm of
$(X-\boldsymbol{\mu})$ to be $\sqrt{d}$:

$\displaystyle\begin{split}\sigma&=\sqrt{\frac{1}{d}\sum_{i}^{d}(x_{i}-\mu)^{2}}\\\
&=\frac{1}{\sqrt{d}}\sqrt{\sum_{i}^{d}(x_{i}-\mu)^{2}}\\\
&=\frac{1}{\sqrt{d}}||X-\boldsymbol{\mu}||_{2}\end{split}$ (2)

We note that, if we consider unit-norm vector
$\frac{1}{\sqrt{d}}\overrightarrow{\mathbbm{1}}$ instead of
$\overrightarrow{\mathbbm{1}}$, it can be shown that $\boldsymbol{\mu}$ is the
projection of $X$ onto $\frac{1}{\sqrt{d}}\overrightarrow{\mathbbm{1}}$
(explaining why $X-\boldsymbol{\mu}$ is orthogonal to
$\overrightarrow{\mathbbm{1}}$):

$\displaystyle\begin{split}\text{proj}(X,\frac{1}{\sqrt{d}}\overrightarrow{\mathbbm{1}})&=\frac{1}{||\frac{1}{\sqrt{d}}\overrightarrow{\mathbbm{1}}||_{2}}\Bigg{(}\mathbf{x}\cdot\frac{1}{\sqrt{d}}\overrightarrow{\mathbbm{1}}\Bigg{)}\frac{1}{\sqrt{d}}\overrightarrow{\mathbbm{1}}\\\
&=\Big{(}\frac{\mathbf{x}\cdot\overrightarrow{\mathbbm{1}}}{\sqrt{d}}\Big{)}\frac{1}{\sqrt{d}}\overrightarrow{\mathbbm{1}}\\\
&=\Big{(}\frac{\mathbf{x}\cdot\overrightarrow{\mathbbm{1}}}{d}\Big{)}\overrightarrow{\mathbbm{1}}\\\
&=\Big{(}\frac{1}{d}\sum_{i}^{d}x_{i}\Big{)}\overrightarrow{\mathbbm{1}}\\\
&=\mu\overrightarrow{\mathbbm{1}}\\\ &=\boldsymbol{\mu}\end{split}$ (3)

From this result, it is straightforward to calculate the projection of $X$
onto the hyperplane $\mathcal{H}$ defined by
$\frac{1}{\sqrt{d}}\overrightarrow{\mathbbm{1}}$:

$\displaystyle\begin{split}\text{proj}_{\mathcal{H}}(X)&=X-\text{proj}(X,\frac{1}{\sqrt{d}}\overrightarrow{\mathbbm{1}})\\\
&=X-\boldsymbol{\mu}\end{split}$ (4)

Finally, we can use the results from Equation 2 and Equation 4 to reformulate
layer normalization in geometric terms:

$\displaystyle\begin{split}\text{LayerNorm}(X)&=\frac{X-\mu}{\sigma}\\\
&=\frac{\text{proj}_{\mathcal{H}}(X)}{\frac{1}{\sqrt{d}}||\text{proj}_{\mathcal{H}}(X)||_{2}}\\\
&=\sqrt{d}\;\frac{\text{proj}_{\mathcal{H}}(X)}{||\text{proj}_{\mathcal{H}}(X)||_{2}}\end{split}$
(5)

Intuitively, layer normalization projects a vector $X\in\mathbb{R}^{d}$ to the
hyperplane $\mathcal{H}$ perpendicular to
$\overrightarrow{\mathbbm{1}}\in\mathbb{R}^{d}$, and normalizes the projection
such that it lies on the surface of a $d-1$ dimensional hyper-sphere of radius
$\sqrt{d}$. A visualization of this process for $d=3$ is shown in Figure 2.

Figure 2:  Layer normalization visualized on 3D data. Left: Original feature
space (from randomly sampled data), with each data point color-coded according
to its position in space. Right: Feature space after layer normalization, note
that all data points lie within the plane perpendicular to the
$\overrightarrow{\mathbbm{1}}$ vector.

In practice, layer normalization includes two additional parameters: a scaling
factor $\gamma$ and a bias term $\beta$. The parameter $\gamma$ scales each
coordinate axis of $\mathbb{R}^{d}$ independently, transforming the hyper-
sphere into a hyper-ellipsoid, and the bias term $\beta$ shifts the center of
said ellipsoid away from the origin. A 2D representation of the entire process
is shown in Figure 3.

Xiong et al. [2020] show that layer normalization should be applied within
each block before the attention and feed-forward module updates and as a final
step before prediction. Doing so scales down the gradient of the feed-forward
module’s weight parameters and keeps the magnitude of the hidden states
bounded with respect to the depth of a given layer, which improves stability
during training and removes the need for a warm-up stage.

###  3.2 Multi-Head Self-Attention

In the previous section, we showed how the layer normalization approach given
by Xiong et al. [2020] enforces data within each layer to be constrained on
the surface of a hyper-sphere, potentially unique to each layer. However,
thanks to the residual nature of transformers, all intermediate layer
representations share the same vector space and thus are essentially
projecting features onto the same hyper-sphere $\mathcal{H}_{S}$. Furthermore,
given that layer normalization is applied before the classification softmax,
the model maximizes the dot-product similarity between a subset of points
within $\mathcal{H}_{S}$ and the word vectors in the embedding matrix
$W_{E}\in\mathbb{R}^{|V|\times d}$ (where $|V|$ denotes the size of the
vocabulary), establishing the meaning of points in $\mathcal{H}_{S}$ in
relation to the words associated with $W_{E}$. To understand how the geometric
intuition behind $\mathcal{H}_{S}$ allows for interpretability, we will first
revisit the self-attention module in transformers [Vaswani et al., 2017].

For a given input sequence $X\in\mathbb{R}^{s\times d}$ of length $s$, the
self-attention mechanism is defined as follows:

$\text{SelfAttention}(X,W_{Q},W_{K},W_{V})=\text{softmax}\Big{(}\frac{QK^{T}}{\sqrt{d}}\Big{)}V$
(6)

where

$\displaystyle\begin{split}Q&=XW_{Q}\\\ K&=XW_{K}\\\ V&=XW_{V}\end{split}$ (7)

Such that $W_{Q}\in\mathbbm{R}^{d\times k}$, $W_{K}\in\mathbbm{R}^{d\times k}$
and $W_{V}\in\mathbbm{R}^{d\times v}$ are projection matrices from the
original model dimension $d$ to intermediate dimension $k$ and value dimension
$v$, respectively. For multi-head attention, multiple projection matrices
$W_{Q}^{i}$, $W_{K}^{i}$, $W_{V}^{i}$ are considered, one for each head
$i\in[1,\dots,h]$ (with $h$ being the number of heads). In this case, the
value dimension $v$ is commonly set equal to $k$ and an extra projection
matrix $W_{O}\in\mathbb{R}^{hk\times d}$ is introduced to combine information
from all heads as follows [Vaswani et al., 2017]:

$\displaystyle\begin{split}\text{MultiHead}(X)&=\text{Concat}([\text{head}_{1},\dots,\text{head}_{h}])W_{O}\\\
\text{where}\;\text{head}_{i}&=\text{SelfAttention}(X,W_{Q}^{i},W_{K}^{i},W_{V}^{i})\end{split}$
(8)

Given that the concatenation happens along the row dimension of each head, it
is possible to re-write multi-head self-attention as follows:

$\displaystyle\begin{split}\text{MultiHead(X)}&=\sum_{i}^{h}\text{SelfAttention}(X,W_{Q}^{i},W_{K}^{i},W_{V}^{i})W_{O}^{i}\\\
\text{where}\;W_{O}&=\text{Concat}[W_{O}^{1},\dots,W_{O}^{h}]\end{split}$ (9)

Such that each $W_{O}^{i}\in\mathbb{R}^{k\times d}$ denotes an element of the
partition of matrix $W_{O}$ alongside the row dimension. Combining Equation 7
and Equation 9 we obtain a single formula for multi-head self-attention:

$\displaystyle\begin{split}\text{MultiHead(X)}&=\sum_{i}^{h}\text{softmax}\Bigg{(}\frac{XW_{Q}^{i}{W_{K}^{i}}^{T}X^{T}}{\sqrt{d}}\Bigg{)}XW_{V}^{i}W_{O}^{i}\\\
&=\sum_{i}^{h}\text{softmax}\Bigg{(}\frac{XW^{i}_{QK}X^{T}}{\sqrt{d}}\Bigg{)}XW^{i}_{VO}\end{split}$
(10)

Where $W_{QK}^{i}\in\mathbb{R}^{d\times d}$ and
$W_{VO}^{i}\in\mathbb{R}^{d\times d}$ are low-rank virtual matrices obtained
by grouping $W_{Q}^{i}W_{K}^{iT}$ and $W_{V}^{i}W_{O}^{i}$ respectively
[Elhage et al., 2021, Dar et al., 2022].

###  3.3 The $W_{QK}$ Matrix

For any given head, the matrix $W^{i}_{QK}$ is commonly interpreted as a bi-
linear form $f:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}$ that
represents the relevance between keys and queries. However, it is also
possible to consider $W^{i}_{QK}$ as a linear transformation that maps inputs
to a query representation $X^{i}_{q}\in\mathbb{R}^{s\times d}$ (similar to
that considered in Brody et al. [2023]):

$XW^{i}_{QK}=X^{i}_{q}$ (11)

With the head’s attention score matrix $A^{i}\in[0,1]^{s\times s}$, for a
given sequence length $s$, obtained as:

$A^{i}=\text{softmax}\Big{(}\frac{X^{i}_{q}X^{T}}{\sqrt{d}}\Big{)}$ (12)

Alternatively, its transpose can be considered as a transformation that maps
inputs to a key representation:

$X{W^{i}_{QK}}^{T}=X^{i}_{k}$ (13)

With the attention score matrix as follows:

$A^{i}=\text{softmax}\Big{(}\frac{X(X^{i}_{k})^{T}}{\sqrt{d}}\Big{)}$ (14)

This process is illustrated for normalized inputs in the bottom-right section
of Figure 3. Essentially, the role of the $W_{QK}$ matrix and the layer
normalization parameters is to find a transformation over $\mathcal{H}_{S}$
such that, when superimposed on itself, brings related terms closer together
and keeps unrelated terms apart.

Figure 3:  Visualization of the self-attention process for a single head. Top
Left: Layer normalization projects the input features on the surface of the
hyper-sphere $\mathcal{H}_{S}$. Top Right: A scaling parameter $\gamma$ is
commonly applied after normalization; it transforms $\mathcal{H}_{S}$ into an
($d-1$)-dimensional ellipsoid. Bottom Left: A bias term $\beta$ is also
applied after normalization; it displaces the ellipsoid away from the origin.
Bottom Right: The input features are transformed to a query representation (in
red) using the $W_{QK}$ matrix and compared against their previous
representation to obtain the self-attention scores.

It is important to mention that for $k<d$, the matrix $W^{i}_{QK}$ cannot be
inverted, as it won’t have a full rank. This implies (by the rank-nullity
theorem) that for each head, there must be a set of $d-k$ query vectors
$Q^{i}_{null}\subset\mathbb{R}^{d}$ that map to the zero vector and, as a
consequence, attend to all keys equally. Conversely, there must also exist a
set of $d-k$ keys $K^{i}_{null}\subset\mathbb{R}^{d}$ that are attended to by
all queries equally, with an attention score of zero.

Note on bias terms: In case the projection given by Equation 7 contains bias
terms $\beta_{q}$, $\beta_{k}\in\mathbb{R}^{1\times k}$, the attention score
matrix from Equation 12 is calculated as follows:

$A^{i}=\text{softmax}\Big{(}\frac{X^{i}_{q}X^{T}+XW^{i}_{Q}\beta_{k}^{T}+\beta_{q}{W^{i}_{k}}^{T}X^{T}+\beta_{q}\beta_{k}^{T}}{\sqrt{d}}\Big{)}$
(15)

In the bias formulation, three new terms are introduced. First,
$W^{i}_{Q}\beta_{k}^{T}\in\mathbb{R}^{d\times 1}$, which can be thought of as
a reference vector for queries, such that queries similar to it get higher
attention scores. Given that the same “bias score” will be broadcasted along
all the different keys of the same query, the network will ignore this term
due to the shift-invariance of the softmax function. More interesting is the
second term $\beta_{q}{W^{i}_{k}}^{T}\in\mathbb{R}^{1\times d}$, which acts as
a reference for keys. Given that its bias score is broadcasted along queries,
it will result in higher attention scores (in all queries) for keys similar to
the reference. Finally, the term $\beta_{q}\beta_{k}^{T}\in\mathbb{R}$ acts as
a global bias and, similar to $W^{i}_{Q}\beta_{k}^{T}$, will be ignored by the
network.

###  3.4 The $W_{VO}$ Matrix and the Residual Stream

To understand the role of the $W_{VO}$ matrix within the transformer, we now
consider the update step after the multi-head attention layer:

$X_{l+1}=X_{l}+\text{MultiHead}(\text{LayerNorm}(X))$ (16)

Note that by plugging in Equation 10 and Equation 12, the layer update can be
re-written as:

$X_{l+1}=X_{l}+\sum_{i}^{h}A^{i}X^{i}_{value}$ (17)

where

$X^{i}_{value}=\text{LayerNorm}(X_{l})W_{VO}^{i}$ (18)

It can be seen that the multi-head attention mechanism consists of the sum of
$h$ individual updates, each one given by one of the attention heads. Within
each head, all words in the sequence propose an update $X^{i}_{value}$, and
these are aggregated according to their attention scores $A^{i}$. In Equation
18, the matrix $W^{i}_{OV}$ acts as a map that takes the normalized inputs in
$\mathcal{H}_{S}$ (adjusting for scale and bias) and outputs a set of updates
in the same space as $W_{E}$, this process is visualised in Figure 4.
Furthermore, we propose that the $W^{i}_{VO}$ matrix is better understood as a
second key-value store [Sukhbaatar et al., 2015, Geva et al., 2021] within the
attention layer. To see why, consider its Singular Value Decomposition (SVD)
[Millidge and Black, 2022]:

$W^{i}_{VO}=U\Sigma V^{T}$ (19)

By substituting in Equation 18, we obtain:

$X^{i}_{value}=(Q_{VO}{K^{i}_{OV}}^{T})V^{i}_{OV}$ (20)

where

$\displaystyle\begin{split}Q_{VO}&=\text{LayerNorm}(X)\\\
K^{i}_{OV}&=(U\Sigma)^{T}\\\ V^{i}_{OV}&=V^{T}\end{split}$ (21)

The left singular vectors, associated with the columns of
$U\Sigma\in\mathbb{R}^{d\times d}$, act as a library of “keys” $K^{i}_{OV}$
against which the normalized features $X_{l}\in\mathcal{H}_{S}$ are compared.
While the corresponding right singular vectors, associated with rows in
$V^{T}\in\mathbb{R}^{d\times d}$, act as the output values $V^{i}_{OV}$ that
define the direction in which to update the information in the residual stream
for a given key. This interpretation is motivated by the results of Millidge
and Black [2022], where it is shown that the right singular vectors $V^{T}$ of
the $W_{VO}$ matrix tend to have interpretable meanings when decoded using
$W_{E}$, with some of the transformer heads consistently representing a single
topic in most of their singular vectors. We would also like to highlight that,
similar to the $W_{QK}$ matrix, the $W_{OV}$ matrix has at least $d-k$
singular values equal to zero. This means that multiple queries $Q_{VO}$ will
map to the zero vector and thus won’t update the information in the residual
stream, allowing the model to skip the update process if necessary.

Note on bias terms: If the value projection in Equation 7 contains a bias term
$\beta_{v}\in\mathbb{R}^{1\times k}$, and the output projection in Equation 8
contains a bias term $\beta_{o}\in\mathbb{R}^{1\times d}$. The layer update in
Equation 17 can be re-written as follows:

$X_{l+1}=X_{l}+\beta_{o}+\sum_{i}^{h}A^{i}X^{i}_{value}+\beta_{v}{W^{i}_{O}}^{T}$
(22)

Here, the term $\beta_{v}{W^{i}_{O}}^{T}\in\mathbb{R}^{1\times d}$ is a bias
on the update direction of head $i$, while $\beta_{o}\in\mathbb{R}^{1\times
d}$ acts as a bias on the entire layer’s update.

Figure 4:  Visualization of the residual update for a single attention head.
For each normalized data point (in gray), there is a corresponding un-
normalized data point in the residual stream (in blue). Data points in the
residual stream are updated according to a given direction (in green)
calculated from the self-attention scores and the update matrix $W_{VO}$.

###  3.5 The Feed Forward Module

We use the same interpretation for the feed-forward module as Geva et al.
[2022, 2021]. In it, the feed-forward module behaves similarly to the $W_{VO}$
matrix in the sense that it also acts as a key-value store [Geva et al., 2021]
that proposes directional updates for features in the residual stream [Geva et
al., 2022]. Similar to the previous section, we will begin by considering the
update step after the feed-forward layer:

$X_{l+1}=X_{l}+\text{FeedForward}(\text{LayerNorm(X)})$ (23)

where

$\text{FeedForward}(X)=f(XW_{1})W_{2}+\beta_{f}$ (24)

Geva et al. [2021] note that, for a hidden dimension $d_{hidden}$, the feed-
forward module weights $W_{1}\in\mathbb{R}^{d\times d_{hidden}}$ and
$W_{2}\in\mathbb{R}^{d_{hidden}\times d}$ act as key and value matrices, plus
a bias term $\beta_{f}\in\mathbb{R}^{1\times d}$. They propose an alternative
formulation of the feed-forward layer:

$\text{FeedForward}(X)=\sum_{i}^{d_{hidden}}f(Xk_{i})\cdot
v_{i}+\beta_{f}=\sum_{i}^{d_{hidden}}m_{i}\cdot v_{i}+\beta_{f}$ (25)

where

$\displaystyle\begin{split}k_{i}&=W_{in}^{T}[:,i]\quad\in\mathbb{R}^{d\times
1}\\\ v_{i}&=W_{out}[i,:]\quad\in\mathbb{R}^{1\times d}\end{split}$ (26)

Such that $W_{1}$ acts as a storage matrix for keys $k_{i}$, $W_{2}$ acts as a
storage matrix for values $v_{i}$, and the activation function $f$ assigns a
weight $m_{i}$ to each value $v_{i}$ depending on the input $X$. In [Geva et
al., 2021], the top-n examples in the training dataset that resulted in the
highest $m_{i}$ coefficients showed interpretable patterns, such that each key
$k_{i}$ in a 16-layer transformer model trained on WikiText-103 [Merity et
al., 2016] was associated with either a syntactical or semantical pattern by
human experts. For the values $v_{i}$, their role is better understood in
terms of the impact that they have on the residual stream (referred to as sub-
updates):

$X_{l+1}=X_{l}+\beta_{f}+\sum_{i}^{d_{hidden}}m_{i}\cdot v_{i}$ (27)

It can be seen that each value $v_{i}$ modifies the residual stream
independently, implying that these share the same space as $W_{E}$. Indeed,
experiments from Geva et al. [2021, 2022] have shown that many values $v_{i}$
are semantically meaningful and can be intervened for applications like zero-
shot toxic language suppression.

To conclude this subsection, we highlight that the attention and feed-forward
modules behave very similarly (see Equation 22 and Equation 27), as both
calculate relevance scores and aggregate sub-updates for the residual stream.
However, the way the scores and updates are calculated is very different. The
attention module relies primarily on dynamic context for its scores and
values, while the feed-forward module relies on static representations.

###  3.6 The Word Embedding Matrix $W_{E}$ and Output Probabilities

Once all the attention and feed-forward updates have been applied, the output
probabilities of the network can be obtained as follows Xiong et al. [2020]:

$p(Y)=\text{softmax}\big{(}\text{LayerNorm}(X_{L})W_{E}^{T}\big{)}$ (28)

In the case where layer normalization has no trainable parameters, Equation 28
can be interpreted as measuring the similarity between the final layer
representation $X_{L}$ when projected to $\mathcal{H}_{S}$, and each of the
embedding vectors in $W_{E}$. Given that all vectors in the projection have
the same norm $\sqrt{d}$, the only relevant factor in deciding the output
probability distribution $p(y^{t})\in[0,1]^{|V|}$, at a given timestep $t$, is
the location of its corresponding vector $x^{t}_{l}$ within $\mathcal{H}_{S}$.
This behavior is very similar to that described by the von Mises-Fisher
distribution [Fisher, 1953], as both represent distributions parameterized by
a reference vector within a hyper-sphere. Nonetheless, in the case of
transformers, the support of the distribution is defined over a discrete set
of vectors in $\mathbb{R}^{d}$, instead of $\mathcal{H}_{S}$ as a whole.

In the case the layer normalization includes scaling and bias parameters
$\gamma$ and $\beta$, the output probabilities are calculated as follows:

$p(Y)=\text{softmax}\big{(}\hat{X_{L}}\Gamma W_{E}^{T}+\beta W_{E}^{T}\big{)}$
(29)

where $\hat{X_{L}}$ is the projection of $X_{L}$ to $\mathcal{H}_{S}$ and
$\Gamma$ is a diagonal matrix such that $\Gamma_{ii}=\gamma_{i}$. The effect
of $\Gamma$ on the representation is that of transforming $\mathcal{H}_{S}$
into an ellipsoid (see Top Right section of Figure 3) while $\beta W_{E}^{T}$
acts as a bias that assigns higher probability to certain tokens independent
of the input.

In both cases (with and without bias and scale parameters), this perspective
aligns with that of iterative refinement [Jastrzebski et al., 2017] discussed
in nostalgebraist [2020], Elhage et al. [2021], Geva et al. [2022], Belrose et
al. [2023], given that intermediate representations $X_{l}$ can always be
converted into output probabilities using Equation 28.

To conclude this section, we would like to highlight that, by considering the
role of layer normalization and how it constrains the representation space, we
can get a geometric intuition behind iterative refinement. We provide a visual
interpretation of this concept in Figure 1.

##  4 Experiments

This section presents our experimental results. All experiments use pre-
trained weights from the 124M parameter version of GPT-2 [Radford et al.,
2019, Karpathy, 2023] unless stated otherwise. Code to replicate all
experiments is available at: https://github.com/santiag0m/traveling-words.

###  4.1 Impact of Layer Normalization on the Word Embeddings

To measure the impact of layer normalization on the position of the embedding
vectors in $W_{E}$, we calculated both the $\ell_{2}$ and cosine distances
between the layer-normalized weights and the following settings:

  * •

Original: The original word embeddings without any modification

  * •

Centered: Original + centering around the mean $\mathrm{E}[w_{e}]$

  * •

Scaled: Original divided by the average vector norm
$\mathrm{E}[||w_{e}||_{2}]$ and multiplied by $\sqrt{d}$

  * •

Centered + Scaled: Original + centering + scaling

Table 1: Distance between the normalized embeddings $\text{LayerNorm}(W_{E})$
and different transformations of the embedding matrix $W_{E}$.

Setting | Mean $\ell_{2}$ Distance | Mean Cosine Distance
---|---|---
Original | 23.747 (0.432) |  $<0.001$ ($<0.001$)
Centered | 24.872 (0.432) | 0.150 (0.035)
Scaled by $\sqrt{d}$ | 2.413 (1.862) |  $<0.001$ ($<0.001$)
Centered + Scaled by $\sqrt{d}$ | 14.591 (1.469) | 0.150 (0.035)

The results in Table 1 show that the mean cosine distance between the original
word embeddings and the embeddings after normalization is close to zero,
meaning that projection onto $\mathcal{H}_{S}$ does not modify the orientation
of the embedding vectors. The results also confirm this when centering is
applied, as the cosine distance increases significantly when the original
vectors are displaced from the origin towards the mean. On the other hand, it
can be seen that the $\ell_{2}$ distance is high for all settings except for
when scaling is applied without centering. Given an average norm of
$\mathrm{E}[w_{e}]=3.959$ and for $\sqrt{d}=27.713$ we can conclude that the
original word embeddings lie between the origin and $\mathcal{H}_{S}$ rather
than on its surface, with different embeddings having different norms.

Variance in the norm of embedding vectors is likely to be a result of the use
of the word embedding matrix as a classification layer later in the network
(see Equation 29). To verify whether this is the case, we select the top and
bottom 5 embedding vectors based on the three following criteria:

  * •

Norm: The norm of the original embedding vector in $W_{E}$

  * •

Scaled Norm: The norm of the embedding vector when scaled by the Layer Norm
parameter $\Gamma$

  * •

Norm + Bias: The norm of the original embedding vector plus the bias scores
obtained from $\beta W_{E}^{T}$

  * •

Scaled Norm + Bias: The sum between the Scaled Norm and the bias scores.

Table 2:  Top 5 and Bottom 5 tokens from the word embedding matrix. Tokens
were sorted according to the relevance of their corresponding embedding
vectors under different measurement settings.

Position | Norm | Scaled Norm | Norm + Bias | Scaled Norm + Bias
---|---|---|---|---
Top 1 | SPONSORED | \xa9\xb6\xe6 | , | the
Top 2 | \x96\x9a | tremend | the | ,
Top 3 | soDeliveryDate | \x96\x9a | . | and
Top 4 | enegger | senal | and | a
Top 5 | Reviewer | millenn | - | in
Bottom 5 | for | - | \xc0 | \x07
Bottom 4 | an | ( | \x07 | \x0f
Bottom 3 | on | “\n” | \x10 | oreAndOnline
Bottom 2 | in | , | \x11 | \x06
Bottom 1 | at | . | \xfe | \xc1

The sorted tokens in Table 2 show that considering only the norm of the
embeddings is not enough, as tokens that are not commonly used (like
‘SPONSORED’ and ‘soDeliveryDate’) have the highest norms, while common words
like ‘for’, ‘an’, ‘on’ and ‘in’ have the lowest norm. After considering the
scaling parameter $\Gamma$, we observe that punctuation signs like the newline
character or the comma ‘,’ have the lowest norm, and that there is no clear
pattern on the top tokens. After considering bias, we see that the
distribution of top tokens clearly shifts, with punctuation symbols and common
words at the top and uncommon bytes at the bottom. Finally, note that when
both scale and bias are considered, the top tokens are consistent with some of
the most common words in the English language: ‘the’, ‘and’, ‘a’ and ‘in’ with
the only exception being the comma character, which is also very common in
natural language, while the bottom tokens are related to uncommon bytes and an
anomalous token.

###  4.2 Probing Attention Heads with Normalized Representations of Common
Nouns

Next, we use the interpretation from subsection 3.3 and 3.4 to probe the
attention heads at layers 0, 5 and 11 of the GPT-2 model using as inputs the
100 most common nouns taken from the Corpus of Contemporary American English
(COCA) [Davies, 2010]. First, we transform the embedding matrix $W_{E}$
according to the normalization parameters specific to each layer (see Figure
3) and then multiply the normalized embeddings $W_{E}^{norm}$ by either
$W_{QK}$ or $W_{VO}$.

Then, we perform decoding steps specific to each matrix after multiplication:

  * •

For $W_{QK}$, we retrieve the top-k closest embedding vectors from
$W_{E}^{norm}$ based on dot product similarity.

  * •

For $W_{VO}$, we add the head-specific and layer-specific output biases (see
Equation 22) to obtain the “update vectors”. These update vectors are then
added to the original embeddings from $W_{E}$ and transformed according to the
normalization parameters from the last layer; then, we retrieve the top-k
closest embeddings from the original $W_{E}$ based on dot product similarity.

####  4.2.1 Query-Key Transformations

In Table 3, we present the results for the Query-Key transformations at layer
0 given the query inputs ‘time’, ‘life’ and ‘world’. We note that some of the
heads preserve the meaning of the query, as is the case for heads 1, 5 and 10,
possibly looking for repetition, while others look for keys that precede it.
Such precedence heads might help to disambiguate the meaning of the words,
with examples like: ‘Showtime’ vs. ‘spacetime’, ‘battery life’ vs. ‘wildlife’
and ‘underworld’ vs. ‘Westworld’. Other heads appear to be looking for
contextual associations, as is the case for head 2, which seems to relate
‘world’ with dates and concepts from the First and Second World wars. When
looking at deeper layers (as shown in Table A.1 & A.2), we were not able to
identify any meaningful patterns on the query transformations, suggesting that
these layers might look for more complex patterns.

Table 3: Transformation of Queries Across Transformer Heads at Layer 0

|  Query $\rightarrow$ Keys
---|---
Head | time | life | world
0 | Level, [?], offenders | battery, Battery, Battery | legraph, Vers, Malf
1 | time, time, Time | Life, life, life | World, world, world
2 | cinematic, Priest, priest | Notre, fetal, abortion | 1914, Churchill, 1916
3 | space, lunch, mid | augh, ertain, ough | under, Nether, Fort
4 | soft, heavy, tool | Middle, Hans, Middle | ether, Unt, Know
5 | time, time, Time | life, Life, Life | world, World, world
6 | Rated, chirop, u | Fukushima, chirop, ulic | ipt, u, Meta
7 | Show, bed, Movie | pro, wild, Wild | Disc, West, West
8 | java, framework, watch | shark, sharks, Wild | edit, ”$:/, movie
9 | stones, pal, cards | Trojan, malware, Wi | Rogers, COUNTY, Rd
10 | time, time, Time | life, life, Life | world, world, World
11 | Wine, a, food | PHI, everal, Span | agus, true, ‘,’

####  4.2.2 Key-Value Transformations

In Table 4, we present the results for the Key-Value transformations for the
same three inputs. For most heads at layer 0, the meaning of the input key is
kept as is. However, when the sum of all the heads is considered, we see a
slight shift in the meaning of the words.

For heads at layer 5 (shown in Table A.3), we see that although most of the
heads preserve the meaning of the input keys ‘life’ and ‘world’ (and around
half of the heads for the input ‘time’), the sum of all heads does change the
word meaning dramatically, and without a clear output pattern. As our
experiment is limited to testing a single input key at a time, it might be
possible that updates in this layer rely more heavily on the composition
between multiple keys, which we did not capture.

Finally, in the last layer (Table A.4), we see that most individual heads map
to seemingly arbitrary values, with only a few preserving the meaning of the
input key. However, when the sum of the heads is considered, the layer
preserves the meaning of the input keys. To test the hypothesis that meaning-
preserving heads dominated the layer update, we measured the norm of the
output values for each head (before adding the layer-specific bias
$\beta_{o}$). We found that, in most cases, these heads do not have higher
norms. Instead, heads promoting common tokens like ‘the’, ‘,’ and ‘and’ had
the highest norms. These results suggest that contrary to our hypothesis, the
heads at the last layer work together to preserve the meaning of the input
keys and mitigate the network’s bias towards common tokens.

Table 4: Transformation of Keys Across Transformer Heads at Layer 0

|  Key $\rightarrow$ Values
---|---
Head | time | life | world
0 | time, Time, time | life, choice, senal | world, World, worlds
1 | time, TIME, time | life, lihood, life | world, Goes, ship
2 | time, [?], Minutes | life, Life, life | world, world, World
3 | time, Time, theless | life, Life, life | world, World, worlds
4 | time, time, Time | life, Life, Life | world, World, world
5 | time, Time, Time | life, Life, Life | world, World, worlds
6 | time, time, Time | life, life, Life | world, world, Feather
7 | time, eless, times | life, Experience, Life | world, World, Abyss
8 | time, iversary, melodies | life, challeng, conservancy | world, worlds, droid
9 | time, time, recall | [?], local, Main | [?], world, local
10 | equivalents, igation, planes | life, ento, planner | world, ento, Tanzania
11 | time, Time, Time | life, Life, +++ | world, World, Trials
Sum | time, etime, watch | Indigo, life, crew | world, Unleashed, World

###  4.3 Singular Value Decomposition of the $W_{VO}$ matrix

To verify whether the key-value interpretation of $W_{VO}$ matrix proposed in
subsection 3.4 is correct, we probe each of its singular vectors (as proposed
in Millidge and Black [2022]). For the left singular vectors $U$ (scaled by
$\Sigma$), we use the normalized embeddings $W_{E}^{norm}$ as a probe, while
for the right singular vectors $V^{T}$, we use the original embeddings
$W_{E}$. Given that all singular values are constrained to be positive, we get
two possible singular vector pairs corresponding to each singular value:
$(u,v)$ and $(-u,-v)$. For ease of analysis, we choose the signed pair with
its $v$ component closest to any of the embeddings $w_{e}\in W_{E}$, using the
dot product similarity.

We did not observe any interpretable pattern for the attention heads at layer
0 and found only one interpretable head at layer 5 (head 10), which referred
to terms in politics and chemistry. However, we found that most heads in layer
11 were interpretable (except for heads 5, 7 and 9) and present the results
for all heads in Appendix B. An illustrative case of these patterns is head 3,
where most of its singular vector mappings are related to jobs or industries.
For example, ‘Dairy’ maps to ‘USDA’ (the United States Department of
Agriculture), ‘engine’ to ‘drivers’, ‘trading’ to ‘Sales’ and so on. Similar
patterns were present in other heads, listed as follows:

  * •

Head 0: Formatting and punctuation symbols (end of text, new line, brackets
and parenthesis)

  * •

Head 1: Gender words

  * •

Head 2: Proper Nouns (Places)

  * •

Head 3: Jobs / Industries

  * •

Head 4: Letters and Numbers

  * •

Head 6: Suffixes and Prefixes related to the ending and beginning of words

  * •

Head 8: Punctuation symbols

  * •

Head 10: Proper Nouns (First and Last names)

  * •

Head 11: The identity function (input similar to the output)

We found that these patterns were consistent with those obtained in the “Key
$\rightarrow$ Value” results from Table A.4, implying that the subject-
specific behavior of the singular vectors is reflected in the input-output
transformations of the attention heads. These results complement previous work
from Millidge and Black [2022], in which only the right singular vectors
$V^{T}$ were considered.

#### SVD of the $W_{QK}$ matrix

In additional experiments on the SVD of the $W_{QK}$ matrix, we found that
some singular vector pairs had clear associations. For example, in head 0 of
layer 0, we found some associations related to programming languages (‘self,
class, =, import’ $\rightarrow$ ‘Python’) and digital cameras (‘Video, 264,
minutes’ $\rightarrow$ ‘Nikon, lineup, shot, camera’) but we could not
identify any specialization for the heads. Surprisingly, we did find that
heads at the last layer had identifiable patterns on their left singular
vectors (associated with the queries) consistent with those listed for the
$W_{VO}$ matrix (punctuation for head 0, gender for head 1, and so on), but no
clear patterns were identified for the right singular vectors.

###  4.4 Visualizing Iterative Refinement

Finally, we visualize how the information in the residual stream is updated
(i.e. the iterative refinement process) leveraging dimensionality reduction
techniques, as shown in Figure 5. For this, we chose the test sentence ‘To
kill two birds with one stone’, as the predictability of its last token,
‘stone’, given the previous context was high (correctly predicted by the
model) and none of the words in the sentence repeated. To project the high
dimensional embeddings into 3D space, we used UMAP [McInnes et al., 2018],
with Laplacian Eigenmap initialization [Belkin and Niyogi, 2001, Kobak and
Linderman, 2021], and we fit the transform using the first 10,000 embedding
vectors from $W_{E}$ to accurately reflect proximity in the original embedding
space. We show the original embedding tokens as reference (in blue) and plot
the trajectory of the second-to-last token, ‘one’, as we process the entire
sequence (with added positional embeddings) throughout the network. For each
layer, we transform the latent representations in the residual stream using
the normalization parameters from the final output layer before projecting
with UMAP. It can be seen that the representation of the second-to-last token
shifts from its original meaning (‘one’) towards the meaning of the next token
(‘stone’). Although the figure also shows the magnitude and direction of each
update in the trajectory, it is important to mention that these quantities
might have been modified due to the dimensionality reduction process.

Figure 5:  UMAP 3D projection of the phrase ‘To kill two birds with one
stone’. The original word embeddings are in blue, the final latent
representation for the second-to-last token (‘one’) in purple, and its
trajectory in red, with each trajectory segment representing an update between
transformer blocks. Note that the latent representation starts close to its
corresponding embedding, ‘one’, and gets closer to that of the next token,
‘stone’, with each update.

##  5 Conclusion

We have presented a new interpretation of transformer models based on the
geometric intuition behind each component and how all these components come
together as the transformation of the meaning of one input token to the next.

First, we showed how layer normalization can be better understood as a
projection of latent features in $\mathbb{R}^{d}$ to a $(d-1)$-dimensional
hyper-sphere and provide experimental evidence that the word embeddings
learned by GPT-2 are distributed toward different directions of the hyper-
sphere. We also showed that the parameters of the final normalization layer
are crucial to obtain high-scoring tokens consistent with high-frequency
tokens in the English language.

Next, we discussed the role of the $W_{QK}$ and $W_{VO}$ matrices as
transformations related to the hyper-sphere, with $W_{QK}$ as an affine
transformation that overlaps queries and keys, and $W_{VO}$ as a key-value map
between the hyper-sphere and the original embedding space. These intuitions
were tested with probing experiments, showing promising results in
understanding the role of query-key attention in earlier layers and extending
the results from Millidge and Black [2022] on the subject-specific nature of
the $W_{VO}$ matrix in attention heads at deeper layers.

Finally, we integrated these ideas and the impact of each component on the
residual stream to provide visual evidence on how the iterative refinement
process works within transformers.

## References

  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Thoppilan et al. [2022] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ , 2022.
  * Bai et al. [2022] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_ , 2022.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report.  OpenAI, 2023.  URL https://cdn.openai.com/papers/gpt-4.pdf.
  * Google [2023] Google.  Palm 2 technical report.  Google AI, 2023.  URL https://ai.google/static/documents/palm2techreport.pdf.
  * Chen et al. [2023] Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, et al.  Symbolic discovery of optimization algorithms.  _arXiv preprint arXiv:2302.06675_ , 2023.
  * Zong et al. [2022] Zhuofan Zong, Guanglu Song, and Yu Liu.  Detrs with collaborative hybrid assignments training.  _arXiv preprint arXiv:2211.12860_ , 2022.
  * Driess et al. [2023] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023.
  * Dar et al. [2022] Guy Dar, Mor Geva, Ankit Gupta, and Jonathan Berant.  Analyzing transformers in embedding space.  _arXiv preprint arXiv:2209.02535_ , 2022.
  * Elhage et al. [2021] Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al.  A mathematical framework for transformer circuits.  _Transformer Circuits Thread_ , 1, 2021.
  * Geva et al. [2022] Mor Geva, Avi Caciularu, Kevin Wang, and Yoav Goldberg.  Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 30–45, 2022.
  * Brody et al. [2023] Shaked Brody, Uri Alon, and Eran Yahav.  On the expressivity role of layernorm in transformers’ attention.  _arXiv preprint arXiv:2305.02582_ , 2023.
  * Windsor [2022] Eric Windsor.  Re-examining layernorm.  https://www.lesswrong.com/posts/jfG6vdJZCwTQmG7kb/re-examining-layernorm, 2022.
  * Millidge and Black [2022] Beren Millidge and Sid Black.  The singular value decompositions of transformer weight matrices are highly interpretable.  https://www.lesswrong.com/posts/mkbGjzxD8d8XqKHzA/the-singular-value-decompositions-of-transformer-weight, 2022.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  _arXiv preprint arXiv:1607.06450_ , 2016.
  * Geva et al. [2021] Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy.  Transformer feed-forward layers are key-value memories.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 5484–5495, 2021.
  * Xiong et al. [2020] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu.  On layer normalization in the transformer architecture.  In _International Conference on Machine Learning_ , pages 10524–10533. PMLR, 2020.
  * Sukhbaatar et al. [2015] Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al.  End-to-end memory networks.  _Advances in neural information processing systems_ , 28, 2015.
  * Merity et al. [2016] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  In _International Conference on Learning Representations_ , 2016.
  * Fisher [1953] Ronald Aylmer Fisher.  Dispersion on a sphere.  _Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences_ , 217(1130):295–305, 1953.
  * Jastrzebski et al. [2017] Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio.  Residual connections encourage iterative inference.  _arXiv preprint arXiv:1710.04773_ , 2017.
  * nostalgebraist [2020] nostalgebraist.  Interpreting gpt: The logit lens.  https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens, 2020.
  * Belrose et al. [2023] Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt.  Eliciting latent predictions from transformers with the tuned lens.  _arXiv preprint arXiv:2303.08112_ , 2023.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Karpathy [2023] Andrej Karpathy.  Github - karpathy/nanogpt: The simplest, fastest repository for training/finetuning medium-sized gpts., 2023.  URL https://github.com/karpathy/nanoGPT.
  * Davies [2010] Mark Davies.  The corpus of contemporary american english as the first reliable monitor corpus of english.  _Literary and linguistic computing_ , 25(4):447–464, 2010.
  * McInnes et al. [2018] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_ , 2018.
  * Belkin and Niyogi [2001] Mikhail Belkin and Partha Niyogi.  Laplacian eigenmaps and spectral techniques for embedding and clustering.  _Advances in neural information processing systems_ , 14, 2001.
  * Kobak and Linderman [2021] Dmitry Kobak and George C Linderman.  Initialization is critical for preserving global data structure in both t-sne and umap.  _Nature biotechnology_ , 39(2):156–157, 2021.

##  Appendix A Attention Head Transformations for Layers 5 and 11

###  A.1 Query-Key Transformations

Table A.1: Transformation of Queries Across Transformer Heads at Layer 5

Query $\rightarrow$ Keys Head time life world 0 depend, annot, reason so, inf,
char Lab, dev, Dev 1 they, themselves, Vers they, Im, depend come, once, haven
2 Nepal, ”:[”, —” ‘….’, ‘…’, Home posted, Logged, ideologically 3 appeared,
actually, had posted, axle, .avascript aryl, Ala, GA 4 attract, CP, contained
misconception, (?, trophy separatists, activists, extremists 5 Plum, rice,
Vers Sniper, too, hides Prim, Bright, am 6 en, annually, – following,
Generator, Library §§, tournaments, StarCraft 7 Wis, def, individual y, ier,
od Af, Gh, agle 8 condition, intensive, inf prol, operation, splend Ard,
marketplace, dev 9 post, market, destinations She, steal, etc strategy, pd,
budget 10 jugg, continuously, Center essim, enter, tast exploration, jugg,
PLAY 11 straight, interview, fucking –, Eva, related spotlight, television, TV

Table A.2: Transformation of Queries Across Transformer Heads at Layer 11

Query $\rightarrow$ Keys Head time life world 0 UNCLASSIFIED, opausal, ster
opausal, backstage, piece routine, cat, ocular 1 assion, upp, pir pir, Virgin,
appa Frontier, theater, onies 2 heid, GI, rict heid, apy, brance region, urgy,
encyclopedia 3 opic, href, Hitchcock susceptibility, space, opic league,
space, opic 4 gy, lots, whatever his, whichever, gy whichever, whatever,
underworld 5 olesterol, tx, erc olesterol, avy, iana wealth, Digest, Market 6
ones, volatile, RIS volatile, olesterol, idency wealth, useum, theatre 7
whichever, ivalent, lower mortal, whichever, living -$, complex, world 8 ove,
HTTP, metaphysical spiritual, metaphysical, bio Endless, metaphysical, Marvel
9 Productions, actic, fare stuff, ience, Productions entertainment, stuff,
World 10 -, code, ing -, core, ola core, Labs, ourse 11 emb, ivan, Union Tour,
etc, iona pires, si, Tour

###  A.2 Key-Value Transformations

Table A.3: Transformation of Keys Across Transformer Heads at Layer 5

Key $\rightarrow$ Values Head time life world 0 BuyableInstoreAndOnline, [?],
time life, advertising, Life world, opathy, qus 1 MON, Sophia, time mallow,
cause, unn world, Cav, fect 2 time, qualified, understatement life, life, Life
world, World, auri 3 )?, ¿), ?’ \\] $=>$, life, \\\” $>$ world, \\] $=>$, %” 4
time, TIME, Sabha Izan, eworld, ieu world, izons, orld 5 destro, time, rall
life, Life, agre world, toget, enthusi 6 time, time, TIME life, Life, life
world, World, WORLD 7 time, corrid, patch life, Life, life world, mathemat,
redes 8 NetMessage, [?], ibu venge, idth, aten ULTS, Magikarp, [?] 9 [?], [?],
amina raviolet, los, SPONSORED Kraft, quickShipAvailable, Berks 10 time,
contrace, Symphony life, Life, life world, World, worlds 11 otle, ide, Ide
framing, plot, plots ittee, rf, pawn Sum externalActionCode, ]), issance ahon,
awa, ]” Magikarp, Hig, ETHOD

Table A.4: Transformation of Keys Across Transformer Heads at Block 11

Key $\rightarrow$ Values Head time life world 0 \n, ”, “ {, ¿, ”# [, [* ,[ 1
player, party, Party youth, House, Youth party, Trump, party 2 Lisp, Ö, ¨ [?],
Quincy, Yemen Scotland, Osborne, Scotland 3 Weather, cinem, weather life,
euth, Life world, Worlds, geop 4 b, k, 2 inav, d, 4 i, V, Rivals 5 Part, Show,
part Well, Well, saw sees, works, View 6 Sub, AM, BR West, West, East Sub,
Under, ob 7 Journal, Air, Online home, Home, house home, Home, internet 8 ‘,’,
the, and ‘,’, the, and the, ‘,’, and 9 interaction, impression, experience
encounter, belief, encounters reservations, Illusion, illusions 10 time, TIME,
Time life, life, LIFE world, world, worlds 11 time, time, Time life, LIFE,
life world, oy, door Sum time, Time, time life, Life, Life world, Worlds,
worlds

##  Appendix B $W_{VO}$ SVD per Head for Layer 11

Table B.5: Left and Right Singular Vectors at Layer 11 - Head 0

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  shenan, cryst, encount |  DragonMagazine, ertodd, soDeliveryDate
1 |  another, Iv, sil |  trave, BuyableInstoreAndOnline, convol
2 |  Sebastian, Luke, humankind |  quickShipAvailable, EStream, MpServer
3 |  rans, thereby, hem |  BuyableInstoreAndOnline, acknow, Buyable
4 |  sectional, [+], Winged |  ThumbnailImage, \ufffd\ufffd\u58eb, Orderable
5 |  abl, isc, Ah |  etheless, olson, llah
6 |  <|endoftext|>, Advertisements, cest |  <|endoftext|>, Advertisements, kindred
7 |  ococ, ilan, guest |  pard, MBA, uid
8 |  ]., ],, ]; |  [, [*, [
9 |  \n\n, ),, cakes |  \n\n, Quote, Quote
10 |  snaps, Bills, Texans |  lineback, Chargers, Packers
11 |  )…, …), ).” |  (\u00a3, (, (?,
12 |  pen, cle, Orioles |  .””, [, .”)
13 |  ](, drm, Updated |  \n\n, [/, [/
14 |  RBI, Field, Triple |  RHP, RBI, Negro
15 |  pod, illus, Maple |  ipeg, aboriginal, ”\u2026
16 |  am, ’m, hearted |  SPONSORED, Newsletter, ….
17 |  Document, whit, Scott |  SPONSORED, tsky, Ras
18 |  gen, idd, anned |  Ukrain, prin, rul
19 |  Ryder, icz, abet |  istries, plet, Gad

Table B.6: Left and Right Singular Vectors at Layer 11 - Head 1

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  Customers, However, Customer |  \u899a\u9192, natureconservancy, racuse
1 |  mint, Anne, Marie |  hers, actress, Denise
2 |  ook, Child, ooks |  parents, Parents, Children
3 |  gow, abad, BEL |  boy, student, Guy
4 |  eries, girl, girls |  Girl, girl, Queen
5 |  Marie, Sue, Patricia |  Woman, woman, woman
6 |  Him, les, LCS |  Person, Persons, Person
7 |  ndra, Joint, rity |  Her, Her, femin
8 |  Coach, recapt, Players |  Players, Coach, coaches
9 |  istries, WAYS, INAL |  god, Allaah, God
10 |  Ens, offspring, statute |  male, males, Woman
11 |  Junction, hole, Abdullah |  girl, daddy, Neighbor
12 |  HR, ig, akings |  Major, Major, minors
13 |  reunion, Madison, mes |  boys, males, Girls
14 |  asting, uba, ynt |  mom, moms, Jim
15 |  ately, ynam, OUS |  doctoral, apprentice, Child
16 |  ifier, Come, Weekly |  class, owners, Class
17 |  Confederation, ATE, ingredient |  Students, Students, Ms
18 |  athon, jen, candidates |  Candidate, candidate, traveler
19 |  Pres, ently, Secure |  character, Characters, Character

Table B.7: Left and Right Singular Vectors at Layer 11 - Head 2

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  orpor, rul, Bolivia |  Adelaide, Edmonton, Calgary
1 |  ball, ERY, hem |  Filipino, Ultron, ANC
2 |  \u30f3\u30b8, else, Lib |  Ruby, Scarborough, Erit
3 |  verb, Lamar, Ankara |  Detroit, Detroit, Wenger
4 |  iana, amacare, edia |  Zoro, Shelby, Tehran
5 |  Gw, otle, Rangers |  \u00ed, Jinn, Texans
6 |  ration, Rim, ially |  Yang, McCain, Yang
7 |  detector, OTOS, Petersen |  Chilean, Pharaoh, ffen
8 |  ald, benefit, ahon |  Petersburg, Henderson, Kessler
9 |  scope, whe, verse |  acio, Mits, Jacobs
10 |  Gators, Laden, SEAL |  Malfoy, Swanson, Romney
11 |  Lilly, \u00e9t, lla |  Greenwood, Collins, Byrne
12 |  ister, ority, isters |  Niagara, Maharashtra, soDeliveryDate
13 |  Paulo, nesota, Clayton |  Loki, \u011f, Finnish
14 |  creen, Cron, Base |  Pike, Krishna, Satoshi
15 |  lake, SP, seeing |  Alberta, Arlington, McKin
16 |  Bowie, ystem, rey |  Bowie, Murray, Utah
17 |  head, ding, ressed |  Bulgar, Warcraft, Crimean
18 |  Venom, elman, lyn |  SJ, Brit, Gordon
19 |  wright, ansas, arta |  NXT, Metroid, Aether

Table B.8: Left and Right Singular Vectors at Layer 11 - Head 3

Rank |  Left Words |  Right Words
---|---|---
0 |  suburbs, restaur, \ufffd |  DragonMagazine, BuyableInstoreAndOnline, \ufffd\u9192
1 |  Dairy, farm, Veget |  USDA, Dairy, cows
2 |  engine, drivers, Motor |  Drivers, drivers, driver
3 |  trading, trade, shoppers |  Sales, retailers, shoppers
4 |  instrument, musical, guitar |  Billboard, halftime, Grammy
5 |  sail, boat, sailing |  sail, sailing, autical
6 |  teachers, teacher, school |  teachers, uberty, curric
7 |  baker, kindergarten, bakery |  baker, SERV, kindergarten
8 |  apparel, prison, recruiting |  Sail, Prison, jail
9 |  shelter, indoors, shelters |  shelters, shelter, Radiant
10 |  tribe, fish, fish |  dred, whales, fisheries
11 |  workers, jobs, job |  workers, worker, subcontract
12 |  Derrick, tribe, Tribal |  Seg, forest, Derrick
13 |  chess, Chess, seating |  Chess, chess, Sheldon
14 |  Soy, Satellite, astronauts |  Soy, Satellite, transmissions
15 |  Anim, visa, Imm |  exhib, Anim, Imm
16 |  medicine, diagnose, doctors |  Doctors, hospital, doctor
17 |  boxing, trainer, spar |  boxing, spar, UFC
18 |  gun, firearm, Sheriff |  ITV, Decoder, Geral
19 |  gambling, tournaments, tournament |  gambling, Gaming, tournaments

Table B.9: Left and Right Singular Vectors at Layer 11 - Head 4

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  them, their, him |  cloneembedreportprint, \u899a\u9192, \u30b5\u30fc\u30c6\u30a3
1 |  iator, ive, ibur |  natureconservancy, Canaver, \u25fc
2 |  if, born, forces |  the, ., ,
3 |  ually, ,., therein |  Buyable, misunder, lehem
4 |  irk, struct, actly |  1, 2, 9
5 |  uku, handle, eenth |  nineteen, seventeen, seventy
6 |  ensional, insk, ploy |  M, M, m
7 |  allowance, \u2605, ther |  ii, Bs, B
8 |  ylon, works, plays |  EDITION, o\u011f, nt
9 |  ysc, oreal, Friend |  B, K, B
10 |  redits, rossover, ameron |  F, K, k
11 |  Tiger, urses, aught |  N, W, C
12 |  aughter, gling, eland |  L, l, L
13 |  othe, cano, ensity |  S, s, S
14 |  ISTORY, hum, pots |  H, H, h
15 |  gers, iegel, ki |  S, s, S
16 |  ya, seq, est |  selves, T, i
17 |  tl, ictionary, latch |  R, R, D
18 |  Fres, pine, delay |  R, u, llah
19 |  Shades, went, culosis |  G, G, S

Table B.10: Left and Right Singular Vectors at Layer 11 - Head 5

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  assail, challeng, achie |  ertodd, \u25fc, \ufffd\u9192
1 |  WARE, padding, req |  \u9f8d\u5951\u58eb, StreamerBot, soDeliveryDate
2 |  uing, anche, Inquis |  heit, MpServer, partName
3 |  ward, ops, actory |  builds, projects, Building
4 |  ary, bell, vis |  ouf, unt, article
5 |  ments, Poo, emo |  Will, Will, terday
6 |  abdom, book, Til |  reads, read, writing
7 |  admission, Fighters, agy |  model, Models, ilib
8 |  line, lines, se |  line, lines, Hold
9 |  iness, less, ood |  udic, ridden, usky
10 |  absence, inar, Miko |  place, Must, must
11 |  hawk, nect, aff |  esson, sees, scene
12 |  ie, een, ennett |  Say, ighting, features
13 |  Peaks, construed, anguages |  finding, find, Find
14 |  ming, mers, pling |  ufact, Put, say
15 |  Authority, urated, disregard |  record, records, Record
16 |  cript, Seen, Crash |  Written, course, arium
17 |  ually, gladly, ously |  tions, show, find
18 |  im, ading, Expand |  image, Image, Image
19 |  NX, W, ees |  swer, \u30c7\u30a3, report

Table B.11: Left and Right Singular Vectors at Layer 11 - Head 6

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  issue, txt, Princ |  isSpecialOrderable, DragonMagazine, \ufffd\ufffd
1 |  mes, same, resa |  guiActiveUn, Yanuk, Beir
2 |  eatured, avier, AMES |  quickShipAvailable, BuyableInstoreAndOnline, RH
3 |  Levine, estone, Bronx |  skirts, Els, Bris
4 |  lder, xit, Sav |  Sov, grap, Al
5 |  xual, ss, soup |  Orient, owship, toile
6 |  rices, glers, lishing |  Uni, Tik, en
7 |  imation, hammer, nels |  BAD, Ze, sa
8 |  saturated, lying, Past |  Ry, AG, Val
9 |  activity, ozy, oko |  Ay, AW, Ay
10 |  ows, aghan, ergy |  Gul, cl, Nex
11 |  yrs, ish, hood |  Wh, Har, Mart
12 |  omp, grandmother, MS |  sidx, Alb, CTR
13 |  ses, ski, doctor |  AD, ython, Ty
14 |  heed, Monthly, angan |  OPS, Tur, Tam
15 |  Agency, VP, lex |  Red, Grey, Redd
16 |  FORE, sil, hing |  wcsstore, uci, Winged
17 |  idences, ining, ahl |  Ste, Pend, hal
18 |  iance, taxpayers, anches |  Fuj, appl, Zamb
19 |  ischer, apo, hiatus |  Zamb, Zer, Nek

Table B.12: Left and Right Singular Vectors at Layer 11 - Head 7

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  shortest, ses, mentally |  iHUD, DragonMagazine, Downloadha
1 |  our, ourselves, we |  ourselves, ours, our
2 |  himself, lements, them |  \u899a\u9192, natureconservancy, ertodd
3 |  etitive, EStream, workshop |  FTWARE, SourceFile, \ufffd\u9192
4 |  \u0627\u0644, holders, mileage |  your, Free, Your
5 |  am, ’m, myself |  my, myself, me
6 |  themselves, auder, ighthouse |  Companies, theirs, THEIR
7 |  stract, hop, \u00a2 |  soDeliveryDate, Civil, civilian
8 |  bage, ros, hyster |  bage, aukee, Free
9 |  shop, acter, Shop |  Humans, ourning, electronically
10 |  ¡+, myself, pse |  my, myself, markets
11 |  Hold, SE, istant |  ilage, roups, usra
12 |  uffs, VG, GG |  verty, Leilan, Soft
13 |  sters, ual, ted |  machine, machine, business
14 |  making, weights, mare |  centrif, istani, culture
15 |  uador, oust, ertain |  us, ours, our
16 |  vable, cam, ophy |  system, System, systems
17 |  exch, velength, un |  Games, abeth, gaming
18 |  latex, Edwards, Conway |  Commercial, Community, community
19 |  ificial, rating, nces |  ificial, System, technology

Table B.13: Left and Right Singular Vectors at Layer 11 - Head 8

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  the, in, a |  \ufffd\ufffd\ufffd, guiActiveUn, cloneembedreportprint
1 |  *., ., determin |  ,, the, -
2 |  and, ,, Un |  arnaev, DragonMagazine, BuyableInstoreAndOnline
3 |  ?”, ?), ?), |  ?’”, TPPStreamerBot, ’,”
4 |  ,’”, GIF, ,” |  Orderable, \ufffd, \ufffd
5 |  .’”, They, .’ |  .’”, ’.”, ).”
6 |  ,’, \u2010, ,’” |  ,’”, ’,, ,’
7 |  ,’, ,’”, ’,’ |  ,’”, ,’, ’,”
8 |  .’, ,’, ’. |  .’, ,’, ’.
9 |  her, she, She |  she, hers, her
10 |  \ufffd, \ufffd, “ |  \ufffd, \ufffd, “
11 |  ”…, ”,”, ”, |  )”,, ),”, ”),
12 |  ).”, ”)., …” |  ).”, .”[, ”).
13 |  us, ),”, our |  ),”, ).”, .”)
14 |  ));, );, ), |  ”,, ’,, ));
15 |  ]., ];, ], |  };, ];, ’;
16 |  …, …”, … |  …], :], …”
17 |  ?], !], .] |  \u2026], !], ?]
18 |  ();, her, He |  hers, ();, His
19 |  \u00ad, \u300f, You |  \u300f, ¿., \u00ad

Table B.14: Left and Right Singular Vectors at Layer 11 - Head 9

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  esthes, Eat, pts |  DragonMagazine, Canaver, natureconservancy
1 |  ups, motors, hinted |  confir, \ufffd, unlaw
2 |  pursue, pursuit, Frie |  ticket, Desire, iferation
3 |  posted, dates, rece |  achievement, unlocking, Hilbert
4 |  differential, prise, ushing |  acceptance, handled, accepting
5 |  Hide, etsu, LET |  optimizations, prioritize, emphasized
6 |  ously, uffer, ca |  opsis, \u30df, stall
7 |  ann, Horn, Specifications |  restraint, notice, surprises
8 |  supremacy, argon, ifier |  ACTIONS, Contin, rue
9 |  ling, ceived, inf |  errors, misunderstanding, accuracy
10 |  ittal, ampton, feld |  denotes, denote, hazard
11 |  inf, andy, ery |  plagiar, mentors, recommending
12 |  Soon, \ufffd, \ufffd |  lax, Talks, Fell
13 |  cia, war, Fighters |  dissatisf, consum, dissatisfaction
14 |  NAS, Schwar, Streamer |  delet, sidx, inem
15 |  Glory, uan, ment |  Reviewed, Congratulations, congratulations
16 |  frey, clay, essional |  quirks, Integration, distinguishing
17 |  uck, marked, Request |  appreciation, Guidelines, guidelines
18 |  prints, forcefully, Cel |  conviction, convictions, impressions
19 |  utic, endez, inging |  disag, bruising, spo

Table B.15: Left and Right Singular Vectors at Layer 11 - Head 10

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  above, former, dm |  \u25fc, Downloadha, Canaver
1 |  Cohen, oku, Corporation |  be, ache, the
2 |  liar, Ross, Irving |  Rossi, Mind, Zen
3 |  Treatment, MT, tubing |  etts, Taylor, Tan
4 |  Torch, dt, Honour |  Divinity, marqu, vine
5 |  ==, Sinn, imitation |  Stafford, Bradford, Halo
6 |  asks, fitted, caution |  BW, BW, Berger
7 |  encer, hero, success |  Gon, Johnny, PATH
8 |  Chung, anke, IRE |  Chennai, Carey, Carmen
9 |  Commodore, iom, attract |  curry, Cunningham, clam
10 |  earth, CS, oyal |  Sov, Trin, paralle
11 |  ramid, el, DIT |  Hilton, diarr, \ufffd\u9192
12 |  ulla, alde, uality |  McInt, alde, Idle
13 |  cam, write, ports |  Cave, Chal, Connie
14 |  buf, anne, Emin |  Dwar, Dwarf, Das
15 |  job, play, job |  buquerque, Liber, reb
16 |  ASC, ector, Order |  Sorceress, Alic, Astro
17 |  ting, enced, te |  Forest, Kan, tree
18 |  ater, Turner, UAL |  \u9f8d\ufffd, Omn, Gamma
19 |  Matrix, RIP, oping |  Fed, STEP, Rand

Table B.16: Left and Right Singular Vectors at Layer 11 - Head 11

Rank |  Top-3 Left Words |  Top-3 Right Words
---|---|---
0 |  8, 9, 6 |  \u899a\u9192, cloneembedreportprint, StreamerBot
1 |  ”, [?], , |  DragonMagazine, cloneembedreportprint, ertodd
2 |  puff, rem, Ey |  \ufffd\ufffd\u58eb, catentry, Flavoring
3 |  air, compressor, exchange |  air, blow, nose
4 |  burn, burning, burns |  burns, burning, burn
5 |  smoke, blowing, sky |  smoke, clouds, airflow
6 |  light, shade, lighting |  light, illumination, Light
7 |  break, breaks, Bre |  breaks, breaker, broken
8 |  finger, air, registrations |  finger, finger, Feet
9 |  rolls, roll, rolled |  Rolls, rolls, Ludwig
10 |  opening, opened, closing |  opened, closes, opening
11 |  ause, blank, generating |  rawdownloadcloneembedreportprint, ause, sburg
12 |  anne, \u0639, sprayed |  \u30fc\ufffd, \ufffd\ufffd, iltration
13 |  ear, audio, Ear |  ear, ears, Ear
14 |  goggles, watched, devotion |  ideos, TPS, goggles
15 |  leaf, slashed, hunger |  gou, ouri, margins
16 |  voices, voic, Hand |  voic, leash, voiced
17 |  short, Short, short |  shorten, shortened, short
18 |  tones, tones, tone |  bells, tone, marrow
19 |  drawn, connected, ieties |  wu, River, Awakening