Challenging Quadratic Attention - A Holistic View On the Rise of Alternative Language Model Architectures

Alexander M. Fichtl Jeremias Bohn Josefin Kelber Edoardo Mosca Georg Groh Simran Arora Aman Timalsina Aaryan Singhal Sabri Eyuboglu Xin-Yi Zhao Ashish Rao Atri Rudra Christopher Ré Alireza Amiri Bavandpour Xin-Ting Huang Mark Rofin Michael Hahn Lower Maximilian Beck Korbinian Pöppel M. Spanring Andreas Auer Oleksandra Prudnikova Michael K Kopp Günter Klambauer Aleksandar Botev Soham De Samuel L. Smith Anushan Fernando G. Muraru Ruba Haroun Leonard Berrada Razvan Pascanu Pier Giuseppe Sessa Robert Dadashi Léonard Hussenot Johan Ferret Sertan Girgin Olivier Bachem Alek Andreev Kathleen Kenealy Thomas Mesnard Cassidy Hardin Surya Bhupati-raju Shreya Pathak L. Sifre Morgane Rivière Mihir Sanjay Kale Juliette Love P. Tafti A. Joulin Noah Fiedel Evan Senter Yu-Tian Chen S. Srinivasan Guillaume Desjardins D. Budden Arnaud Doucet S. Vikram A. Paszke Trevor Gale Sebastian Borgeaud Charlie Chen Andy Brock Antonia Paterson Jenny Brennan Meg Risdal Rajani Gundluru N. Devanathan Paul Mooney Nilay Chauhan Phil Culliton Luiz Gustavo E. Martins David Bandy Glenn Huntsperger A. Cameron Tris Zucker L. Warkentin Minh Peran Zoubin Giang Clément Ghahramani Koray Farabet Kavukcuoglu Tom Brown B. Mann N. Ryder M. Subbiah J. Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sas-try A. Askell S. Agarwal A. Herbert-Voss G. Krueger T. Henighan R. Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Chris Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess J. Clark Christopher Berner Sam McCandlish A. Radford I. Sutskever D. Amodei Jerry Tworek Heewoo Jun Qim-ing Yuan Henrique Pondé Oliveira Pinto J. Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman A. Ray Raul Puri Michael Petrov H. Khlaaf Girish Sas-try Pamela Mishkin Brooke Chan Mikhail Pavlov Alethea Power L. Kaiser Mo Bavarian P. Tillet F. Such Dave Cummings Matthias Plappert Fotios Chantzis Elizabeth Barnes W. H. Guss Alex Nichol A. Paino N. Tezak Jie Tang Igor Babuschkin S. Balaji Shantanu Jain William Saunders Andrew N Carr Jan Leike Josh Achiam Vedant Misra Evan Morikawa Matthew Knight M. Brundage M. Murati Katie Mayer Peter Welinder Bob McGrew Wojciech Zaremba Evaluating K. Choromanski Valerii Likhosherstov David Dohan Xing-You Song Andreea Gane Tamás Sarlós Peter G. Hawkins J. Davis Afroz Mohiuddin David Belanger Lucy J. Colwell Adrian Weller Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot K. Cobbe V. Kosaraju Jacob Hilton Reiichiro Nakano Xin Dong Y. Fu Shizhe Diao Wonmin Byeon Zi Chen A. Mahabaleshwarkar Matthijs Van Keirsbilck Shih-Yang Liu Min-Hung Chen Yoshi Suhara Y. Lin Jan Kautz Pavlo Molchanov Hymba Daniel Y. Fu Khaled Tri Dao Kamal Saab Armin W. Thomas Hun-gry Tianxiao Fu Hao-Feng Huang Xue-Fei Ning Genghan Zhang Bo Chen Tian-Qi Wu Hong-Yi Wang Zix-iao Huang Shi-Yao Li Shen-Gen Yan Aaron Grattafiori Abhimanyu Dubey Abhinav Jauhri Abhinav Pandey Abhishek Kadian Ahmad Al-Dahle Aiesha Letman Akhil Mathur A. Schelten Alex Vaughan Amy Yang Angela Fan Anirudh Goyal A. Hartshorn Ao-Bo Yang Archi Mitra Archie Sravankumar A. Korenev Arthur Hinsvark Arun Rao Aston Zhang Aurélien Rodriguez Austen Gregerson Ava Spataru Baptiste Rozière B. Biron Binh Tang Bobbie Chern Char-lotte Caucheteux Chaya Nayak Chloe Bi Chris Marra Chris McConnell Christian Keller C. Touret Chunyang Wu C. Wong Cristian Canton Ferrer Cyrus Nikolaidis Damien Al-lonsius D. Song D. Pintz D. Livshits Danny Wyatt David Esiobu Dhruv Choudhary Dhruv Mahajan Diego Garcia-Olano Diego Perino Dieuwke Hupkes Egor Lakomkin Ehab A. AlBadawy E. Lobanova Emily Dinan E. Smith Filip Radenovic Francisco Guzmán Frank W. Zhang Gabriel Synnaeve Gabrielle Lee G. Lewis Govind Thattai G. Nail Grégoire Mi-alon Guan Pang Guillem Cu-curell Hailey Nguyen Hannah Korevaar Hu Xu Hugo Touvron Imanol Iliyan Zarov Arrieta Ibarra Isabel Kloumann Is-han Misra Ivan Evtimov Jack Zhang Jade Copet Jaewon Lee Jan Geffert Jana Vranes Jason Park Jay Mahadeokar Jeet Shah Jelmer van der Linde Jennifer Billock Jenny Hong Jen-Ya Lee Jeremy Fu Jian-Feng Chi Jian-Yu Huang Jia-Wen Liu Jie Wang Jie-Cao Yu Joanna Bitton J. Spisak Jongsoo Park Joseph Rocca J. Johnstun Joshua Saxe Jun-Teng Jia Kalyan Vasuden Alwala Karthik Prasad K. Upasani Kate Plawiak Ke Li K. Heafield Kevin Stone Khalid El-Arini Krithika Iyer Kshi-tiz Malik Kuen-ley Chiu Kunal Bhalla Kushal Lakhotia Lauren Rantala-Yeary L. van der Maaten Lawrence Chen Liang Tan Liz Jenkins Louis Martin Lovish Madaan L. Malo Lukas Blecher Lukas Landzaat L. de Oliveira Madeline Muzzi Mahesh Pasupuleti Mannat Singh Manohar Paluri Marcin Kardas M. Tsimpoukelli Mathew Oldham Mathieu Rita Maya Pavlova M. Kambadur Mike Lewis Mitesh Min Si Kumar Singh Mona Hassan Naman Goyal Narjes Torabi Nikolay Bash-lykov N. Bo-goychev Niladri S. Chatterji Ning Zhang Olivier Duchenne Onur Çelebi Patrick Alrassy Pengchuan Zhang Petar Pengwei Li Peter Weng Prajjwal Bhargava Pratik Dubal Punit Praveen Krishnan S. Koura Pu-Xin Xu Qing He Qingxiao Dong R. Srinivasan R. Ganapathy Ramon Calderer Ricardo Silveira Cabral Robert Stojnić R. Raileanu Rohan Mah-eswari Rohit Girdhar Rohit Patel Romain Sauvestre Ron-nie Polidoro Roshan Sumbaly Ross Taylor Ruan Silva Rui Hou Rui Wang Saghar Hosseini Sahana Chennabasappa Sanjay Singh Sean Bell Seo-hyun Sonia Kim Sergey Edunov Shao-Liang Nie Sharath Narang S. Raparthy Sheng Shen Sheng-Ye Wan Shruti Bhosale Shun Zhang Simon Vandenhende Soumya Batra Spencer Whitman Sten Sootla S. Collot Suchin Gururangan S. Borodinsky Tamar Herman Tara Fowler Tarek Sheasha Thomas Georgiou Thomas Scialom Tobias Speckbacher Todor Mihaylov Tong Xiao Ujjwal Karn Vedanuj Goswami Vibhor Gupta Vignesh Ramanathan Viktor Kerkez Vincent Gonguet Virginie Do Vish Vogeti Vítor Albiero Vladan Petro-vic Weiwei Chu Wen-Han Xiong Wen-Yin Fu Zoe Chen Aaditya Papakipos Aayushi Singh Abha Sri-vastava A. Jain A. Kelsey S. Adithya Adolfo Gangidi Ahuva Victoria Goldstand Ajay A. Menon Alex Sharma Alex Boesen-berg Allie Baevski Amanda Feinstein Amit Kallet Amos San-gani Anam Teo Andrei Yunus Andres Lupu Andrew Alvarado A. Caples Andrew Gu Andrew Ho Andrew Poulton A. Ryan Annie Dong Annie Franco A. Goyal Aparajita Saraf Arkabandhu Chowdhury Ashley Gabriel Ashwin R. Bharambe Assaf Eisenman Azadeh Yaz-dan Beau James Ben Maurer B. Leonhardi Bernie Huang Beth Loyd B. de Paola Bhargavi Paranjape Bing Liu Bo Wu B. Ni Braden Han-cock Bram Wasti Brandon Spence B. Stojkovic Brian Gamido Britt Montalvo Carl Parker Carly Burton Catalina Mejia Ce Liu Changhan Wang Changkyu Kim Chao Zhou Chester Hu Ching-Hsiang Chu Chris Cai Chris Tindal Christoph Fe-71 Elaine Montgomery E. Presani Emily Hahn Emily Wood Eric-Tuan Lê Erik Brinkman E. Arcaute Evan Dunbar Evan Smoth-ers Fei Sun F. Kreuk

Semantic Scholar