EPTQ (E8) achieves accuracy competitive with or surpassing QTIP on Llama-2 models, while EPTQ (FE8) surpasses competing VQ methods in decode throughput on most models, achieving up to 38% faster inference than FP16, with both variants reducing quantization time by over 80% compared to competing approaches.
Ha-Myung Park, Jungeun Park, Chaewon Jeong et al.· Proceedings of the 35th ACM...· 0 citations
How can we efficiently retrieve historical meteorological satellite images that are partially similar to a given query image? With advancements in satellite technology, it has become possible to acquire large volumes of high-resolution meteorological images. To efficiently retrieve past satellite images with similar we...
Two-bit Post-Training Quantization (PTQ) offers an attractive path to deploying Large Language Models under strict memory budgets, yet scalar quantization suffers from catastrophic performance collapse at this extreme bit-width. While Vector Quantization (VQ) methods resolve the accuracy issue, existing approaches suff...
Ha-Myung Park, Jungeun Park, Chaewon Jeong et al.· Proceedings of the 35th ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.