Llama-Mobile: Efficient 2.7-Bit Quantization of VLMs
This work presents a framework for quantizing VLMs for efficient inference on resource-constrained hardware, which combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format.