Vision-centric generative AI models: A software-hardware perspective
TL;DR
It is argued that progress in vision generative AI has been driven by output quality, with hardware evolving reactively to accommodate growing model demands, making generative AI deployment sustainable and accessible across a much broader range of platforms.
Abstract
Vision generative artificial intelligence (AI) has emerged as one of the most rapidly advancing areas of deep learning. The explosion of multimodal models has made them widely associated with text-to-image applications running on large datacentres. However, vision generative models are equally needed in applications that operate under strict hardware constraints at the edge, including autonomous vehicles, agricultural sensors, and mobile devices. In this Perspective, we argue that progress in vision generative AI has been driven by output quality, with hardware evolving reactively to accommodate growing model demands. We quantify the parameter cost and energy efficiency of these models across a range of accelerator platforms, and map four generative model families against seven real-world application domains. Finally, we advocate a software-hardware co-design approach, where deployment constraints are considered from the start of the design process, ensuring that the"right model"runs on the"right hardware"to serve the"right application", making generative AI deployment sustainable and accessible across a much broader range of platforms.