Automatic prompt engineering using multimodal large language models for the analysis of biological research images.

Abstract

Large language models (LLMs) are being applied across diverse fields due to their capability to derive various insights from complex data. In biotechnology, where complex multimodal data including images is rapidly expanding, LLMs offer powerful capabilities for data analysis. However, unlocking the full potential of these models depends critically on prompt engineering, which is often a labor-intensive process that requires specialized expertise and lacks reproducibility. To address these challenges, we developed novel automatic prompt engineering (APE) approaches tailored for multimodal tasks in the bioscience and bioengineering domains. This study introduces two types of approaches: the Batch APE method, an efficient method for optimizing prompts for powerful black-box models, and a fine-tuning method with supervised fine-tuning (SFT) and direct preference optimization (DPO) on a local vision-language model (VLM). These methods were systematically evaluated across four diverse scientific datasets of microscopic images of protein crystals, human cell images, molecular structure images, and medical radiography images of Chest X-ray. Experimental results demonstrated that Gemini with Batch APE and SFT-based local LLM alignment generally outperformed baseline APE techniques, though DPO alone showed inconsistent results across datasets. Furthermore, qualitative analysis of the generated prompts revealed key characteristics of prompts that enhance image classification performance in multimodal LLMs. These findings highlight the potential of advanced APE to improve the utility of both local and black-box multimodal models for specialized scientific applications, particularly in domains where fine-tuning was restricted or infeasible.