From Lab to Reality: An Eye-tracking Study of How Users are Influenced to Search in the Era of GenAI

Abstract

This work proposes two complementary eye tracking methodologies focused on understanding information access. The first study examines the earliest stage of search, the formation of an information need, and the formulation of a query. The second study presents participants with a pre-typed query on a Search Engine Results Page (SERP) containing GenAI content positioned above the traditional ten blue links. Across both studies, the planned analysis examines how user interactions differ from patterns reported in prior literature, and outlines future research directions aimed at deepening our understanding of search behavior in the era of GenAI. By linking visual attention to query behavior across traditional and GenAI-enhanced interfaces, this research contributes toward more intuitive and adaptive search systems that better support evolving information-seeking behavior. The design space for search engine interfaces is vast, but interaction techniques and user interfaces for information access with LLMs remain under-researched and poorly understood. Between 2020 and 2024, 750 preprints related to LLMs were published on arXiv in the field of Information Retrieval, with only 22 mentioning "user interface" in their abstracts [3]. This research will contribute to filling this gap by understanding user behavior, preferences, and cognitive effort in human-AI interactions. Specifically, the goal of this research is to explore how users seek information in the era of GenAI via a series of user lab studies. These studies will be conducted to compare traditional search engines with GenAI-based systems. Based on these lab studies, the aim is to develop new methods that better fulfill users' information needs. Key research questions for this research include: •[RQ1:] How do eye fixations on specific words read before a search session impact query formulations?•[RQ2:] How do eye fixations on specific words read before a search session, together with curiosity about a topic, influence query variations? •[RQ3:] How do individuals read information prior to formulating queries?•[RQ4:] How does user interaction with new search engine interfaces differ from the patterns of scanning search engine interfaces described in the literature? •[RQ5:] How to evaluate search behaviors in the context of information seeking?•[RQ6:] What are the fundamental factors Information Retrieval researchers should investigate when conducting user studies in the era of Generative AI? The work I have described in this research is done under a reasoned approach, but may not account for all user variables such as domain expertise, use of different devices [1], other demographics, anomalous state of knowledge, and cognitive biases [2]. Researchers have attempted to overcome the complexity of understanding human search behavior by creating personas and simulating user behavior. However, this raises concerns that simulated users may not accurately reflect real human behaviors, and assessing the validity of simulated users still remains a challenge [1]. Therefore, conducting both types of studies separately and in combination is important, as each approach can provide valuable insights and collectively contribute to the body of knowledge to understand and improve human-GenAI interactions. The challenge we expect to face when conducting this research is frequent updates to search interfaces. How can we standardize our studies to account for these changes? How do we analyze user behavior when interfaces keep evolving? Should we test multiple interface versions, present our findings, and then provide recommendations? An additional aspect is determining how participants interact with different search tools based on different types of tasks. Do they rely on traditional search engines, generative AI systems, or a combination of both?