A Model-based Testing Technique for Amazon Lex Task-based Chatbots

Reverse Engineering of Chatbot Behaviour for Automated Testing

A reverse engineering technique is presented for extracting a behavioural model of a deployed chatbot through the automated exploration of its functionality via Large Language Models through the automated exploration of its functionality via Large Language Models.

DEVELOPMENT AND ACCEPTABILITY OF AN INTELLIGENT CHATBOT FOR STUDENT SERVICES IN EASTERN SAMAR STATE UNIVERSITY SALCEDO CAMPUS

The study successfully validated that the intelligent chatbot efficiently bridges communication barriers, automates repetitive administrative inquiries, and improves service accessibility, confirming that the application is highly usable, practical, and effective as an inclusive, assistive communication tool.

Refused in Chat, Written in Code: Workflow-Level Jailbreak Construction in IDE Coding Agents

Large language models are increasingly deployed as IDE-integrated coding agents that decompose tasks, generate and edit files, run code, and refine outputs over many turns. Yet their safety is still often evaluated as if they were chatbots: one harmful prompt, one response, judged in isolation. We introduce workflow-level jailbreak construction, a failure mode in which a harmful objective is assembled across ordinary stages of a software-development workflow rather than generated through a single direct prompt. Using GitHub Copilot in Visual Studio Code, we study four closed-weight backends: Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, and Gemini 3.5 Flash. Across 204 prompts from Hammurabi's Code, HarmBench, and AdvBench , the models show near-complete refusal under direct chat, CSV-read, and single-step code-fix baselines, with only 8/816 successful responses in each baseline condition. Under the full workflow, however, the same prompts and backends produce 816/816 unsafe teaching-shot completions, all independently confirmed by two expert evaluators under a strict rubric. These results show that conversational refusal benchmarks can substantially overstate the safety of deployed coding agents and motivate defenses that reason about safety across multi-turn IDE workflows and their generated artifacts, not only individual chat turns.

RAG-test: a framework to evaluate RAG chatbot performance and measure productivity gains

Exploring the Effect of Multiple Languages on Code Suggestion Using ChatGPT 5.2

: Artificial Intelligence (AI)-based code suggestion tools are becoming increasingly relevant in the software development community. While several studies analyse the quality and security of these tools, relatively few systematic studies have investigated how different input text languages can affect the effectiveness of code generation models. Similar research has been conducted with other tools, addressing multiple programming languages and different languages and challenge platforms. However, it is important to expand this knowledge base by considering additional scenarios. In this study, we analyse the performance of ChatGPT 5.2 in generating Python solutions for 70 programming problems extracted from the Beecrowd repository, considering three languages: Portuguese, English and Spanish. These 70 challenges were selected to encompass a range of difficulty levels for mathematical problems. Each question was submitted to the model five times in each language and the accuracy rate was evaluated using the platform’s validation tests. It was observed that Portuguese prompts generally showed a slight advantage in terms of the accuracy rate compared to prompts in other languages. However, the analysis of variance (ANOVA) indicates that this variation was not significant at the 5% level. The aim of this article is to extend current knowledge about the impact of languages on code generation by demonstrating that the language used in the problem description may or may not affect the ability of AI models to provide correct solutions. Finally, this article discusses the influence of potential linguistic biases, as well as opportunities for future research to address other languages, different problem modalities and various AI tools.

Optimizing RAG-Based Academic Chatbot Performance Using Prompt Engineering

Background: The increasing use of artificial intelligence in higher education has encouraged institutions to develop academic chatbots that provide faster access to official information. However, Retrieval-Augmented Generation (RAG)-based chatbots still require optimization to ensure accurate, relevant, and context-grounded responses.Aims: This study aims to optimize the performance of a RAG-based academic chatbot by applying zero-shot and few-shot prompt engineering strategies.Methods: A comparative experiment was conducted using 45 in-context academic questions for RAGAS-based quantitative evaluation, while 5 out-of-context questions were used as a qualitative robustness check. The system was developed using Python, LangChain, FAISS, OpenAI, and Streamlit, and evaluated using the RAGAS metrics: faithfulness, answer relevancy, context precision, and context recall.Results: The baseline system achieved an average RAGAS score of 0.8421. After prompt engineering was applied, zero-shot prompting achieved 0.8697, while few-shot prompting achieved 0.8565.Conclusion: Zero-shot prompting produced the best overall performance and showed the strongest improvement in faithfulness and context recall. These findings indicate that clear prompt instructions can improve the reliability of RAG-based academic chatbot responses for academic information services.