SDBench is introduced, a generic framework for generating evaluation datasets for domain-specific LLMs that reduces the reliance on expert man-power while ensuring that the collected data is uniformly distributed.
This study discloses the model efficiency for different tasks, causes of failed fs tasks, and techniques for mitigating LLM failures, and will open source \phi-Bench to facilitate public research on using LLMs for fs development.
ESF-Bench is introduced, a challenging Enterprise Slot Filling benchmark consisting of 810 multi-turn samples and 6530 slots over 8 unique domains and curated using a taxonomy of the 57 most challenging slot-filling scenarios observed during real-world enterprise deployments.
Toby Liang, Gopal Sarda, Sagar Davasam et al.· 0 citations
A comparative study of eight candidate methods by combining the base model, SFT, RAG, and ICL using a ticket database of 60,000 question-answer pairs from Texas Advanced Computing Center suggests that combining ICL, RAG, and a powerful base model is a promising solution for LLM-enabled user support in computing centers...
Ming-Kai Zheng, Fang-Ru Linghu, Sikan Li et al.· Practice and Experience in A...· 0 citations
The findings show that, while LLMs achieve promising results, they struggle with harder problems and with programming languages that have fewer available resources for training, and they often fail due to fundamental and easily avoidable errors that underscore the unreliability of automatically generated code.
Rodrigo Pato Nogueira, Marco Vieira, João R. Campos· Empirical Software Engineeri...· 1 citation
In this work we introduce a novel application of Large Language Models (LLMs) towards the classification of malicious Windows Portable Executable (WinPE) files. Rather than fine-tuning an LLM based on the raw binary of the samples serving as the corpus, in this case we leverage our in depth knowledge of the underlying...
Timothy Miskell, Yan Luo, Peilong Li et al.· International Conference on...· 0 citations
An automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok is presented and a substantial gap between correctness and quality attributes is revealed.
Seyed Mohammad Mahdi Ghalandarian, Majid Bazargani, Masoumeh Taromirad· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
