Evaluating and Improving Syntax Error Diagnostics in Modern Programming Languages
Abstract
Syntax errors come up a lot when someone is trying to compile code. The messages a compiler or interpreter shows are one way the system tells the user what went wrong. These warnings and errors are part of how syntax analysis is reported back to the programmer. This work compares how three tools report syntax errors. The tools are Python 3.13.5, Java 21.0.11, and GNU C++ 14.2.0. The goal was to look at the wording and the usefulness of what each one prints. To make the test fair, ten related syntax-error cases were set up. The same kinds of mistakes were used each time. But the source text was adjusted to match each language grammar. The code was not kept identical across all languages. In total, thirty diagnostics were gathered. The data came from running the language implementations directly. No IDE was used. For each diagnostic, six checks were applied. The checks were: clarity, accuracy, where the error was found, specificity, consistency, and usefulness. For the first five checks, a rubric with five levels was used. Clarity, accuracy, location, specificity, and usefulness each got a level from that scale. Consistency was judged in a different way. It was scored at the language level, based on how regular the diagnostic format looked across the ten cases. The results indicate that every tool caught the intended malformed programs. Still, the results were not the same in detail. The kind of info and the way it was written changed across tools. On the six-part scoring scale, Python averaged 4.50. Java averaged 4.63. C++ averaged 4.57. All three tools gave a source location. More variation showed up in specificity. It also showed up in how many alternative messages appeared for one error. These findings describe only what happened under these conditions. They are tied to the exact versions tested. They also depend on the chosen test cases and the scoring method used. Overall, the study adds to work on programming language theory. It links grammar and parsing behavior to the diagnostic layer that reports detected syntax problems.