Language model tokenizers are typically selected with minimal evaluation, despite the fact that their design choices directly impact model capabilities. This can be partly attributed to a limited understanding of which tokenizer properties affect which aspects of downstream performance. We introduce TokEval, a framework of tokenizer evaluation metrics that goes beyond standard measures like fertility and compression rate to capture linguistically and structurally meaningful properties, e.g., UTF-8 character boundary integrity and digit place-value boundary alignment for mathematics. To validate whether these metrics are predictive of downstream model performance, we conduct controlled language model pretraining experiments, varying solely the tokenizers'training data mixture, pretokenization strategy, and training algorithm. We evaluate the resulting models on bits-per-byte (a tokenizer-agnostic version of perplexity) and several benchmarks, spanning linguistic understanding, mathematical reasoning, and code generation. Our experiments suggest that different intrinsic properties have different impacts on model abilities: information-theoretic metrics predict language modeling abilities (Spearman rho up to 0.80), while structure-sensitive metrics, such as those measuring digit and line-break handling, correlate with task accuracy. We hope TokEval enables more principled tokenizer evaluation, replacing pretraining sweeps with intrinsic measurement wherever the two agree.
This work proposes a multi-dimensional, primitive based framework for dynamic contrast-enhanced MRI reconstruction that disentangles the underlying anatomy, the dynamic contrast enhancement, and residual motion into separate temporal basis functions, thereby enabling a geometrical interpretation of the representation.
Veronika Spieker, Wenqi Huang, Cemre Ariyurek et al.· 0 citations
This work investigates the use of auxiliary k-space magnitude information for accelerated steady-state dynamic MRI reconstruction, and demonstrates strong consistency of k-space magnitudes across time-frames.
Mahdi Saberi, Y. Alçalar, Merve Gülle et al.· 0 citations
Analysis of the pre-trained embedding geometry of a small sentence transformer (SBERT) and classic SLM reveals that pre-trained embedding geometry is associated with classification performance and reveals a counterintuitive finding that a structured input that would help a human reader does not improve the SLM performance.
Emma Ceccherini, Daniel Lawson, Anjulika Salhan· 0 citations
Reach audiences
Advertise in front of researchers, engineers, and readers.
TSN4PI, a unified framework for tracking the evolution of political ideologies on social media, includes two core modules: the PIDN and the PIPN, which employs temporal graph neural networks to predict future ideological shifts, enabling comprehensive analysis of ideology presence, intensity, and evolution.
This work presents a novel world model formulation where the reward prediction only depends on a subset of structured, symbolic components of the whole latent state, and demonstrates the strong generalisation properties of this approach over purely neural methods.
Isidoro Tamassia, Lennert De Smet, Giuseppe Marra· 0 citations
Large language models can generate executable programs, which makes it possible to search directly over procedural content generators rather than individual levels. We study this approach in Sokoban, Zelda, Dangerous Dave, and Lode Runner. Each run evolves complete Python generators through language-model mutation and crossover. We introduce Continual Abstraction Discovery, or CAD, which extracts reusable primitives from high-fitness programs into a run-specific helper module. A 2x2 experiment crosses CAD with access to a fixed hand-written domain API. The completed data set contains 160 complete runs, with at least ten 50-generation runs in every cell. CAD raises mean final best fitness in all eight domain and API comparisons. Across all CAD runs, learned libraries are adopted by most later programs and repeatedly rediscover validation, reachability, and structural utilities. These results support that discovering reusable primitives improves evolutionary program search for content generators.
Matthew Siper, A. Khalifa, Julian Togelius· 0 citations
An advanced system capable of automatically detecting complicated appendicitis from ultrasound images was developed and was explained with gradient-weighted class activation mapping (Grad-CAM), which creates a heatmap of the regions responsible for the model's prediction of the infected areas.
Fahad Ahammed, Omar Faruq Shikdar, Navid Zaman et al.· 0 citations
This paper investigates the feasibility of using residual learning to improve unsteady aerodynamic load prediction for aeroelastic applications. The machine learning technique selected for the study is the long short-term memory (LSTM) neural network, which is used for its suitability for sequential data with aerodynamic memory effects. The approach is investigated for the NLR 7301 airfoil benchmark using high-fidelity CFD lift data for prescribed pitch and plunge motions in the transonic flow regime in the presence of shock motion. An analytical unsteady aerodynamic model based on the Wagner function is used as a physics-based baseline, and the neural network is trained to learn the difference between the CFD lift coefficient and the Wagner prediction. The residual model is compared with a direct neural-network model trained to predict the CFD lift coefficient. The comparison includes feature and normalization studies, external benchmark cases, and leave-one-out and leave-family-out generalization tests across a range of sinusoidal and non-sinusoidal motions. The residual model performs best when its inputs align with the Wagner formulation variables, generally giving lower error and more consistent performance across training runs, though the direct model remains more accurate for some high-frequency cases. The residual model also generalizes better in the leave-one-out and leave-family-out tests, with a smaller increase in error than the direct model when entire motion families are withheld from training. Overall, the results indicate that residual learning shows promise as a modular approach for augmenting classical low-order aerodynamic theories, especially when the physics baseline removes a structured part of the aerodynamic response and leaves a lower-variance correction for the neural network to learn.
The results resolve the open question raised in the literature concerning the sharp arm-dependent regret--instability frontier and develop a new offline top-prefix representation that removes path dependence from online decisions.
Accurate building height information at the individual footprint scale is essential for material stock accounting and post-disaster damage assessments yet remains difficult to obtain at city scale in the Global South where airborne LiDAR coverage is rare and commercial very high-resolution imagery is cost-prohibitive or unavailable. While recent works have demonstrated building height estimation using freely available Sentinel imagery, the resolution ceiling of resulting products is still coarse for material stock analysis. This study incorporates products derived from data freely accessible under scientific research licenses, TerraSAR-X StripMap and PlanetScope, alongside Sentinel-1 to predict building heights in a large city in Brazil. To account for the spatial autocorrelation in the training set, features from all sources are integrated in a geographically weighted random forest model, returning an RMSE of 5.34 m and R2 of 0.756 against a LiDAR reference dataset. Local feature importance showed predictor dominance to vary consistently across intra-urban contexts, with footprint geometry dominating for low-rise buildings, shadow-derived height for taller and more isolated structures, and spectral reflectance for the tallest buildings in the set. Sentinel-1 backscatter and InSAR occupy complementary spatial niches, with no single sensor uniformly preferable across the set. Results provide optioneering guidance and insight over satellite-derived products predictive relevance in distinct contexts, which global machine learning or neural network models cannot offer.
Guilherme Iablonovski, P. Frison, Tatiana Silva da Silva· 0 citations
The proposed Diff-DDoS framework, a three-phase framework for realistic attack synthesis and robust detection using tabular diffusion models, supports tabular diffusion models for stress-testing and hardening intrusion detectors in data-scarce 5G cyber-physical deployments.
Bilal Hussain, Xiao Tang, Qinghe Du et al.· 0 citations
A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.
MIT News · Artificial Intelligence· news.mit.eduAug 24, 2026
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
