A benchmark study systematically benchmark six state‐of‐the‐art MLPs, namely CHGNet, EquiformerV2 in two training variants, MatterSim, SevenNet, and MACE, on representative Li‐ and Na‐based superionic conductors to highlight the importance of accurately capturing atomic forces in reasonably producing ion transport properties in all‐solid‐state battery materials.