Enclave-Anchored Hybrid Quantum-Safe Key Exchange for Untrusted Platforms

Abstract

The transition to Post-Quantum Cryptography (PQC) is often reduced to a primitive-replacement problem, yet the security of PQC schemes remains dependent on the integrity of the underlying host. Quantum-secure protocol frameworks like Muckle offer robust hybrid security by combining classical, post-quantum and quantum key exchanges with guarantees for Post-Compromise Security (PCS) and Forward Secrecy (FS); these guarantees collapse, however, in the face of a compromised operating system capable of exfiltrating keys, manipulating nonces, or rolling back the protocol state. We propose MuckleTEE, a hardware-software co-design that hardens hybrid key exchange against a fully compromised host by confining only its security-critical state to a Trusted Execution Environment (TEE), while delegating heavy cryptographic operations and networking to the untrusted domain. The construction is generic over KEM-based hybrid key exchange; we instantiate it on a signature-free variant, which lets the enclave authenticate without generating post-quantum signatures, leaving only passive certificate verification inside the trusted boundary. Benchmarks on hardware Intel SGX show the TEE boundary adds 7.6% CPU-time overhead, and a formal analysis in the extended HAKE framework shows that MuckleTEE preserves session-key confidentiality, long-term identity-key secrecy, and post-compromise security against host compromise; recovery after enclave compromise requires a subsequent enclave-only fresh secret.