Ourong Lin

Difficulty-Aware Hybrid Routing for Cost-Efficient Long-Context LLM Inference

The Difficulty-Aware Hybrid Router is proposed, a lightweight framework that jointly leverages query-level, context-level, and draft-level features to classify each request into one of three inference tiers: direct answering by a small model, small-model answering with lightweight verification, or full processing by a large model.