Efficient Iterative Retrieval with Heterogeneous Batching
Orthrus is a serving system that performs heterogeneous batching within a unified inference loop that addresses the primary challenge lies in unifying embedding and generation workloads with conflicting computational patterns while optimizing batch composition for high performance.
Dohyun Park, Hubertus Franke, Daniel G. Waddington et al.
· 0 citations