Skip to content

Riemannian Structure and Optimization for a Class of Low-Parametric Orthogonal Matrices

Sep 2026 · arXiv.org · Vol abs/2609.27982 · 0 citations · 27 references
Computer Science Mathematics

TL;DR

This paper derives the essential Riemannian tools and proposes efficient algorithms for their implementation of orthogonal two-factor matrices -- a flexible family of structured matrices formed by block-diagonal factors interleaved with fixed permutations.

Abstract

In this paper, we are concerned with matrices formed by block-diagonal factors interleaved with fixed permutations -- a flexible family of structured matrices. This class has recently drawn interest in deep learning architectures for its balanced expressivity-efficiency trade-off, yet efficient computational strategies for working with it remain to be found. We approach this problem through Riemannian geometry and examine under what conditions this class admits a smooth manifold structure. For the practically important case of orthogonal two-factor matrices, we derive the essential Riemannian tools and propose efficient algorithms for their implementation. The algorithms leverage automatic differentiation, support parameter sharing within each factor, and avoid explicit dense matrix construction. We test them within the Riemannian optimization framework on the best matrix approximation problem and for parameter-efficient fine-tuning of large language models. Beyond the two-factor setting, we study the geometric and matrix-theoretic properties of factorizations with a larger number of block-diagonal factors.

View source

Similar papers

Preprint Aug 2026

Closest Normal Matrix Found Again Using Riemannian Optimization

We propose an approach based on Riemannian optimization to compute a nearest normal matrix to a given one. The problem can be formulated as the minimization of a smooth function either on the manifold $U(n)$ of unitary matrices of size n or on the flag manifold $U (n)/U (1)^n$. The flag manifold is particularly suitabl...

V. Noferini, M. Zhukov · 2 citations · ⚡1
Preprint Sep 2026

Riemannian optimization on low-rank tensor train manifolds for the Gross-Pitaevskii equation

This work is concerned with the numerical computation of Gross-Pitaevskii ground states using the tensor train (TT) format. We regard the problem as the minimization of the Gross-Pitaevskii energy functional under a unit mass constraint, and propose a first-order Riemannian optimization scheme that preserves both the m...

Ivan Bioli, Carlo Marcati, M. Rakhuba · 0 citations
Preprint Oct 2026

Curvature, Scaling, and Complexity in Riemannian Projection-Free Optimization

Projection-free methods replace projections by linear minimization oracles, making their convergence fundamentally sensitive to the geometry of the feasible region. We develop a geometric framework for this dependence in Riemannian Frank--Wolfe optimization. A power-type double-geodesic modulus quantifies the intrinsic...

B. T. Nguyen · 0 citations
Open access Aug 2026

STOD: Sparse Tensor Train Optimization via Orthogonal Decomposition for High-Dimensional Learning

This paper proposes a novel Tensor Train (TT)-based tensor-on-tensor regression optimization framework for variable selection based on mode-1 hyperslice sparsity, and designs an alternating iterative algorithm equipped with a preconditioned metric to efficiently solve the proposed model.

Xiao-Yu Li, Zi-Yan Luo · 0 citations
Preprint Sep 2026

Regular simplex tensors: optimization landscape, conjecture proof, and beyond

The concept of tensor eigenpairs has attracted increasing research attention in the past decades. Recent works have focused on a special class termed regular simplex tensors, which are constructed from an equiangular tight frame of n vectors in (n-1) dimensional space for n>= 3 and order m>= 3. Existing works focus on...

Lei Wang · 0 citations

Related blog posts

MIT News · Artificial Intelligence Sep 29, 2026

Who we become when we talk to machines

Professor Sherry Turkle’s new book, “Artificial Intimacy,” offers a withering critique of chatbots and the antisocial dynamics she believes they encourage.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.