Generalised Hardness of Approximation and the SCI Hierarchy –On Determining the Boundaries of Training Algorithms in AI

Abstract

<jats:p> Generalised hardness of approximation (GHA) is the phenomenon that one can easily compute an <jats:inline-formula> <jats:alternatives> <jats:tex-math>$$\epsilon $$</jats:tex-math> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>ϵ</mml:mi> </mml:math> </jats:alternatives> </jats:inline-formula> -approximation to a solution of a computational problem for <jats:inline-formula> <jats:alternatives> <jats:tex-math>$$\epsilon> \epsilon _1 > 0$$</jats:tex-math> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>ϵ</mml:mi> <mml:mo>></mml:mo> <mml:msub> <mml:mi>ϵ</mml:mi> <mml:mn>1</mml:mn> </mml:msub> <mml:mo>></mml:mo> <mml:mn>0</mml:mn> </mml:mrow> </mml:math> </jats:alternatives> </jats:inline-formula> , but for <jats:inline-formula> <jats:alternatives> <jats:tex-math>$$\epsilon < \epsilon _1$$</jats:tex-math> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>ϵ</mml:mi> <mml:mo><</mml:mo> <mml:msub> <mml:mi>ϵ</mml:mi> <mml:mn>1</mml:mn> </mml:msub> </mml:mrow> </mml:math> </jats:alternatives> </jats:inline-formula> (the approximation threshold) it suddenly becomes hard, for example, non-computable or intractable (non-polynomial time). In this paper we demonstrate the phenomenon that GHA happens when using AI techniques for solving inverse problems, namely training neural networks (NNs) to optimally perform on the training data. In particular, for any non-zero underdetermined linear inverse problem the following phase transition can occur: For a certain family of training sets <jats:inline-formula> <jats:alternatives> <jats:tex-math>$$\Omega $$</jats:tex-math> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>Ω</mml:mi> </mml:math> </jats:alternatives> </jats:inline-formula> , one can prove the existence of optimal NNs for solving the inverse problem for each <jats:inline-formula> <jats:alternatives> <jats:tex-math>$$\mathcal {T}\in \Omega $$</jats:tex-math> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mi>T</mml:mi> <mml:mo>∈</mml:mo> <mml:mi>Ω</mml:mi> </mml:mrow> </mml:math> </jats:alternatives> </jats:inline-formula> , however, these optimal neural networks can only be computed to a certain accuracy <jats:inline-formula> <jats:alternatives> <jats:tex-math>$$\epsilon _1 > 0$$</jats:tex-math> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:msub> <mml:mi>ϵ</mml:mi> <mml:mn>1</mml:mn> </mml:msub> <mml:mo>></mml:mo> <mml:mn>0</mml:mn> </mml:mrow> </mml:math> </jats:alternatives> </jats:inline-formula> . Below the approximation threshold <jats:inline-formula> <jats:alternatives> <jats:tex-math>$$\epsilon _1$$</jats:tex-math> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:msub> <mml:mi>ϵ</mml:mi> <mml:mn>1</mml:mn> </mml:msub> </mml:math> </jats:alternatives> </jats:inline-formula> , not only does it become intractable to compute the NNs, it becomes impossible regardless of computing power, and no randomised algorithm can solve the problem with probability better than 1/2. Moreover, despite the existence of a stable optimal NN, any attempts of computing it below two times the approximation threshold <jats:inline-formula> <jats:alternatives> <jats:tex-math>$$2\epsilon _1$$</jats:tex-math> <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>2</mml:mn> <mml:msub> <mml:mi>ϵ</mml:mi> <mml:mn>1</mml:mn> </mml:msub> </mml:mrow> </mml:math> </jats:alternatives> </jats:inline-formula> will yield an unstable NN. Our results use and extend the current mathematical framework of the Solvability Complexity Index (SCI) hierarchy and initiate a program for analysing the GHA phenomenon throughout computational mathematics and AI. GHA generalises the phenomenon of hardness of approximation in discrete computations to arbitrary computational problems. </jats:p>