We introduce a novel generic mathematical formulation of AutoML, resting on formal definitions of hyperparameter optimization (HPO) and meta-learning. In light of this formulation, we decompose various algorithms and show that HPO does not really address the AutoML problem, more than “classical” machine learning algorithms, while meta-learning does. In some sense, the objective of AutoML is to beat No Free Lunch theorems, which is the charter of meta-learning not that of HPO. Other branches of machine learning such as transfer learning and ensemble learning are also reviewed, re-formulated and unified. Our framework allows us to gain a clear global view on the naturally involved hierarchy of algorithms and problems in the field and provides us with a set of formal/algebraic language and tools to facilitate and inspire future research. We show that these tools can already help to gain interesting insights by analyzing existing domains and methods in our framework.
Supplementary notes can be added here, including code and math.