Supervised learning

In a general supervised learning problem, we have two spaces of objects X and Y and would like to learn a function h: X → Y, which outputs an object y ∈ Y, given x ∈ X. The function h is sometimes called a score function or hypothesis. It is an element of some space of possible functions H called the hypothesis space. We assume a given sample {(x₁, y₁), . . . , (x_n, y_n)}, where x_i ∈ X is an input and y_i ∈ Y is the corresponding response that we wish to get from h(x_i). We consider probabilistic learning models and assume that there is a joint probability distribution P(x, y) and that the sample is made of independent and identically distributed (iid) variables drawn from that distribution. That is, we model