Least Square Fitting

Let us assume that y = f_{c1,..., ck}(x₁,..., x_m) is a real-valued function of (x₁,..., x_m) which depends upon kparameters c₁,..., c_k. These parameters are unknown to us. However, suppose that we can perform repeated experiments that for given values of (x₁,..., x_m) allows us to measure output values for y. How can we estimate the parameter c₁,..., c_k that best correspond with this information?

Let us assume that the experiment measuring the value y = f_c(x)for specific input values x = (x₁,..., x_m) is repeated n-times. We will then obtain a system of equations

fc1,..., ck(x11, x12, ... , x1m)	=	y1
fc1,..., ck(x21, x22, ... , x2m)	=	y2
$$\vdots$$
fc1,..., ck(xn1, xn2, ... , xnm)	=	yn

where x_{$\imath$$\jmath$} and y_$\imath$ are the measurements of x_$\jmath$ and y in the $\imath$ experiment. These experimental results gives us information about the unknown coefficients c_$\imath$'s.

Since we may perform the experiments as many times as we wish, we may end up with more relations of the type above than unknowns (that is, n is larger than k). The larger the n, the more information we have collected about the coefficients. However, even if the experiments are carried out with great care, they unavoidably will contain some error. The question remains: how may we estimate judiciously the coefficients c₁,..., c_k using the collected information about y = f_c(x)? What is the best fit?

A very common method to respond to this question is known as the method of least-squares. The idea is simple: per realization of the experiment, we measure the fitting error by the distance from the real number f_c(x₁, x₂, ^... , x_m) and the observed value of y. The best fit for the distance though will also lead to a best fit for the square of the distance. To avoid absolute values, we change our viewpoint slightly and measure the fitting error by (f_{c1,..., ck}(x₁, x₂, ^... , x_m) - y)². The error function that considers all the information obtained from the n experiments is then

E(c₁,..., c_k) = $$\sum_{i=1}^{n}$$((f_{c1,..., ck}(x_i1, x_i2, ^... , x_im) - y_i)² ,

where (x_i1, x_i2, ^... , x_im) and y_i are the i-th input for x and value of the measurement for y.

This turns out to be a function of c = (c₁,..., c_k). Mathematically, our best fit problem is now reduced to finding the value of c which produces a minimum for this error function E. The details of how this can be done depends intrinsically upon the assumed form of the function f, and its relation to the parameters c₁,..., c_k.