# DM825 - Introduction to Machine Learning Sheet 2, Spring 2011 [pdf format]

Prepare exercises 1.3, 1.11, 1.14, 1.24, 3.8 from book [B1] of the course literature and the exercises below for discussion in class on Wednsday 13th April 2011.

[Wait Monday’s lecture to approach exercises 3.8 and 1 below.]

Exercise 1

Suppose that a fair-looking coin is tossed three times and lands heads each time. Show that a classical maximum likelihood estimate of the probability of landing heads would give 1, implying that all future tosses will land heads. By contrast, show that a Bayesian approach with a prior of 0.5 for the probability of heads would lead to a much less extreme conclusion on the posterior probability of observing heads.

Exercise 2. Linear Regression and k nearest neighbor The files q2x.dat and q2y.dat contain the inputs (x(i)) and outputs (y(i)) for a regression problem, with one training example per row.

1. [i.] Implement the linear regression (y = βT x) on this dataset using the normal equations (which is done in R automatically via the `lm` function) and plot on the same figure the data and the straight line resulting from your fit (in R, plot the points and then pass the fitted linear model to `abline`). Compare your result with the implementation via the sequential gradient algorithm from the past exercise sheet. (Remember to include the intercept term.)
2. Implement a k-nearest neighbor regression (in R install package FNN and read the documentation of `knn.reg`). Use some randomly chosen x values as test points. Plot the training and predicted points for k=3. Further, show graphically the behavior of the square error as k increases from k=0 to the size of the training set that you decided.
3. Use a cross validation procedure to select and assess the best model between the linear regression and the k-nearest neighbor. Use a 5-fold cross validation procedure on the training data to decide the best value of parameters in the linear regression.