DM825 - Introduction to Machine Learning
Sheet 3, Spring 2013 [pdf format]

Exercise 1

Redo exercise 1 from Sheet 1 using logistic regression (transform the response label -1 to 0). Alternatively use logistic regression on these data []. Although, as we will see, logistic regression can be impleemnted in R via glm, you are asked here to implement the method by yourself. For the optimization you can reuse the gradient descent method developed in previous exercises or you can use optim.

Exercise 2

In exercise 3 of Sheet 2 use 1/2 of the data for training the models, 1/4 of the data to select the model (k-nearest neighbor or linear regression) and 1/4 to assess the performance of the best model selected.

Exercise 3 Bayesian prediction

In class we saw an example with binary variables. Often however we encounter discrete variables that can take on one of K possible mutually exclusive states. A way to handle this situation is to express such variables by a K-dimensional vector x in which one of the xk elements equals to 1 and all remaining elements equal 0. Consider a sample described by m multinomial random variables (X1, X2, …, Xm), where XiMult(θ) for each m, and where the Xi are assumed conditionally independent given θ. Let θ ∼ Dir(α). Now consider a random variable XnewMult(θ) that is assumed conditionally independent of (X1, X2, …, Xm) given θ. Compute the predictive distribution:

p(xnew | x1 , x2 ,…, xN ,α)

by integrating over θ.