625.664留学生代做、代写Python语言、代做Java，c/c++编程设计

日期：2020-02-18 10:24

625.664 Computational Statistics

Problem Set 3

Associated Reading: Chapter 2: 2.2.2-2.2.2.2

Chapter 4: Introduction - Section 4.2

Complete the problems either by hand or using the computer and upload your final document

to the Blackboard course site. All final submittals are to be in PDF form. Please document any

code used to solve the problems and include it with your submission.

1. Problem 2.5. Only do parts (a), (b), (c), and (e).

2. Consider the following mixture of two normal densities:

p(x; θ) = π?(x; μ1, σ21) + (1 ? π)?(x; μ2, σ22)

where θ = (π, μ1, μ2, σ21, σ22) is the collection of all the parameters driving the model, with

π denoting the the mixing proportion (and not the usual number), and ? is the standard

density for a normal random variable. Direct use of maximum likelihood is made difficult

here because of the expression of the likelihood function. So, we wish to simplify this

problem by use of the EM algorithm.

Let Zi be the random variable that denotes the class membership of of an observed xi

in

a given sample. Clearly, zi

is the missing data, with Zi ∈ {0, 1} and Zi ～ Bernoulli(π).

The complete-data density for Y = (X, Z) is given by

p(xi, zi|θ) = [π?(x; μ1, σ21)]zi

[(1 ? π)?(x; μ2, σ22)](1?zi)

(a) Show that the complete-data log-likelihood function is

l(θ; x, z) = log(π)∑ni=1zi +

∑ni=1zilog(?(xi; μ1, σ21)) + log(1 ? π)(n ?∑ni=1zi)+

∑ni=1(1 ? zi) log(?(xi; μ2, σ22)).

For the Expectation step of the EM algorithm, we need Q(θ|θ

(k)) = E[l(θ; y)|x, θ(k)].

(b) Find Q(θ|θ

(k)) in terms of E[Zi|xi, θ(k)], again it is best to leave your answer as the

sum of four main terms. Also, if is fine to leave your answer in terms of the function

?(xi; μ(k)1,(σ21)(k)).

(c) Show that, given θ

(k) and xi,E[Zi|xi, θ(k)] = P[Zi = 1|xi, θ(k)] = π

(k)?(xi; μ(k)1,(σ21)(k))π(k)?(xi; μ(k)1,(σ21)(k)) + (1 ? π(k))?(xi; μ(k)2,(σ22)(k))

For the Maximization step of the EM algorithm, we need to find the new set of parameters

θ(k+1) that maximizes Q(θ|θ(k)). This can be done in a straightforward manner by solving

δQ(θ|θ(k))δθ = 0

for each of the parameters in θ = (π, μ1, σ21, μ2, σ22). To simplify your notation let η

(k)i =E[Zi|xi, θ(k)] and η

(k) =∑ni=1 η(k)i.

(d) First show that

(e) Now, use the fact that

(f) Suppose that you have observed the sample x1, x2, . . . , xn from the above distribution.

Write the psuedocode required to estimate θ using the EM algorithm. Be sure to include

some form of convergence criteria.