3. In this problem, you are required to use spark.ml API. As in Problem 2, consider 3

objects:

(1) The first object, denoted by OA, is a ball centered at (0, 0, 0) of radius 1. As a

set of points, we write OA = {(x, y, z) | x

2 + y

2 + z

2 ≤ 1}.

(2) The second object, denoted by OB, is a cylinder defined by OB = {(x, y, z) |

x

2 + y

2 ≤ 4, 2 ≤ z ≤ 4}.

(3) The third object, denoted by OC, is an ellipsoid

OC = {(x, y, z) |(x ? 2)2

Note that OA overlaps with OC a little bit.

Create a dataset in the following way:

(1) Each record in the dataset corresponds to a point contained in the union of OA,

OB and OC, which has a “features” part which is made of the xyz coordinates

2

of that point and a “label” part which tells which of OA, OB or OC this point

is contained in. Note that since OA ∩ OC is nonempty, if the point happens to

locate in OA ∩ OC, you still can only label it as OA or OC, but not both.

(2) The dataset you create should contain at least 500000 records. You should generate

the records randomly in the following way:

i. Each time, choose OA, OB or OC randomly. Suppose we choose OX (X is A,

B or C).

ii. Randomly create a point P contained in OX (think of how to do it). Now

the features of the newly created record is the coordinates of P and the

corresponding label is “OX”.

iii. After creating all the records, you should load and transform the dataset to

a spark Dataframe.

You are required to do the following work.

(1) Do classifications using both logistic regression and decision tree classifier. You

should try several different training/test split ratio on your dataset and for each

trained model, evaluate your model and show the accuracy of the test.

(2) Use K-means clustering to make cluster analysis on your data. Now only the

“feature” part of your data matters. Set the number K of clusters to 2, 3 and 4

respectively and make a comparison. Show the location of the centroids for each

case.

(3) Provide a visualization of the results of your classifications and cluster analysis.

In your report, you should provide both your codes and your demonstration of the

results. Take screenshots whenever necessary.

版权所有：留学生编程辅导网 2018 All Rights Reserved 联系方式：QQ:99515681 电子信箱：99515681@qq.com

免责声明：本站部分内容从网络整理而来，只供参考！如有版权问题可联系本站删除。