AIML427代写、代写Python，Java编程

日期：2023-04-25 08:34

2023 AIML427 Big Data: Assignment 2

This assignment has 100 marks and is at on 11:59pm, Monday, 8th May 2023. Please submit

your answers as a single .pdf file. Make sure you read the Assessment section before writing

the report. This assignment contributes 25% to your overall course grade.

Any questions about Parts 1 or 2 should be directed to Bach; any questions about Parts 3

should go to Qi.

1 Manifold Learning [40 marks]

In class, we discussed a variety of different manifold learning methods, which we broadly

categorised as “classic” statistical methods or “modern” ML-based methods. In this question,

you are expected to further explore the differences between these classes of methods, and com-

pared to PCA (as a linear dimensionality reduction method). You should use no more than 3

pages to answer the following questions.

1. Find a reasonably high-dimensional (at least 100 dimensions) dataset that is interesting

to you. It should also have at least 100 instances, but preferably more. Describe the

dataset (name, related task/what it is used for, number of features and instances, refer-

ence) and justify your choice.

2. Using your choice of library, apply PCA to the dataset, and present your results. You

should show visualisation(s) of the PCs found and also comment on the explained vari-

ance.

3. Pick one of the “classic” manifold learning methods and apply it to the same data. Show

visualisation(s) and compare the results to that of PCA (e.g. for an embedding with two

dimensions). Highlight any differences between the two methods, and hypothesise why

they may have occurred.

4. Pick one of the “modern” manifold learning methods and apply it to the same data.

Show visualisation(s), compare and contrast the results to the two previous methods,

with analysis of any differences seen.

5. Finally, pick one of the two manifold learning methods for further analyse. Your method

will have “tunable parameters” — parameters that you can change to get different re-

sults. Pick one such parameter, and explore how sensitive the embedding is to changes

in this parameter. You should explain the role of this parameter in the manifold learning

algorithm, how you tested its effect, and show the results found.

1

2 Clustering [30 marks]

The NCI60 dataset (from the Stanford NC160 Cancer Microarray Project) consists of p = 6,830

gene expression measurements for each of n = 64 cancer cell lines. (Sourced from An Introduc-

tion to Statistical Learning).

In this question, you will be clustering the genes, rather than individual cancer cell lines.

This can be seen as a form of feature clustering — i.e. what genes are most related?

For each clustering method, you will need to visualise the clustering results (partition) for

that method. Given that there are 64 dimensions for each gene, for visualising the clustering

results, you should use PCA to reduce the dimensionality to 2D so that you can plot the found

clusters.

It is recommended you use either R or Python for this question, as they both have libraries

to interact with this data, as shown below. You should use no more than 3 pages to answer the

following questions.

2.1 R:

library(ISLR)

nci.data = NCI60$data

X = scale(t(nci.data))

P = X %*% prcomp(X)$rotation

X will be the numpy array of interest — note that we have transposed our data so our rows

are the genes. P is the principal components of the data. You will use the the first 2 PCs, i.e.

the first 2 columns of P, to visualise the clusters.

2.2 Python:

For Python (or any other language), you will need to first download nci60 data.csv from

the course homepage.

import pandas as pd

from sklearn.preprocessing import scale

from sklearn.decomposition import PCA

nci_data = pd.read_csv("nci60_data.csv", index_col=0)

X = scale(nci_data.T)

P = PCA().fit_transform(X)

X will be the data matrix of interest — note that we have transposed our data so our rows

are the genes. P is the principal components of the data. You will use the the first 2 PCs, i.e.

the first 2 columns of P, to visualise the clusters.

2.3 Tasks:

1. Carry out hierarchical clustering with Euclidean distance and complete linkage.

(a) Describe the resulting clustering for 3 to 6 clusters.

(b) Plot the first 2 principal components against each other with the colour argument

set equal to the cluster labels. What can you deduce/observe about the clustering?

2

2. Repeat the cluster analysis using correlation-based distance and complete linkage. NB:

you will need to precompute the correlations and pass them into your clustering method.

Compare the clusters with those found above.

3. Finally, carry out K-means clustering for 3 to 6 clusters. Compare the clusters of K-means

with that of the above two approaches. Which of the hierarchical clustering results is

more similar to that of K-means? Why?

3 Regression [30 marks]

In the lecture, we considered the case in which the features/predictors appeared only linearly

in the regression model. The simplest type of nonlinearity we could add to the model is pair-

wise interactions of the features. If xj and xk are distinct features, this means we also consider

xjxk as a feature. Pairwise interactions are rather straightforward to implement in R:

X = model.matrix(balance ～ . ? ., Credit)[,?1] (1)

becomes the new design matrix. The construction . ? . means consider all pairwise multiplica-

tions of distinct features.

Repeat the analysis for the Credit dataset (we have done it in the lecture on Week 7) with

pairwise interactions of the features. You will find it convenient to set grid = 10∧seq(3,?1, 100)

and thresh = 1e? 10.

Answer the following questions:

1. How many predictors are there, i.e. what is p?

2. How did you generate your training and test sets?

3. Select the tuning parameter for the ridge regression model using cross-validation, and

show the process.

4. Select the tuning parameter for the lasso regression model using cross-validation, and

show the process. How many features have been selected by the lasso?

5. Compare and discuss the final form of the model from the linear regression, ridge re-

gression, and lasso regression.

6. Compare the test errors for the linear model, ridge regression model, and lasso model.

7. Plot a comparison of the test predictions for the three approaches.

NB Please show how you generated your test and training sets – in particular the RNG

seed you used – so we can replicate your results. Remember to report this in the following

questions when applicable.

Assessment

Format: You can use any font to write the report, with a minimum of single spacing and 11

point size (hand writing is not permitted unless with approval from the lecturers). Reports

exceeding the maximum page limit will be penalised. Any additional material such as code or

figures/tables can be included in an appendix, which will not count towards the page limit.

3

Communication: a key skill required of a scientist is the ability to communicate effectively.

No matter the scientific merit of a report, if it is illegible, grammatically incorrect, mispunctu-

ated, ambiguous, or contains misspellings, it is less effective and marks will be deducted.

Marking Criteria: The final report will be submitted to Turnitin for a plagiarism check. Late

submissions without a pre-arranged extension will be penalised as per the course outline.

The usual mark checking procedures in place for all assessment apply to this report. The

assessment of the reports will account of the understanding of big data, clarity and accuracy

of answer, presentation, organisation, layout and referencing.

Submission: You are required to submit a single .pdf report through the web submission

system from the AIML427 course website by the due time.