Resources / Assignments (/COMP9321/18s2/resources/20310) / Week 6 (/COMP9321/18s2/resources/20312)
/ Assignment Description
Assignment Description
Data Service for World Bank Economic Indicators
In this assignment, you are asked to develop a Flask-Restplus data service that allows a client to read and store some publicly available
economic indicator data for countries around the world, and allow the consumers to access the data through a REST API.
IMPORTANT : For this assignment , you will be asked to access the given Web content programmatically. Some Web hosts do not allow
their web pages to be accessed programmatically, some hosts may block your IP if you access their pages too many times. During the
implementation, download a few test pages and access the content locally - try not to perform too many programmatically.
The source URL: http://api.worldbank.org/v2/ (http://api.worldbank.org/v2/)
(http://api.worldbank.org/v2/) Documentations on API Call Structure:
(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)
https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure
(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)
(https://datahelpdesk.worldbank.org/knowledgebase/articles/898581-api-basic-call-structure)
The World Bank indicators API provides 50 year of data over more than 1500 indicators for countries around the world.
List of indicators can be found at: http://api.w o rldbank.org/v2/indicators (http://api.worldbank.org/v2/indicators)
List of countries can be found at: (http://api.worldbank.org/v2/countries) http://api.worldbank.org/v2/countries
(http://api.worldbank.org/v2/countries)
As the documentation shows , you can construct URLs specifying different parameters to acquire necessary data. Use a custom URL
to get information about a specific indicator. For example, the data on the GDP of all countries from 2012 to 2017 can be acquired via
this URL: (http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017) http://api.worldbank.org
/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json (http://api.worldbank.org/v2/countries/all/indicators
/NY.GDP.MKTP.CD?date=2012:2017&format=json)
For this assignment we are interested in annual values of a user specified indicator from 2012 to 2017 for one or all countries. The
content of the file is quite straightforward. The data service specification will ask you to 'import' this data into a 'data storage'. You
should inspect the content of the file carefully and decide on a data model and storage method. You will find that a JSON format is
suitable to accessing data and publishing it.
You should use mLab : MongoDB as a service ( https://mlab.com/ (https://mlab.com/) ) for your data storage. This way
examiners can run your implementation to access your data stored remotely.
Assignment Specification
Specification Make Submission Check Submission Collect Submission
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
1 of 11 18/9/18, 09:33
The data service should use JSON format to publish its data and implement following operations.
1- Import a collection from the data service
This operation can be considered as an on-demand 'import' operation. The service will download the JSON data for all countries
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json) respective to the year 2012 to
2017 and identified by the indicator id given by the user and process the content into an internal data format and store it in the
database ( mLab ).
Parameters should be given to the endpoint (in the payload) by the user:
indicator_id : an indicator (http://api.worldbank.org/v2/indicators) http://api.worldbank.org/v2/indicators
(http://api.worldbank.org/v2/indicators)
After importing the collection, the service should return a response containing at least the following information:
location : the URL with which the imported collection can be retrieved
collection_id : a unique identifier automatically generated
creation_time : the time the collection stored in the database
Example:
HTTP operation: POST /<collections>
Input Payload:
{ "indicator_id" : "NY.GDP.MKTP.CD" }
Returns: 201 Created
{
"location" : "/<collections>/<collection_id>",
"collection_id" : "<collection_id>",
"creation_time": "2018-04-08T12:06:11Z",
"indicator" : "<indicator>"
}
The return response contains the location of the data entry created as the result of the processing the import. What is shown
above is just a suggestion. You can decide what else is appropriate. You do not have to return the actual imported content with
POST. Just return the location.
You should return appropriate responses in case of invalid indicators or any invalid attempts to use the endpoint ( e.g. If the input
indicator id doesn't exist in the data source, return error 400)
If an input contains a n indicator that already has been imported before, you should still return the location of the data entry - but
with status code 200 OK (instead of 20 1 Created).
A POINT TO PONDER ABOUT: An `asynchronous POST'?? If a POST takes too long, you may not want the client to wait. What
you would do? You do not need to address this in the assignment.
The source API has pagination; in order to get all of data you need to send many request to import a single collection; however,
you are required to get only first two pages instead of all:
http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json&page=2
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json&page=2)
The data entries inside the collection must be converted as described below:
Data entry conversion:
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
2 of 11 18/9/18, 09:33
Here is an example of source data entry as it is in the source API (http://api.worldbank.org/v2/countries/all/indicators
/NY.GDP.MKTP.CD?date=2012:2017&format=json) :
{
"indicator": {
"id": "NY.GDP.MKTP.CD",
"value": "GDP (current US$)"
},
"country": {
"id": "1A",
"value": "Arab World"
},
"countryiso3code": "",
"date": "2016",
"value": 2500164034395.78,
"unit": "",
"obs_status": "",
"decimal": 0
}
However, you do not need to store all of its attributes; instead convert it to a JSON format as below:
{
"country": "Arab World",
"date": "2016",
"value": 2500164034395.78
}
And as a result a collection should be formatted and stored in the database (mLab) as follow:
{
"collection_id" : "<collection_id>",
"indicator": "NY.GDP.MKTP.CD",
"indicator_value": "GDP (current US$)",
"creation_time" : "<creation_time>"
"entries" : [
{ "country": "Arab World", "date": "2016", "value": 2500164034395.78 },
{ "country": "Australia", "date": "2013", "value": 780016444034.00 },
...
]
}
2- Deleting a collection with the data service
This operation deletes an existing collection from the database. The interface should look like as below:
HTTP operation: DELETE /<collections>/{collection_id}
Returns: 200 OK
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
3 of 11 18/9/18, 09:33
{
"message" :"Collection = <collection_id> is removed from the database!"
}
3 - Retrieve the list of available collections
This operation retrieves all available collections. The interface should look like as like below:
HTTP operation: GET /<collections>
Returns: 200 OK
[
{
"location" : "/<collections>/<collection_id_1>",
"collection_id" : "collection_id_1",
"creation_time": "<time>",
"indicator" : "<indicator>"
},
{
"location" : "/<collections>/<collection_id_2>",
"collection_id" : "collection_id_2",
"creation_time": "<time>",
"indicator" : "<indicator>"
},
...
]
4 - Retrieve a collection
This operation retrieves a collection by its ID . The response of this operation will show the imported content from world bank API for all
6 years. That is, the data model that you have designed is visible here inside a JSON entry's content part.
The interface should look like as like below:
HTTP operation: GET /<collections>/{collection_id}
Returns: 200 OK
{
"collection_id" : "<collection_id>",
"indicator": "NY.GDP.MKTP.CD",
"indicator_value": "GDP (current US$)",
"creation_time" : "<creation_time>"
"entries" : [
{"country": "Arab World", "date": "2016", "value": 2500164034395.78 },
{"country": "Australia", "date": "2016", "value": 780016444034.00 },
...
]
}
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
4 of 11 18/9/18, 09:33
5 - Retrieve economic indicator value for given country and a year
The interface should look like as like below:
HTTP operation: GET /<collections>/{collection_id}/{year}/{country}
Returns: 200 OK
{
"collection_id": <collection_id>,
"indicator" : "<indicator_id>",
"country": "<country>,
"year": "<year">,
"value": <indicator_value_for_the_country>
}
6 - Retrieve top/bottom economic indicator values for a given year
The interface should look like as like below:
HTTP operation: GET /<collections>/{collection_id}/{year}?q=<query>
Returns: 200 OK
{
"indicator": "NY.GDP.MKTP.CD",
"indicator_value": "GDP (current US$)",
"entries" : [
{
"country": "Arab World",
"date": "2016",
"value": 2500164034395.78
},
...
]
}
The <query> is an optional parameter which can be either of following:
top<N> : Return top N countries sorted by indicator value
bottom<N> : Return bottom N countries sorted by indicator value
where N can be an integer value between 1 and 100. For example, a request like " GET /<collections>/< id>/2015?query=top10 "
should returns top 10 countries according to the collection_id.
Updates:
REASON : the source API will ban your (and tutors') IP addresses if you send too many requests
The source API has pagination; in order to get all of data you need to send many request to import a single collection; however,
you are required to get only first two pages instead of all:
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
5 of 11 18/9/18, 09:33
Resource created 14 days ago (Tuesday 04 September 2018, 08:15:08 PM), last modified a day ago (Monday 17 September 2018, 08:14:48 AM).
http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json&page=1
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json&page=1)
http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json&page=2
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json&page=2) or to get all data
use : http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json&per_page=2000
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json&per_page=2000)
In Q6: top1 should return the highest numeric value
Your code must implemented in flask-restplus and provide swagger doc for testing the endpoints.
Comments ! " (/COMP9321/18s2/forums/search?forum_choice=resource/20832) # (/COMP9321/18s2/forums/resource/20832)
$ Add a comment
Nan Li (/users/z5172069) about 8 hours ago (Tue Sep 18 2018 01:57:34 GMT+1000 (AEST))
Hi tutor,
I have a question on Q1, about the format of the data stored in the Mlab.
For the "creation_time", my value is a dictionary {"$data": "2018-09-17T15:50.583Z"}, not a string as
"<creation_time>" in the description. But my return value displayed to the customer is the same as the string format.
Is this ok?
Reply
Tony Tran (/users/z5113238) about 9 hours ago (Tue Sep 18 2018 00:12:52 GMT+1000 (AEST))
Hi there I was having trouble figuring out how to input a payload in q1 when there is no parameter section for the
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
6 of 11 18/9/18, 09:33
request: HTTP operation: POST /<collections>
Reply
Sarah Agbulos (/users/z5059829) about 12 hours ago (Mon Sep 17 2018 21:53:49 GMT+1000 (AEST))
is <collections> a fixed string that we define or defined by the user/tester when making requests?
my current implementation assumes that <collections> is a user provided string and handles accordingly e.g. if the
collection name provided doesn't exist in the database, an error returns or if the collection name provided does exist
in the database it behaves as normal.
should i be setting it instead to some arbitrary name such as "collection1" and assume that the user will not be trying
to make api calls to any other collection name?
Reply
Yuexuan Liu (/users/z5093599) about 14 hours ago (Mon Sep 17 2018 19:21:07 GMT+1000 (AEST))
In marking scheme, it said
"Endpoints must return appropriate messages/http status codes in any case"
what does it mean, do we have to handle all status codes for get/post/delete
For example, for GET, we have to handle 200,204,301,303,304,400,404?
Reply
Bojing Fu (/users/z5142286) about 15 hours ago (Mon Sep 17 2018 18:27:27 GMT+1000 (AEST))
Requirement said that we should store a collection like the following format, but the "collection_id" only available
after I do insertion. Must I modify the "_id" instead of "collection_id"?
This picture is requirement:
This picture is the collection stored in mlab:
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
7 of 11 18/9/18, 09:33
Reply
Tuan Nguyen (/users/z5055824) about 17 hours ago (Mon Sep 17 2018 16:16:05 GMT+1000 (AEST))
question: Do we need to build a frontend, or the code will be test on swagger / Postman
Reply
Yuxing Wu (/users/z5061719) about 21 hours ago (Mon Sep 17 2018 12:44:29 GMT+1000 (AEST))
Do we have to use the format for the list of entries in the field 'entries' like this:
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
8 of 11 18/9/18, 09:33
{"country": "Arab World", "date": "2016", "value": 2500164034395.78 }
{"country": "Arab World", "date": "2014", "value": 2500543234395.78 }
...
Or can it output in the response like this:
{
"country": "Arab World",
"date": "2016",
"value": 2500164034395.78
}
{
"country": "Arab World",
"date": "2014",
"value": 2500543234395.78
}
...
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 19 hours ago (Mon Sep 17 2018 14:27:28 GMT+1000 (AEST))
It must be JSON, and how it looks is not important
Reply
Qiming Ye (/users/z5057721) about 22 hours ago (Mon Sep 17 2018 11:48:50 GMT+1000 (AEST))
Does the order of JSON field matter? Do we have to follow the given example?
{ "location" : "/<collections>/<collection_id>", "collection_id" : "<collection_id>", "creation_time": "2018-0
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) about 22 hours ago (Mon Sep 17 2018 11:51:55 GMT+1000 (AEST)),
last modified about 22 hours ago (Mon Sep 17 2018 11:52:10 GMT+1000 (AEST))
Field order is not important
Reply
Bojing Fu (/users/z5142286) a day ago (Sun Sep 16 2018 23:20:32 GMT+1000 (AEST))
The Updates said that we need to get two pages instead of all. Does it mean I need to get two pages' data via these
two URLs from data source (world bank)? And I have to merge these two pages into one?
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
9 of 11 18/9/18, 09:33
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) a day ago (Mon Sep 17 2018 08:11:16 GMT+1000 (AEST))
Yes; however, you can get the collection completely if you can do it in only one request
Reply
Tuan Nguyen (/users/z5055824) a day ago (Mon Sep 17 2018 03:09:25 GMT+1000 (AEST)), last modified a day ago (Mon Sep 17
2018 03:09:51 GMT+1000 (AEST))
essentially, each page contain 50 entries, 2 pages contained 100 entries. Therefore, to fetch 2 pages woth of info
(100 entries) you can do:
http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json& per_page=
(http://api.worldbank.org/v2/countries/all/indicators/NY.GDP.MKTP.CD?date=2012:2017&format=json&page=2)
100
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) a day ago (Mon Sep 17 2018 08:12:14 GMT+1000 (AEST))
Thanks for the URL
Reply
Courteney Home (/users/z5116348) a day ago (Sun Sep 16 2018 21:14:19 GMT+1000 (AEST)), last modified a day ago (Sun Sep 16
2018 21:15:32 GMT+1000 (AEST))
What is the expected response in Question 6 for if a user specifies a top/bottom amount greater than the amount of
entries? For example, lets say there are 20 entries with the year 2014 in a collection. If I specify q=top21 I could
return an error (400 Bad Request?) or I could return all 20 entries (sorted).
Also, for getting the bottom<N> values I have sorted in ascending order and for the top<N> values I have sorted in
descending order. Is this correct?
Edit: should we be including entries with null values in our response?
Reply
Mohammadali Yaghoubzadehfard (/users/z5138589) a day ago (Mon Sep 17 2018 08:13:18 GMT+1000 (AEST))
No, in that case, you need to return only the existing entries.
if you have 10, then only return 10
Reply
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
10 of 11 18/9/18, 09:33
Load More Comments
Assignment Description | COMP9321 18s2 | WebCMS3 https://webcms3.cse.unsw.edu.au/COMP9321/18s2/resources/20832
11 of 11 18/9/18, 09:33
版权所有：留学生编程辅导网 2020 All Rights Reserved 联系方式：QQ:99515681 微信：codinghelp 电子信箱：99515681@qq.com
免责声明：本站部分内容从网络整理而来，只供参考！如有版权问题可联系本站删除。
