{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "hm20a-J7nLQL" }, "source": [ "# CUHK STAT5106 Mid-Term Take Home Exam\n", "## 25 Oct 2024 (Fri), 2200 - 27 Oct 2024 (Sun), 2159\n", "48 hours are given, but 9 hours are expected to be finished.
\n", "This is open-book; open-notes; and open-web exam. Welcome to utilize any resources.
\n", "Please follow the instructions step-by-step:\n", "- Please access into Blackboard \\> STAT5106 \\> Course Content \\> Mid-Term Take Home Exam\n", "- Access the link below, copy the ipynb to anywhere such as local computer, or personal cloud drive, so this becomes savable.
\n", "For the best experience - using **colab and your google drive account**.\n", "- Code and complete all questions in this ipynb.\n", "- Please keep on saving for protecting what you have input.
\n", "Finally you need to upload this to Blackboard account.\n", "- Upload this in Mid-Term Take Home Exam page, liked how you have submitted assignments.\n", "\n", "For any queries, please submit your questions on BLACKBOARD called **MID-TERM discussion room**. During the mid-term exam period - in the 48 hours. For fair the publicity to all students.\n", "\n", "Alan will answer most questions around the following time point:\n", "- 1000, 26 Oct 2024\n", "- 2200, 26 Oct 2024\n", "- 1000, 27 Oct 2024\n", "\n", "Some of them I would not answer if your question is not suitable - such as asking the full answers.\n", "\n", "DON'T email to tutors or me - we won't answer in the period.\n", "\n", "Thanks and good luck,
\n", "Alan" ] }, { "cell_type": "markdown", "metadata": { "id": "c5OJIueQuyPh" }, "source": [ "# Warm up Question (20%)\n", "\n", "What is the name of your lecturer ? (Adding prefix if necessary)" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "cellView": "form", "id": "GRfIGwWYvTAh" }, "outputs": [], "source": [ "Your_Answer = \"ALAN LO\" #@param {type:\"string\"}" ] }, { "cell_type": "markdown", "metadata": { "id": "kAKvAWCOotH4" }, "source": [ "# Finding the nearest available parking space\n", "\n", "
\n", "
\n", "\n", "You need to park your car. It can be difficult to find an available parking space during busy times. Let's look at the data behind the parking space availability app.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "cSea-HnRSxKV" }, "source": [ "# Task 1. Get Carpark location from data.gov.hk API (15%)\n", "\n", "The API page is here. ([In Traditional Chinese](https://data.gov.hk/tc-data/dataset/hk-dpo-datagovhk1-carpark-info-vacancy/resource/f4c792c6-071c-4a64-888b-afeea33d5ad7), [In English](https://data.gov.hk/en-data/dataset/hk-dpo-datagovhk1-carpark-info-vacancy/resource/01752c62-a6b6-4ddc-bf2d-25efccadc143))\n", "\n", "and the data dictionary is [here](https://resource.data.one.gov.hk/opendata/carpark/Parking_Vacancy_Data_Specification.pdf).\n", "\n", "**Write a script for calling API** to get all 471 (up to 2024-10-18) carparks.\n", "- In calling API, set the \"data\" parameter to be \"vacancy\"\n", "- set the \"vehicleTypes\" parameter to be \"privateCar\". We focus on private carpark spaces only.\n", "\n", "---\n", "\n", "Expected Results for a single carpark:\n", "\n", "```\n", "# js_row = js_result[0]\n", "\n", "{'park_Id': '10',\n", " 'name': 'Kai Tak Cruise Terminal Car Park 1',\n", " 'nature': 'commercial',\n", " 'carpark_Type': 'multi-storey',\n", " 'address': {'buildingName': 'Kai Tak Cruise Terminal',\n", " 'streetName': 'Shing Fung Road',\n", " 'buildingNo': '33',\n", " 'floor': '1',\n", " 'subDistrict': 'Kowloon Bay',\n", " 'dcDistrict': 'Kwun Tong District',\n", " 'region': 'KLN'},\n", " 'displayAddress': '1st floor, Kai Tak Cruise Terminal, 33 Shing Fung Road, Kowloon Bay, KLN',\n", " 'district': 'Kwun Tong District',\n", " 'latitude': 22.3062049,\n", " 'longitude': 114.21309471,\n", " 'contactNo': '+852 3465 6888, 09:30-18:00 Mon-Fri, except public holiday',\n", "\n", "# ... Skipping in the middle ... #\n", "\n", " 'LGV': {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, 'space': 0},\n", " 'HGV': {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, 'space': 0},\n", " 'coach': {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, 'space': 0},\n", " 'motorCycle': {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, 'space': 0},\n", " 'creationDate': '2016-08-16 10:03:56',\n", " 'modifiedDate': '2024-01-03 13:28:25',\n", " 'publishedDate': '2018-12-12 12:22:47',\n", " 'lang': 'en_US'}\n", "```\n", "\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "eznQytp966Cz" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: requests in c:\\python312\\lib\\site-packages (2.32.3)Note: you may need to restart the kernel to use updated packages.\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "WARNING: Ignoring invalid distribution ~ (c:\\Python312\\Lib\\site-packages)\n", "WARNING: Ignoring invalid distribution ~ip (c:\\Python312\\Lib\\site-packages)\n", "DEPRECATION: Loading egg at c:\\python312\\lib\\site-packages\\vboxapi-1.0-py3.12.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330\n", "WARNING: Ignoring invalid distribution ~ (c:\\Python312\\Lib\\site-packages)\n", "WARNING: Ignoring invalid distribution ~ip (c:\\Python312\\Lib\\site-packages)\n", "WARNING: Ignoring invalid distribution ~ (c:\\Python312\\Lib\\site-packages)\n", "WARNING: Ignoring invalid distribution ~ip (c:\\Python312\\Lib\\site-packages)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Requirement already satisfied: charset-normalizer<4,>=2 in c:\\python312\\lib\\site-packages (from requests) (3.3.2)\n", "Requirement already satisfied: idna<4,>=2.5 in c:\\python312\\lib\\site-packages (from requests) (3.7)\n", "Requirement already satisfied: urllib3<3,>=1.21.1 in c:\\python312\\lib\\site-packages (from requests) (2.2.2)\n", "Requirement already satisfied: certifi>=2017.4.17 in c:\\python312\\lib\\site-packages (from requests) (2024.7.4)\n" ] } ], "source": [ "%pip install requests" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "id": "hBysnKoh6579" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'park_Id': '10',\n", " 'privateCar': [{'lastupdate': '2024-10-26 01:05:32',\n", " 'vacancy': 109,\n", " 'vacancy_type': 'A'}]}\n", "'length of json: 472'\n" ] } ], "source": [ "# to fullfill question\n", "# https://api.data.gov.hk/v1/carpark-info-vacancy?data=vacancy&vehicleTypes=privateCar\n", "import requests\n", "from pprint import pprint\n", "\n", "url = \"https://api.data.gov.hk/v1/carpark-info-vacancy\"\n", "params = {\n", " \"data\": \"vacancy\",\n", " \"vehicleTypes\": \"privateCar\"\n", "}\n", "\n", "response = requests.get(url, params=params)\n", "js_result = response.json()['results']\n", "\n", "pprint(js_result[0])\n", "pprint(\"length of json: \"+str(len(js_result)))" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "id": "u3gqYUGH65OM" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "{'HGV': {'space': 0, 'spaceDIS': 0, 'spaceEV': 0, 'spaceUNL': 0},\n", " 'LGV': {'space': 0, 'spaceDIS': 0, 'spaceEV': 0, 'spaceUNL': 0},\n", " 'address': {'buildingName': 'Kai Tak Cruise Terminal',\n", " 'buildingNo': '33',\n", " 'dcDistrict': 'Kwun Tong District',\n", " 'floor': '1',\n", " 'region': 'KLN',\n", " 'streetName': 'Shing Fung Road',\n", " 'subDistrict': 'Kowloon Bay'},\n", " 'carpark_Type': 'multi-storey',\n", " 'coach': {'space': 0, 'spaceDIS': 0, 'spaceEV': 0, 'spaceUNL': 0},\n", " 'contactNo': '+852 3465 6888, 09:30-18:00 Mon-Fri, except public holiday',\n", " 'creationDate': '2016-08-16 10:03:56',\n", " 'displayAddress': '1st floor, Kai Tak Cruise Terminal, 33 Shing Fung Road, '\n", " 'Kowloon Bay, KLN',\n", " 'district': 'Kwun Tong District',\n", " 'facilities': ['disabilities', 'evCharger'],\n", " 'gracePeriods': [{'minutes': 10}],\n", " 'heightLimits': [{'height': 2}],\n", " 'lang': 'en_US',\n", " 'latitude': 22.3062049,\n", " 'longitude': 114.21309471,\n", " 'modifiedDate': '2024-01-03 13:28:25',\n", " 'motorCycle': {'space': 0, 'spaceDIS': 0, 'spaceEV': 0, 'spaceUNL': 0},\n", " 'name': 'Kai Tak Cruise Terminal Car Park 1',\n", " 'nature': 'commercial',\n", " 'openingHours': [{'excludePublicHoliday': False,\n", " 'periodEnd': '23:00',\n", " 'periodStart': '07:00',\n", " 'weekdays': ['MON',\n", " 'TUE',\n", " 'WED',\n", " 'THU',\n", " 'FRI',\n", " 'SAT',\n", " 'SUN',\n", " 'PH']}],\n", " 'opening_status': 'OPEN',\n", " 'park_Id': '10',\n", " 'paymentMethods': ['octopus', 'visa'],\n", " 'privateCar': {'hourlyCharges': [{'covered': 'covered',\n", " 'excludePublicHoliday': True,\n", " 'periodEnd': '23:00',\n", " 'periodStart': '07:00',\n", " 'price': 15,\n", " 'remark': '',\n", " 'type': 'hourly',\n", " 'usageMinimum': 1,\n", " 'weekdays': ['MON',\n", " 'TUE',\n", " 'WED',\n", " 'THU',\n", " 'FRI']},\n", " {'covered': 'covered',\n", " 'excludePublicHoliday': False,\n", " 'periodEnd': '23:00',\n", " 'periodStart': '07:00',\n", " 'price': 20,\n", " 'remark': '',\n", " 'type': 'hourly',\n", " 'usageMinimum': 1,\n", " 'weekdays': ['SAT', 'SUN', 'PH']}],\n", " 'space': 112,\n", " 'spaceDIS': 0,\n", " 'spaceEV': 0,\n", " 'spaceUNL': 0},\n", " 'publishedDate': '2018-12-12 12:22:47',\n", " 'renditionUrls': {'banner': 'https://sps-opendata.pilotsmartke.gov.hk/rest/getRendition/fs-1%3A693265207413252869411532657339312395903827562313.JPG/banner.png',\n", " 'square': 'https://sps-opendata.pilotsmartke.gov.hk/rest/getRendition/fs-1%3A693265207413252869411532657339312395903827562313.JPG/square.png',\n", " 'thumbnail': 'https://sps-opendata.pilotsmartke.gov.hk/rest/getRendition/fs-1%3A693265207413252869411532657339312395903827562313.JPG/thumbnail.png'},\n", " 'website': 'http://www.kaitakcruiseterminal.com.hk/'}\n", "'length of json: 478'\n" ] } ], "source": [ "# to fullfill answer\n", "# https://api.data.gov.hk/v1/carpark-info-vacancy\n", "# to notice that the url is differ to the question provided\n", "import requests\n", "from pprint import pprint\n", "\n", "url = \"https://api.data.gov.hk/v1/carpark-info-vacancy\"\n", "\n", "response = requests.get(url)\n", "js_result = response.json()['results']\n", "\n", "pprint(js_result[0])\n", "pprint(\"length of json: \"+str(len(js_result)))" ] }, { "cell_type": "markdown", "metadata": { "id": "PHa-82SOXNsv" }, "source": [ "# Task 2. Load the json extracts into pandas data frame (40%)\n", "\n", "With the following instructions:\n", "- The following columns are needed:\n", " - park_Id\n", " - name\n", " - displayAddress\n", " - district\n", " - latitude\n", " - longitude\n", " - opening_status\n", " - facilities\n", " - paymentMethods\n", " - modifiedDate\n", "- 2A: Write a function for getting URL of carpark image.\n", " - and insert new column called \"url_image\"\n", "- 2B: Write a python script for scrapping the dates of public holidays.\n", "- 2C: Write a function for getting carpark today information, such as:\n", " - Carpark opening time of today: \"periodStart\"\n", " - Carpark closing time of today: \"periodEnd\"\n", " - Carpark Hourly Charge: \"price\"\n", " - Carpark Total Parking Space: \"space\" \n", "- 2D: Combining all the stuffs, writing python script for outputting a pandas DataFrame.\n", "\n", "---\n", "\n", "Expected Output:\n", "\n", "![Example Image](https://drive.google.com/uc?id=1Jbjqx7ghZTYyyigml_tqJK2JxDTD5Etp)\n", "\n", "Note the dataset is with dimensions of 471 rows and 17 columns." ] }, { "cell_type": "markdown", "metadata": { "id": "elhOaT-NdLBS" }, "source": [ "## 2A. Write a function: Get URL of image of each carpark (10%)\n", "We search anyone URL in the dictionaries 'renditionUrls', with the format:\n", "\n", "```\n", "'renditionUrls': {'square': 'https://sps-opendata.pilotsmartke.gov.hk/rest/getRendition/fs-1%3A693265207413252869411532657339312395903827562313.JPG/square.png',\n", "'thumbnail': 'https://sps-opendata.pilotsmartke.gov.hk/rest/getRendition/fs-1%3A693265207413252869411532657339312395903827562313.JPG/thumbnail.png',\n", "'banner': 'https://sps-opendata.pilotsmartke.gov.hk/rest/getRendition/fs-1%3A693265207413252869411532657339312395903827562313.JPG/banner.png'}\n", "```\n", "\n", "**Write a function \"get_url_image\"** with\n", "- input as \"js_row\", the json item for a single carpark\n", "- return output as the first URL item in the dictionary\n", "\n", "Besides, in the function you need\n", "- If dictionary 'renditionUrls' cannot be found, return None.\n", "- If 'renditionUrls' is found but not the URL, return None as well.\n", "\n", "---\n", "\n", "Expected Output:\n", "\n", "\n", "```\n", "# get_url_image(js_row)\n", "'https://sps-opendata.pilotsmartke.gov.hk/rest/getRendition/fs-1%3A693265207413252869411532657339312395903827562313.JPG/square.png'\n", "```\n", "\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "9q8VyUgFdXPu" }, "outputs": [], "source": [ "def get_url_image(js_row):\n", " # ... your answer\n", " url_image = ''\n", " return url_image" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "iu7Isd3H7LeD" }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": { "id": "FhQml8q_oqsq" }, "source": [ "## 2B. Write a script: Get Public Holidays in Hong Kong (10%)\n", "\n", "**Write a script for webscrapping with package BeautifulSoup** through the website: [General holidays for 2024](https://www.gov.hk/en/about/abouthk/holiday/2024.htm)\n", "\n", "save those dates in format 'YYYY-mm-dd' packed with a list in the variable 'ph_dates', as following:\n", "\n", "```\n", "# ph_dates\n", "\n", "['2024-01-01',\n", " '2024-02-10',\n", " '2024-02-12',\n", " '2024-02-13',\n", " '2024-03-29',\n", " '2024-03-30',\n", " '2024-04-01',\n", " '2024-04-04',\n", " '2024-05-01',\n", " '2024-05-15',\n", " '2024-06-10',\n", " '2024-07-01',\n", " '2024-09-18',\n", " '2024-10-01',\n", " '2024-10-11',\n", " '2024-12-25',\n", " '2024-12-26']\n", "```\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "IH1C4Tse7Osh" }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "yxGrHxqZ7Opt" }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": 6, "metadata": { "id": "raUQhaQ47OkJ" }, "outputs": [], "source": [ "# ph_dates = ...your answer..." ] }, { "cell_type": "markdown", "metadata": { "id": "oABeEM7Iep8D" }, "source": [ "## 2C. Write a function: Get information for each carpark, according to the date inputted. (10%)\n", "\n", "The hours opening/closing, hourly charge, and parking spaces can be found in dictionaries 'privateCar' and nested dictionary 'hourlyCharges', such as the following example:\n", "\n", "```\n", " 'privateCar': {'hourlyCharges': [{'weekdays': ['MON',\n", " 'TUE',\n", " 'WED',\n", " 'THU',\n", " 'FRI'],\n", " 'excludePublicHoliday': True,\n", " 'periodStart': '07:00',\n", " 'periodEnd': '23:00',\n", " 'price': 15,\n", " 'type': 'hourly',\n", " 'covered': 'covered',\n", " 'usageMinimum': 1,\n", " 'remark': ''},\n", " {'weekdays': ['SAT', 'SUN', 'PH'],\n", " 'excludePublicHoliday': False,\n", " 'periodStart': '07:00',\n", " 'periodEnd': '23:00',\n", " 'price': 20,\n", " 'usageMinimum': 1,\n", " 'type': 'hourly',\n", " 'covered': 'covered',\n", " 'remark': ''}],\n", " 'spaceUNL': 0,\n", " 'spaceEV': 0,\n", " 'spaceDIS': 0,\n", " 'space': 112},\n", "```\n", "\n", "**Write a function \"get_todayinfo\"** with\n", "- input as\n", " - \"js_row\", the json item for a single carpark\n", " - \"ph_dates\", the list of dates in string of public holidays\n", " - \"today_date_str\", the date in string with format \"YYYY-mm-dd\"\n", "- return a dictionary with\n", " - \"periodStart\", time when carpark is opened\n", " - \"periodEnd\", time when carpark is closed\n", " - \"price\", hourly charge\n", " - \"space\", no. of parking spaces in the carpark.\n", "\n", "Besides, in the function you need\n", "- Set to None if \"periodStart\", \"periodEnd\", or \"price\" have not been mentioned in data\n", "- Set to 0 if \"space\" has not been mentioned in data.\n", "- default value of parameter \"today_date_str\" is assigned to be None. Then today date is assigned in the function with variable \"today_date_str\"\n", "\n", "Be CAUTION: For some carparks, the keys of some dictionaries may be missed. You need to set default value once you cannot call some value in the dictionary.\n", "\n", "---\n", "\n", "Expected Outputs:\n", "\n", "```\n", "# get_todayinfo(js_row, ph_dates)\n", "\n", "{'periodStart': '07:00',\n", " 'periodEnd': '23:00',\n", " 'price': 15,\n", " 'space': 112,\n", " 'today': '2024-10-18',\n", " 'today_weekday': 'FRI'}\n", "\n", "# get_todayinfo(js_row, ph_dates, '2024-12-25')\n", "\n", "{'periodStart': '07:00',\n", " 'periodEnd': '23:00',\n", " 'price': 20,\n", " 'space': 112,\n", " 'today': '2024-12-25',\n", " 'today_weekday': 'PH'}\n", "```\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "id": "64hacEppffjy" }, "outputs": [ { "ename": "SyntaxError", "evalue": "invalid syntax (772246852.py, line 1)", "output_type": "error", "traceback": [ "\u001b[1;36m Cell \u001b[1;32mIn[7], line 1\u001b[1;36m\u001b[0m\n\u001b[1;33m def get_todayinfo(...):\u001b[0m\n\u001b[1;37m ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n" ] } ], "source": [ "def get_todayinfo(...):\n", "\n", " #... your answer...\n", " dict_todayinfo = {}\n", "\n", " return dict_todayinfo\n" ] }, { "cell_type": "markdown", "metadata": { "id": "lMPED8aQpbzh" }, "source": [ "## 2D. Putting all the things together - Output the pandas data frame (10%)\n", "\n", "**Write a script to get the full dataset merged**.\n", "\n", "Expected Output:\n", "\n", "![Example Image](https://drive.google.com/uc?id=1Jbjqx7ghZTYyyigml_tqJK2JxDTD5Etp)\n", "\n", "Note the dataset is with dimensions of 471 rows and 17 columns.\n", "\n", "Save the dataset with name \"df_carpark\"." ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "id": "FciJdfFS7oKz" }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
park_Idnamenaturecarpark_TypeaddressdisplayAddressdistrictlatitudelongitudecontactNo...paymentMethodsprivateCarLGVHGVcoachmotorCyclecreationDatemodifiedDatepublishedDatelang
010Kai Tak Cruise Terminal Car Park 1commercialmulti-storey{'buildingName': 'Kai Tak Cruise Terminal', 's...1st floor, Kai Tak Cruise Terminal, 33 Shing F...Kwun Tong District22.306205114.213095+852 3465 6888, 09:30-18:00 Mon-Fri, except pu......[octopus, visa]{'hourlyCharges': [{'weekdays': ['MON', 'TUE',...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...2016-08-16 10:03:562024-01-03 13:28:252018-12-12 12:22:47en_US
112Amoy Plazacommercialmulti-storey{'region': 'KLN', 'dcDistrict': 'Kwun Tong Dis...77 Ngau Tau Kok Raod, Kowloon Bay, KLNKwun Tong District22.324701114.2167532756 3666...[cash, octopus]{'privileges': [{'weekdays': ['MON', 'TUE', 'W...{'privileges': [{'weekdays': ['MON', 'TUE', 'W...{'privileges': [{'weekdays': ['MON', 'TUE', 'W...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'monthlyCharges': [{'price': 1100, 'type': 'm...2016-09-05 17:44:102024-07-12 21:55:172017-08-15 09:32:17en_US
227Yau Lai Shopping Centre Carparkgovernmentmulti-storey{'region': 'KLN', 'dcDistrict': 'Kwun Tong Dis...LG1 & LG2, Yau Lai Shopping Centre, 9 Yau Tong...Kwun Tong District22.298512114.23740123530667...[octopus]{'privileges': [{'weekdays': ['MON', 'TUE', 'W...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'hourlyCharges': [{'weekdays': ['MON', 'TUE',...2016-09-27 09:15:232022-07-22 15:47:422016-12-30 12:21:58en_US
330Telford Plaza D Carparkcommercialmulti-storey{'streetName': '33 Wai Yip Street', 'subDistri...33 Wai Yip Street, Kowloon Bay, KLNKwun Tong District22.322289114.2120432997 2262...[octopus, visa, master]{'hourlyCharges': [{'weekdays': ['MON', 'TUE',...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...2016-11-25 15:45:542018-01-31 17:20:142017-02-13 09:41:31en_US
431Telford Plaza I Carparkcommercialmulti-storey{'region': 'KLN', 'dcDistrict': 'Kwun Tong Dis...33 Wai Yip Street, Kowloon Bay, KLNKwun Tong District22.322686114.2135352997 2262...[octopus, visa, master]{'hourlyCharges': [{'weekdays': ['MON', 'TUE',...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...{'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '...2016-11-25 15:49:172018-01-31 17:19:102017-02-13 09:42:13en_US
..................................................................
473tdstt93Hoi Yu Street Car Park No 1NaNNaNNaNHoi Yu Street Quarry BayEastern22.290065114.214859...NaNNaNNaNNaNNaNNaNNaNNaNNaNen_US
474tdc150p1Ying Tung Estate CarparkNaNNaNNaN12 YING TUNG ROADIslands22.296614113.9530832702 8911...NaNNaNNaNNaNNaNNaNNaNNaNNaNen_US
475tdstt97On Sau RoadNaNNaNNaNOn Sau Road, KowloonKwun Tong22.325902114.231010...NaNNaNNaNNaNNaNNaNNaNNaNNaNen_US
476tdstt100Hoi Wah Road (II) Car ParkNaNNaNNaNHoi Wah Road, Area 16, Tuen Mun, New TerritoriesTuen Mun22.381421113.97392995428756...NaNNaNNaNNaNNaNNaNNaNNaNNaNen_US
477tdc177p1KTR350 Car ParkNaNNaNNaN350 Kwun Tong Road, Kwun Tong, KowloonKwun Tong22.315039114.21800021168259...NaNNaNNaNNaNNaNNaNNaNNaNNaNen_US
\n", "

478 rows × 27 columns

\n", "
" ], "text/plain": [ " park_Id name nature carpark_Type \\\n", "0 10 Kai Tak Cruise Terminal Car Park 1 commercial multi-storey \n", "1 12 Amoy Plaza commercial multi-storey \n", "2 27 Yau Lai Shopping Centre Carpark government multi-storey \n", "3 30 Telford Plaza D Carpark commercial multi-storey \n", "4 31 Telford Plaza I Carpark commercial multi-storey \n", ".. ... ... ... ... \n", "473 tdstt93 Hoi Yu Street Car Park No 1 NaN NaN \n", "474 tdc150p1 Ying Tung Estate Carpark NaN NaN \n", "475 tdstt97 On Sau Road NaN NaN \n", "476 tdstt100 Hoi Wah Road (II) Car Park NaN NaN \n", "477 tdc177p1 KTR350 Car Park NaN NaN \n", "\n", " address \\\n", "0 {'buildingName': 'Kai Tak Cruise Terminal', 's... \n", "1 {'region': 'KLN', 'dcDistrict': 'Kwun Tong Dis... \n", "2 {'region': 'KLN', 'dcDistrict': 'Kwun Tong Dis... \n", "3 {'streetName': '33 Wai Yip Street', 'subDistri... \n", "4 {'region': 'KLN', 'dcDistrict': 'Kwun Tong Dis... \n", ".. ... \n", "473 NaN \n", "474 NaN \n", "475 NaN \n", "476 NaN \n", "477 NaN \n", "\n", " displayAddress district \\\n", "0 1st floor, Kai Tak Cruise Terminal, 33 Shing F... Kwun Tong District \n", "1 77 Ngau Tau Kok Raod, Kowloon Bay, KLN Kwun Tong District \n", "2 LG1 & LG2, Yau Lai Shopping Centre, 9 Yau Tong... Kwun Tong District \n", "3 33 Wai Yip Street, Kowloon Bay, KLN Kwun Tong District \n", "4 33 Wai Yip Street, Kowloon Bay, KLN Kwun Tong District \n", ".. ... ... \n", "473 Hoi Yu Street Quarry Bay Eastern \n", "474 12 YING TUNG ROAD Islands \n", "475 On Sau Road, Kowloon Kwun Tong \n", "476 Hoi Wah Road, Area 16, Tuen Mun, New Territories Tuen Mun \n", "477 350 Kwun Tong Road, Kwun Tong, Kowloon Kwun Tong \n", "\n", " latitude longitude contactNo \\\n", "0 22.306205 114.213095 +852 3465 6888, 09:30-18:00 Mon-Fri, except pu... \n", "1 22.324701 114.216753 2756 3666 \n", "2 22.298512 114.237401 23530667 \n", "3 22.322289 114.212043 2997 2262 \n", "4 22.322686 114.213535 2997 2262 \n", ".. ... ... ... \n", "473 22.290065 114.214859 \n", "474 22.296614 113.953083 2702 8911 \n", "475 22.325902 114.231010 \n", "476 22.381421 113.973929 95428756 \n", "477 22.315039 114.218000 21168259 \n", "\n", " ... paymentMethods \\\n", "0 ... [octopus, visa] \n", "1 ... [cash, octopus] \n", "2 ... [octopus] \n", "3 ... [octopus, visa, master] \n", "4 ... [octopus, visa, master] \n", ".. ... ... \n", "473 ... NaN \n", "474 ... NaN \n", "475 ... NaN \n", "476 ... NaN \n", "477 ... NaN \n", "\n", " privateCar \\\n", "0 {'hourlyCharges': [{'weekdays': ['MON', 'TUE',... \n", "1 {'privileges': [{'weekdays': ['MON', 'TUE', 'W... \n", "2 {'privileges': [{'weekdays': ['MON', 'TUE', 'W... \n", "3 {'hourlyCharges': [{'weekdays': ['MON', 'TUE',... \n", "4 {'hourlyCharges': [{'weekdays': ['MON', 'TUE',... \n", ".. ... \n", "473 NaN \n", "474 NaN \n", "475 NaN \n", "476 NaN \n", "477 NaN \n", "\n", " LGV \\\n", "0 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", "1 {'privileges': [{'weekdays': ['MON', 'TUE', 'W... \n", "2 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", "3 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", "4 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", ".. ... \n", "473 NaN \n", "474 NaN \n", "475 NaN \n", "476 NaN \n", "477 NaN \n", "\n", " HGV \\\n", "0 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", "1 {'privileges': [{'weekdays': ['MON', 'TUE', 'W... \n", "2 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", "3 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", "4 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", ".. ... \n", "473 NaN \n", "474 NaN \n", "475 NaN \n", "476 NaN \n", "477 NaN \n", "\n", " coach \\\n", "0 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", "1 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", "2 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", "3 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", "4 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... \n", ".. ... \n", "473 NaN \n", "474 NaN \n", "475 NaN \n", "476 NaN \n", "477 NaN \n", "\n", " motorCycle creationDate \\\n", "0 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... 2016-08-16 10:03:56 \n", "1 {'monthlyCharges': [{'price': 1100, 'type': 'm... 2016-09-05 17:44:10 \n", "2 {'hourlyCharges': [{'weekdays': ['MON', 'TUE',... 2016-09-27 09:15:23 \n", "3 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... 2016-11-25 15:45:54 \n", "4 {'spaceUNL': 0, 'spaceEV': 0, 'spaceDIS': 0, '... 2016-11-25 15:49:17 \n", ".. ... ... \n", "473 NaN NaN \n", "474 NaN NaN \n", "475 NaN NaN \n", "476 NaN NaN \n", "477 NaN NaN \n", "\n", " modifiedDate publishedDate lang \n", "0 2024-01-03 13:28:25 2018-12-12 12:22:47 en_US \n", "1 2024-07-12 21:55:17 2017-08-15 09:32:17 en_US \n", "2 2022-07-22 15:47:42 2016-12-30 12:21:58 en_US \n", "3 2018-01-31 17:20:14 2017-02-13 09:41:31 en_US \n", "4 2018-01-31 17:19:10 2017-02-13 09:42:13 en_US \n", ".. ... ... ... \n", "473 NaN NaN en_US \n", "474 NaN NaN en_US \n", "475 NaN NaN en_US \n", "476 NaN NaN en_US \n", "477 NaN NaN en_US \n", "\n", "[478 rows x 27 columns]" ] }, "execution_count": 10, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "import json\n", "\n", "# Load and parse the JSON file\n", "with open('js_result.json', 'r') as file:\n", " test_json = json.load(file)\n", "\n", "\n", "# Translate the JSON data to a pandas DataFrame\n", "test_df = pd.DataFrame(test_json)\n", "\n", "test_df" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Fqb-beuy7oIR" }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7vuMCITU7oF9" }, "outputs": [], "source": [ "# df_carpark = ...your answer..." ] }, { "cell_type": "markdown", "metadata": { "id": "lF63KHwqznj4" }, "source": [ "# Task 3. Get Real-time Vacancy for each carpark (15%)\n", "\n", "Besides the carpark information, the API can also be called for the real-time vacancies.\n", "\n", "Please re-check the following:\n", "- The API page is here. ([In Traditional Chinese](https://data.gov.hk/tc-data/dataset/hk-dpo-datagovhk1-carpark-info-vacancy/resource/f4c792c6-071c-4a64-888b-afeea33d5ad7), [In English](https://data.gov.hk/en-data/dataset/hk-dpo-datagovhk1-carpark-info-vacancy/resource/01752c62-a6b6-4ddc-bf2d-25efccadc143))\n", "- and the data dictionary is [here](https://resource.data.one.gov.hk/opendata/carpark/Parking_Vacancy_Data_Specification.pdf).\n", "\n", "**Write a script for obtaining the full pandas data frame**, with the following instructions:\n", "- In calling API, set the \"data\" parameter to be \"vacancy\"\n", "- set the \"vehicleType\" parameter to be \"privateCar\" as well\n", "- 3 columns are in dataset\n", " - \"park_Id\", for joining the carpark info dataset.\n", " - \"vacancy\", the no. of available carpark spaces\n", " - \"lastupdate\", Last Update time of the \"vacancy\" number.\n", "- Only \"vacancy_type\" = \"A\", the vacancy no. will be taken into account.\n", "- In default, set \"vacancy\" as 0, and \"lastupdate\" as None, if the carpark does not have this information.\n", "- Name the dataset as \"df_vacancy\".\n", "\n", "---\n", "\n", "Expected JSON for a single carpark\n", "\n", "```\n", "# js_result[0]\n", "\n", "{'park_Id': '10',\n", " 'privateCar': [{'vacancy_type': 'A',\n", " 'vacancy': 29,\n", " 'lastupdate': '2024-10-18 16:04:59'}]}\n", "```\n", "\n", "Expected Output\n", "\n", "![Example Image](https://drive.google.com/uc?id=1I2NhcYdpQedgSw53k1nq5xKBeA6KKBXu)\n", "\n", "Note the vacancy dataset is still with 471 rows, same as the carpark info dataset.\n", "\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "cB9zDNPG7vXd" }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "MhBS62lS7vVF" }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "IeuVM5-P7vPL" }, "outputs": [], "source": [ "# df_vacancy = ...your answer..." ] }, { "cell_type": "markdown", "metadata": { "id": "mdELHx582xC9" }, "source": [ "# Task 4. Join both datasets (10%)\n", "\n", "Finally, **write a script to join df_carpark and df_vacancy**, with the following instruction.\n", "- Applying \"Left-Join\"\n", "- on the common column \"park_Id\"\n", "- saving as \"df_out\"\n", "\n", "---\n", "\n", "Expected Output\n", "\n", "![Example Image](https://drive.google.com/uc?id=1Nlc9PQ74rlqjQT_Oj200m0Vpg4om39lv)\n", "\n", "Note the dataset is with dimensions of 471 rows and 19 columns.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LYIZvKU37xxR" }, "outputs": [], "source": [ " # df_out = ...your answer..." ] }, { "cell_type": "markdown", "metadata": { "id": "fG6fLzPr3FiO" }, "source": [ "\n", "\n", "Finally we can make the following !\n", "\n", "![Example Image](https://drive.google.com/uc?id=1-I14TdnpkGwwObWs1ZNGt2Ufb2NMIBSH)\n", "\n", "The dataset can be used in Tableau dashboard which will be introduced in the lesson later." ] }, { "cell_type": "markdown", "metadata": { "id": "k1eYGlV6x8r4" }, "source": [ "# Congratuations ! That's all for the mid-term exam !
\n", "# Please save this and upload this jupyter notebook to Blackboard.
\n", "# Thanks !" ] } ], "metadata": { "colab": { "provenance": [], "toc_visible": true }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.4" } }, "nbformat": 4, "nbformat_minor": 0 }