1 line
12 KiB
Plaintext
1 line
12 KiB
Plaintext
{"nbformat_minor": 2, "cells": [{"source": "# HW3 Q4 [10 pts]\n\n\n\n## Important Notices\n\n<div class=\"alert alert-block alert-danger\">\n WARNING: Do <strong>NOT</strong> add any cells to this Jupyter Notebook, because that will crash the autograder.\n</div>\n\n\nAll instructions, code comments, etc. in this notebook **are part of the assignment instructions**. That is, if there is instructions about completing a task in this notebook, that task is not optional. \n\n\n\n<div class=\"alert alert-block alert-info\">\n You <strong>must</strong> implement the following functions in this notebook to receive credit.\n</div>\n\n`user()`\n\n`load_data()`\n\n`exclude_no_pickuplocations()`\n\n`exclude_no_tripdistance()`\n\n`include_fare_range()`\n\n`get_highest_tip()`\n\n`get_total_toll()`\n\nEach method will be auto-graded using different sets of parameters or data, to ensure that values are not hard-coded. You may assume we will only use your code to work with data from NYC Taxi Trips during auto-grading. You do not need to write code for unreasonable scenarios. \n\nSince the overall correctness of your code will require multiple function to work together correctly (i.e., all methods are interdepedent), implementing only a subset of the functions likely will lead to a low score.\n\n### Helper functions\n\nYou are permitted to write additional helper functions, or use additional instance variables so long as the previously described functions work as required. ", "cell_type": "markdown", "metadata": {}}, {"source": "#### Pyspark Imports\n<span style=\"color:red\">*Please don't modify the below cell*</span>", "cell_type": "markdown", "metadata": {}}, {"execution_count": 26, "cell_type": "code", "source": "import pyspark\nfrom pyspark.sql import SQLContext\nfrom pyspark.sql import *", "outputs": [], "metadata": {}}, {"source": "#### Define Spark Context\n<span style=\"color:red\">*Please don't modify the below cell*</span>", "cell_type": "markdown", "metadata": {}}, {"execution_count": 2, "cell_type": "code", "source": "sc\nsqlContext = SQLContext(sc)", "outputs": [], "metadata": {}}, {"source": "### Student Section - Please compete all the functions below", "cell_type": "markdown", "metadata": {}}, {"source": "#### Function to return GT Username", "cell_type": "markdown", "metadata": {}}, {"execution_count": 3, "cell_type": "code", "source": "def user():\n \"\"\"\n :return: string\n your GTUsername, NOT your 9-Digit GTId \n \"\"\" \n return 'icanlapan3'", "outputs": [], "metadata": {}}, {"source": "#### Function to load data", "cell_type": "markdown", "metadata": {}}, {"execution_count": 39, "cell_type": "code", "source": "def load_data(gcp_storage_path):\n \"\"\"\n :param gcp_storage_path: string (full gs path including file name e.g gs://bucket_name/data.csv) \n :return: spark dataframe \n \"\"\"\n \n ################################################################\n # code to load yellow_tripdata_2019-01.csv data from your GCP storage bucket#\n\n df = spark.read.option (\"inferSchema\" , \"true\").option (\"header\" , \"true\") \\\n .csv(gcp_storage_path)\n \n # Convert to decimal trip_distance, fare_amount, tip_amount, tolls_amount\n \n df = df.withColumn(\"trip_distance\",df[\"trip_distance\"].cast('Decimal'))\\\n .withColumn(\"fare_amount\",df[\"fare_amount\"].cast('Decimal'))\\\n .withColumn(\"tip_amount\",df[\"tip_amount\"].cast('Decimal'))\\\n .withColumn(\"tolls_amount\",df[\"tolls_amount\"].cast('Decimal'))\n \n \n ################################################################\n \n return df", "outputs": [], "metadata": {}}, {"source": "#### Function to exclude trips that don't have a pickup location", "cell_type": "markdown", "metadata": {}}, {"execution_count": 41, "cell_type": "code", "source": "def exclude_no_pickuplocations(df):\n \"\"\"\n :param nyc tax trips dataframe: spark dataframe \n :return: spark dataframe \n \"\"\"\n ################################################################\n # code to exclude trips with no pickup locations #\n # Note: Exclude nulls, blanks and zeros # \n ################################################################\n \n df.createOrReplaceTempView(\"data\")\n df2 = spark.sql(\"\"\"\n select *\n from data \n where ((PULocationID is not null) or (PULocationID != \"\"))\n \"\"\")\n return df2\n", "outputs": [], "metadata": {}}, {"source": "#### Function to exclude trips with no distance", "cell_type": "markdown", "metadata": {}}, {"execution_count": 73, "cell_type": "code", "source": "def exclude_no_tripdistance(df):\n \"\"\"\n :param nyc tax trips dataframe: spark dataframe \n :return: spark dataframe \n \"\"\"\n ################################################################\n # code to exclude trips with no trip distances #\n # Note: Exclude nulls, blanks and zero # \n ################################################################\n \n df.createOrReplaceTempView(\"data\")\n df2 = spark.sql(\"\"\"\n select *\n from data \n where ((trip_distance is not null) or (trip_distance != \"\") or (trip_distance > 0))\n \"\"\")\n \n return df2", "outputs": [], "metadata": {}}, {"source": "#### Function to include fare amount between the range of 20 to 60 Dollars", "cell_type": "markdown", "metadata": {}}, {"execution_count": 44, "cell_type": "code", "source": "def include_fare_range(df):\n \"\"\"\n :param nyc tax trips dataframe: spark dataframe \n :return: spark dataframe \n \"\"\"\n \n ################################################################\n # code to include trips with only within the fare range of #\n # 20 to 60 dollars (including 20 and 60 dollars) # \n ################################################################\n \n df.createOrReplaceTempView(\"data\")\n df2 = spark.sql(\"\"\"\n select *\n from data \n where fare_amount >= 20 \n and fare_amount <= 60\n \"\"\")\n return df2", "outputs": [], "metadata": {}}, {"source": "#### Function to get the highest tip amount", "cell_type": "markdown", "metadata": {}}, {"execution_count": 65, "cell_type": "code", "source": "def get_highest_tip(df):\n \"\"\"\n :param nyc tax trips dataframe: spark dataframe \n :return: decimal (rounded to 2 digits) (NOTE: DON'T USE FLOAT)\n \"\"\"\n \n ################################################################\n # code to get the highest tip #\n # # \n ################################################################\n \n df.createOrReplaceTempView(\"data\")\n df2 = spark.sql(\"\"\"\n select round(max(tip_amount),2) as max_tip\n from data\n \"\"\")\n\n return df2.collect()[0]['max_tip']", "outputs": [], "metadata": {}}, {"source": "#### Function to get total toll amount", "cell_type": "markdown", "metadata": {}}, {"execution_count": 71, "cell_type": "code", "source": "def get_total_toll(df):\n \"\"\"\n :param nyc tax trips dataframe: spark dataframe \n :return: decimal (rounded to 2 digits) (NOTE: DON'T USE FLOAT)\n \"\"\"\n \n ################################################################\n # code to get total toll amount #\n ################################################################\n \n df.createOrReplaceTempView(\"data\")\n df2 = spark.sql(\"\"\"\n select round(sum(tolls_amount),2) as total_toll\n from data\n \"\"\")\n\n return df2.collect()[0]['total_toll']", "outputs": [], "metadata": {}}, {"source": "### Run above functions and print\n\n#### Uncomment the cells below and test your implemented functions", "cell_type": "markdown", "metadata": {}}, {"source": "#### Load data from yellow_tripdata_2019-01.csv ", "cell_type": "markdown", "metadata": {}}, {"execution_count": 40, "cell_type": "code", "source": "gcp_storage_path = \"gs://icanlapan3/yellow_tripdata_2019-01.csv\"\ndf = load_data(gcp_storage_path)\ndf.printSchema()", "outputs": [{"output_type": "stream", "name": "stdout", "text": "root\n |-- VendorID: integer (nullable = true)\n |-- tpep_pickup_datetime: timestamp (nullable = true)\n |-- tpep_dropoff_datetime: timestamp (nullable = true)\n |-- passenger_count: integer (nullable = true)\n |-- trip_distance: decimal(10,0) (nullable = true)\n |-- RatecodeID: integer (nullable = true)\n |-- store_and_fwd_flag: string (nullable = true)\n |-- PULocationID: integer (nullable = true)\n |-- DOLocationID: integer (nullable = true)\n |-- payment_type: integer (nullable = true)\n |-- fare_amount: decimal(10,0) (nullable = true)\n |-- extra: double (nullable = true)\n |-- mta_tax: double (nullable = true)\n |-- tip_amount: decimal(10,0) (nullable = true)\n |-- tolls_amount: decimal(10,0) (nullable = true)\n |-- improvement_surcharge: double (nullable = true)\n |-- total_amount: double (nullable = true)\n |-- congestion_surcharge: double (nullable = true)\n\n"}], "metadata": {}}, {"source": "#### Print total numbers of rows in the dataframe", "cell_type": "markdown", "metadata": {}}, {"execution_count": 6, "cell_type": "code", "source": "df.count()", "outputs": [{"execution_count": 6, "output_type": "execute_result", "data": {"text/plain": "7667792"}, "metadata": {}}], "metadata": {}}, {"source": "#### Print total number of rows in the dataframe after excluding trips with no pickup location", "cell_type": "markdown", "metadata": {}}, {"execution_count": 31, "cell_type": "code", "source": "df_no_pickup_locations = exclude_no_pickuplocations(df)\ndf_no_pickup_locations.count()", "outputs": [{"execution_count": 31, "output_type": "execute_result", "data": {"text/plain": "7667792"}, "metadata": {}}], "metadata": {}}, {"source": "#### Print total number of rows in the dataframe after exclude trips with no distance", "cell_type": "markdown", "metadata": {}}, {"execution_count": 74, "cell_type": "code", "source": "df_no_trip_distance = exclude_no_tripdistance(df)\ndf_no_trip_distance.count()", "outputs": [{"execution_count": 74, "output_type": "execute_result", "data": {"text/plain": "7667792"}, "metadata": {}}], "metadata": {}}, {"source": "#### Print total number of rows in the dataframe after including trips with fair amount between the range of 20 to 60 Dollars", "cell_type": "markdown", "metadata": {}}, {"execution_count": 45, "cell_type": "code", "source": "df_include_fare_range = include_fare_range(df)\ndf_include_fare_range.count()", "outputs": [{"execution_count": 45, "output_type": "execute_result", "data": {"text/plain": "1011593"}, "metadata": {}}], "metadata": {}}, {"source": "#### Print the highest tip amount", "cell_type": "markdown", "metadata": {}}, {"execution_count": 70, "cell_type": "code", "source": "max_tip = get_highest_tip(df)\nprint(max_tip)", "outputs": [{"output_type": "stream", "name": "stdout", "text": "787\n"}], "metadata": {}}, {"source": "#### Print the total toll amount", "cell_type": "markdown", "metadata": {}}, {"execution_count": 72, "cell_type": "code", "source": "total_toll = get_total_toll(df)\nprint(total_toll)", "outputs": [{"output_type": "stream", "name": "stdout", "text": "2525415\n"}], "metadata": {}}], "nbformat": 4, "metadata": {"kernelspec": {"display_name": "PySpark", "name": "pyspark", "language": "python"}, "language_info": {"mimetype": "text/x-python", "nbconvert_exporter": "python", "version": "2.7.14", "name": "python", "file_extension": ".py", "pygments_lexer": "ipython2", "codemirror_mode": {"version": 2, "name": "ipython"}}}} |