python_packages_data_analysis_kf.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "python_packages_data_analysis_kf.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMHde1Z4EIifyRruF6SeHWU",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/stephenfrein/py_packages_data_analysis/blob/master/python_packages_data_analysis_kf.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Qr8WXOAZ6Mdu",
        "colab_type": "text"
      },
      "source": [
        "# Useful Python Packages for Data Analysis\n",
        "\n",
        "***\n",
        "\n",
        "# General Notes on This Session\n",
        "\n",
        "This is a Jupyter notebook running in Google's Colab environment that we will use to practice with some Python packages that are useful for data analysis.\n",
        "\n",
        "You can write and execute your Python code right in the browser here. No additional setup is required.\n",
        "\n",
        "Because of the large number of people here, our interaction during the session will be limited. If you get stuck on something, please do your best for now and I promise to help you out later.\n",
        "\n",
        "If you get an error with the code I supplied, make sure you have *run all prior code.*\n",
        "\n",
        "The main packages we will cover today are *pandas* (used for manipulating tabular data) and *matplotlib* / *seaborn* (both used to create graphs).\n",
        "\n",
        "We could easily spend hours on each of these packages and so can only do a quick tour during our time today."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l3TeC_HQ8sJ5",
        "colab_type": "text"
      },
      "source": [
        "# Pandas\n",
        "\n",
        "The pandas library is essential for data analysis in Python. It allows you to maniulate tabular data structures, such as you would find in a relational database or spreadsheet.\n",
        "\n",
        "The name comes from \"panel data\" - a term for used for data sets that track multiple variables over time.\n",
        "\n",
        "Some things we'll do with pandas:\n",
        "*   Load data\n",
        "*   Explore/summarize the data\n",
        "*   Subset the data\n",
        "*   Group the data\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_Z5yTUpaGjSm",
        "colab_type": "text"
      },
      "source": [
        "# Load Data\n",
        "\n",
        "Let's get some data first. We can load data from (and write back to) a variety of formats, including:\n",
        "*   text files (CSV, fixed-width)\n",
        "*   JSON\n",
        "*   HTML\n",
        "*   MS Excel\n",
        "*   SQL\n",
        "\n",
        "...and lots of others as well.\n",
        "\n",
        "We can also pull data from a filesystem (just give the path) or a URL.\n",
        "\n",
        "To get things started, we'll load some CSV data about the Titanic from a URL.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X09stpEIFWnw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# it's conventional to alias pandas as pd once imported\n",
        "import pandas as pd\n",
        "url=\"https://drive.google.com/uc?export=download&id=1uBVSWxr_20BbtBW0ls5dPS5fU_QeHIAt\"\n",
        "# pandas will read this data into a DataFrame, the typical pandas data structure\n",
        "# df is a common abbreviation used in DataFrame variables \n",
        "titanic_df=pd.read_csv(url)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nc0K5QbyKH35",
        "colab_type": "text"
      },
      "source": [
        "# Explore Our Data\n",
        "\n",
        "Now that we've loaded our data, let's take a look at it. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8WXI35XZLNrZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# let's see the first rows - could also do tail()\n",
        "titanic_df.head(n=10)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "F4KhaHCh_NTy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# what's the type of each column?\n",
        "titanic_df.dtypes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Tso18vC8SBsx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# let's see some stats about the values in each column\n",
        "titanic_df.describe(include=\"all\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YROjhZGsAdkg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# let's see the passengers oldest to youngest\n",
        "titanic_df.sort_values(by='Age', ascending=False)\n",
        "# try adding .head(n=5)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jrcIghsRORSc",
        "colab_type": "text"
      },
      "source": [
        "#  Subset the Data\n",
        "\n",
        "Sometimes we only need parts of the data set. We can grab just the rows and columns that we need.\n",
        "\n",
        "There are many, many ways to do this sort of thing in Pandas. If you need to search or subset a different way, you can surely find a way to do it.\n",
        "\n",
        "We'll focus on loc[], which uses a loc[ (rows) , (columns) ] format to specify the data to be returned.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ttjQ_75SB2iv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# let's find the first-class passengers who were over 60\n",
        "# this addresses rows only - getting all columns by default\n",
        "titanic_df.loc[(titanic_df['Pclass']==1) & (titanic_df['Age'] > 60)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3J_GXzPJT1ED",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# see just the names and ages for first 3 rows (note zero-based indexing for start and end)\n",
        "titanic_df.loc[0:2,['Name','Age']]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gd1QxS4gVC8s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# now name and age for all rows - notice the colon is basically a wildcard here\n",
        "titanic_df.loc[:,['Name','Age']]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fobYhUFIFEGA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# let's combine the two approaches - show the passenger class, name, and age\n",
        "# for surviving passengers over 65\n",
        "titanic_df.loc[(titanic_df['Survived']==1) & (titanic_df['Age'] > 65), ['Pclass','Name','Age']]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GBagj6h5aKQE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# you can save any of these subsets as a new data frame\n",
        "older_survivors = titanic_df.loc[(titanic_df['Survived']==1) & (titanic_df['Age'] > 65), ['Pclass','Name','Age']]\n",
        "older_survivors"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jTvFELzSMHkd",
        "colab_type": "text"
      },
      "source": [
        "# Exercise #1\n",
        "\n",
        "Now, you try. You can find a modified set of COVID-19 data from Johns Hopkins at https://drive.google.com/uc?export=download&id=1v2reITVBrhkoSDleHBOStvsjwDh_zAzh. Pull it into a pandas data frame called covid_df and look at the first rows to get a feel for the data. Then, create a new data frame called 'monday_us_confirmed_cases' that is just confirmed cases (case type is 'Confirmed') from the United States (iso2 value of 'US') from Monday (4/27/2020). What is the average population count across all the counties included in the data set? What is the greatest number of cases in any single county? What county is that?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2MDG0jQhMbPz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# set up a url and pull the data into a data frame\n",
        "\n",
        "\n",
        "# show the first rows \n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wAZW2O9EeWDG",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# create a data frame for just confirmed cases in the US on 4/26/2020\n",
        "\n",
        "# get summary stats on that data frame - numeric columns only\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CzVhYA9TsYsA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# which county has that maximum number of cases?\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CTCRJpgrohaX",
        "colab_type": "text"
      },
      "source": [
        "# Grouping\n",
        "\n",
        "Grouping allows us to put our data into categorical buckets and then operate on those buckets. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vaydhbVCoxmA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# was the tragedy of Titanic shared equally by passengers of all classes?\n",
        "titanic_df.groupby('Pclass').mean()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rr-rLJiPJlU5",
        "colab_type": "text"
      },
      "source": [
        "# Exercise #2\n",
        "\n",
        "By Monday the 27th, how many COVID cases had been recorded by each of the states with the top 5 highest number of infections?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "96l-1b2RB51m",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# # in case you never completed exericse 1 above\n",
        "#url=\"https://drive.google.com/uc?export=download&id=1v2reITVBrhkoSDleHBOStvsjwDh_zAzh\"\n",
        "#covid_df=pd.read_csv(url)\n",
        "#monday_us_confirmed_cases = covid_df.loc[(covid_df['Case_Type']=='Confirmed') & (covid_df['iso2'] == 'US') & (covid_df['Date'] == '4/27/2020')]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JrpFLGyerAac",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# use a group by to get state totals, summing up the records for each state, and sort by decreasing number of cases, limiting to top 5\n",
        "# don't forget that you can chain calls together\n",
        "\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MZDTBcLVKcZm",
        "colab_type": "text"
      },
      "source": [
        "Remember the number of cases for the state that has the most - we'll need that to double-check ourselves a bit later."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_lOsNO7LuZDk",
        "colab_type": "text"
      },
      "source": [
        "# Plotting with Matplotlib\n",
        "\n",
        "The matplotlib package is the most commonly used way to plot data from pandas data frames and probably Python data in general.\n",
        "\n",
        "It was inspired by and based partly upon a mathematical computing and graphics environment called MATLAB."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aSla1o0bwRKV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# let's create a new data frame from our original COVID data for confirmed cases in just those five states we identified above\n",
        "# we'll use a new \"isin\" method to subset that is less cumbersome for this situation\n",
        "#                              rows-------------------------------------------------------------------------------------------------------------------------------------  columns--------------------------\n",
        "hard_hit_states = covid_df.loc[covid_df['Province_State'].isin(['New York','New Jersey','Massachusetts','Illinois','California']) & (covid_df['Case_Type']=='Confirmed'), ['Cases','Date','Province_State']]\n",
        "hard_hit_states.head()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bl04SHQS6HUl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# we want to plot by date - does pandas know that Date is a date?\n",
        "hard_hit_states.dtypes"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4tYRwO0S6X7r",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# make it a date - then go back and run line above\n",
        "hard_hit_states['Date'] =  pd.to_datetime(hard_hit_states['Date'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "g9lDKX2i4NvL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# numpy is a library used for various numeric operations - pandas is actually built on it\n",
        "import numpy as np\n",
        "# pivot the data frame - each date gets a row, the states become columns, and the sum of the cases become the cell values  \n",
        "hard_hit_pivot = pd.pivot_table(hard_hit_states, values='Cases', index=['Date'], columns=['Province_State'], aggfunc=np.sum)\n",
        "hard_hit_pivot.tail()\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KrABP3071dN1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# need this line to create plot inside a Jupyter notebook like this one\n",
        "%matplotlib inline\n",
        "# conventional to import as plt\n",
        "import matplotlib.pyplot as plt\n",
        "# draw the plot\n",
        "hard_hit_pivot.plot()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E8ab8u2T98uX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# make plot bigger with width, height in inches\n",
        "plt.rcParams['figure.figsize'] = [20, 10]\n",
        "# get a reference to the plot area and add a marker\n",
        "hard_hit_plot = hard_hit_pivot.plot(marker=\"o\")\n",
        "# set the x-axis limits \n",
        "hard_hit_plot.set_xlim(pd.Timestamp('2020-03-15'), pd.Timestamp('2020-04-27'))\n",
        "# add a title\n",
        "hard_hit_plot.set_title(\"COVID-19 Cases in Hardest Hit States\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BXpouN-5LgCC",
        "colab_type": "text"
      },
      "source": [
        "#Exercise #3 \n",
        "\n",
        "Plot the daily growth in cases (\"Difference\") for each of these states. Create a pivoted version of the difference data called \"hard_hit_diffs_pivot\" as a parallel to \"hard_hit_pivot\" from the prior example. This name is important because it will appear in the code later on."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LMa9h2uEMTRw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# you can steal most of the code above for the cases plot - you just need to make a handful of key edits\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TKfg4hFe41-2",
        "colab_type": "text"
      },
      "source": [
        "# Smoothing Things Out with Resampling\n",
        "\n",
        "The above day-by-day graph is pretty jagged, making it difficult to see the overall trend in new cases for these states. It would be desirable to smooth out the jitter in the lines above so that we have a clearer picture of the trends.\n",
        "\n",
        "Luckily, pandas makes it easy to resample time-series data so that we can smooth things out and see a less noisy version."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WUsM67HV9eVy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# # in case you didn't get to complete exercise 3 above, we need to create the hard_hit_diffs_pivot data set\n",
        "# hard_hit_diffs = covid_df.loc[covid_df['Province_State'].isin(['New York','New Jersey','Massachusetts','Illinois','California']) & (covid_df['Case_Type']=='Confirmed'), ['Difference','Date','Province_State']]\n",
        "# hard_hit_diffs['Date'] =  pd.to_datetime(hard_hit_diffs['Date'])\n",
        "# hard_hit_diffs_pivot = pd.pivot_table(hard_hit_diffs, values='Difference', index=['Date'], columns=['Province_State'], aggfunc=np.sum)\n",
        "# hard_hit_diffs_pivot"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TWbA-Qrj5LSf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# new data frame\n",
        "weekly_changes = pd.DataFrame()\n",
        "# resample each of the state columns to be a weekly ('W') average\n",
        "weekly_changes['New York'] = hard_hit_diffs_pivot['New York'].resample('W').mean()\n",
        "weekly_changes['New Jersey'] = hard_hit_diffs_pivot['New Jersey'].resample('W').mean()\n",
        "weekly_changes['Massachusetts'] = hard_hit_diffs_pivot['Massachusetts'].resample('W').mean()\n",
        "weekly_changes['Illinois'] = hard_hit_diffs_pivot['Illinois'].resample('W').mean()\n",
        "weekly_changes['California'] = hard_hit_diffs_pivot['California'].resample('W').mean()\n",
        "# focus on complete weeks so don't go past 4/26\n",
        "weekly_changes = weekly_changes.truncate(before='2020-01-01', after='2020-04-26')\n",
        "weekly_changes.tail()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XhV0vidx-tOr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# now plot the smoother graph\n",
        "hard_hit_diffs_smooth_plot = weekly_changes.plot(marker=\"o\")\n",
        "hard_hit_diffs_smooth_plot.set_xlim(pd.Timestamp('2020-03-15'), pd.Timestamp('2020-04-26'))\n",
        "hard_hit_diffs_smooth_plot.set_title(\"COVID-19 New Cases Per Week Smoothed\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yaq3XU3ASWZj",
        "colab_type": "text"
      },
      "source": [
        "# Plotting with Seaborn\n",
        "\n",
        "Seaborn is a data visualization library built on top of matplotlib. It focuses on having a simple interface and attractive defaults. Basically, it tries to expose matplotlib capabilities more easily and make things look nicer out-of-the-box.\n",
        "\n",
        "The name comes from a character in the TV series \"The West Wing.\" The author of the package just seems to like the show."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Na7Xe9aOSg1b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# common to import as sns - the initials of the character from that show\n",
        "import seaborn as sns\n",
        "# set default style, color palette, etc.\n",
        "sns.set(style=\"white\")\n",
        "# creation relational plot (basically a scatterplot)\n",
        "# sizes gives a relative scale on which things are drawn\n",
        "splot = sns.relplot(x=\"Long\", y=\"Lat\", hue=\"Province_State\", size=\"Cases\", \n",
        "            sizes=(20,1000), legend=None, data=monday_us_confirmed_cases)\n",
        "splot.fig.set_size_inches(20, 12)\n",
        "# focus axes on contiguous US states\n",
        "plt.ylim(25, 50)\n",
        "plt.xlim(-125,-65)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rb6Gxvlq3Z04",
        "colab_type": "text"
      },
      "source": [
        "# Lastly, Check Out Bokeh\n",
        "\n",
        "Bokeh is another plotting library that emphasizes interactivity. It allows you do pan/zoom, save graphics to disk, and build other kinds of interactions. Check it out at https://docs.bokeh.org/en/latest/."
      ]
    }
  ]
}