Describing Distributions¶

Setup¶

Import our modules again:

In [1]:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

And load the MovieLens data. We're going to pass the memory_use='deep' to info, so we can see the total memory use including the strings.

In [2]:

movies = pd.read_csv('ml-25m/movies.csv')
movies.info(memory_usage='deep')

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 62423 entries, 0 to 62422
Data columns (total 3 columns):
 #   Column   Non-Null Count  Dtype 
---  ------   --------------  ----- 
 0   movieId  62423 non-null  int64 
 1   title    62423 non-null  object
 2   genres   62423 non-null  object
dtypes: int64(1), object(2)
memory usage: 9.6 MB

In [3]:

ratings = pd.read_csv('ml-25m/ratings.csv')
ratings.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 25000095 entries, 0 to 25000094
Data columns (total 4 columns):
 #   Column     Dtype  
---  ------     -----  
 0   userId     int64  
 1   movieId    int64  
 2   rating     float64
 3   timestamp  int64  
dtypes: float64(1), int64(3)
memory usage: 762.9 MB

Quickly preview the ratings frame:

In [4]:

ratings

Out[4]:

	userId	movieId	rating	timestamp
0	1	296	5.0	1147880044
1	1	306	3.5	1147868817
2	1	307	5.0	1147868828
3	1	665	5.0	1147878820
4	1	899	3.5	1147868510
...	...	...	...	...
25000090	162541	50872	4.5	1240953372
25000091	162541	55768	2.5	1240951998
25000092	162541	56176	2.0	1240950697
25000093	162541	58559	4.0	1240953434
25000094	162541	63876	5.0	1240952515

25000095 rows × 4 columns

Movie stats:

In [5]:

movie_stats = ratings.groupby('movieId')['rating'].agg(['mean', 'count'])
movie_stats

Out[5]:

	mean	count
movieId
1	3.893708	57309
2	3.251527	24228
3	3.142028	11804
4	2.853547	2523
5	3.058434	11714
...	...	...
209157	1.500000	1
209159	3.000000	1
209163	4.500000	1
209169	3.000000	1
209171	3.000000	1

59047 rows × 2 columns

In [6]:

movie_info = movies.join(movie_stats, on='movieId')
movie_info

Out[6]:

	movieId	title	genres	mean	count
0	1	Toy Story (1995)	Adventure\|Animation\|Children\|Comedy\|Fantasy	3.893708	57309.0
1	2	Jumanji (1995)	Adventure\|Children\|Fantasy	3.251527	24228.0
2	3	Grumpier Old Men (1995)	Comedy\|Romance	3.142028	11804.0
3	4	Waiting to Exhale (1995)	Comedy\|Drama\|Romance	2.853547	2523.0
4	5	Father of the Bride Part II (1995)	Comedy	3.058434	11714.0
...	...	...	...	...	...
62418	209157	We (2018)	Drama	1.500000	1.0
62419	209159	Window of the Soul (2001)	Documentary	3.000000	1.0
62420	209163	Bad Poems (2018)	Comedy\|Drama	4.500000	1.0
62421	209169	A Girl Thing (2001)	(no genres listed)	3.000000	1.0
62422	209171	Women of Devil's Island (1962)	Action\|Adventure\|Drama	3.000000	1.0

62423 rows × 5 columns

Normal Distribution¶

I want to visualize an array of random, normally-distributed numbers.

We'll first generate them:

In [7]:

numbers = pd.Series(np.random.randn(10000) + 5)
numbers

Out[7]:

0       3.996778
1       4.119017
2       4.360605
3       4.850503
4       6.244754
          ...   
9995    4.614897
9996    5.742292
9997    4.271137
9998    4.117684
9999    4.199830
Length: 10000, dtype: float64

And then describe them:

In [8]:

numbers.describe()

Out[8]:

count    10000.000000
mean         4.998850
std          1.006624
min          1.473666
25%          4.305575
50%          4.982345
75%          5.690335
max          9.202777
dtype: float64

And finally visualize them:

In [9]:

plt.hist(numbers, bins=25)

Out[9]:

(array([6.000e+00, 8.000e+00, 3.500e+01, 5.500e+01, 1.220e+02, 2.280e+02,
        4.070e+02, 6.110e+02, 8.810e+02, 1.033e+03, 1.175e+03, 1.184e+03,
        1.125e+03, 9.890e+02, 8.230e+02, 5.580e+02, 3.160e+02, 1.990e+02,
        1.360e+02, 6.000e+01, 2.800e+01, 1.400e+01, 6.000e+00, 0.000e+00,
        1.000e+00]),
 array([1.47366561, 1.78283007, 2.09199454, 2.401159  , 2.71032347,
        3.01948793, 3.3286524 , 3.63781687, 3.94698133, 4.2561458 ,
        4.56531026, 4.87447473, 5.18363919, 5.49280366, 5.80196812,
        6.11113259, 6.42029706, 6.72946152, 7.03862599, 7.34779045,
        7.65695492, 7.96611938, 8.27528385, 8.58444831, 8.89361278,
        9.20277725]),
 <a list of 25 Patch objects>)

Average Movie Rating¶

To start looking at some real data, let's look at the distribution of average movie rating:

In [10]:

movie_info['mean'].describe()

Out[10]:

count    59047.000000
mean         3.071374
std          0.739840
min          0.500000
25%          2.687500
50%          3.150000
75%          3.500000
max          5.000000
Name: mean, dtype: float64

Let's make a histogram:

In [11]:

plt.hist(movie_info['mean'])
plt.show()

C:\Users\michaelekstrand\Anaconda3\lib\site-packages\numpy\lib\histograms.py:839: RuntimeWarning: invalid value encountered in greater_equal
  keep = (tmp_a >= first_edge)
C:\Users\michaelekstrand\Anaconda3\lib\site-packages\numpy\lib\histograms.py:840: RuntimeWarning: invalid value encountered in less_equal
  keep &= (tmp_a <= last_edge)

And with more bins:

In [12]:

plt.hist(movie_info['mean'], bins=50)
plt.show()

Movie Count¶

Now we want to describe the distribution of the ratings-per-movie (movie popularity).

In [13]:

movie_info['count'].describe()

Out[13]:

count    59047.000000
mean       423.393144
std       2477.885821
min          1.000000
25%          2.000000
50%          6.000000
75%         36.000000
max      81491.000000
Name: count, dtype: float64

In [14]:

plt.hist(movie_info['count'])
plt.show()

In [15]:

plt.hist(movie_info['count'], bins=100)
plt.show()

That is a very skewed distribution. Will it make more sense on a logarithmic scale?

We don't want to just log-scale a histogram - it will be very difficult to interpret. We will use a point plot.

The value_counts() method counts the number of times each value appers. The resulting series is indexed by value, so we will use its index as the x-axis of the plot. Indexes are arrays too!

In [16]:

hist = movie_info['count'].value_counts()
plt.scatter(hist.index, hist)
plt.yscale('log')
plt.ylabel('Number of Movies')
plt.xscale('log')
plt.xlabel('Number of Ratings')

Out[16]:

Text(0.5, 0, 'Number of Ratings')

Penguins¶

Let's load the Penguin data (converted from R):

In [17]:

penguins = pd.read_csv('penguins.csv')
penguins

Out[17]:

	species	island	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g	sex	year
0	Adelie	Torgersen	39.1	18.7	181.0	3750.0	male	2007
1	Adelie	Torgersen	39.5	17.4	186.0	3800.0	female	2007
2	Adelie	Torgersen	40.3	18.0	195.0	3250.0	female	2007
3	Adelie	Torgersen	NaN	NaN	NaN	NaN	NaN	2007
4	Adelie	Torgersen	36.7	19.3	193.0	3450.0	female	2007
...	...	...	...	...	...	...	...	...
339	Chinstrap	Dream	55.8	19.8	207.0	4000.0	male	2009
340	Chinstrap	Dream	43.5	18.1	202.0	3400.0	female	2009
341	Chinstrap	Dream	49.6	18.2	193.0	3775.0	male	2009
342	Chinstrap	Dream	50.8	19.0	210.0	4100.0	male	2009
343	Chinstrap	Dream	50.2	18.7	198.0	3775.0	female	2009

344 rows × 8 columns

Now we'll compute a histogram. There are ways to do this automatically, but for demonstration purposes I want to do the computations ourselves:

In [18]:

spec_counts = penguins['species'].value_counts()
plt.bar(spec_counts.index, spec_counts)
plt.xlabel('Species')
plt.ylabel('# of Penguins')

Out[18]:

Text(0, 0.5, '# of Penguins')

What if we want to show the fraction of each species? We can divide by the sum:

In [19]:

spec_fracs = spec_counts / spec_counts.sum()
plt.bar(spec_counts.index, spec_fracs)
plt.xlabel('Species')
plt.ylabel('Fraction of Penguins')

Out[19]:

Text(0, 0.5, 'Fraction of Penguins')

In [ ]: