MovieLens Time Series

This notebook demonstrates basic time series analysis with the MovieLens data.

Setup

Import libraries:

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import statsmodels.api as sm
import statsmodels.formula.api as smf

Load the MovieLens 25M data:

movies = pd.read_csv('ml-25m/movies.csv').set_index('movieId')
movies.info(memory_usage='deep')

<class 'pandas.core.frame.DataFrame'>
Int64Index: 62423 entries, 1 to 209171
Data columns (total 2 columns):
 #   Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
 0   title   62423 non-null  object
 1   genres  62423 non-null  object
dtypes: object(2)
memory usage: 9.6 MB

ratings = pd.read_csv('ml-25m/ratings.csv', dtype={
    'movieId': 'i4',
    'userId': 'i4',
    'rating': 'f4',
    'timestamp': 'i4'
})
ratings.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 25000095 entries, 0 to 25000094
Data columns (total 4 columns):
 #   Column     Dtype  
---  ------     -----  
 0   userId     int32  
 1   movieId    int32  
 2   rating     float32
 3   timestamp  int32  
dtypes: float32(1), int32(3)
memory usage: 381.5 MB

MovieLens represents time with UNIX timestamps: seconds since the UNIX epoch. Convert those to a Pandas DateTime:

ratings['timestamp'] = pd.to_datetime(ratings['timestamp'], unit='s')

Basic Aggregations

In order to do time-series operations, we need to index by time series, and sort the index:

rts = ratings.set_index('timestamp').sort_index()
rts.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 25000095 entries, 1995-01-09 11:46:49 to 2019-11-21 09:15:03
Data columns (total 3 columns):
 #   Column   Dtype  
---  ------   -----  
 0   userId   int32  
 1   movieId  int32  
 2   rating   float32
dtypes: float32(1), int32(2)
memory usage: 476.8 MB

monthly_ratings = rts.resample('1M')['rating'].count()
monthly_ratings

timestamp
1995-01-31         3
1995-02-28         0
1995-03-31         0
1995-04-30         0
1995-05-31         0
               ...  
2019-07-31     99159
2019-08-31    107210
2019-09-30    125523
2019-10-31     96364
2019-11-30     66464
Freq: M, Name: rating, Length: 299, dtype: int64

monthly_ratings.plot()
plt.ylabel('# of Ratings')
plt.xlabel('Month')
plt.show()

sns.lineplot(data=monthly_ratings)
plt.ylabel('# of Ratings')
plt.xlabel('Month')
plt.show()

Selecting Subsets

Let’s do a range select for 2010:

rs_2010 = rts.loc['2010-01-01':'2010-12-31']
rs_2010.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 792436 entries, 2010-01-01 00:01:13 to 2010-12-31 23:58:34
Data columns (total 3 columns):
 #   Column   Non-Null Count   Dtype  
---  ------   --------------   -----  
 0   userId   792436 non-null  int32  
 1   movieId  792436 non-null  int32  
 2   rating   792436 non-null  float32
dtypes: float32(1), int32(2)
memory usage: 15.1 MB

We can also index by time interval:

rs_jul2010 = rts.loc['2010-07']
rs_jul2010.info()

<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 66320 entries, 2010-07-01 00:23:42 to 2010-07-31 23:57:34
Data columns (total 3 columns):
 #   Column   Non-Null Count  Dtype  
---  ------   --------------  -----  
 0   userId   66320 non-null  int32  
 1   movieId  66320 non-null  int32  
 2   rating   66320 non-null  float32
dtypes: float32(1), int32(2)
memory usage: 1.3 MB

Diff and Lag

Let’s compute the month-over-month growth:

month_growth = monthly_ratings.diff()
month_growth

timestamp
1995-01-31        NaN
1995-02-28       -3.0
1995-03-31        0.0
1995-04-30        0.0
1995-05-31        0.0
               ...   
2019-07-31    13083.0
2019-08-31     8051.0
2019-09-30    18313.0
2019-10-31   -29159.0
2019-11-30   -29900.0
Freq: M, Name: rating, Length: 299, dtype: float64

The first value is NaN, because there is no “previous” value.

Let’s plot this:

month_growth.plot()

<matplotlib.axes._subplots.AxesSubplot at 0x1b90de21eb0>

Monthly Unique Visitors

A common metric for web sites is monthly unique visitors. We can’t quite compute that, because we have no records of people who just came to the web site but did not rate anything. But we can compute monthly unique active users, where a user is active if they rated a movie:

uniques = rts.resample('1M')['userId'].nunique()
uniques.plot()
plt.ylabel('Unique Active Users')
plt.xlabel('Month')
plt.show()

We can show the monthly growth (increase/decrease in # of unique users):

plt.axhline(0, color='grey')
uniques.diff().plot()
plt.ylabel('Growth (Increase in Unique Users)')
plt.xlabel('Month')
plt.show()