Tuning Example
Contents
Tuning Example#
This notebook demonstrates hyperparameter tuning
It uses the CHI Papers data.
All of these are going to optimize for accuracy, the metric returned by the score function on a classifier.
Setup#
scikit-optimize isn’t in Anaconda, so we’ll install it from Pip:
%pip install scikit-optimize
Requirement already satisfied: scikit-optimize in c:\users\michaelekstrand\anaconda3\lib\site-packages (0.8.1)
Requirement already satisfied: pyaml>=16.9 in c:\users\michaelekstrand\anaconda3\lib\site-packages (from scikit-optimize) (20.4.0)
Requirement already satisfied: scikit-learn>=0.20.0 in c:\users\michaelekstrand\anaconda3\lib\site-packages (from scikit-optimize) (0.23.1)
Requirement already satisfied: numpy>=1.13.3 in c:\users\michaelekstrand\anaconda3\lib\site-packages (from scikit-optimize) (1.18.5)
Requirement already satisfied: scipy>=0.19.1 in c:\users\michaelekstrand\anaconda3\lib\site-packages (from scikit-optimize) (1.5.0)
Requirement already satisfied: joblib>=0.11 in c:\users\michaelekstrand\anaconda3\lib\site-packages (from scikit-optimize) (0.16.0)
Requirement already satisfied: PyYAML in c:\users\michaelekstrand\anaconda3\lib\site-packages (from pyaml>=16.9->scikit-optimize) (5.3.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in c:\users\michaelekstrand\anaconda3\lib\site-packages (from scikit-learn>=0.20.0->scikit-optimize) (2.1.0)
Note: you may need to restart the kernel to use updated packages.
Import our general PyData packages:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
And some SciKit Learn:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV
from sklearn.pipeline import Pipeline
from sklearn.metrics import classification_report, accuracy_score
from sklearn.naive_bayes import MultinomialNB
And finally the Bayesian optimizer:
from skopt import BayesSearchCV
We want predictable randomness:
rng = np.random.RandomState(20201130)
In this notebook, I have SciKit-Learn run some tasks in parallel. Let’s configure the (max) number of parallel tasks in one place, so you can easily adjust it based on your computer’s capacity:
NJOBS = 8
Load Data#
We’re going to load the CHI Papers data from the CSV file, output by the other notebook:
papers = pd.read_csv('chi-papers.csv', encoding='utf8')
papers.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 13422 entries, 0 to 13421
Data columns (total 6 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 id 13333 non-null object
1 title 13416 non-null object
2 authors 13416 non-null object
3 date 13422 non-null object
4 abstract 12926 non-null object
5 year 13422 non-null int64
dtypes: int64(1), object(5)
memory usage: 629.3+ KB
Let’s treat empty abstracts as empty strings:
papers['abstract'].fillna('', inplace=True)
papers['title'].fillna('', inplace=True)
For our purposes we want all text - the title and the abstract. We will join them with a space, so we don’t fuse the last word of the title to the first word of the abstract:
papers['all_text'] = papers['title'] + ' ' + papers['abstract']
We’re going to classify papers as recent if they’re newer than 2005:
papers['IsRecent'] = papers['year'] > 2005
And make training and test data:
train, test = train_test_split(papers, test_size=0.2, random_state=rng)
Let’s make a function for measuring accuracy:
def measure(model, text='all_text'):
preds = model.predict(test[text])
print(classification_report(test['IsRecent'], preds))
And look at the class distribution:
sns.countplot(train['IsRecent'])
<matplotlib.axes._subplots.AxesSubplot at 0x26bb8bdde80>
Classifying New Papers#
Let’s classify recent papers with k-NN on TF-IDF vectors:
base_knn = Pipeline([
('vectorize', TfidfVectorizer(stop_words='english', lowercase=True, max_features=10000)),
('class', KNeighborsClassifier(5))
])
base_knn.fit(train['all_text'], train['IsRecent'])
Pipeline(steps=[('vectorize',
TfidfVectorizer(max_features=10000, stop_words='english')),
('class', KNeighborsClassifier())])
And measure it:
measure(base_knn)
precision recall f1-score support
False 0.39 1.00 0.56 1034
True 0.91 0.01 0.02 1651
accuracy 0.39 2685
macro avg 0.65 0.51 0.29 2685
weighted avg 0.71 0.39 0.23 2685
Tune the Neighborhood#
Let’s tune the neighborhood with a grid search:
tune_knn = Pipeline([
('vectorize', TfidfVectorizer(stop_words='english', lowercase=True, max_features=10000)),
('class', GridSearchCV(KNeighborsClassifier(), param_grid={
'n_neighbors': [1, 2, 3, 5, 7, 10]
}, n_jobs=NJOBS))
])
tune_knn.fit(train['all_text'], train['IsRecent'])
Pipeline(steps=[('vectorize',
TfidfVectorizer(max_features=10000, stop_words='english')),
('class',
GridSearchCV(estimator=KNeighborsClassifier(), n_jobs=8,
param_grid={'n_neighbors': [1, 2, 3, 5, 7,
10]}))])
What did it pick?
tune_knn.named_steps['class'].best_params_
{'n_neighbors': 10}
And measure it:
measure(tune_knn)
precision recall f1-score support
False 0.51 0.90 0.65 1034
True 0.87 0.45 0.60 1651
accuracy 0.62 2685
macro avg 0.69 0.67 0.62 2685
weighted avg 0.73 0.62 0.62 2685
SVD Neighborhood#
Let’s set up SVD-based neighborhood, and use random search to search both the latent feature count and the neighborhood size:
svd_knn_inner = Pipeline([
('latent', TruncatedSVD(random_state=rng)),
('class', KNeighborsClassifier())
])
svd_knn = Pipeline([
('vectorize', TfidfVectorizer(stop_words='english', lowercase=True)),
('class', RandomizedSearchCV(svd_knn_inner, param_distributions={
'latent__n_components': stats.randint(1, 50),
'class__n_neighbors': stats.randint(1, 25)
}, n_iter=60, n_jobs=NJOBS, random_state=rng))
])
svd_knn.fit(train['all_text'], train['IsRecent'])
Pipeline(steps=[('vectorize', TfidfVectorizer(stop_words='english')),
('class',
RandomizedSearchCV(estimator=Pipeline(steps=[('latent',
TruncatedSVD(random_state=RandomState(MT19937) at 0x26BB86DF140)),
('class',
KNeighborsClassifier())]),
n_iter=60, n_jobs=8,
param_distributions={'class__n_neighbors': <scipy.stats._distn_infrastructure.rv_frozen object at 0x0000026BB8B0BEB0>,
'latent__n_components': <scipy.stats._distn_infrastructure.rv_frozen object at 0x0000026BB945A070>},
random_state=RandomState(MT19937) at 0x26BB86DF140))])
What parameters did we pick?
svd_knn['class'].best_params_
{'class__n_neighbors': 21, 'latent__n_components': 21}
And measure it on the test data:
measure(svd_knn)
precision recall f1-score support
False 0.75 0.47 0.58 1034
True 0.73 0.90 0.81 1651
accuracy 0.74 2685
macro avg 0.74 0.69 0.69 2685
weighted avg 0.74 0.74 0.72 2685
SVD with scikit-optimize#
Now let’s cross-validate with SciKit-Optimize:
svd_knn_inner = Pipeline([
('latent', TruncatedSVD()),
('class', KNeighborsClassifier())
])
svd_bayes_knn = Pipeline([
('vectorize', TfidfVectorizer(stop_words='english', lowercase=True)),
('class', BayesSearchCV(svd_knn_inner, {
'latent__n_components': (1, 50),
'class__n_neighbors': (1, 25)
}, n_jobs=NJOBS, random_state=rng))
])
svd_bayes_knn.fit(train['all_text'], train['IsRecent'])
C:\Users\michaelekstrand\Anaconda3\lib\site-packages\skopt\optimizer\optimizer.py:449: UserWarning: The objective has been evaluated at this point before.
warnings.warn("The objective has been evaluated "
C:\Users\michaelekstrand\Anaconda3\lib\site-packages\skopt\optimizer\optimizer.py:449: UserWarning: The objective has been evaluated at this point before.
warnings.warn("The objective has been evaluated "
C:\Users\michaelekstrand\Anaconda3\lib\site-packages\skopt\optimizer\optimizer.py:449: UserWarning: The objective has been evaluated at this point before.
warnings.warn("The objective has been evaluated "
C:\Users\michaelekstrand\Anaconda3\lib\site-packages\skopt\optimizer\optimizer.py:449: UserWarning: The objective has been evaluated at this point before.
warnings.warn("The objective has been evaluated "
C:\Users\michaelekstrand\Anaconda3\lib\site-packages\skopt\optimizer\optimizer.py:449: UserWarning: The objective has been evaluated at this point before.
warnings.warn("The objective has been evaluated "
C:\Users\michaelekstrand\Anaconda3\lib\site-packages\skopt\optimizer\optimizer.py:449: UserWarning: The objective has been evaluated at this point before.
warnings.warn("The objective has been evaluated "
Pipeline(steps=[('vectorize', TfidfVectorizer(stop_words='english')),
('class',
BayesSearchCV(estimator=Pipeline(steps=[('latent',
TruncatedSVD()),
('class',
KNeighborsClassifier())]),
n_jobs=8,
random_state=RandomState(MT19937) at 0x26BB86DF140,
search_spaces={'class__n_neighbors': (1, 25),
'latent__n_components': (1,
50)}))])
What parameters did we pick?
svd_bayes_knn['class'].best_params_
OrderedDict([('class__n_neighbors', 25), ('latent__n_components', 20)])
And measure it:
measure(svd_bayes_knn)
precision recall f1-score support
False 0.74 0.43 0.55 1034
True 0.72 0.91 0.80 1651
accuracy 0.72 2685
macro avg 0.73 0.67 0.67 2685
weighted avg 0.73 0.72 0.70 2685
Naive Bayes#
Let’s give the Naive Bayes classifier a whirl:
nb = Pipeline([
('vectorize', TfidfVectorizer(stop_words='english', lowercase=True, max_features=10000)),
('class', MultinomialNB())
])
nb.fit(train['all_text'], train['IsRecent'])
Pipeline(steps=[('vectorize',
TfidfVectorizer(max_features=10000, stop_words='english')),
('class', MultinomialNB())])
measure(nb)
precision recall f1-score support
False 0.89 0.50 0.64 1034
True 0.75 0.96 0.85 1651
accuracy 0.78 2685
macro avg 0.82 0.73 0.74 2685
weighted avg 0.81 0.78 0.77 2685
Summary Accuracy#
What does our test accuracy look like for our various classifiers?
models = {
'kNN': base_knn,
'kNN-CV': tune_knn,
'kNN-SVD-Rand': svd_knn,
'kNN-SVD-Bayes': svd_bayes_knn,
'NB': nb
}
all_preds = pd.DataFrame()
for name, model in models.items():
all_preds[name] = model.predict(test['all_text'])
acc = all_preds.apply(lambda ds: accuracy_score(test['IsRecent'], ds))
acc
kNN 0.391806
kNN-CV 0.623836
kNN-SVD-Rand 0.735940
kNN-SVD-Bayes 0.724022
NB 0.783240
dtype: float64
acc.plot.bar()
plt.ylabel('Accuracy')
plt.show()
