Tricks with Boolean Series

This notebook discusses Boolean series, and various useful things you can do with them.

First let’s import our libraries:

import pandas as pd
import numpy as np

rng = np.random.default_rng(20201103)

Computing Probability

If we have a logical series:

xb = pd.Series([True, False, True, True, True, False, False, True, False, True])
xb

0     True
1    False
2     True
3     True
4     True
5    False
6    False
7     True
8    False
9     True
dtype: bool

We can count the number of True values with sum:

xb.sum()

6

We can count the fraction of True values, or the probability of True, with mean():

xb.mean()

0.6

The NumPy equivalents also work:

np.mean(xb)

0.6

Creating with Logical Operations

If you do a logical operation (==, <, >, <=, or >=) on a series, comparing it to either a fixed value or another series, you will get a Boolean series.

This is very useful for creating the outcome vector for a logistic regression, as the boolean series will be treated as 1 (True) and 0 (False), and can be predicted with either StatsModels or SciKit-Learn’s logistic regression or other classifier.

For example, if we draw some random numbers:

xs = pd.Series(rng.random(size=1000))
xs

0      0.163352
1      0.333021
2      0.054169
3      0.662442
4      0.517284
         ...   
995    0.882045
996    0.897472
997    0.161948
998    0.850922
999    0.222618
Length: 1000, dtype: float64

xs.hist()

<matplotlib.axes._subplots.AxesSubplot at 0x22e1f5277f0>

xs.describe()

count    1000.000000
mean        0.491382
std         0.285871
min         0.000805
25%         0.235146
50%         0.489305
75%         0.737922
max         0.997661
dtype: float64

We can get a series that is True when the number is at least 0.8:

xs_ge = xs >= 0.8
xs_ge

0      False
1      False
2      False
3      False
4      False
       ...  
995     True
996     True
997    False
998     True
999    False
Length: 1000, dtype: bool

We can verify that values start at 0.8:

xs[xs_ge].min()

0.8050954039782324

Since the values are drawn uniformly in the range \([0,1)\), then approximately 20% of the values should be at least 0.8 (\(P(X \ge 0.8) = 0.2\)). Let’s check:

xs_ge.mean()

0.183

Logical Operations

The bitwise negation operator, ~, negates a boolean series:

~xb

0    False
1     True
2    False
3    False
4    False
5     True
6     True
7    False
8     True
9    False
dtype: bool

The np.logical_not function also does:

np.logical_not(xb)

0    False
1     True
2    False
3    False
4    False
5     True
6     True
7    False
8     True
9    False
dtype: bool

We can combine them with bitwise and (&) or or (|).

Let’s find all the values between 0.8 and 0.9:

xs_in_range = xs_ge & (xs < 0.9)
xs[xs_in_range].describe()

count    87.000000
mean      0.849424
std       0.028517
min       0.805095
25%       0.826880
50%       0.845260
75%       0.878704
max       0.897472
dtype: float64

We can find everything except \([0.2,0.8)\):

xs_lohi = xs_ge | (xs < 0.2)
xs[xs_lohi].hist()

<matplotlib.axes._subplots.AxesSubplot at 0x22e1edadeb0>

Look, we cut out the middle!