Recently while setting up Python on a new device, I instinctively installed the missingno library, along with essentials like pandas and numpy. This made me reflect on why this underrated library has become as indispensable to me as these Python staples. Let's dive into how this little-known library became a favorite of mine, and perhaps might become yours too!

Introduction

Handling missing data is a common yet critical challenge in data analysis. The missingno library in Python offers a visually intuitive way to understand the distribution and patterns of missing data. In this article, we'll explore how to use missingno to analyze missing data effectively.

Installation

First, install missingno using pip:

pip install missingno

Importing Libraries

Let's start by importing the necessary libraries:

import pandas as pd
import numpy as np
import missingno as msno

Loading a Dataset

# Sample dataframe (fetch it from github, or read your own)

path = "https://raw.githubusercontent.com/Akshaysehgal2005/Datasets/main/titanic_extended.csv"
df = pd.read_csv(path)

print(df.shape) #(1310, 14)
df.head()

Visualizing Missing Data

The missingno library offers various visualization tools to analyze missing data. Let's explore some of them -

1. Matrix

The matrix provides a heatmap-like representation of the data's completeness:

msno.matrix(df)

1. In this matrix, the white lines represent missing values. This visualization helps in spotting trends in the occurrence of missing values.

2. The vertical line graph on the far right indicates the count of data points in each row, also marking the minimum and maximum counts. For instance, the number 13 signifies a row with a maximum of 13 data points at that position, while a 0 indicates a row at that position with no data points, representing an empty row.

2. Bar Chart

The bar chart shows the completeness of the dataset:

msno.bar(df)

This visualization is quite practical, offering a straightforward count of missing values by column.

3. Heatmap

The heatmap shows the correlation of missingness between different columns:

msno.heatmap(df)

A positive value indicates a positive correlation between the missingness of two columns, and vice versa. This concept is called Nullity Correlation.

Nullity correlation is a measure that assesses the relationship between the presence of missing values in different variables or columns in a dataset. It is a specific type of correlation that focuses on the patterns of missing (null) data rather than the actual data values.

Similar to a standard correlation coefficient, nullity correlation values typically range from -1 to 1.

• A value close to +1 indicates that if one column has missing data, the other column is likely to have missing data as well.

• A value close to -1 suggests that if one column has missing data, the other column is likely not to have missing data.

• A value around 0 indicates no particular correlation between the missingness of the two columns.

Here, we observe an inverse nullity correlation of 0.2 between the variables Body and Boat, indicating that rows with one variable occasionally lack data in the other, and vice versa.

4. Dendrogram

The dendrogram helps to understand the clustering of missing data:

Columns clustered together have a similar pattern of missing data.

What’s next?

Once you have analyzed the gaps in your dataset, you could use a ton of methods for handling the missing data.

• Flat remove rows with any missing data

• Remove rows for specific variables containing missing data

• Drop variables with high volumes of missing data

• Impute missing data for numeric variables

• Direct replacements using aggregates like mean, median, mode, etc.

• Estimate and fill in missing data using ML (e.g. nearest neighbor imputation)

• etc.

The possibilities are endless and each dataset and use-case will require its own strategy to handle information loss due to missing data. Once your dataset is cleaned up, you might want to use missingno to visualize it again which might look something like this.

Nothing is more satisfying after an hour of data cleaning than the visualization above!

Conclusion

In conclusion, missingno is a powerful tool for a preliminary check of missing data in your dataset. It helps in understanding the pattern and extent of missingness, which is crucial for any data preprocessing steps. By integrating missingno it into your data analysis workflow, you can make more informed decisions about how to handle missing values in your dataset.

Thanks to Aleksey Bilogur for authoring and maintaining missingno for all these years!

References

• https://github.com/ResidentMario/missingno

• https://www.kaggle.com/code/akshaysehgal/handling-missing-data-like-a-boss

• https://scikit-learn.org/stable/modules/impute.html

Here are a few examples of what the 2D lattices and the “gaps” in them look like -

The yellow points represent 1s and the rest represent 0s. Notice that in a few places where the regular structure should persist, a few 1s are missing. The goal of this puzzle is to write code that can identify these “gaps”. Here is the starter code to create these 2D arrays in python -

import numpy as np

arr = np.array([[1, 0, 0, 0, 1, 0, 0, 0, 0], #one gap here 
                [0, 0, 1, 0, 0, 0, 1, 0, 0],
                [1, 0, 0, 0, 1, 0, 0, 0, 1],
                [0, 0, 1, 0, 0, 0, 0, 0, 0], #one gap here
                [1, 0, 0, 0, 1, 0, 0, 0, 1]])

NOTE: There is one assumption here. There should exist at least one complete pattern in the lattice/array. Not every instance of the pattern can have a gap in it!

And, let’s say you are given a function to generate such “impure” lattices on demand, so you can test your algorithm properly!

import random
import numpy as np
import matplotlib.pyplot as plt

def generate_random_lattice(gaps=0.1):
    
    #Generate random tile pattern
    n,m = random.randrange(3,9,2), random.randrange(3,9,2)
    tile = np.zeros((n,m)).astype(int)
    tile[::tile.shape[0]-1, ::tile.shape[1]-1] = 1  #update corners
    tile[tile.shape[0]//2, tile.shape[1]//2] = 1    #update center
    tile = tile[:-1, :-1]                           #remove bottom edge
    
    #Create pure lattice
    x,y = np.random.randint(2,10), np.random.randint(2,10)
    lattice = np.tile(tile, (x,y))
    
    #Add impurities / gaps
    ones_shape = lattice[lattice.nonzero()].shape
    lattice[lattice.nonzero()] = np.random.binomial(n=1, p=1-gaps, size=ones_shape)
    
    #Plot lattice
    plt.imshow(lattice)
    return lattice

arr = generate_random_lattice()

Do think about how you would solve this puzzle algorithmically, or better yet, attempt it before scrolling down to the (potential) solution!

There are a million ways to solve this problem, but I would like to explore one that involves an operation which is is very frequently used in the world of Data Science and Machine Learning: Convolutions.

Solution: Using Convolutions

The folks with a background in ML/Data Science would already know what the term “Convolution” is, as it is foundational to Computer Vision and Image Processing. Convolution Neural Networks (CNNs) are a class of neural network architectures and at their core, they use the operation named Convolution.

I will not go a lot deeper into what the Convolution operation or CNNs are but here is a quick overview.

What is a 2D Convolution?

A 2D convolution operation involves a matrix/image and a filter. The filter is moved/convolved over the matrix and a dot product is performed, which is returned in the output matrix. In an implementation, since these are independent operations, these can be done in parallel in a vectorized manner.

We can use other operations instead of a dot product as well! The idea here is that if the pattern in the filter is found in the image exactly, it should return a very high value. The analysis of the output of the operation reveals what the filter and the operation is designed to discover.

There is additional complexity here due to how padding can be leveraged etc, but I will skip that for this post. There is a lot of great material on this online, for example, this one. Here is a visual representation of what this operation looks like (image source: wikipedia)

An interesting thing here is that Convolutions are used for identifying “patterns” in an image, but they can also be used to find a lack of a given pattern, which is the case in this coding puzzle.

With this knowledge, let’s design an algorithm of how we can detect and mark these gaps given an “imperfect lattice” or an array with a repeating pattern with some imperfections.

Here is the pseudo-code -

1. Identify the smallest convolution window size by analyzing the proximity of 1s, row-wise and column-wise.

2. Estimate the number of such repeating patterns in the array over both axes.

3. Fetch all the convolution windows by iterating over the array.

4. Take the sum of each convolution window to search for “gaps”. The largest sum would indicate a complete pattern, while anything less would indicate incomplete patterns.

5. Fetch a copy of the complete pattern and subtract it (with broadcasting) from each convolution window to get the “gaps” as -1’s.

6. Check the location of the -1’s and mark them in the original array.

Let’s begin implementing it. First, let’s start by generating an example to work with.

arr = generate_random_lattice() #using the function above
print(arr.shape)

(19, 17)

Let’s start by analyzing the proximity of 1s over the 2 axes. When working on anything related to “distance between elements” or “span of a specific element” in arrays, I find using numpy.cumsum() is quite common. Because we have gaps in the lattice, we can get the pattern over the 2 axes by just taking a sum as below -

print(arr.sum(1))
print(arr.sum(0))

[4 0 0 3 0 0 5 0 0 3 0 0 5 0 0 4 0 0 4] #pattern of x-2-x-2-x
[4 0 2 0 4 0 3 0 4 0 3 0 4 0 2 0 2]     #pattern of x-1-x-1-x

We can see that this shows us the structure of the pattern over both axes while “ignoring” the gaps in the lattice. Let’s use this to get the shape of the convolution window that we want to fetch from the array.

xdims = np.unique((arr.sum(1)!=0).cumsum(), return_counts=True)[1].max()*2+1
ydims = np.unique((arr.sum(0)!=0).cumsum(), return_counts=True)[1].max()*2+1

pattern_shape = (xdims, ydims)
print(pattern_shape)

(7, 5)

With this, now we know that the smallest pattern shape that we can create convolution windows is (7,5). Now, we could create a perfect pattern of this shape and just use that to find imperfections in our lattice, but where is the fun in that? Let’s first find out how many sliding window operations are needed over both axes, for a pattern of this shape to repeat and fit into the overall array.

num_xpatterns = int(arr.shape[0]/(pattern_shape[0]-1))
num_ypatterns = int(arr.shape[1]/(pattern_shape[1]-1))

print((num_xpatterns, num_ypatterns))

(3, 4)

This is straightforward. Since the pattern is not completely repeating, but shares an edge, we need to remove the last row/column of pixels from the pattern shape and then calculate how many times we need to move. Here, we need to repeat the (7,5) window 3 times to the right and 4 times downwards.

Let’s now break the original array into these 3 X 4 = 12 matrices of (7,5) shape sliding windows. I will use numpy.lib.stride_tricks2 for this.

#Calculating the stride and shape that needs to be taken with stride_tricks
shp = (num_xpatterns, num_ypatterns, xdims, ydims)
strd = (arr.strides[0]*(xdims-1), arr.strides[1]*(ydims-1), arr.strides[0], arr.strides[1])

#Generating rolling windows/convolutions over the image to separate the patterns.
convolve_pattern = np.lib.stride_tricks.as_strided(arr, shape=shp, strides=strd)
convolve_pattern.shape

(3, 4, 7, 5)

As expected, we get a tensor of the shape (3,4,7,5) which means 3X4 matrices of shape (7,5). Let’s fetch one of them to check what it looks like.

convolve_pattern[0,-1,:,:]

array([[1, 0, 0, 0, 0],   #<- one gap here
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 1, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [1, 0, 0, 0, 1]])

This corresponds to this window of the array -

Great next, we just calculate the sum of each of these windows to find the gaps!

pattern_sums = convolve_pattern.sum(axis=(-1,-2))
pattern_sums

array([[4, 5, 5, 4],
       [5, 5, 5, 4],
       [5, 5, 5, 4]])

So, in this (3,4) matrix, the values where the sum is 5 are examples of perfect patterns, while the ones < 5 are patterns with gaps. Let’s fetch a perfect pattern from this view.

pattern_sums = convolve_pattern.sum(axis=(-1,-2))
idx = np.unravel_index(np.argmax(pattern_sums), pattern_sums.shape)
truth_pattern = convolve_pattern[idx]
print(truth_pattern)

array([[1, 0, 0, 0, 1],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 1, 0, 0],
       [0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0],
       [1, 0, 0, 0, 1]])

Nice, we are almost there! Now, since we have this (7,5) pattern, we can subtract it from the convolved view of the array (3,4,7,5) with broadcasting to get the gap locations!

gaps = convolve_pattern - truth_pattern[None, None, :, :]
gaps[0,-1,:,:]

array([[ 0,  0,  0,  0, -1],  #<- gap marked with -1
       [ 0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0]])

Then we can use the beautiful concept of numpy views vs copies3, to overwrite the 0s of these gap locations on the memory of the original matrix with -1s. Overwriting the locations on the (3,4,7,5) view also overwrites the original location in the (19,17) array since they share the same memory! This is the power of understanding how to work with views vs copies!

for i in np.argwhere(gaps==-1):
    convolve_pattern[tuple(i)]=-1

Thats it! Here is the complete code and the function in action!

#IDENTIFYING IMPERFECT PATTERNS IN A LATTICE

import numpy as np
import matplotlib.pyplot as plt

def find_gaps(x, verbose=False):
    #Identifying the size and shape of the repeating pattern
    xdims = np.unique((x.sum(1)!=0).cumsum(), return_counts=True)[1].max()*2+1
    ydims = np.unique((x.sum(0)!=0).cumsum(), return_counts=True)[1].max()*2+1
    pattern_shape = (xdims, ydims)

    #Calculating number of rolling windows that exist with that pattern
    num_xpatterns = int(x.shape[0]/(pattern_shape[0]-1))
    num_ypatterns = int(x.shape[1]/(pattern_shape[1]-1))
    
    #Calculating the stride and shape that needs to be taken with stride_tricks
    shp = (num_xpatterns, num_ypatterns, xdims, ydims)
    strd = (x.strides[0]*(xdims-1), x.strides[1]*(ydims-1), x.strides[0], x.strides[1])

    #Generating rolling windows/convolutions over the image to separate the patterns.
    convolve_pattern = np.lib.stride_tricks.as_strided(x, shape=shp, strides=strd)

    #Assuming at least 1 untouched pattern exists, finding that pure pattern
    pattern_sums = convolve_pattern.sum(axis=(-1,-2))
    idx = np.unravel_index(np.argmax(pattern_sums), pattern_sums.shape)
    truth_pattern = convolve_pattern[idx]

    #Printing Debugging info
    if verbose==True:
        print('x shape:',x.shape)
        print('pattern shape:',pattern_shape)

        print('convolved shape:',convolve_pattern.shape)

        print('')
        print('pattern sums')
        print(pattern_sums)

        print('')
        print('true pattern')
        print(truth_pattern)
    
    #Identifying the gaps by subtracting the convolved image with the truth pattern
    gaps = convolve_pattern - truth_pattern[None, None, :, :]

    #Setting the gaps as -1 directly into the location of memory of the original image
    for i in np.argwhere(gaps==-1):
        convolve_pattern[tuple(i)]=-1
    
    plt.imshow(x)
    return x

arr = generate_random_lattice()   #Generate lattice
o = find_gaps(x, verbose=False)   #Find gaps in lattice

Thanks for reading all the way to the end! My goal here was to demonstrate the power of Convolution operations and Numpy in solving unique scenarios. Hope this was entertaining and educational for you, the reader!

In fact, infinite sequences are natural to programming languages such as Haskell. Here is a great video from Computerphile to demonstrate that.

But did you know that you can work with infinite sequences quite efficiently in Python as well? You can write while loops but a pythonic way would be to create iterators that you can use. In fact, itertools has some very interesting out-of-the-box methods called "infinite iterators" for you to work with infinite sequences.

There are 3 infinite iterators in itertools that you can work with -

Working with Infinite Iterators

Let's start with a simple example of generating prime factors for an infinite sequence of multiples of 5 greater than 100 -

#!pip install sympy
from itertools import count, islice
from sympy.ntheory import primefactors

itr = count(start=100,step=5)      #<-- infinite sequence
f = lambda x: (x, primefactors(x)) #<-- fx for prime factors

factors = map(f, itr)              #<-- map the function

list(islice(factors, 10))          #<-- get first 10 elements

 (105, [3, 5, 7]),
 (110, [2, 5, 11]),
 (115, [5, 23]),
 (120, [2, 3, 5]),
 (125, [5]),
 (130, [2, 5, 13]),
 (135, [3, 5]),
 (140, [2, 5, 7]),
 (145, [5, 29])]

You can see the power of this. The map() + count() structure can get super complex super, fast with more advanced operations, and all of this, without defining a close set of inputs. Then during inference, you could use itertools.islice or just a simple for loop to infer them as per need with lazy execution.

Here is another example of summing 2 infinite sequences -

iter1 = count(10,2)         #<-- inf seq start 10, step 2
iter2 = count(5,3)          #<-- inf seq start 5, step 3
g = lambda x, y: (x,y,x+y)  #<-- sum function

inf_sum = map(g, iter1, iter2) #<-- map the function

list(islice(inf_sum, 10))   #<-- get first 10 elements

[(10, 5, 15),
 (12, 8, 20),
 (14, 11, 25),
 (16, 14, 30),
 (18, 17, 35),
 (20, 20, 40),
 (22, 23, 45),
 (24, 26, 50),
 (26, 29, 55),
 (28, 32, 60)]

You have a few more options to build infinite sequences using itertools.cycle and itertools.repeat -

from itertools import cycle

iter3 = itertools.cycle(['A','B','C'])
list(islice(iter3, 10))

## ['A', 'B', 'C', 'A', 'B', 'C', 'A', 'B', 'C', 'A']

from itertools import repeat

iter4 = itertools.repeat('hello world') #<-- pass any object
list(islice(iter4, 5))

## ['hello world', 'hello world', 'hello world', 'hello world', 'hello world']

And, you can mix and match these iterators with your functions using map().

I worked with these iterators for creating a batch-wise Numpy computation. My Numpy operation was taking a ton of memory due to a large number of input vectors and so I needed a strategy where I could break the large computation task into smaller batches. The function I wrote would take a batch of vectors, run a vectorized operation on them, save the output, and then fetch the next batch. This would work infinitely until it met a break condition.

This was a very high-level overview of the world of infinite sequences using Python. Hope this gives you some ideas when working on your next project involving such functions!

The principle is simple - Embed words/sentences into an N-dimensional vector space where the distance between the vectors of 2 words encodes their semantic similarity. In practice, if done correctly, you would find that the embedding of the word “car” and “bike” would be clustered closer to each other than the embedding of “car” and “spaceship”. Given good vector representations or embeddings of your data, you could explore a large variety of downstream models or use them as features for secondary tasks! But this same concept is expandable to other types of data - images, tabular, graphs, etc.

Flashback - At the time, I was working with a database of “Skillsets” which were structured as a hierarchy, for example, Programming → Python → Pandas → Dataframe. And, during my quest to “master” the world of Representation learning, I came across this very unique challenge of representing Hierarchical Data. I wanted to know how I could create embeddings for these “skills” using the hierarchical relationships between them and use these as part of my downstream estimators.

Enter Poincaré Embeddings!

This post will focus on implementing these rather than exploring the maths behind them but here is a quick overview. So feel free to skip this section!!

“However, while complex symbolic datasets often exhibit a latent hierarchical structure, state-of-the-art methods typically learn embeddings in Euclidean vector spaces, which do not account for this property. For this purpose, we introduce a new approach for learning hierarchical representations of symbolic data by embedding them into hyperbolic space – or more precisely into an n-dimensional Poincaré ball.”

— Nickel, M., & Kiela, D. (2017). Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems, 30.

Poincaré embeddings allow you to create hierarchical embeddings in a non-euclidean space. The vectors on the outside of the Poincaré ball are lower in the hierarchy compared to the ones in the center.

The transformation to map a Euclidean metric tensor to a Riemannian metric tensor is an open d-dimensional unit ball.

Distances between 2 vectors in this non-euclidean space are calculated as -

Once plotted, your hierarchical data might look like this when mapped over a Poincaré ball.

The research paper for Poincaré embeddings is wonderfully written and you will find some easy implementations in popular libraries for them as well. Needless to say, they are underrated.

Two popular implementations that you can use are found in -

• gensim.models.poincare

• tensorflow_addons.PoincareNormalize

Implementing Poincaré Embeddings!

Let’s start by generating some hierarchical data. If you have access to such a dataset already, feel free to use that. But for our exploration of the concept, I find that the synset relationships in NLTK’s Wordnet are perfect!

Here is a function I wrote to generate this data (inspired by this repo) -

def generate_data(root='mathematics.n.01'):
    """
    Script to generate sample DAG data using input root
    """
    #INSTALL THE CORPUS WORDNET (UNCOMMENT FIRST RUN)
    #import nltk
    #nltk.download('wordnet')
    #nltk.download('omw-1.4')

    #IMPORT DEPENDENCIES
    import random
    from nltk.corpus import wordnet as wn

    #SET ROOT/TARGET AS MAMMAL(NOUN)
    target = wn.synset(root)
    print('target', target.name())

    #FIND ALL NOUNS IN WORDNET
    words = wn.words()
    nouns = set([])
    for word in words:
        nouns.update(wn.synsets(word, pos='n'))  
    print(len(nouns), 'nouns')

    #FETCH ALL HYPERNYMS WITH PATH TO TARGET
    hypernyms = []
    for noun in nouns:
        paths = noun.hypernym_paths()
        for path in paths:
            try:
                pos = path.index(target)
                for i in range(pos, len(path)-1):
                    hypernyms.append((noun, path[i]))
            except Exception:
                continue            
    hypernyms = list(set(hypernyms))
    print(len(hypernyms), 'hypernyms')

    #SHUFFLE AND SAVE ALL RELATIONSHIP TUPLES
    random.shuffle(hypernyms)
    data = [(n1.name(), n2.name()) for n1, n2 in hypernyms]
    print(len(data), 'relations')
    return data

#generating data with root as "bird"
data = generate_data('bird.n.01')

# target bird.n.01
# 82115 nouns
# 3435 hypernyms
# 3435 relations

#Show 20 sample datapoints
print(data[:20])

[('australian_turtledove.n.01', 'columbiform_bird.n.01'),
 ('snowy_egret.n.01', 'bird.n.01'),
 ('american_gallinule.n.01', 'gallinule.n.01'),
 ('banded_stilt.n.01', 'stilt.n.03'),
 ('plover.n.01', 'shorebird.n.01'),
 ('great_bowerbird.n.01', 'passerine.n.01'),
 ('jaeger.n.01', 'coastal_diving_bird.n.01'),
 ('blue_jay.n.01', 'new_world_jay.n.01'),
 ('new_world_jay.n.01', 'passerine.n.01'),
 ('auklet.n.01', 'aquatic_bird.n.01'),
 ('hedge_sparrow.n.01', 'passerine.n.01'),
 ('shoveler.n.02', 'anseriform_bird.n.01'),
 ('peahen.n.01', 'gallinaceous_bird.n.01'),
 ('pheasant_coucal.n.01', 'coucal.n.01'),
 ('goldfinch.n.02', 'oscine.n.01'),
 ('old_squaw.n.01', 'sea_duck.n.01'),
 ('cochin.n.01', 'domestic_fowl.n.01'),
 ('corncrake.n.01', 'aquatic_bird.n.01'),
 ('least_sandpiper.n.01', 'shorebird.n.01'),
 ('warbler.n.02', 'oscine.n.01')]

This hierarchical data is structured as a list of 3,435 “child → parent” tuples. Let’s quickly plot this to see what these might look like.

Also, try using other roots such as mathematics.n.01 or mammal.n.01

import networkx as nx
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(18,14))
g = nx.Graph(data)
nx.draw(g, with_labels=True)

Not very readable but the local and global structure is somewhat visible.

Let’s use Gensim to implement Poincare Embeddings. Want to know how easy it is? Just 2 lines of code as marked below -

import numpy as np
from gensim.models.poincare import PoincareModel

#MODEL TRAINING
model = PoincareModel(data, size=50, negative=10)  #<---
model.train(epochs=30)                             #<---

#FETCH VECTORS AND STORE
vectors = {}
keys = model.kv.index_to_key
for k in keys:
    vectors[k] = model.kv.get_vector(k)

#STACKING EMBEDDINGS TO A NDARRAY
embeddings = np.stack(vectors.values())

print(len(keys),'keys')
print(embeddings.shape[0], 'embeddings')
print(embeddings.shape[1], 'dimensions')

872 keys
872 embeddings
50 dimensions

Great, so we have 872 nodes in our hierarchy and each one is represented with a 50-dimensional vector in this non-euclidean space. Let’s use the Poincare distance implementation from Gensim to see if the hierarchical relationships are encoded in the model.

print('owl vs hawk:', model.kv.similarity('owl.n.01','hawk.n.01'))

print('eagle vs bald_eagle:', model.kv.similarity('eagle.n.01','bald_eagle.n.01'))
print('golden_eagle vs bald_eagle:', model.kv.similarity('golden_eagle.n.01','bald_eagle.n.01'))

print('owl vs barn_owl:', model.kv.similarity('owl.n.01','barn_owl.n.01'))
print('barn_owl vs spotted_owl:', model.kv.similarity('barn_owl.n.01','spotted_owl.n.01'))
print('barn_owl vs mandarin_duck:', model.kv.similarity('barn_owl.n.01','mandarin_duck.n.01'))

owl vs hawk: 0.31635084567961813

eagle vs bald_eagle: 0.8757754918787964
golden_eagle vs bald_eagle: 0.8758848440323967

owl vs barn_owl: 0.6982827095728305
barn_owl vs spotted_owl: 0.738832541978296
barn_owl vs mandarin_duck: 0.1329321668369148

Excellent! Related nodes are closer to each other in this space as expected! Let’s maybe use Principle Component Analysis to plot this space in 3D just to get a sense of how this is clustered.

NOTE: We are using PCA to project a non-euclidean space into a 3 dimensional euclidean space, so take it with a grain of salt!

import pandas as pd
import plotly.express as px
from sklearn.decomposition import PCA

pca = PCA(3)
components = pca.fit_transform(embeddings)

keys = model.kv.index_to_key
cols = ['component1','component2','component3']
plot_df = pd.DataFrame(components, columns=cols, index=keys)

# PLOTTING ONLY 100 SAMPLE VECTORS
plot_df = plot_df.head(100)   #<---

px.scatter_3d(plot_df, 
              x='component1', y='component2', z='component3', 
              text=plot_df.index, opacity=0.5, height=900, width=900)

And as expected, we see the outline of the sphere where these embeddings exist. Also, we see that the outer edge of the sphere has a higher density of nodes due to the nature of how these embeddings are mapped onto this curved space. The clusters of related nodes are quite visible in the additional zoomed-in views.

Want to use these in your neural networks?

Worry not! You can use, tensorflow_addons.layers.PoincareNormalize and add this layer over your Embedding layer!

from tensorflow.keras import layers, Model, utils
import tensorflow_addons as tfa

X = np.random.random((100,10))
y = np.random.random((100,))


inp = layers.Input((10,))
x = layers.Embedding(500, 5)(inp)
x = tfa.layers.PoincareNormalize(axis=-1)(x)  #<-------
x = layers.Flatten()(x)
out = layers.Dense(1)(x)

model = Model(inp, out)
model.compile(optimizer='adam', loss='binary_crossentropy')
utils.plot_model(model, show_shapes=True, show_layer_names=False)

model.fit(X, y, epochs=3)

So there you have it! Poincaré Embeddings in all their glory. It’s a powerful concept that I hope you can now use as part of your model experiments when you have to work with hierarchical data!

Here are 10 ways you can implement rolling windows/n-grams in python!

Let’s start with a toy example as usual -

# Rolling window size / n in ngram
n = 3

# Example sentence / sequence
doc = "The quick brown fox jumps over the lazy dog"
tokens = doc.split(' ')
print(tokens)

['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog']

So, we have a humble yet overused sequence of tokens, and we need to iterate over this with a rolling window of size 3, otherwise known as tri-grams.

Let’s get rolling (pun intended)!

1. Classic for loop

All we need to do here is iterate over the range of length of the sequence and use basic list indexing as follows -

out = []
for i in range(len(tokens)):
    out.append(tokens[i:i+n])

out

[['The', 'quick', 'brown'],
 ['quick', 'brown', 'fox'],
 ['brown', 'fox', 'jumps'],
 ['fox', 'jumps', 'over'],
 ['jumps', 'over', 'the'],
 ['over', 'the', 'lazy'],
 ['the', 'lazy', 'dog'],
 ['lazy', 'dog'],
 ['dog']]

You can choose to skip the sublists which are smaller than the window size with a simple if condition.

2. List Comprehension

I know this might be a tiny bit of cheating, but some of us just love using list comprehensions everywhere, which are just another way of writing the above code.

[tokens[i:i+n] for i in range(len(tokens))]

[['The', 'quick', 'brown'],
 ['quick', 'brown', 'fox'],
 ['brown', 'fox', 'jumps'],
 ['fox', 'jumps', 'over'],
 ['jumps', 'over', 'the'],
 ['over', 'the', 'lazy'],
 ['the', 'lazy', 'dog'],
 ['lazy', 'dog'],
 ['dog']]

3. ZIP *

This might be my go-to method as I find it quite pythonic, but here is how you can use zip along with the unpacking operator * -

list(zip(*[tokens[i:] for i in range(n)]))

[('The', 'quick', 'brown'),
 ('quick', 'brown', 'fox'),
 ('brown', 'fox', 'jumps'),
 ('fox', 'jumps', 'over'),
 ('jumps', 'over', 'the'),
 ('over', 'the', 'lazy'),
 ('the', 'lazy', 'dog')]

How does this work? What's happening here is that list comprehension creates L, L[1:], L[2:] which then get zipped, meaning the first element of each of these (which are the 1st, 2nd, and 3rd elements of L) get clubbed together and the second elements get clubbed together, and so on. The unpacking operator just unpacks the list containing L, L[1:], L[2:] for zip to work with. Here is a helpful diagram that I made to explain this intuitively.

4. Pandas Rolling Objects

You might have used this for performing rolling averages or sums, but did you know that you can iterate over a pandas rolling object? Here is how you can do this -

import pandas as pd
list(map(list,pd.Series(tokens).rolling(n)))

[['The'],
 ['The', 'quick'],
 ['The', 'quick', 'brown'],
 ['quick', 'brown', 'fox'],
 ['brown', 'fox', 'jumps'],
 ['fox', 'jumps', 'over'],
 ['jumps', 'over', 'the'],
 ['over', 'the', 'lazy'],
 ['the', 'lazy', 'dog']]

All we have to do is to map a list class to typecast the iterable rolling object. Or you could use a list comprehension as well! However, if you want to get a “forward rolling window” style output, it just takes an extra step of defining the window type.

import pandas as pd

indexer = pd.api.indexers.FixedForwardWindowIndexer(window_size=n)
list(map(list,pd.Series(tokens).rolling(indexer)))

[['The', 'quick', 'brown'],
 ['quick', 'brown', 'fox'],
 ['brown', 'fox', 'jumps'],
 ['fox', 'jumps', 'over'],
 ['jumps', 'over', 'the'],
 ['over', 'the', 'lazy'],
 ['the', 'lazy', 'dog'],
 ['lazy', 'dog'],
 ['dog']]

5. Numpy Stride Tricks

This one is a slightly more advanced method as it involves accessing a numpy array’s contiguous memory storage directly to create a “view”, but it’s super powerful and pretty much used universally behind the scenes for a lot of numpy high-level functions.

import numpy as np

arr = np.array(tokens)
shape = (arr.shape[0] - n + 1, n)            # (7, 3)
strides = (arr.strides[0], arr.strides[0])   # (20, 20) bytes

np.lib.stride_tricks.as_strided(arr, shape=shape, strides=strides)

array([['The', 'quick', 'brown'],
       ['quick', 'brown', 'fox'],
       ['brown', 'fox', 'jumps'],
       ['fox', 'jumps', 'over'],
       ['jumps', 'over', 'the'],
       ['over', 'the', 'lazy'],
       ['the', 'lazy', 'dog']], dtype='<U5')

The “shape” is the expected output shape for the view of this numpy array and the “strides” are the numbers of bytes numpy has to move in each axis to reach the next element. Use this method at your own risk as it can cause memory corruption if now used properly!!

6. NLTK ngrams

You can’t talk about n-grams without talking about NLTK. Most of us have learned our first implementations of n-grams using NLTK, and it can be used for any standard iterator!

from nltk import ngrams
list(ngrams(tokens, n))

[('The', 'quick', 'brown'),
 ('quick', 'brown', 'fox'),
 ('brown', 'fox', 'jumps'),
 ('fox', 'jumps', 'over'),
 ('jumps', 'over', 'the'),
 ('over', 'the', 'lazy'),
 ('the', 'lazy', 'dog')]

7. more_itertools library

This is a purpose-built library that extends the classic itertools library with some interesting functions. One of these is the more_itertools.windowed method.

#pip install more_itertools
import more_itertools
list(more_itertools.windowed(tokens, n))

[('The', 'quick', 'brown'),
 ('quick', 'brown', 'fox'),
 ('brown', 'fox', 'jumps'),
 ('fox', 'jumps', 'over'),
 ('jumps', 'over', 'the'),
 ('over', 'the', 'lazy'),
 ('the', 'lazy', 'dog')]

8. toolz library

Another example of a library built as an extension to itertools, which comes inbuilt with the handy sliding_window method.

#pip install toolz
import toolz
list(toolz.sliding_window(n, tokens))

[('The', 'quick', 'brown'),
 ('quick', 'brown', 'fox'),
 ('brown', 'fox', 'jumps'),
 ('fox', 'jumps', 'over'),
 ('jumps', 'over', 'the'),
 ('over', 'the', 'lazy'),
 ('the', 'lazy', 'dog')]

9. Itertools islice

Implementing this in itertools is a slight bit more complex, but very handy to learn as it exposes you to work with some underrated yet powerful itertools methods such as islice and tee.

from itertools import islice, tee
list(zip(*(islice(s, i, None) for i, s in enumerate(tee(tokens, n)))))

[('The', 'quick', 'brown'),
 ('quick', 'brown', 'fox'),
 ('brown', 'fox', 'jumps'),
 ('fox', 'jumps', 'over'),
 ('jumps', 'over', 'the'),
 ('over', 'the', 'lazy'),
 ('the', 'lazy', 'dog')]

10. Scikit Learn CountVectorizer

Lastly, this is kinda a cheat and mostly applicable to text documents/sentences but it’s something that data scientists use quite regularly as part of their modeling pipelines. The trick is to define your CountVectorizer with the ngram_range as (n,n) thus only fetching n-grams and not the uni-grams, bi-grams, etc.

from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer(ngram_range=(n,n))
analyzer = cv.build_analyzer()
analyzer(doc)

['the quick brown',
 'quick brown fox',
 'brown fox jumps',
 'fox jumps over',
 'jumps over the',
 'over the lazy',
 'the lazy dog']

The output returned is not a list of tokens but a string and notice that the input to the analyzer is also the original document rather than the token list.

So that’s 10 ways of quickly implementing rolling window iteration or n-grams in a given sequence using Python! Hope this has been useful for you, the reader!

Let's set up a toy example.

import pandas as pd

df = pd.DataFrame({"name": ["Rick","Morty","Beth","Summer","Jerry"], 
                   "age": [70, 14, 35, 17, 35]})
print(df)

     name  age
0    Rick   70
1   Morty   14
2    Beth   35
3  Summer   17
4   Jerry   35

Let’s say we want to create labels for the Smith family members based on age. A classic way might be to write a function with all of the necessary conditions and then apply that function row-wise.

def age_group(row):
    if row.age >= 0 and row.age < 20:
        return '0 - 20 yrs'
    elif row.age >= 20 and row.age < 40:
        return '20 - 40 yrs'
    elif row.age >= 40 and row.age < 60:
        return '40 - 60 yrs'
    elif row.age >= 60:
        return '60+ yrs'
    else:
        return 'invalid age'
    
df['apply_age'] = df.apply(age_group,1)
print(df)

     name  age    apply_age
0    Rick   70      60+ yrs
1   Morty   14   0 - 20 yrs
2    Beth   35  20 - 40 yrs
3  Summer   17   0 - 20 yrs
4   Jerry   35  20 - 40 yrs

But this is ugly, and slow, mostly because a pd.DataFrame.apply method in pandas is not vectorized! So is there a more pythonic way to do this, which is readable, cleaner, and faster at the same time?

Introducing Numpy’s Select method -

This is how it works. You pass the numpy.select method a list of conditions, a list of values for each of those conditions (if, elif), and a default value (else).

numpy.select(condlist, choicelist, default)

So how does it work in action?

import numpy as np

#Conditions
c1 = (df.age >= 0) & (df.age < 20)
c2 = (df.age >= 20) & (df.age < 40)
c3 = (df.age >= 40) & (df.age < 60)
c4 = (df.age >= 60)

#Choices
values = ['0 - 20 yrs', '20 - 40 yrs', '40 - 60 yrs', '60+ yrs']

#Default
default = 'invalid age'

df['select_age'] = np.select([c1,c2,c3,c4], values, default=default)
print(df)

     name  age   select_age
0    Rick   70      60+ yrs
1   Morty   14   0 - 20 yrs
2    Beth   35  20 - 40 yrs
3  Summer   17   0 - 20 yrs
4   Jerry   35  20 - 40 yrs

I hope you agree with me when I say this is significantly cleaner and more readable, especially when you imagine working with 10s of 100s such conditions with higher complexity!

But, as you might remember, I claimed earlier that this is also much faster than the traditional way. That is because it is vectorized unlike the pd.DataFrame.apply which is applying your function one row at a time.

How fast you ask?

Let’s scale the data and test this out with a %%timeit -

df = pd.concat([df]*10000)  #Repeating the dataframe 10k times

%%timeit for apply_method
539 ms ± 3.48 ms per loop

%%timeit for np_select_method
3.84 ms ± 52.7 µs per loop

That’s 140x faster at just 50k rows and this only gets better and better as the data size increases!

So next time you are working with a ton of if’s and elif’s, know that there is a more pythonic way to do this which your fellow Data Scientists will thank you for!

1. Introduction

2. Syntax
   2.1 Adding more groups/levels
   2.2 Adding more variables/features

3. Grouping

4. Aggregation
   4.1 In-built aggregation methods
   4.2 Custom functions with pandas apply
   4.3 Multiple aggregations using agg method
   4.4 Custom functions with agg method

5. Transform

6. Advanced Usage
   6.1 Sequential/local grouping of a dataframe
   6.2 Re-indexing to a fixed date range for each group

7. Other ways of grouping data
   7.1 Using collections' defaultdict
   7.2 Using numpy's split function
   7.3 Using itertools' groupby

8. References

1. Introduction

"Groupby" is probably one of the most basic data pre-processing steps that a Data Scientist should master as soon as possible. Interestingly enough, you find it in almost every scripting language that claims to work well with databases.

Most of us would have been introduced to the SQL GROUPBY statement which allows a user to summarize or aggregate a given dataset. Python brings the pandas groupby method to the table, which is highly pythonic in its syntax and equally versatile, if not more. But the utility of a groupby is much more than just aggregation. In this notebook, I will showcase a few examples, where you could really exploit this method for various other use cases.

#SQL Query to groupby Col1 and Col2
#and get mean and sum of col3 and col 4 respectively

SELECT Col1, Col2, mean(Col3), sum(Col4)
FROM Table
GROUP BY Col1, Col2

Before we can start writing code, let's explore the basics behind a groupby operation. The core concept behind any groupby operation is a three-step process called Split-Apply-Combine.

• Split: Splitting the data into groups based on some criteria

• Apply: Applying a function to each group independently

• Combine: Combining the results into a data structure

Here is a diagram to make this more intuitive.

2. Syntax

The syntax for using a groupby method in Pandas comprises 2 parts. The first is a grouper object and the second is the aggregator. The general structure looks like the following -

        df.groupby(['groups'])['cols'].aggregations()
        |____________________||_____________________|
                   |                     |
            grouper(split)  aggregation(apply & combine)

This will get more clear as we take an example from actual data. So, let's start by loading a dataset to work with. For this notebook, I will use the Titanic dataset which can either be downloaded from Kaggle or directly loaded using the visualization library called Seaborn.

#Load dependencies
import pandas as pd
import numpy as np
import seaborn as sns

titanic = sns.load_dataset('titanic').dropna() #drop missing data

print(titanic.shape)
titanic.head()

Question: What is the total number of passengers from each class who survived?

#Step 1: Create grouper
grouper = titanic.groupby(['class'])

#Step 2: Filter column and apply aggregation
grouper['survived'].sum().reset_index()

You would usually do this in a single statement as the following:

titanic.groupby(['class'])['survived'].sum().reset_index()

Note: The reset_index() helps bring the grouping columns from index, back as a column in a dataframe.

Question: What was the average fare for passengers from each town?

titanic.groupby(['embark_town'])['fare'].mean().reset_index()

2.1 Adding more groups/levels

We can pass a list of features in the groupby() to increase the levels for grouping the data as below.

Question: What was the average fare for male vs female passengers from each town?

titanic.groupby(['embark_town','sex'])['fare'].mean().reset_index()

2.2 Adding more variables/features

Similarly, we can select a list of variables for which you need to apply the aggregate function.

Question: What was the average fare and age for male vs female passengers from each town?

titanic.groupby(['embark_town','sex'])[['fare', 'age']].mean().reset_index()

3. Grouping

Before we go further and try other, more complex scenarios, let's try to understand the data structures we are working with, so that we can be much more creative with our approaches and get a deeper understanding of how they work.

You can imagine the pipeline of the above code to be as -

• Step 1: Create a grouper object with titanic.groupby(['embark_town']) which splits data into the relevant groups

• Step 2: Select the column 'fare' from each of those groups

• Step 3: Apply mean() on this column for each of the groups, combine and then return the aggregated dataset

Let's see what the grouper object looks like for a better understanding.

grouper = titanic.groupby(['embark_town'])

#Print dtype for each of the elements in the grouper
[(type(k),type(g)) for k,g in grouper]

[(str, pandas.core.frame.DataFrame),
 (str, pandas.core.frame.DataFrame),
 (str, pandas.core.frame.DataFrame)]

So, this shows that if we try to iterate over the grouper object, it’s nothing but a tuple with the key and a dataframe.

Let's see what each of those is.

#Print shape for the dataframe groups
[(k,g.shape) for k,g in grouper]

[('Cherbourg', (65, 15)), ('Queenstown', (2, 15)), ('Southampton', (115, 15))]

The key for each of the tuples/groups is the value from the grouper column (in this case the embark_town) and the value is just the complete dataframe filtered for that value! If we try to print one of the dataframe from this grouper, you can see that all the rows in this slice of data contain Queenstown as the embark_town, as shown below.

print(list(grouper)[1][0]) #print key
list(grouper)[1][1]        #print dataframe

Similarly, let's see what the grouper object looks like for multiple grouping features. The 'key' in this case is just a tuple with all the group combinations, which, after aggregation, gets set as the index of the final output.

grouper = titanic.groupby(['embark_town','sex'])

#Print dtype for each of the elements in the grouper
[(k,g.shape) for k,g in grouper]

[(('Cherbourg', 'female'), (34, 15)),
 (('Cherbourg', 'male'), (31, 15)),
 (('Queenstown', 'female'), (1, 15)),
 (('Queenstown', 'male'), (1, 15)),
 (('Southampton', 'female'), (53, 15)),
 (('Southampton', 'male'), (62, 15))]

#The grouping columns become the index after the groupby aggregation
titanic.groupby(['embark_town','sex'])['age'].mean()

embark_town  sex   
Cherbourg    female    35.352941
             male      39.774194
Queenstown   female    33.000000
             male      44.000000
Southampton  female    30.952830
             male      37.595484
Name: age, dtype: float64

4. Aggregation

There are multiple ways of aggregating your grouper object.

1. The first part of this section is to understand that you can perform aggregations on multiple columns, OR, perform multiple aggregations themselves on different columns, or a combination of both.

2. Second, you can use apply() or agg() to write your own custom aggregators but Pandas makes it much easier by providing a ton of in-built aggregators such as sum() or mean() as we discussed in the above examples.

Let's try to go through a few scenarios and explore how we can use these aggregations.

4.1 In-built aggregation methods

Pandas provides a ton of aggregation methods to quickly get the statistics you are looking for. Below are a few of the common ones that are used and more details on these can be found in the official pandas documentation.

4.2 Custom functions with pandas apply

This is by far the most popular way of applying a custom function to a dataframe, or in this case, applying it on each of the dataframe slices for groups defined by the grouper. The behavior of the apply() method with groupby is similar to the standard one.

You can apply it to each row (or column) of a dataframe input (if you have more than one column for aggregation) or to a series (if you have single column for aggregation). Within the function, you can actually either work directly with individual series or just write your own lambda function. Here are a few ways using the apply function.

Question: Get the unique set of ages for all each age category (who column) from each town.

titanic.groupby(['embark_town', 'who'])['age'].apply(set).reset_index()

Question: Get the range (min-max) of ages for each age category (who column) from each town.

titanic.groupby(['embark_town', 'who'])['age'].apply(lambda x: x.max()-x.min()).reset_index()

Question: Get the mean fare-by-age ratio for each age category (who column) from each town.

titanic.groupby(['embark_town', 'who']).apply(lambda x: (x['fare']/x['age']).mean())

embark_town  who  
Cherbourg    man       3.146083
             woman     3.614103
Queenstown   man       2.045455
             woman     2.727273
Southampton  child    28.956893
             man       1.410745
             woman     2.593897
dtype: float64

4.3 Multiple aggregations using agg method

Sooner or later, you would find it necessary to work with multiple aggregations over multiple columns at once. This is where the agg() method comes in. Here is a quick example of how you can use multiple in-built functions over multiple columns at once.

The general way to do this is to create a dictionary with the requirements and pass it to the agg() function. There are a few ways to structure the dictionary -

##Single function per column
{
 'column1': 'function1', 
 'column2': 'function2'
}

##Multiple functions per column
{
 'column1': ['function1', 'function2'], 
 'column2': ['function3', 'function4']
}

Question: Get the mean of fare, AND median of age for each age category (who column) from each town

#Define aggregations as a dictionary
g = {'fare':'mean', 
     'age':'median'
    }

titanic.groupby(['embark_town', 'who']).agg(g).reset_index()

Question: Get the sum & mean of fare, AND median, min, and max of age for each age category (who column) from each town

#Define aggregations as a dictionary
g = {'fare':['sum', 'mean'], 
     'age':['median', 'min', 'max']
    }

titanic.groupby(['embark_town', 'who']).agg(g).reset_index()

As of Pandas >= 0.25, another way to define the agg function is to define each column with ('column', 'function').

Let's demonstrate that with an example.

Question: Get the sum & mean of fare, AND min and max of age for each age category (who column) from each town, but rename columns

#Define aggregations directly as columns and tuples
titanic.groupby(['embark_town', 'who']).agg(A=('fare', 'sum'), 
                                            B=('fare', 'mean'),
                                            C=('age', 'min'),
                                            D=('age', 'max')).reset_index()

4.4 Custom functions with agg method

As you might think, just modifying the aggregate functions to include lambda functions is a way to create your own custom functions applied to specific columns. Here are a few examples.

#Define aggregations as a dictionary
g = {'fare':lambda x: x.sum(), 
     'age' :lambda x: x.max()
    }

titanic.groupby(['embark_town', 'who']).agg(g).reset_index()

5. Transform

Apart from just aggregating, you can use groupby to transform columns based on the grouper object. This requires using transform() function and returns the same number of rows as the original dataset, but the functions are applied based on the grouping defined. Let's consider the following point.

Question: Create a new column that returns the average fare for the age group (who column) the passenger belongs to.

titanic.groupby('who')['fare'].transform(lambda x: x.mean())

1      88.817429
3      88.817429
6      69.821026
10     77.379485
11     88.817429
         ...    
871    88.817429
872    69.821026
879    88.817429
887    88.817429
889    69.821026
Name: fare, Length: 182, dtype: float64

Notice that the output series is the length of the original titanic dataframe, but contains only 3 unique values [88.8, 69.8, 77.3], one for each of the ['woman', 'man', 'child']. This makes the grouping object highly versatile in the way you would use it for data preprocessing.

6. Advanced Usage

Let's introduce a few advanced cases where you end up using groupby for data preprocessing.

6.1 Sequential/local grouping of a dataframe

The grouper object doesn’t need to explicitly come from the dataframe. As long as the length of the grouper is the same as the number of rows in the dataframe, you can assign any grouper to groupby the rows by.

df = pd.DataFrame({'A':[1,2,3,4,8,10,12,13],
                   'B':[1,2,2,3,1,3,2,3]
                  })

#custom grouping
even_odd = ['even' if i%2==0 else 'odd' for i in df['A']]

df.groupby(even_odd)['B'].mean()

even    2.2
odd     2.0
Name: B, dtype: float64

Question: Get the sum of the value column of the given dataframe based on the sequentially occurring groups category i.e, in [1,1,2,2,1,1] the first group of 1's should be a separate group from the second set of 1's.

We can solve this creating a custom grouper, by shifting the column value by 1 and comparing them with original. If not equal, it will swap the boolean value. Then we can take a cumsum over the boolean to get groups where the value changes consecutively. Here is the solution for a similar problem I solved on Stack Overflow.

df = pd.DataFrame({'A':[1,1,2,2,2,1,1,3,3], #<- column to group on
                   'B':[1,7,2,4,1,8,2,1,3]  #<- column to aggregate
                  })

grouper = (df['A']!=df['A'].shift()).cumsum()

df.groupby(grouper).agg({'A':'mean','B':'sum'}).reset_index(drop=True)

6.2 Re-indexing to a fixed date range for each group

Question: A dataframe only contains rows for a few dates for each id. The goal is to re-index the dataframe for a fixed date range, but for each of the id individually. Also, fill in the missing data with 0 values.

Here we can create a custom reindex using pandas.date_range. Then, after setting the original date column as index, we can apply pandas.DataFrame.reindex along with groupby on the id column to reindex with the new date range for group, while filling empty values as 0.

d = {'id': [11, 11, 11, 11, 13, 13, 13],
     'date': ['2017-06-01','2017-06-03','2017-06-05','2017-06-06','2017-06-01','2017-06-02','2017-06-07'],
     'value': [1, 7, 8, 2, 9, 2, 11]
    }

df = pd.DataFrame(d)
df['date'] = pd.to_datetime(df['date'])
print("Input dataframe:")
df

#custom date range
idx = pd.date_range('2017-06-01','2017-06-07')

#set original date column as index
df.set_index('date', inplace=True)

#grouby and apply pd.DataFrame.reindex to apply new index and fill value as 0
df.groupby('id').apply(pd.DataFrame.reindex, idx, fill_value=0).drop('id',1).reset_index()

7. Other ways of grouping data

Here I discuss 3 ways that are popularly used to group data depending on the data structures and libraries you are already working with.

• Grouping using collections.defaultdict

• Using numpy.split() to group an array

• Chunking into groups using itertools.groupby()

Let's say we have a list of tuples with keys and values that we need to group.

data = list(zip(np.random.randint(0,4,(10,)), np.random.randint(0,100,(10,))))
print(data)

[(1, 41), (1, 30), (2, 70), (3, 82), (2, 68), (0, 18), (3, 97), (1, 37), (3, 8), (0, 51)]

7.1 Using collections' defaultdict

A useful way of grouping data is to use defaultdict. Defaultdict can store the grouping values as keys, and store the values as a list of values (or a custom function on them)

from collections import defaultdict

d = defaultdict(list)

for k,v in data:
    d[k].append(v)
    
grouped_data = dict(d)

print(grouped_data)

{1: [41, 30, 37], 2: [70, 68], 3: [82, 97, 8], 0: [18, 51]}

7.2 Using numpy's split function

Another way of splitting an array into a list of sub-arrays based on a grouping key is by using np.split along with the indexes for each group returned by np.unique. Only important thing is, the arrays need to be sorted explicitly.

import numpy as np

#sorted numpy array (sorted by the grouping column)
a = np.array(data)
a = a[np.argsort(a[:, 0])]

#Take the index positions for the unique values using return_index 
#and start from the second one to split the data
groups = np.split(a[:,1], np.unique(a[:,0], return_index=True)[1][1:])
print(groups)

[array([18, 51]), array([41, 30, 37]), array([70, 68]), array([82, 97,  8])]

7.3 Using itertools' groupby

Itertools provides a groupby api which is actually a sequential/local grouping method.

It's powerful for getting groups from "AAABBBAACCC" as "AAA BBB AA CCC". But in order to get groups as "AAAAA BBB CCC", it’s necessary to first sort the data by the grouping key.

import itertools

items = sorted(data, key=lambda x:x[0])
grouper = itertools.groupby(items, key=lambda x:x[0])
groups = [list(g) for k, g in grouper]
groups

[[(0, 18), (0, 51)],
 [(1, 41), (1, 30), (1, 37)],
 [(2, 70), (2, 68)],
 [(3, 82), (3, 97), (3, 8)]]

8. References

• http://www.scipy-lectures.org/packages/statistics/index.html#hypothesis-testing-comparing-two-groups

• https://www.simple-talk.com/sql/t-sql-programming/sql-group-by-basics/

• http://www.shanelynn.ie/summarising-aggregation-and-grouping-data-in-python-pandas/

• http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.groupby.html