Making sense of XKCD

- 5 mins


Who doesn’t love XKCD? A while back, around the start of my grad school, I came across a relevant machine learning comic from Randall (see below). Today, we’re going to throw some NLP and topic modeling into a pile and hope something fun comes out the other end.

Test caption.

First things first, lets download some of these comics. Altogether, Randall Munroe (author of XKCD) has been extremely productive and has created over 2100 comics since the inception of XKCD on January 1st 2006! Looking at this, he is scarily consistent, averaging about 13 comics a month. The first month is an outlier, I imagine he had a butt load of comics he had drawn and released them all onto the internet at once. But I truly admire his consistency!

Next, lets gather some more data. Thankfully, the folks at Explained XKCD have annotated each of these comics with a transcript as well as their own explanations for each. This, along with the actual images, meant I had a rich dataset to work with of roughly unstructured text and their corresponding comics.

We have data, now what?

The title of this section accurately describes how I felt at this stage in the project. With work, I usually have a goal or end project, but I wanted to use this dataset to learn some new skills. So, I thought to try a bit of topic modelling. I really like this technique because it essentially allows the data to “self-organize”! Basically, based on the co-occurance of specific words, we can identify several topics that are commonly used in XKCD comics. After surprisingly little actual coding, I was able to build a Latent Ditrecht Allocation (LDA) model that identified 10 topics of interest.

Creating the model itself was pretty simple:

  1. Lemmatizing all words identified in both transcript and explanations of the comic. Gensim / NLTK libraries helped out a lot here!
  2. Create a dictionary, and remove all words that appear relatively infrequently.
  3. Use gensim to build an LDA model across 10 topics, passing over the data 20 times.

And with that, we get the following topics revealed.

Topic Number Words What I think it means?
0 megan, black, ponytail, friend, game, panel, year, beret, number, phone People and timelines
1 stapl, momentum, len, megan, cat, repair, beret, ponytail, pointer, black ??
2 cabl, jack, geek, ingredi, pump, verb, guest, sister, factori, weekend Cables?
3 barrel, bowl, polar, weekend, screenshot, float, dozen, flowchart, flash, sleep Barrels
4 gold, megan, michael, pool, sheet, love, backpack, feynman, peni, bird Gold?
5 band, fluid, song, wikipedia, megan, bush, word, game, monster, music ??
6 elev, twitter, megan, reddit, cabl, friend, kernel, river, user, lake ??
7 skateboard, senat, steve, firefli, escal, summer, paul, democrat, republican, adam Politics
8 chess, pokemon, knight, hors, basketbal, sister, player, captur, piec, board Chess and pokemon
9 virus, backup, keyboard, chip, belt, dougla, drone, black, beret, ghost ??

Looking through some of these, there were some that were surprisingly accurate. For example, barrels is a recurring theme in his works, and was deserving of a whole topic on its own. Other topics such as 2, 5, 6 and 9 were very rarely observed.

Lets search by topic!

From here, I started playing around to see how we could sort and search for these topics among all of these comics. In particular, I wanted to share this with all of you. So, using the LDA model for each of these topics, I wanted to create a web app with SLIDERS! Realistically, you can use the sliders to identify the mix of these topics you want, and with my website, I’ll identify the top five comics that match your search. Kinda like the DJ booth of XKCD comics!

I built this pretty rapidly using Flask to act as the backend + server side rendering engine. Heroku allows you to deploy flask applications with ease, and for free so you can find it all on there.

Overall, I was pretty impressed with this dataset. In the end, I wanted to put something out before diving too much into the data science-y aspects of it, as I could have diven much deeper! For anyone curious, here are a few more ideas I had / might pursure with this dataset:

Nishanth Merwin

Nishanth Merwin

A researcher tackling biology one keystroke at a time

comments powered by Disqus
rss facebook twitter github youtube mail spotify lastfm instagram linkedin google google-plus pinterest medium vimeo stackoverflow reddit quora quora