Yannick Weiss

Crawling Dota 2 API with Go


The small script hosted on gist gatherdata.go walks through all the played Dota 2 matches and counts the game modes. It can be run with: go run gatherdata.go -key YOUR_API_KEY

Why so slow?

The Dota 2 API allows you to get the complete match data of all games played. The problem lies people play a lot of matches. Probably more than 2 million per day.


I used Apple Numbers to create the charts and exported it as image by taking a screen shot 😃.


Signal Crash

An earlier version had the ability to CRL-C but the code to handle the interrupt signal did cause a crash. I removed it and print out the collected data periodically to the console. Investigation is still ongoing.

UPDATE: Go 1.5.1 fixed this crash.

Request Results in EOF Error

Because the net/http creates its own pool of connections that stay open it triggers the rate limiting of the server and he closes all connections. To fix this disable Keep-Alive or reuse the http client.

Rate Limiting

The server responds with a 500 code when he wants you to slow down.