In the era of big data, there are many daunting challenges as well as exciting opportunities for IT research. While some are classic challenges taken to unprecedented scales, others are completely new. In this era, it is crucial to have a common dataset for validating research ideas, prototypes, as well as to publish and share results without sharing your proprietary information. The VE Datapak is a data product created just for that purpose.
The VE Datapak contains data that can uncover insights into how large numbers of individual players collaborate (i.e. 'chat') or compete (i.e. 'kill') with each other in thirteen different virtual universes in a massively multiplayer online game ("MMOG"). While some users play solely in one universe, others play across multiple universes. The time span for game play is four weeks of wall clock time, on a granularity of minutes. With each update, the timespan grows continually by four more weeks.
Please Note: The latest copy of this document can always be found at paralleldomains.com/readme.html, and the VE Datapak may be found at data.paralleldomains.com.
The VE Datapak has been created by recording activity on thirteen servers hosting a popular networked game called Minecraft. The dataset is heavily anonymized.
Servers are identified by random strings, such as [509fb8211dbf9d6b7e000004]. Players are identified by random strings, such as [baba27973f7f497e51d553f0b233ba7df42ac8ae]. For researchers studying game addiction or behaviors, this dataset will provide valuable information on game play. For confidentiality, we provide the time zone from which the players logged on but obfuscate their precise location.
All data records have been consolidated into a single csv file. Each record documents a different temporal interval, either a total count of an event in that time period, or an average of a system performance metric. Each comma separated line has 8 fields. For each record, fields irrelevant to the event are left blank.
The first three fields are always populated: start_t, stop_t and server_id. The time data is reported in minutes in Coordinated Universal Time (UTC). All time intervals in VE Datapak lasts a minute or more. The start day has been anonymized. A time value of 0 means 00 hour, 00 minute on the first day of data collection.
There are only two activity types involving two players: Chat and KilledBy, both of which define one-directional relationship. Those records look like the following example, where the two records come from the same one-minute interval: 24780-24781 on the same server [509fb8211dbf9d6b7e000004]. The fields of player_a and player_b are populated. The field "key" describes the action: KilledBy for the first example and Chat for the other.
********* Two-Player Events ********** 24780, 24781, 509fb8211dbf9d6b7e000004, 1bbe90a19292f745edd527085838d6b0b77d67a5, baba27973f7f497e51d553f0b233ba7df42ac8ae, KilledBy, , 1 24780, 24781, 509fb8211dbf9d6b7e000004, 56dfed03423fb02bdabd40367bf5310f5f00cc34, baba27973f7f497e51d553f0b233ba7df42ac8ae, Chat, , 1
In the first record, player_a was KilledBy player_b, i.e. [1bbe90a1...] was killed by [baba2797...]. The meta field follows, but for KilledBy events there is no meta data. The last count field shows that this KilledBy event took place once. Using this convention, it's easy to parse out that the 2nd record record says: on the same server during the same one-minute interval, player_a [56dfed03...] Chatted with player_b [baba2797...] once. Please note - Chat is a one-way event, where player_a spoke to player_b. If player_b spoke back, it will be counted as another one-way event.
Game playing events about a single player look like the following:
********* Single-Player Events ********** 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , BlockBreak, DIRT, 5 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , BlockPlace, SAND, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, FURNACE, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, IRON_HELMET, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, IRON_SWORD, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, STONE_PICKAXE, 1 24780, 24782, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , Death, , 2
In this case, the player_b field is empty because the event involves only one player - player_a. In the first record the key field says "BlockBreak", and the meta field tells which type of block is broken - in this case "DIRT". The final count field indicates 5 dirt blocks have been broken. The next record is a "BlockPlace" event, in which the player placed one block of "SAND" type. The following records are "CraftedItems" events, showing that the player [baba2797] crafted a "FURNACE" (a key in-game crafting tool), an "IRON_HELMET" (armor), an "IRON_SWORD" (weapon) and a "STONE_PICKAXE" (mining tool). The last record says that the player died twice (please note: these death events are single-player deaths, which exclude deaths caused by being killed by another player.). For "Death" events no types are recorded, although many possible causes exist, such as creeper explosion, etc.
Another type of event are "LOGIN" events. A record looks like:
********* Player Login Events ********** 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , Logins, America/Los_Angeles, 1
Login events are not reliable for telling when a player starts to play, because a player can stay logged in while being away from the computer. We use login events to tell from which time zone the player is from. In this case, the player is located in the US west coast time zone (America/Los_Angeles). Please note: the player's geoIP resolved location is hidden and will never be provided for confidentiality.
Finally, there are events relating to server operating state. These fall into two types: TickRate and Used Memory in Megabytes.
********* System Operations Events ********** 25320, 25380, 509fb8211dbf9d6b7e000004, , , SysOps, TickRate, 15.96 25320, 25380, 509fb8211dbf9d6b7e000004, , , SysOps, UsedMemMB, 5836.89
Since these events are server specific, player_a and player_b fields are always empty. The key field is always "SysOps", and the meta field tells which system operating state is collected. These system operation states do not vary too much, we average those to hourly granularity. TickRate is a Minecraft specific way to report server lag. The optimal number is above or around 20. When a system is underload and system lag increases, the TickRate will drop. A TickRate of 16 or above is considered to be very playable. When under severe load, the TickRate can be as low as 3 to 5. System memory usage is very straight forward, it in some ways correlate with server load, the more game playing activity there is, the more memory is typically in use.
24780, 24781, 509fb8211dbf9d6b7e000004, 1bbe90a19292f745edd527085838d6b0b77d67a5, baba27973f7f497e51d553f0b233ba7df42ac8ae, KilledBy, , 1 24780, 24781, 509fb8211dbf9d6b7e000004, 56dfed03423fb02bdabd40367bf5310f5f00cc34, baba27973f7f497e51d553f0b233ba7df42ac8ae, Chat, , 1 24780, 24781, 509fb8211dbf9d6b7e000004, 66a71bd3be9dcd739fe188b2d927b519be6c52f9, baba27973f7f497e51d553f0b233ba7df42ac8ae, Chat, , 1 24780, 24782, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , Death, , 2 24780, 24782, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, b520e0573c0ae9c3995c7cf46015e1040533439c, KilledBy, , 2 24840, 24841, 509fb8211dbf9d6b7e000004, cd3fcbe9ef2f6685426136166c58e1dd913c3f39, baba27973f7f497e51d553f0b233ba7df42ac8ae, Chat, , 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , BlockBreak, DIRT, 5 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , BlockBreak, GRAVEL, 5 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , BlockBreak, OTHER, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , BlockBreak, SAND, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , BlockPlace, SAND, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, FURNACE, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, IRON_HELMET, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, IRON_SWORD, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, STONE_PICKAXE, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, TORCH, 2 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, WOOD, 3 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , CraftedItems, WOOD_BUTTON, 1 25320, 25321, 509fb8211dbf9d6b7e000004, baba27973f7f497e51d553f0b233ba7df42ac8ae, , Logins, America/Los_Angeles, 1
Examples are easier to tell with a hero in the picture. In the above case, it's [baba2797...]. During the minute of 24780-24781, two players talked to our hero (they are: [56dfed03...] and [66a71bd3...]). During that same minute, our hero killed [1bbe90a1...] once.  Nine hours later (540 minutes later), our hero came back to the same server to play. In a minute (25320-25321), our hero mined for blocks (dirt, gravel, sand and other types) and crafted a furnace, and very efficiently built weapon, armor etc. It seems that our hero had an active Player v. Player (PvP) game play first, and an active survival mode of game play next.
From the login events, we can tell that our hero is on US west coast (the time zone designation is America/Los_Angeles). Because 24780 is per minute in UTC time, we can convert that to the player's local time as: (24780 / 60) % 24 = 5AM UTC, which is about 9PM US west coast time. Similarly, 25320 converts to 2PM UTC and 6AM in US west coast time.
By following some relationships had by our hero [baba2797...] we can find out:
Similar relationships can be built by following how players chat with each other, or according to which players typically play during the same time on the same server(s).
The VE Datapak already provides data on 70,000 unique players from 201 IANA Time Zones. In the IANA standard database on time zones, the whole world consists of 408 time zones.
The servers included in the VE Datapak are very different from each other. Solely from audience perspective, some are geographic region specific, some are language specific (e.g. English speaking vs. Spanish speaking countries), while others are specific to a narrow set of game-playing styles. It should be extremely interesting for both researchers and game developers to pose hypotheses about why some players chose to play on certain servers. More interestingly, we note about 2 to 3 thousand players (among the 70,000 cohort) played on more than one servers included in the VE Datapak, more than 1,000 players changed time zone at least once during the 4-week recording period.
For researchers that prefer to work with a smaller dataset, we have prepared the VE Datapak in two versions: a full version and a small version. Compared to the "full" version, the "small" version contains half of the servers and half of the recording period. The "small" version is less expensive and does not come with the option to receive monthly update.
The VE Datapak will continue to grow as we expand our capacity. The "small" version will always remain an up-to-date subset of the growing "full" version. For users that have purchased the monthly update option, updates to the full version will be downloadable from a link automatically emailed to you through our monthly news letter.
Please note: you are more than welcome to first purchase the smallest version to play with. If you decide to purchase a larger version, you only pay the difference in the price.
The VE Datapak already provides data on 70,000 players working alone, collaborating, and competing. Using the VE Datapak, almost all graph related solutions can take a firm footing in a practical scenario. On the theoretical side, you can test the scalability of your code that finds cliques and strongly connected components. You can also work on new algorithms that find cliques of players staying together for more than a period of 10 days, for instance. You can look for friend circles with obvious leaders or friend circles with more or less equal peers. You can work on methods to efficiently and quickly identify players that as a clan compete with players from other clans, while the clans form and dissolve over time autonomously.
For game developers or psychologists interested in studying in-game behaviors, Minecraft is an interesting subject because there are so many different game-playing styles on typical Minecraft servers. Minecraft also seems to have better gender parity than many other games. It would be very interesting to study how playing styles together with player interactions affect long-term game playing. In addition, how would one characterize player experiences? Whether a player will return to play more on the same server?
In support of these big data research efforts, we created the VE Datapak with a license that provides unrestricted rights to publish your research results. We would also encourage that the VE Datapak be used in educational settings, where students of all majors can learn the trade of computational thinking.
Referencing the challenges identified in "The Top 10 Challenges in Extreme-Scale Visual Analytics" by Wong et. al, the VE Datapak is helpful for the following research topics:
In the framework of Ten Challenging Problems in Data Mining Research by Yang and Wu, the VE Datapak is applicable to the following areas:
Specifically referencing the "Massive Multiplayer Online Gaming: A Research Framework for Military Training and Education" by Bonk and Dennen, the VE Datapak is useful for the following primary experiments identified in the report:
We have developed and will continue to develop a suite of sample Python scripts for working with the dataset. These sample code has been made open source through BSD license on github. Those scripts provide many fine grained data processing needs that range from simply counting number of unique players in the dataset, to reporting number of unique players that each server sees from different IANA timezones. If you would like to participate in our effort to improve the sample codes, please join us!
The player names, location and server names will never be provided with the VE Datapak. If you really just want to refer to things with names, we suggest that you try some scripts to auto generate names.
One useful name generation tool is Chris Pound's Language Generation Machine, which generates names centered around a given theme. It's very fascinating!
In the world of large scale networked games, cheating or pirating using a hacked game clients is an everyday thing. We cannot avoid it, nor should we. In a way, isn't a lot of today's cyber security R&D dedicated to detecting such behavior.
Jokes aside, there is a key event that has to do with how Minecraft handles abnormal behavior from a game client. Those scenarios include: when a client logs on but cannot provide the certificates to authenticate, or a client is causing severe server lag. The first case is more likely someone using a pirated game client, while the second case is often related to a player using a hacked client to cheat in the game by "flying" and causing server lag for all players on the server.
Regardless of which reason, when abnormal behavior happens, or simply when the server is not allowing new players to join the game, the Minecraft server will "kick" the player off the server. These kick events are captured, and reported in VE Datapak. The following is a sample record of that. In this example, on the same server our hero [baba2797...] plays, player [bac6018a...] got kicked 7 times during a seven-minute period. It's normal for a player to get kicked once in a while, but this level of kick activity definite incidates potential issues. We don't know what's going on in this case, but if you can figure out please let us know.
The player_b and meta fields are always left blank.
********* Sample Kick Events ********** 39000, 39007, 509fb8211dbf9d6b7e000004, bac6018ac2951c436437dc3fd4bbc99f72f7379b, , Kicks, , 7


Interested in learning more about a collaborative structured dictionary? Follow us on twitter or sign up for e-mail updates.