Inside MoPaQ - Chapter 1: Introduction to MPQs

C H A P T E RO N E

INTRODUCTION TO MPQS

MPQ, or MoPaQ, is a proprietary archive format created by Mike O'Brien, the man hailed as Blizzard's multiplayer engine genius, back in 1996 as a general purpose archive for use with Diablo, and named narcissistically for its creator - "Mike O'brien PaCK". The copyrights to it, however, are held by Havas Interactive, Blizzard's parent company, and it may continue to be used now that Mike O'Brien has left Blizzard. MPQs apparently excelled in their role in Diablo, because Havas has turned back to them time and time again for Starcraft, Warcraft 2: BNE, Diablo 2, Lords of Magic (by Sierra, another company owned by Havas), and possibly others that I'm not aware of.
An archive is a file that contains other files inside it, usually in a compressed state. Havas uses MPQs to hold all sorts of things ranging from installing files to game data. It is the game data that is particularly useful. Those MPQs contain everything including graphics, sounds, animations, levels, strings, hard data, and storyline information. Obviously, the potential for customization is astounding. But, in order to make use of MPQs, you need to understand them.

Before There Was MPQ...

Long before the MPQ format was around, there was the WAR (Warcraft ARchive) format, which was the archive that Warcraft 2, and possibly even 1, used to store its data. This fledgling format was simple, unrefined, and had all the look of a rookie file format (believe me, I know). Files in the archive were referenced by ordinal (number), and whether or not to use compression was the only option.
However simple it might have been, it got the job done. It provided a quick and dirty method to store lots of files in a compressed state. But, soon deficiencies began to pop up. Referencing the files by ordinal meant that a long list of file entries had to be kept and consulted when a programmer needed to use one of the files, and as the list got longer and longer, the work got more and more tedious. The simplicity of the file format meant that any halfway decent hacker could figure out the format in some 15 minutes, and do all sorts of strange things to it.
While these problems might not have seemed all that bad at the time, the nature of Diablo - persistent characters, wide-spread internet play via Battle.net - made these problems unacceptable.

Why MPQ?

As mentioned previously, the MPQ format was made to fix some notable design flaws in the WAR format. But, they also wanted to add some brand new features to MPQ. In general, the goals Blizzard had for the MPQ format can be summed up as follows:

Security. Blizzard most certainly did not want people hacking into their games the way anyone could with Warcraft 2. Perhaps they even knew ahead of time that the MPQ format would be used for Starcraft levels. Either way, security was the top priority, as is evident by the excruciating efforts they went to to maintain it.

Efficiency. MPQs would be required to perform a variety of tasks from simple preloading of data to real-time streaming. For preloading data, time was not of the essence. But it is another story for real-time streaming, where data must be decompressed as fast as it is played. Consequently, speed was mandatory.

Multilinguality. From the very beginning, Blizzard planned to release its games on the worldwide market, so they wanted the porting to different languages to be as easy as possible. In an innovative move, they decided to build multilingual capabilities right into the MPQ format itself.

Expandability. It is obvious that holding all the data for a game in a single, massive archive is just ludicrous. Not only would it be inefficient and slow, but it would also be heck to apply after-purchase patches if the patches modified the archive. Blizzard knew this; and, consequently, made the ability to simply, efficiently, and elegantly draw data from multiple archives one of the mandates for the MPQ format.

Storm

Rather than duplicate common code in program modules, most programmers prefer to put their common code into shared libraries. Shared libraries are code modules that contain functions which any program may use. Doing this avoids redundancy, and can save a lot of program size.
Following this practice, Blizzard uses a shared library called Storm (Storm.dll on PCs, Storm.bin on Macs). This library is used by all modern Blizzard games to store important functions like MPQ reading (discussed in chapter 3), Battle.net, and even some graphics routines. When Blizzard releases a new game, it adds functions to Storm without changing old functions. This means that an old game can use a new version of Storm just fine. But, like all shared libraries, Storm makes its functions available to any program that wants to use them, which is not very good for security. It is for this reason that Storm only contains MPQ reading functions. The MPQ writing functions are Blizzard's prized possessions, and they're not going to let just anyone use them.

Starcraft Campaign Editor

It is common knowledge that the Starcraft Campaign Editor (also known as StarEdit) makes Starcraft levels (SCM/SCX). But it is less common knowledge that Starcraft levels are actually MPQs. That means that StarEdit actually creates MPQs, and therefore has the MPQ creation functions.
But StarEdit is not a shared library like Storm, and it doesn't make its functions available to just anyone who wants them. In fact, StarEdit doesn't make its functions available, period. But that doesn't mean that it is impossible to use them. With a lot of unorthodox code and some really gnarly hacking, it is possible. But most programmers are not able to perform such feats. That is why the MPQ API Library was designed; to take care of this extreme hacking, and allow programs to use these MPQ creation functions easily. Both StarEdit and the MPQ API Library are discussed further in chapter 4.

Back to the Table of Contents


C H A P T E RO N E INTRODUCTION TO MPQS
	MPQ, or MoPaQ, is a proprietary archive format created by Mike O'Brien, the man hailed as Blizzard's multiplayer engine genius, back in 1996 as a general purpose archive for use with Diablo, and named narcissistically for its creator - "Mike O'brien PaCK". The copyrights to it, however, are held by Havas Interactive, Blizzard's parent company, and it may continue to be used now that Mike O'Brien has left Blizzard. MPQs apparently excelled in their role in Diablo, because Havas has turned back to them time and time again for Starcraft, Warcraft 2: BNE, Diablo 2, Lords of Magic (by Sierra, another company owned by Havas), and possibly others that I'm not aware of. An archive is a file that contains other files inside it, usually in a compressed state. Havas uses MPQs to hold all sorts of things ranging from installing files to game data. It is the game data that is particularly useful. Those MPQs contain everything including graphics, sounds, animations, levels, strings, hard data, and storyline information. Obviously, the potential for customization is astounding. But, in order to make use of MPQs, you need to understand them.
	Before There Was MPQ...
		Long before the MPQ format was around, there was the WAR (Warcraft ARchive) format, which was the archive that Warcraft 2, and possibly even 1, used to store its data. This fledgling format was simple, unrefined, and had all the look of a rookie file format (believe me, I know). Files in the archive were referenced by ordinal (number), and whether or not to use compression was the only option. However simple it might have been, it got the job done. It provided a quick and dirty method to store lots of files in a compressed state. But, soon deficiencies began to pop up. Referencing the files by ordinal meant that a long list of file entries had to be kept and consulted when a programmer needed to use one of the files, and as the list got longer and longer, the work got more and more tedious. The simplicity of the file format meant that any halfway decent hacker could figure out the format in some 15 minutes, and do all sorts of strange things to it. While these problems might not have seemed all that bad at the time, the nature of Diablo - persistent characters, wide-spread internet play via Battle.net - made these problems unacceptable.
	Why MPQ?
		As mentioned previously, the MPQ format was made to fix some notable design flaws in the WAR format. But, they also wanted to add some brand new features to MPQ. In general, the goals Blizzard had for the MPQ format can be summed up as follows: Security. Blizzard most certainly did not want people hacking into their games the way anyone could with Warcraft 2. Perhaps they even knew ahead of time that the MPQ format would be used for Starcraft levels. Either way, security was the top priority, as is evident by the excruciating efforts they went to to maintain it. Efficiency. MPQs would be required to perform a variety of tasks from simple preloading of data to real-time streaming. For preloading data, time was not of the essence. But it is another story for real-time streaming, where data must be decompressed as fast as it is played. Consequently, speed was mandatory. Multilinguality. From the very beginning, Blizzard planned to release its games on the worldwide market, so they wanted the porting to different languages to be as easy as possible. In an innovative move, they decided to build multilingual capabilities right into the MPQ format itself. Expandability. It is obvious that holding all the data for a game in a single, massive archive is just ludicrous. Not only would it be inefficient and slow, but it would also be heck to apply after-purchase patches if the patches modified the archive. Blizzard knew this; and, consequently, made the ability to simply, efficiently, and elegantly draw data from multiple archives one of the mandates for the MPQ format.
	Storm
		Rather than duplicate common code in program modules, most programmers prefer to put their common code into shared libraries. Shared libraries are code modules that contain functions which any program may use. Doing this avoids redundancy, and can save a lot of program size. Following this practice, Blizzard uses a shared library called Storm (Storm.dll on PCs, Storm.bin on Macs). This library is used by all modern Blizzard games to store important functions like MPQ reading (discussed in chapter 3), Battle.net, and even some graphics routines. When Blizzard releases a new game, it adds functions to Storm without changing old functions. This means that an old game can use a new version of Storm just fine. But, like all shared libraries, Storm makes its functions available to any program that wants to use them, which is not very good for security. It is for this reason that Storm only contains MPQ reading functions. The MPQ writing functions are Blizzard's prized possessions, and they're not going to let just anyone use them.
	Starcraft Campaign Editor
		It is common knowledge that the Starcraft Campaign Editor (also known as StarEdit) makes Starcraft levels (SCM/SCX). But it is less common knowledge that Starcraft levels are actually MPQs. That means that StarEdit actually creates MPQs, and therefore has the MPQ creation functions. But StarEdit is not a shared library like Storm, and it doesn't make its functions available to just anyone who wants them. In fact, StarEdit doesn't make its functions available, period. But that doesn't mean that it is impossible to use them. With a lot of unorthodox code and some really gnarly hacking, it is possible. But most programmers are not able to perform such feats. That is why the MPQ API Library was designed; to take care of this extreme hacking, and allow programs to use these MPQ creation functions easily. Both StarEdit and the MPQ API Library are discussed further in chapter 4.