C H A P T E RT H R E E

S
TORM

The Storm library, or Storm for short, is an operating system unto itself. It is a gargantuan library of functions, and may be one of the most complicated shared libraries ever to not be made by Microsoft. It contains enough functions to make calling the native API almost unnecessary. In fact, this was one of its two purposes for being. Storm contains all the reusable functions written by Blizzard - such as the MPQ functions. But it also wraps OS specific calls, such as those in the GDI, DirectX, QuickDraw, etc. The reason for this is simple - to ease porting from one OS to another. After all, why spend thousands of man-hours rerouting thousands of OS function calls in the Windows source to Mac equivalents; why not just have all the calls go to a single function, and change that one function when the time to port to another OS comes?
By the version of Storm that ships with Starcraft (the version that I will refer to primarily in this document) has accumulated about 275 exported functions (functions which it makes available to anyone who wants them). You see, when a new game that uses Storm is released, the old functions in Storm aren't updated. Rather, the new functions are added on to the old version. This assures backward compatibility with games written for an older version of Storm.
These 275 functions are grouped into about 20 function sets (commonly referred to as subsystems on Windows, and managers on the Mac. I will call them subsystems in this document). A partial list of these is shown below:

  • Memory Subsystem - Routines for common memory functions, including allocating new memory, freeing allocated memory, filling memory, and more. That's right, Storm does its own memory management, including built-in error-checking and other robust features. This subsystem is equivalent to the functions with the 'mem' prefix on PCs.
  • String Subsystem - Functions to work with strings, such as copying, merging, searching, etc. These functions are, for the most part, equivalent to the 'str' functions.
  • File Subsystem - Functions for accessing the file system. These functions have the capability to read from (but not write to) either straight data files on disk, or MPQ archives. The MPQ reading capability aside, the manner in which these functions work is highly OS specific.
  • Network Subsystem - Functions to access remote computer system, through IPX, modem, TCP/IP, and direct cable. Functions for communicating with the server and/or other players during gameplay. Uses highly OS specific calls.
  • Error Subsystem - Functions for trapping and dealing with errors. Most of these functions do not have any OS equivalents.
  • Registry Subsystem - Functions to save persistent data to the computer. Uses the Registry on Windows systems, and the Preferences system on Macs.
  • Bitmap Subsystem - Functions for loading and displaying bitmaps from files. Uses OS specific calls.

Now, it would be quite easy to fill a full-size book with documentation for every one of these functions. Unfortunately, only about 40 of these have been identified and documented so far (although I believe that every useful MPQ function has been identified), and I don't have near enough time to take my dissassembler and debugger to every one of the rest (such a task would take months even if worked on diligently). Besides, this document is specifically made for MPQs. Consequently, only MPQs will be discussed.

Using the Storm API

WARNING: THE REST OF THIS CHAPTER IS SPECIFIC TO THE WINDOWS PLATFORM!

Click here to download the Storm Interface Library for Windows

As I said before, the functions in Storm are available to anyone who wants to use them. However, Blizzard isn't just going to hand them over without a fight.
To summarize two days I spent recently: Storm uses a downright depraved method to lock out hackers like us from using it. I spent at least 10 hours working hard trying to disarm the stupid thing, a task of which I'm proud to have succeeded in. But, I'll spare you the rather lengthy and complicated details of what I did, and get right down to business.
The culmination of my efforts matching wits with Mike O'Brien is what I call the Storm Interface Library. It is composed of a header file and an import library, which I synthesized with great effort to disarm the Storm booby-traps (download the Library from the link above). If you recall from DLLs 101, an import library contains the import tables that are used when a program is compiled to connect the program to a DLL. What this means is that all you have to do is include Storm.lib (in the Storm Interface Library) with the modules to link in your program and #include the Storm.h header file and you're good to go. Kinda makes me mad that I had to go to all that work to make the Storm Interface Library, and you can use it so easily :-p
And now, without further ado, let us get on to the Storm functions themselves.

Opening an MPQ Archive - SFileOpenArchive
BOOL WINAPI SFileOpenArchive(LPCSTR lpFileName, DWORD dwMPQID, DWORD dwUnknown, HANDLE *lphMPQ);
 
Parameter What it is
 lpFileName [in] A pointer to NULL terminated string that holds the path of the MPQ to open. SFileOpenArchive will crash if this is NULL.
 dwMPQID [in] An ID value that is saved internally in Storm for the MPQ. What this parameter is used for is not clear at this time.
 dwUnknown Unknown. Should always be NULL.
 lphMPQ [out] Pointer to a HANDLE variable that, upon successful completion, will hold the HANDLE of the MPQ. SFileOpenArchive will fail if this is NULL.

Before you can read from an MPQ, you have to open it. For this, you must use SFileOpenArchive. It will open an archive and give you a HANDLE that you can later use for calls to SFileOpenFileEx and SFileCloseArchive.
The first parameter, lpFileName, is simply the name of the MPQ to be open, and must not be NULL. The second parameter, dwMPQID is the ID that Storm will assign to the MPQ internally. This does not change the MPQ, and it is unclear as to what it actually does do. The third parameter, dwUnknown, is just that, and, for all we know, unused; it may, therefore, be ignored. The final parameter, lphMPQ, is a pointer to a HANDLE that you must have declared previously. If SFileOpenArchive completes successfully, this HANDLE will be that of the MPQ.
If SFileOpenArchive succeeds, its return value will be nonzero. But, there are several situations which may cause SFileOpenArchive to fail. If it does fail, it will return with a value of FALSE. In this situation, you can call GetLastError to get information about why it failed. If lpFileName is a 0-length string or lphMPQ is NULL, GetLastError will return ERROR_INVALID_PARAMETER. If the file lpFileName does not exist, GetLastError will return ERROR_FILE_NOT_FOUND. In some very rare cases, GetLastError may return some other cryptic error value.

Closing an Archive - SFileCloseArchive
BOOL WINAPI SFileCloseArchive(HANDLE hMPQ);
 
Parameter What it is
 hMPQ [in] The HANDLE of the MPQ to close, which was acquired earlier with SFileOpenArchive. SFileCloseArchive will fail (or worse) if this is NULL or a HANDLE not obtained with SFileOpenArchive.

Once you have opened an MPQ archive, you must remember to close it when you're done! SFileCloseArchive is the natural compliment of SFileOpenArchive, closing an archive that was opened earlier.
As with SFileOpenArchive, SFileCloseArchive returns a nonzero value to indicate success, and FALSE if it should fail. However, in this case, GetLastError wouldn't provide any useful information. But, there should only be one reason for SFileCloseArchive to fail - if the hMPQ parameter is invalid - so you can only assume that that was the reason.

Opening a File Inside an MPQ - SFileOpenFileEx
BOOL WINAPI SFileOpenFileEx(HANDLE hMPQ, LPCSTR lpFileName, DWORD dwSearchScope, HANDLE *lphFile);
 
Parameter What it is
 hMPQ [in] The HANDLE of the MPQ previously opened with SFileOpenArchive that contains the file you want to open. SFileOpenFileEx will fail or crash if this is NULL or a HANDLE not obtained with SFileOpenArchive.
 lpFileName [in] A pointer to a NULL terminated string that holds name of the file within the MPQ to be opened. SFileOpenFileEx will crash if this is NULL.
 dwSearchScope [in] Specifies where to look for a file when opening a file on the hard drive. Must be NULL when working with MPQs.
 lphFile [out] A pointer to a HANDLE variable that, upon successful completion, will hold the HANDLE of the requested file. SFileOpenFileEx will fail if this is NULL.

Just because you have opened an MPQ with SFileOpenArchive doesn't mean you can start reading from it immediately. Remember, MPQs are nothing more than archives - they contain other files. Before you can read anything from an MPQ, you must open one (or more) of the files inside. SFileOpenFileEx is the function you will use for this task; it will open the requested file in an MPQ and return a HANDLE to it.
Once again, SFileOpenFileEx will return a nonzero value on success and FALSE on failure, and you can call GetLastError to get the reason why. If lpFileName is a 0-length string or lphFile is NULL, GetLastError will return ERROR_INVALID_PARAMETER. If the file does not exist in the MPQ, GetLastError will report ERROR_FILE_NOT_FOUND. In some rare cases, GetLastError may report ERROR_FILE_INVALID, and on extremely rare occasions, it may return some other obscure error value.
IMPORTANT NOTE: When you call SFileCloseArchive to close an MPQ, all the open files inside that MPQ are also closed, and the HANDLEs you received from SFileOpenFileEx become invalid. If you call SFileReadFile, SFileGetFileSize, SFileSetFilePointer, or SFileCloseFile with one of these invalid HANDLEs, the call will fail, and Storm may even crash.

Closing a File Inside an MPQ - SFileCloseFile
BOOL WINAPI SFileCloseFile(HANDLE hFile);
 
Parameter What it is
 hFile [in] The HANDLE of the file to close, which was acquired earlier with SFileOpenFileEx. SFileCloseFile will fail (or worse) if this is NULL or a HANDLE not obtained with SFileOpenFileEx.

Just like SFileCloseArchive is to SFileOpenArchive, SFileCloseFile is the natural compliment of SFileOpenFileEx, closing an already open file.
Also like SFileCloseArchive, SFileCloseFile will return a nonzero value on success, FALSE on failure, and GetLastError will provide exactly zero help. Fortunately, even more than SFileCloseArchive, SFileCloseFile should only fail if hFile is NULL or an invalid HANDLE.

Reading from a File in an MPQ - SFileReadFile
BOOL WINAPI SFileReadFile(HANDLE hFile, LPVOID lpBuffer, DWORD nNumberOfBytesToRead, LPDWORD lpNumberOfBytesRead, LPOVERLAPPED lpOverlapped);
 
Parameter What it is
 hFile [in] The HANDLE of the file to read from, which was acquired earlier with SFileOpenFileEx. SFileReadFile will crash if this is NULL or a HANDLE not obtained with SFileOpenFileEx.
 lpBuffer [out] A pointer to a buffer in memory where SFileReadFile will place the data read from the file. This buffer must be at least as large as nNumberOfBytesToRead. SFileReadFile will fail if this is NULL.
 nNumberOfBytesToRead [in] The number of bytes for SFileReadFile to read from the file. SFileReadFile may crash if this is larger than the size of lpBuffer.
 lpNumberOfBytesRead [out] A pointer to a DWORD that will hold the number of bytes actually read from the file. The number of bytes read will never be more than nNumberOfBytesToRead, but may be less if the number of unread bytes in the file is less than nNumberOfBytesToRead. It is not recommended to let this be NULL.
 lpOverlapped [in] A pointer to an OVERLAPPED structure. This is used for asynchronous reading of files on a disk, and must be NULL when reading files in MPQs.

Of course, the whole reason you went to the trouble of opening an MPQ and a file inside it is to read from the file. Well, once you have obtained a valid file HANDLE from SFileOpenFileEx, this is the function for the job. SFileReadFile will read a specified number of bytes, and then advance the file pointer. This means that if you have a file, and you call SFileReadFile to read half the file you would get the first half of the file, but if you called SFileReadFile again you would get the second half of the file. If you needed to get the first half again, you would need to call SFileSetFilePointer.
SFileReadFile will give a nonzero return value on success, or FALSE on failure. However, you must remember that just because it returned a nonzero value doesn't mean it actually read anything; it only means that no errors occurred. If there are less unread bytes (bytes from the file pointer to the end of the file) in the file hFile than the number requested, SFileReadFile will read less than the number of requested bytes; and If the file pointer for the file hFile is at the end of the file, SFileReadFile will read nothing, set lpNumberOfBytesRead to 0, and return TRUE. Consequently, it's very important to check lpNumberOfBytesRead.

Getting a File's Size - SFileGetFileSize
DWORD WINAPI SFileGetFileSize(HANDLE hFile, LPDWORD lpFileSizeHigh);
 
Parameter What it is
 hFile [in] The HANDLE of the file whose size is to be determined. SFileGetFileSize will crash if this is NULL or a HANDLE not obtained with SFileOpenFileEx.
 lpdwFileSizeHigh [out] A pointer to a DWORD that, upon successful completion, will hold the high 32-bits (a DWORD) of the file's size. However, it is not possible for a file in an MPQ to be this large (over 4 gigabytes), so this is virtually unused, and may safely be NULL.

Although it is entirely possible to read from a file of unknown length without causing a crash, this is generally considered extremely bad programming practice, and most programmers prefer to know the size before they start reading from a file. SFileGetFileSize is the trick here, as it can retrieve the size of a file already opened with SFileOpenFileEx.
There are some major trouble spots in SFileGetFileSize. First is the ambiguous way that it deals with errors. When SFileGetFileSize succeeds, it returns the file's size (which can be 0!). But, when an error occurs, it will return 0xffffffff, which is the same thing it would return for a 4,294,967,295 byte (4 GB). Fortunately, this isn't a big problem, as you'll probably never actually see a file that large. The second problem, however, is more dangerous, and is concerned with SFileGetFileSize's lack of error checking. SFileGetFileSize doesn't check whether hFile is valid or even NULL. That means that if you give it an invalid HANDLE for hFile, down goes the program, and down goes the computer. So, the bottom line is this: use this function with care.

Moving the File Pointer - SFileSetFilePointer
DWORD WINAPI SFileSetFilePointer(HANDLE hFile, long nDistanceToMove, long *lpDistanceToMoveHigh, DWORD dwMoveMethod);
 
Parameter What it is
 hFile [in] The HANDLE of the file whose file pointer is to be moved. SFileSetFilePointer will crash if this is NULL or a HANDLE not obtained with SFileOpenFileEx.
 nDistanceToMove [in] The low-order 32-bits of the number of bytes for SFileSetFilePointer to move the file pointer, with positive numbers moving the pointer forward and negative numbers moving the pointer backward. This value can also be 0.
 lpDistanceToMoveHigh [in] A pointer to the high-order 32-bits of the distance for SFileSetFilePointer to move the file pointer. But, because MPQs do not support files this large, this is unused and must be NULL or SFileSetFilePointer will fail.
 dwMoveMethod [in] Specifies the relative location the file pointer will be moved to. Must be one of these following values in Windows.h:
FILE_BEGIN The file pointer will be set to nDistanceToMove bytes from the beginning of the file. nDistanceToMove must be positive.
FILE_CURRENT The file pointer will be set to nDistanceToMove bytes from the current location of the file pointer.
FILE_END The file pointer will be set to nDistanceToMove bytes from the end of the file. nDistanceToMove must be negative.

Many times it is sufficient to read through an entire file sequentially - from the very beginning to the very end. Other times, however, it is necessary to move straight to some location in the file and read only a small portion of the file. Reading from an arbitrary location in the file required that you move move the file pointer, a task performed by SFileSetFilePointer. The file pointer dictates where the next read operation will read data from, or the next write operation will write data to in a file; each read or write operation will then move the file pointer to the end of the read/written region.
SFileSetFilePointer does not actually move the file pointer to the absolute position
nDistanceToMove in the file. Rather, SFileSetFilePointer moves the file pointer to a position relative to either the beginning or end of the file, of the current file pointer position. For example, let's say you have a file 1000 bytes in length. When the file is first opened, it's file pointer is set to 0, referring to the very first byte. But, you then read 100 bytes from the file. You then call SFileSetFilePointer with a nDistanceToMove of 500. If you set dwMoveMethod to FILE_BEGIN in the call, the file pointer would be set to 500. If you had dwMoveMethod as FILE_CURRENT, the file pointer would be 600, because the file pointer was moved to byte 100 when you read from the file. But, if you set dwMoveMethod to FILE_END, SFileSetFilePointer would fail, because it would try to set the file pointer to 1499 (the last byte if the file, 999, plus 500) which doesn't exist. This system is easy to use if you understand it.
On failure, SFileSetFilePointer returns 0xffffffff, which carries the same pitfalls as SFileGetFileSize. But, on successful completion, SFileSetFilePointer returns the new absolute position of the file pointer for the file hFile. This means that you can simply get the current position of the file pointer by calling SFileSetFilePointer with a nDistanceToMove of 0, and dwMoveMethod set to FILE_CURRENT. In fact, it is for this very reason that there isn't a SFileGetFilePosition function.
Just like SFileGetFileSize, SFileSetFilePointer will recklessly use any hFile HANDLE you give it, without performing any error-checking whatsoever. That means that, once again, it is your responsibility to make sure it gets a valid file HANDLE, or else it's good-night-computer.

Choosing a Language - SFileSetLocale
LCID WINAPI SFileSetLocale(LCID lcNewLocale);
 
Parameter What it is
 lcNewLocale [in] The language code (LCID) that SFileSetLocale will make the new default. The following codes are ones that I've found in Starcraft MPQs:
0 Language Neutral/English
0x407 German
0x409 English
0x40a Spanish
0x40c French
0x410 Italian
0x416 Portuguese

SFileSetLocale is the functional manifestation of the simple, elegant multilinguality system Blizzard created in MPQs. Thanks to it, despite the underlying complexity, a single function call assures that all files read from an MPQ are of the intended language. It's only parameter is the new language for the entire Storm MPQ subsystem. It will never fail, and it returns the language code you gave it, making its return value worthless.
The way MPQ's multilinguality works is this: Each file in an MPQ has a language code, and there can be multiple files with the same name as long as they have different language codes. When a call to SFileOpenFileEx is made, SFileOpenFileEx looks for a file with the same language code as the one stored by the last call to SFileSetLocale (or a language code of 0, if SFileSetLocale has never been called). If a file with matching language code cannot be found, SFileOpenFileEx will open the language neutral (having a language code of 0) version of the file instead.

Putting it All Together - DumbExtractor

Click here to download DumbExtractor and source code

By now you should have a pretty good idea of what all the Storm MPQ functions do and how to use them. But, of course, no good programming chapter would be complete without one more thing - a sample program!
DumbExtractor is a simple console application that I whipped up in about 20 minutes, which can be run simply by double-clicking on it. All it does is start up, ask you for an archive and file to extract, and extract the specified file, all with a healthy dose of status reports and comments (I would suggest you download it from the link above and look it over before you continue reading this chapter). But don't let its simple operation fool you; DumbExtractor is as robust as anything, and can handle just about any error you can throw at it.
DumbExtractor works in a straightforward way, as it's intended to be as simple and comprehensible as possible. Out of necessity, it uses the Storm Interface Library. The first  task it performs is to allocate a read buffer. This is done first because I wanted to get as many things that could go wrong as possible out of the way before we go bothering the user, even though there is the slim possibility that the file to extract might be 0 bytes large (in which case we haven't really lost anything in allocating the memory, since any computer that can't spare 4 MBs of RAM for a couple seconds is in desperate need of life-support anyway). After making sure we have memory to work with, DumbExtractor then prompts the user for an archive and a file to extract, all of which is performed with the simple cout and cin.
Once all the preparatory duties are out of the way, we can get into the real Storm stuff. Our first call is to SFileSetLocale, to set the language. I suppose this is sort of unnecessary, considering my language is English, and that is what Storm defaults to anyway. But, it could also be argued that it's good programming practice, so think what you will about it. The next thing we will do is to open the requested archive using SFileOpenArchive, followed by opening the file with SFileOpenFileEx.
Once we're sure that the file exists in the archive and opened it, we can then open the output file on the hard drive using the Win32 API function CreateFile. Then, we get the file's size with SFileGetFileSize. The reason I waited to check whether the file is not empty (has more than 0 bytes) until after creating the file on the hard drive despite the fact that doing so adds something else that could go unnecessarily wrong, is because even if the file is 0 bytes, it's supposed to be 0 bytes. That means we have to deliver it as intended - as a 0 byte file.
Finally, we get to the inner loop, where we marshal the data from the MPQ file to the output file using SFileReadFile and the Win32 API WriteFile. Here I use a BOOL variable to remember whether there were any reading or writing errors. The reason I didn't just end the program right away when an error occurs is that I didn't want to duplicate any code and just let things end naturally instead (I'll leave the question of whether I actually achieved this goal up to the philosophers). That means I need to wait for the output file to be closed before I can delete it. Finally, once the extraction is complete, we can shut everything down, closing the files and MPQ, and freeing the read buffer we allocated.

The Rest is Attitude

Okay, you got me; that is indeed a quote from book Writing Solid Code, by Microsoft Press (which is a good book, by the way). Anyway, in this chapter, I've shown you everything you need in order to make any manner of MPQ extractor; from DumbExtractor to MPQView, to something much more spectacular, the foundation - and flexibility - is all there in the simple Storm functions. A program's personality is not what it does, but what it is - the user interface. The Storm functions are all there. What you choose to make of them is just attitude.

Back to the Table of Contents

Web site and content copyright © 2000 Justin Olbrantz(Quantam) unless otherwise noted. All rights reserved.