ShadowFlare's Realm
http://sfsrealm.hopto.org/cgi-bin/yabb2/YaBB.pl
ShadowFlare's Realm Forums >> Support >> WinMPQ 1.64 - problem adding 400MB file to MPQ
http://sfsrealm.hopto.org/cgi-bin/yabb2/YaBB.pl?num=1208299453

Message started by Kato on Apr 15th, 2008, 11:44pm

Title: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by Kato on Apr 15th, 2008, 11:44pm

:-[  I'm trying to update the MPQ for my Diablo2 mod.  One of the files is over 400MB and the tool will not add it to the existing MPQ.  I've been able to add smaller versions of the same file which were 300MB in size.

:-/  Is there any way to get this to work?  Thanks!

Title: Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by ShadowFlare on Apr 16th, 2008, 7:24am

What happens exactly?

Title: Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by Kato on Apr 16th, 2008, 2:03pm

;)  Good to hear from you!

::)  I startup WinMPQ and open the MPQ file.  Then I do an Add command to add the 400MB .bin file.  It looks like it's loading the file.  Lots of disk activity and hour-glassing.  Then at the end of it, it stops with no error messages or anything like that.  I look through the list of files and I do not see the file I tried to add.

Be aware that this MPQ already has 3350 files loaded in it and I did set the upper file limit to 4096 just to be sure it was enough.  Like I said before, it works with a slightly smaller 300MB file.  

Q:  Does WinMPQ handle large files?  Is there a limit to the file size that can be Added?

Title: Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by ShadowFlare on Apr 17th, 2008, 2:45am

Hmm, the upper limit should theoretically be more like 2 GB or 4 GB for files that are added with WinMPQ.

So the file does not show up after reloading the archive either?

Title: Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by Kato on Apr 17th, 2008, 3:39am

:-[  No, the file does not show up at all.

I turned on the task manager and watched as it attempted to add the 400MB file.  WinMPQ took a chunk of memory about 420MB, tried to load it for about 3 seconds and then stopped - no file added.  

Then I did the Add command on the 300MB file (which I knew worked).  WinMPQ took about 320 MB of memory and about 980MB of page file usage and did the Add which took about half a minute.

Have you ever tried adding a 400MB file?

Title: Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by ShadowFlare on Apr 17th, 2008, 5:24am

Nope, I have not, since there aren't typically files that large to add for the applicable games.  From what you've mentioned, I'd guess there is a possibility of it running out of memory and being unable to complete it.  My mpq library was not originally designed with the intention of being used for adding files that large, so it may be inefficient with its memory usage in some cases.  How much ram does your system have?

On a side note, what are you doing that you are needing to add a file that is that large?  I don't remember any of the game data files ever being even close to that.

Title: Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by Kato on Apr 17th, 2008, 10:10pm

8-)  Okay . . . I've found a solution.  My PC only has 1 GB of RAM.

I installed WinMPQ on another PC that has 2 GB of RAM and tried to Add the 400 MB file.  It worked.  It took 830 MB of RAM while running and about 1.3 GB of page file space over the course of 8 minutes.  The WinMPQ program seems to require an amount of memory equal to twice the size of the file it is attempting to Add to the MPQ.  In the end, it did the job.  So, there is the solution . . . get more RAM.

;)  As to the contents of the 400 MB file, it's a cubemain.bin file containing Horadric cube recipes for my Zy-El mod for Diablo 2 (don't know if you're familiar with that game).  I've always prided myself on pushing the game to its limits and one of these limits is generating inordinate numbers of cube recipes.  Well, the game can handle the large number of recipes but the WinMPQ tool could not without enough RAM.

Thank you for your time and input.

Title: Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by ShadowFlare on Apr 18th, 2008, 11:17am

I suppose that makes sense, since my mpq library currently reads the entire file into memory before compressing and compresses the entire file into memory before writing it to the mpq.

If I ever rewrite that part of the code (or even add onto it for that matter), I'll probably still make it do that for smaller files, but for larger files write it out incrementally to disk in a temporary file, then add that into the mpq archive.

Title: Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by Kato on Apr 19th, 2008, 10:36pm

8-)  Bought an extra 1GB of RAM for my home PC and got it all installed (now 2GB RAM in total).  

:(  However, I still could not Add the 400MB .bin file to any MPQ.  It was the same results as before when I only had 1GB of RAM in total.  Perhaps there is some buffer size limit as to how big a file it can contain before compressing it?  Is there an optional setting that I'm overlooking?  I don't know now.  It works on the PC at work but I don't want to have to use that one for doing my "hobby".

:-?  My PC does indicate that I have the extra RAM and I've tested it by turning on a whole bunch of programs that chewed up more than 1GB of RAM to be sure that the new memory is accessible.  My PC does run noticeable faster on bootup as well as switching between programs.  So, the extra RAM was not for nothing.

Guess I'm stuck having to port the .bin and MPQ to work in order to add it.  I'll have to make do.  

Thanks for your time.  

Title: Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by ShadowFlare on Apr 20th, 2008, 2:56am

Have you tweaked your page file settings at all in Windows?  That could potentially cause some difference in how much memory is perceived to be available to a program, and the PC at work is probably less likely to have had any of those settings tweaked.

Title: Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Post by Kato on Apr 20th, 2008, 4:50am

:-?  I'm running Windows XP SP2 on a P-4 3.0 Ghz PC now with 2GB RAM.  I'm not an expert at optimization - are you talking about the virtual memory page?  Where is the page file setting?

::)  Yeah, at work, we have NetSup do that hardware tweaking stuff - I'm an app programmer - hardware's not my strong suit.

::)  I also tried swapping the new 1GB card to a different slot.  My PC's a little older and is still using DDR memory.  I now have a 1GB with 2x512MB DIMM's.  Didn't make any difference.  I'm thinking of looking at my kids computer which is newer and uses DDR2 memory but I think it's only got 1GB.  Man!  Bought memory for the wrong computer . . . hahahaha, just my luck!  ;D

===========================================

:D  (Throws hands up in the air!!!)  It worked!!!!

:o  Just as a lark, I thought I'd try Adding the file on my kid's computer with only 1 GB RAM and IT WORKED!!!   ::)  I still can't run it on my PC - the older one.  Modern technology, I guess or somewhere me messing up the settings on my dinosaur!  

ShadowFlare's Realm » Powered by YaBB 2.2.1!
YaBB © 2000-2008. All Rights Reserved.