Page Index Toggle Pages: 1
Topic Tools
Hot Topic (More than 10 Replies) WinMPQ 1.64 - problem adding 400MB file to MPQ (Read 7542 times)
Kato
Peon
*
Offline



Posts: 6
Joined: Apr 15th, 2008
WinMPQ 1.64 - problem adding 400MB file to MPQ
Apr 15th, 2008 at 10:44pm
Post Tools
Embarrassed  I'm trying to update the MPQ for my Diablo2 mod.  One of the files is over 400MB and the tool will not add it to the existing MPQ.  I've been able to add smaller versions of the same file which were 300MB in size.

Undecided  Is there any way to get this to work?  Thanks!
  
Back to top
 
IP Logged
 
ShadowFlare
Administrator/Lord of Nightmares
*****
Offline


The source of all chaos

Posts: 125
Location: The Sea of Chaos
Joined: May 11th, 2002
Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Reply #1 - Apr 16th, 2008 at 6:24am
Post Tools
What happens exactly?
  

-ShadowFlare

"My mind is my power. My power is my mind.
When uncorrupted by other elements, my mind is my pure power."

"I am a darkness beyond blackest pitch, deeper than the deepest night.
I am the Sea of Chaos. The source of all chaos.
That is how you describe me...
I am... the Lord of Nightmares."
Back to top
WWW  
IP Logged
 
Kato
Peon
*
Offline



Posts: 6
Joined: Apr 15th, 2008
Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Reply #2 - Apr 16th, 2008 at 1:03pm
Post Tools
Wink  Good to hear from you!

Roll Eyes  I startup WinMPQ and open the MPQ file.  Then I do an Add command to add the 400MB .bin file.  It looks like it's loading the file.  Lots of disk activity and hour-glassing.  Then at the end of it, it stops with no error messages or anything like that.  I look through the list of files and I do not see the file I tried to add.

Be aware that this MPQ already has 3350 files loaded in it and I did set the upper file limit to 4096 just to be sure it was enough.  Like I said before, it works with a slightly smaller 300MB file.   

Q:  Does WinMPQ handle large files?  Is there a limit to the file size that can be Added?
  
Back to top
 
IP Logged
 
ShadowFlare
Administrator/Lord of Nightmares
*****
Offline


The source of all chaos

Posts: 125
Location: The Sea of Chaos
Joined: May 11th, 2002
Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Reply #3 - Apr 17th, 2008 at 1:45am
Post Tools
Hmm, the upper limit should theoretically be more like 2 GB or 4 GB for files that are added with WinMPQ.

So the file does not show up after reloading the archive either?
  

-ShadowFlare

"My mind is my power. My power is my mind.
When uncorrupted by other elements, my mind is my pure power."

"I am a darkness beyond blackest pitch, deeper than the deepest night.
I am the Sea of Chaos. The source of all chaos.
That is how you describe me...
I am... the Lord of Nightmares."
Back to top
WWW  
IP Logged
 
Kato
Peon
*
Offline



Posts: 6
Joined: Apr 15th, 2008
Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Reply #4 - Apr 17th, 2008 at 2:39am
Post Tools
Embarrassed  No, the file does not show up at all.

I turned on the task manager and watched as it attempted to add the 400MB file.  WinMPQ took a chunk of memory about 420MB, tried to load it for about 3 seconds and then stopped - no file added.   

Then I did the Add command on the 300MB file (which I knew worked).  WinMPQ took about 320 MB of memory and about 980MB of page file usage and did the Add which took about half a minute.

Have you ever tried adding a 400MB file?
  
Back to top
 
IP Logged
 
ShadowFlare
Administrator/Lord of Nightmares
*****
Offline


The source of all chaos

Posts: 125
Location: The Sea of Chaos
Joined: May 11th, 2002
Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Reply #5 - Apr 17th, 2008 at 4:24am
Post Tools
Nope, I have not, since there aren't typically files that large to add for the applicable games.  From what you've mentioned, I'd guess there is a possibility of it running out of memory and being unable to complete it.  My mpq library was not originally designed with the intention of being used for adding files that large, so it may be inefficient with its memory usage in some cases.  How much ram does your system have?

On a side note, what are you doing that you are needing to add a file that is that large?  I don't remember any of the game data files ever being even close to that.
  

-ShadowFlare

"My mind is my power. My power is my mind.
When uncorrupted by other elements, my mind is my pure power."

"I am a darkness beyond blackest pitch, deeper than the deepest night.
I am the Sea of Chaos. The source of all chaos.
That is how you describe me...
I am... the Lord of Nightmares."
Back to top
WWW  
IP Logged
 
Kato
Peon
*
Offline



Posts: 6
Joined: Apr 15th, 2008
Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Reply #6 - Apr 17th, 2008 at 9:10pm
Post Tools
Cool  Okay . . . I've found a solution.  My PC only has 1 GB of RAM.

I installed WinMPQ on another PC that has 2 GB of RAM and tried to Add the 400 MB file.  It worked.  It took 830 MB of RAM while running and about 1.3 GB of page file space over the course of 8 minutes.  The WinMPQ program seems to require an amount of memory equal to twice the size of the file it is attempting to Add to the MPQ.  In the end, it did the job.  So, there is the solution . . . get more RAM.

Wink  As to the contents of the 400 MB file, it's a cubemain.bin file containing Horadric cube recipes for my Zy-El mod for Diablo 2 (don't know if you're familiar with that game).  I've always prided myself on pushing the game to its limits and one of these limits is generating inordinate numbers of cube recipes.  Well, the game can handle the large number of recipes but the WinMPQ tool could not without enough RAM.

Thank you for your time and input.
  
Back to top
 
IP Logged
 
ShadowFlare
Administrator/Lord of Nightmares
*****
Offline


The source of all chaos

Posts: 125
Location: The Sea of Chaos
Joined: May 11th, 2002
Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Reply #7 - Apr 18th, 2008 at 10:17am
Post Tools
I suppose that makes sense, since my mpq library currently reads the entire file into memory before compressing and compresses the entire file into memory before writing it to the mpq.

If I ever rewrite that part of the code (or even add onto it for that matter), I'll probably still make it do that for smaller files, but for larger files write it out incrementally to disk in a temporary file, then add that into the mpq archive.
  

-ShadowFlare

"My mind is my power. My power is my mind.
When uncorrupted by other elements, my mind is my pure power."

"I am a darkness beyond blackest pitch, deeper than the deepest night.
I am the Sea of Chaos. The source of all chaos.
That is how you describe me...
I am... the Lord of Nightmares."
Back to top
WWW  
IP Logged
 
Kato
Peon
*
Offline



Posts: 6
Joined: Apr 15th, 2008
Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Reply #8 - Apr 19th, 2008 at 9:36pm
Post Tools
Cool  Bought an extra 1GB of RAM for my home PC and got it all installed (now 2GB RAM in total).   

Sad  However, I still could not Add the 400MB .bin file to any MPQ.  It was the same results as before when I only had 1GB of RAM in total.  Perhaps there is some buffer size limit as to how big a file it can contain before compressing it?  Is there an optional setting that I'm overlooking?  I don't know now.  It works on the PC at work but I don't want to have to use that one for doing my "hobby".

Huh  My PC does indicate that I have the extra RAM and I've tested it by turning on a whole bunch of programs that chewed up more than 1GB of RAM to be sure that the new memory is accessible.  My PC does run noticeable faster on bootup as well as switching between programs.  So, the extra RAM was not for nothing.

Guess I'm stuck having to port the .bin and MPQ to work in order to add it.  I'll have to make do.   

Thanks for your time.
  
Back to top
 
IP Logged
 
ShadowFlare
Administrator/Lord of Nightmares
*****
Offline


The source of all chaos

Posts: 125
Location: The Sea of Chaos
Joined: May 11th, 2002
Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Reply #9 - Apr 20th, 2008 at 1:56am
Post Tools
Have you tweaked your page file settings at all in Windows?  That could potentially cause some difference in how much memory is perceived to be available to a program, and the PC at work is probably less likely to have had any of those settings tweaked.
  

-ShadowFlare

"My mind is my power. My power is my mind.
When uncorrupted by other elements, my mind is my pure power."

"I am a darkness beyond blackest pitch, deeper than the deepest night.
I am the Sea of Chaos. The source of all chaos.
That is how you describe me...
I am... the Lord of Nightmares."
Back to top
WWW  
IP Logged
 
Kato
Peon
*
Offline



Posts: 6
Joined: Apr 15th, 2008
Re: WinMPQ 1.64 - problem adding 400MB file to MPQ
Reply #10 - Apr 20th, 2008 at 3:50am
Post Tools
Huh  I'm running Windows XP SP2 on a P-4 3.0 Ghz PC now with 2GB RAM.  I'm not an expert at optimization - are you talking about the virtual memory page?  Where is the page file setting?

Roll Eyes  Yeah, at work, we have NetSup do that hardware tweaking stuff - I'm an app programmer - hardware's not my strong suit.

Roll Eyes  I also tried swapping the new 1GB card to a different slot.  My PC's a little older and is still using DDR memory.  I now have a 1GB with 2x512MB DIMM's.  Didn't make any difference.  I'm thinking of looking at my kids computer which is newer and uses DDR2 memory but I think it's only got 1GB.  Man!  Bought memory for the wrong computer . . . hahahaha, just my luck!  Grin

===========================================

Cheesy  (Throws hands up in the air!!!)  It worked!!!!

Shocked  Just as a lark, I thought I'd try Adding the file on my kid's computer with only 1 GB RAM and IT WORKED!!!   Roll Eyes  I still can't run it on my PC - the older one.  Modern technology, I guess or somewhere me messing up the settings on my dinosaur!
« Last Edit: Apr 20th, 2008 at 4:54am by Kato »  
Back to top
 
IP Logged
 
Page Index Toggle Pages: 1
Topic Tools