mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

I'm going to try something out here and see how it works. I've put together a (Windows) patch for the latest changes and bug fixes, and I'm posting it here so people can try out the fixes before the official DROD:JtRH 2.0.15 release. If everything seems good over the next week or so, then we'll release this for real. Otherwise, you can report any problems and we can tweak things first so (1) we don't release new builds across several OSes and sites that still have old bugs in them (or new bugs like 2.0.14 had - yikes!), and (2) you don't have to wait another couple months before getting a patch that fixes those problems. When reporting bugs in this version, please state you're using the 2.0.15 candidate and which revision number.

This patch should be stable, just like previous patches, but backup your drod.exe and DROD dir (especially the four .dat files in the Data subdirectory) just in case. If you don't, you may lose your game data. If you don't understand what I mean about what files you should back up, then I'd recommend against you trying out this unstable patch.

Oh, and here's the list of changes (+ means change or improvement and * means bug fix):

+ Huge player export speedup.
+ Reduced memory allocation and memory fragmentation during import significantly.
+ Made saved game and demo verification during import much faster.
+ Updated several third-party library versions (libCURL 7.15.2, libpng 1.2.12, SDL_ttf 2.0.8, and SDL 1.2.11, which now uses the Windib driver instead of DirectX (5?) by default).
+ Now "Wait 0" script command halts script execution till next turn (Hint: do this at the beginning of a script if you don't want it to start doing anything until after the player makes his first move in a room).
+ Seep in the open on room start now immediately die.
+ Can now make an NPC look like a decoy. (rev7)
+ Now CaravelNet rankings can be displayed during level exit sequence
+ Can now delete SS holds from the hold list
+ Added INI options: Startup->ValidateSavesOnImport and LogErrors.
+ Now stairs ending the hold are highlighted in the room editor. (rev10)
+ Enlarged the Change Location dialog. (rev13)
+ Reduced delay when choosing a different save on the Restore Screen (rev7)
+ Reduced delay going to and from editor screens. (rev9)
+ CaravelNet: Now player may view room conquer demos if they are flagged as having conquered the room on the server. (rev14)
+ Added command line API for text localization (-exportTexts and -importTexts). (rev14)
+ F7 displays move counter display in-game. Alt-F7 toggles framerate and sound memory usage display. (rev14)
* Now a door can't be made to have multiple associations with one orb, allowing orb lightning to empty tiles.
* Room conquer demos weren't recorded when leaving an initially empty room after undoing or restarting.
* Script "Wait for Monster stabbed" event didn't catch rock golems breaking.
* Fixed small text.dat file bloat when playtesting, deleting holds and player profiles.
* Player sound preferences not always set correctly on startup.
* Fixed game display issues: incorrect persistence of user-clicked tile highlighting, burning fuse non-display, shadow of a wall under an obstacle can be cast onto the obstacle, missing error highlight.
* Removed erroneous saves to disk during demo playback or playtesting.
* CaravelNet interface more robust to errors.
* Game more robust to bad data files.
* Fixed demo display assertions.
* Fixed "end on room exit" script bug. (rev7)
* Fixed rev6 bug: rooms disappearing on restore screen. (rev7)
* Fixed rev7 bug: Bad data may be introduced into the .dats on import, causing hold list to appear empty or player progress to become unavailable. (rev8)
* Fixed crash when playing a room with tailless serpents. (rev8)
* Fixed speech memory leak. (rev9)
* F1-help in main editor screen now works. (rev9)
* Fixed sword position error highlighting not updating when reflecting a room in the room editor. (rev9)
* Fixed crash when starting a game on stairs that end hold. (rev11)
* Drod now always quits when confirming the "Really Quit?" dialogue. (rev11)
* Fixed two songs playing simultaneously. (rev13)
* Fixed bug: can remove level's main entrance room to another level. (rev12)
* Fixed conquer demo error when saved in clean rooms. (rev13)
* Evil eye: now can spot decoys when player is invisible and outside smell range. (rev14)
* Editor: couldn't start playtest on a level entrance w/ arrow or potion. Now can start playtest w/ sword on monster. Serpent editing got stuck after tooltip popped up. (rev14)
* Fixed editor placement bugs. (rev14)
* Fixed assertion firing when decoy NPC moves. (rev14)
* Settings Screen: Keyboard UI now works on the check boxes.
* Demos Screen: Wrong (previous) CaravelNet demos list sometimes showed up. (rev14)
* Demo Screen: Pressing F7 to show move count doesn't display move count immediately. (rev14)
* Some demos were considered corrupt (related to 2.0.15's new method of not immediately marking the current room explored on entrance). (rev14)
* Fixed bug: when making a copy of a hold, copied speech voice data wasn't being correctly owned by the new hold. (rev15)
* Fixed bug: Stopped a way to disconnect a serpent's body from its tail in the room editor. (rev15)

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 02-19-2007 02:04 AM]

kie
Level: Master Delver

Rank Points: 221
Registered: 02-26-2005
IP: Logged

I'll give it a try. Backing up as I write...


____________________________
cheers, kie

Citrus
Level: Smitemaster

Rank Points: 1228
Registered: 01-08-2006
IP: Logged

mrimer wrote:
+ Seep in the open on room start now immediately die.

www... I liked that bug. Okay, I remove all rooms that had that in my hold.

____________________________
Call me Citrus.

kie
Level: Master Delver

Rank Points: 221
Registered: 02-26-2005
IP: Logged

mrimer wrote:
+ Huge player export speedup.

WOW! Did an export before patch which took around 17 minutes, export after patch < 1 minute!
I haven't noticed any bugs yet...

____________________________
cheers, kie

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

mrimer wrote:
+ Huge player export speedup.

Yay - took less than a minute here at work. But...

When importing my player file from home, DROD.exe went *poof* when the process' Virtual Size (as displayed by Process Explorer) hit 2GB and got killed by Windows - and the progress bar wasn't even half through...

While the amount of memory used topped at 635MB, DROD had exhausted all it's address space when importing - not good...

If you want, I can put up the export and/or my DAT files on my server and mail you a link to it...

EDIT: Mailed you the link anyway - a single email can't hurt, can it?

That wouldn't be the XML parser creating and destroying lots of small objects thus totally fragmenting the heap?

If so, maybe importing only one hold at a time (which might need a change to the way things are exported - think concatenating several player files that only contain one hold each) would keep the memory usage down?

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

---

[Last edited by Briareos at 07-26-2006 08:16 AM]

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

Briareos wrote:
When importing my player file from home, DROD.exe went *poof* when the process' Virtual Size (as displayed by Process Explorer) hit 2GB and got killed by Windows - and the progress bar wasn't even half through...

While the amount of memory used topped at 635MB, DROD had exhausted all it's address space when importing - not good...

If you want, I can put up the export and/or my DAT files on my server and mail you a link to it...

EDIT: Mailed you the link anyway - a single email can't hurt, can it?

That wouldn't be the XML parser creating and destroying lots of small objects thus totally fragmenting the heap?

If so, maybe importing only one hold at a time (which might need a change to the way things are exported - think concatenating several player files that only contain one hold each) would keep the memory usage down?

Eep! I haven't seen this before -- the game should be able to handle importing a player file of that size. Thanks for the email. I'm d/ling your .dats and .player file now. I'll check them out and let you know how it goes for me.

Edit: Altered how imported data are handled to greatly reduce import memory footprint

Edit: Found out how to avoid allocating the entire size of the decompressed file during parsing (e.g. almost 90MB for you)

Edit: I've found a place where I can free up 1-3x the size of the .player file at the beginning of the import process (e.g. 10-20MB for you)

Edit: Plugged a small memory leak importing each saved game

Watching all those demos and saved games validate, I felt like I was playing Progress Quest. I've added a new INI option to 2.0.15(rev2), Startup->ValidateSavesOnImport, that may be turned off to skip validation of the move sequences of saved games and demos when a player is imported or hold upgraded. This will, of course, speed up import, but saved games and demos recorded in older versions of holds may appear broken.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 07-27-2006 07:09 AM]

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

mrimer wrote:
Eep! I haven't seen this before -- the game should be able to handle importing a player file of that size. Thanks for the email. I'm d/ling your .dats and .player file now. I'll check them out and let you know how it goes for me.

Thanks! Yeah, I'm just trying it with an older, smaller export I have from work - that one is "just" 3.4MB in size and takes the Virtual Size of the DROD process to a whopping 1.159MB...

Of course, you could also port DROD to 64-bit Linux and I could install the AMD 64-bit Debian port here, which together should fix the lack of address space problem. But maybe hunting down the cause of this might be the better solution...

np: SCSI-9 - Teplyi Dym (The Line Of Nine)

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

trick
Level: Legendary Smitemaster
Rank Points: 2580
Registered: 04-12-2003
IP: Logged

Briareos wrote:
Of course, you could also port DROD to 64-bit Linux and I could install the AMD 64-bit Debian port here, which together should fix the lack of address space problem. But maybe hunting down the cause of this might be the better solution...

On the other hand, DROD has run fine in 64-bit Linux for several months now ..

Before you ask, the port was done mainly because it makes it easier for me to debug it (I run 64-bit Linux myself). There probably won't be an official 64-bit Linux release of DROD for end users, at least not yet. The 32-bit version works fine in 64-bit Linux too, after all.

- Gerry

---

[Last edited by trick at 07-27-2006 03:30 AM]

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

mrimer wrote:
Edit: Altered how imported data are handled to greatly reduce import memory footprint

Edit: Found out how to avoid allocating the entire size of the decompressed file during parsing (e.g. almost 90MB for you)

Edit: I've found a place where I can free up 1-3x the size of the .player file at the beginning of the import process (e.g. 10-20MB for you)

Edit: Plugged a small memory leak importing each saved game

Yay!

Watching all those demos and saved games validate, I felt like I was playing Progress Quest.

Well, Ridge Racer on the Playstation let you play Galaga while it was loading, so why shouldn't DROD let you, errr, "play" Progress Quest while importing?

(Now if only my ordered copy of Titan Quest would finally show up - I bet the guys at customs are having a blast with it... )

I've added a new INI option to 2.0.15(rev2), Startup->ValidateSavesOnImport, that may be turned off to skip validation of the move sequences of saved games and demos when a player is imported or hold upgraded. This will, of course, speed up import, but saved games and demos recorded in older versions of holds may appear broken.

Great - that's something I can definitely live with.

*steps on the suggestion box*

How about a menu option to run that validation at any time? With a nice fat "Cancel" button on top?

I guess saving the version the demo was tested with locally and skipping demos that were already checked with this version would let you more or less continue where it left off as well...

*steps off the box again, leaving the crushed remains in his wake*

Errr, oops...

EDIT: I assume the patch attached in the first post is still the first one? Your edited list of improvements is a bit misleading in this case...

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

---

[Last edited by Briareos at 07-27-2006 08:01 AM]

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

Yeah, so I look deep into how importing is done and improved the process, both memory- and speed-wise. Now, instead of taking 2+ GB and four hours to import and validate all your demos+saved games, it took ~300MB and 20 minutes on my computer. Turning off full validation would make it just a few minutes long. I'm happy about these improvements, but it's hard for me to tell whether the changes are correct. I'm attaching a new patch to the first post. Would you try importing your player file (if still needed), look around at some of the areas you should have imported new progress for, and telling me whether everything looks as you'd expect it to?

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

mrimer wrote:
Would you try importing your player file (if still needed), look around at some of the areas you should have imported new progress for, and telling me whether everything looks as you'd expect it to?

Oh, being able to import the player file was exactly what I needed...

Yeah, it's currently checking the saved game/demo integrity (I believe I managed to fumble turning off full validation) and this time the Virtual Size topped at 1,195MB - that I first did an export of my player might have contributed to that value. I'll try making a backup of my DROD directory, deleting my player and re-importing it as well...

As for disabling full validation - right above the "ValidateSavesOnImport=0" I have

LastNews=<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>Error Message</TITLE>
<META http-equiv=Content-Type content="text/html; charset=UTF-8">
...

complete with line breaks and stuff, thanks to me starting DROD from the patch setup instead of through the batch file that sets the correct http_proxy settings, so getting the news line got an error page instead - perhaps DROD should check if the reply has a HTTP status code of 200?

EDIT: Checking 2 holds that I've downloaded prior to importing everything looks like it should...


____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

---

[Last edited by Briareos at 07-27-2006 11:20 AM]

Pilchard VIII
Level: Smiter
Rank Points: 392
Registered: 01-21-2006
IP: Logged

It's probably a small thing, but I had patched up to 2.0.15, and was playing Timo006's Slayer Killing Hold. And when I got to I think it was 3E, whenever I moved up or left, the little noise that is made when Beethro moves didn't play.

____________________________
The DROD Dragon

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

Briareos wrote:
As for disabling full validation - right above the "ValidateSavesOnImport=0" I have

LastNews=<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>Error Message</TITLE>
<META http-equiv=Content-Type content="text/html; charset=UTF-8">
...

complete with line breaks and stuff, thanks to me starting DROD from the patch setup instead of through the batch file that sets the correct http_proxy settings, so getting the news line got an error page instead - perhaps DROD should check if the reply has a HTTP status code of 200?

Woah...I didn't realize our HTTP code wasn't doing that! I've added a check for 200.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

I've made these fixes and attached a new patch above (rev3):

* Removed erroneous saves to disk during demo playback or playtesting, causing delay.
* CaravelNet interface more robust to errors.
* Fixed demo display assertions.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

Pilchard VIII wrote:
It's probably a small thing, but I had patched up to 2.0.15, and was playing Timo006's Slayer Killing Hold. And when I got to I think it was 3E, whenever I moved up or left, the little noise that is made when Beethro moves didn't play.

Thanks for reporting this. Unfortunately, I can't reproduce this problem.

I have some helpful hints for reporting bugs you'd like to get fixed. Please describe a specific situation where the problem happens. For example, double-check what room it happens in and what moves must be performed to make the problem happen. For example, if these beeps only happen once you set off the bomb or kill the slayer, this is helpful to know. It's impossible to be too specific, so describe the problem with as much detail as you can.

By double-check, I also mean you might exit the game, restart, and try to make the problem happen again. This will help focus on where the problem's actually happening. Alternately, if you can't make the problem happen more than once no matter what you try, it'll probably be difficult or impossible for anyone else to track it down either.

Also, creating a demo that demonstrates the problem in play with F4 or F5, exporting it from the Demos Screen (press F6), and attaching it here would be very helpful.

Similarly, if you spot a problem with the level editor, you may make and post a little hold that exhibits the problem. Following these suggestions will make it easier for us to track down the problem and fix it.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 07-27-2006 06:42 PM]

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

mrimer wrote:
I've made these fixes and attached a new patch above (rev3):

* Removed erroneous saves to disk during demo playback or playtesting, causing delay.
* CaravelNet interface more robust to errors.
* Fixed demo display assertions.

Ummm... with rev3 importing the player file that I've put on my server or another one I made at work today hit's 2GB of virtual memory after about 15 seconds and the process goes poof, again...

EDIT: using rev2, the process' Virtual Size ballooned to 1968MB pretty quickly, but stayed there during the "Importing data..." phase - so something still is quite not right. I'll upload my changed files when the demo/saves verification is done...

EDIT: another fun fact - Sysinternal's Process Explorer shows a "Virtual Size" of 1967MB for the DROD process, while Task Manager only lists a "VM Size" of 622MB; still, I'm more inclined to trust Process Explorer on this as DROD reliably blew up when PE listed a Virtual Size of ~2GB... so in case you were using Task Manager to check the memory usage maybe try it again with Process Explorer...

np: SCSI-9 - Endlich (The Line Of Nine)

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

---

[Last edited by Briareos at 07-27-2006 08:40 PM]

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

Briareos wrote:
Ummm... with rev3 importing the player file that I've put on my server or another one I made at work today hit's 2GB of virtual memory after about 15 seconds and the process goes poof, again...

EDIT: using rev2, the process' Virtual Size ballooned to 1968MB pretty quickly, but stayed there during the "Importing data..." phase - so something still is quite not right. I'll upload my changed files when the demo/saves verification is done...

Okay, please do. I'm not sure why the import is grabbing so much memory on your end -- when I use the files you gave me, the app only reaches ~400MB virtual. If you've got some larger or updated files, I'll try those instead. Just email me the address when they're ready.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

mrimer wrote:
When I use the files you gave me, the app only reaches ~400MB virtual.

<redundant>
As I said in my last edit to my previous post - don't trust Task Manager, watch the process in Process Explorer...
</redundant>

EDIT: On closer inspection, what taskman calls "VM Size" is called "Private Bytes" in PE, whereas "Virtual Size" doesn't seem to have an equivalent in taskman...

Also, it looks like the integrity check is going to run for quite a while...

EDIT: Also, to quote this post from the cygwin mailing list:

"procexp's "virtual size" is simply a representation of the amount of virutal memory that has been allocated to the process. Virtual memory is not real memory and it only means that X number of pages have been allocated, it says absolutely nothing about the actual memory used by the process, and you should ignore it completely unless you have a specific reason to need to know about it.

If I parse that correctly, this would mean that DROD needs almost 2GB of address space (the maximum a process can normally get under 32-bit Windows), but only uses 622MB - I guess the rest is going to waste due to memory fragmentation, i.e. lots of small holes between the allocated areas that are too small for DROD to be used but of course exhaust address space due to the lack of a compacting garbage collector...

EDIT: Seems to be the perfect use case for Valgrind - too bad it's only available for Linux... and I'm not sure if both DROD and Valgrind run under CoLinux...

np: SCSI-9 - Sweets And Love (The Line Of Nine)

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

---

[Last edited by Briareos at 07-27-2006 09:20 PM]

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

Okay, I've uploaded my new files at the same location that I mailed you; I've just moved the old files to a folder named "old" and put the new ones in a folder named "new"...

What scares me is that my data.dat shrunk from 160MB to 64MB and my drod2_0.dat shrunk from 65 to 1.2MB - can this be right?

I'll probably try copying back my old data files and import my newest export over them, or deleting the player and re-importing or something...

By the way - could you perhaps tell me what is stored in each of the .dat files?

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

Briareos wrote:
Okay, I've uploaded my new files at the same location that I mailed you; I've just moved the old files to a folder named "old" and put the new ones in a folder named "new"...

What scares me is that my data.dat shrunk from 160MB to 64MB and my drod2_0.dat shrunk from 65 to 1.2MB - can this be right?

I don't understand. Do you mean that after importing your .player file into fresh .dats, this is the resultant size of those dats? If that's the case, I'd say to forgot to import the holds.

By the way - could you perhaps tell me what is stored in each of the .dat files?

Sure -- player stores basically player GUIDs and settings, text stores text strings, data stores imported media (images, voice...), and drod2_0 stores holds, demos, saved games, and speech records. Not obvious partitionings -- I've cleaned up the .dat organization in 3.0.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

coppro
Level: Smitemaster
Rank Points: 1309
Registered: 11-24-2005
IP: Logged

mrimer wrote:

Briareos wrote:
Okay, I've uploaded my new files at the same location that I mailed you; I've just moved the old files to a folder named "old" and put the new ones in a folder named "new"...

What scares me is that my data.dat shrunk from 160MB to 64MB and my drod2_0.dat shrunk from 65 to 1.2MB - can this be right?

I don't understand. Do you mean that after importing your .player file into fresh .dats, this is the resultant size of those dats? If that's the case, I'd say to forgot to import the holds.

By the way - could you perhaps tell me what is stored in each of the .dat files?

Sure -- player stores basically player GUIDs and settings, text stores text strings, data stores imported media (images, voice...), and drod2_0 stores holds, demos, saved games, and speech records. Not obvious partitionings -- I've cleaned up the .dat organization in 3.0.

I know that text.dat stores all the text data for the game - menu options and stuff (essentially allowing for different versions of text.dat to allow for different languages - an excellent idea), but does it doesn't include speech data, right?

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

I repeat, speech data is in drod2_0.dat.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

mrimer wrote:
I don't understand. Do you mean that after importing your .player file into fresh .dats, this is the resultant size of those dats? If that's the case, I'd say to forgot to import the holds.

I wish it were so, but this is what's left of the DAT files I started out with (from the "old" directory) after several attempts of importing the player files I've uploaded, some of which succeeded and some of which crashed DROD...

Which reminds me - how likely are the DAT files to survive a crash while importing?

I guess I'll start over with those old files, as these are from right before experimenting with importing since my first problem was exporting... *phew*

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

---

[Last edited by Briareos at 07-27-2006 10:52 PM]

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

Blech. I'm trying out Process Explorer to view earmarked virtual pages and now I'm seeing the same thing you are. I don't know how much fragmentation is occurring in the DB library, but I don't think it's that much, actually. Still, I have no idea why 100MB+ of paging gets reserved for every 10MB of memory actually allocated, so maybe it's a lot. The short answer then is that I don't know how to fix this problem. (After all, when has a game ever had to deal with 50000 save files?) A workaround you might use is to keep track of progress across two sets of .dats: each with half of the published holds imported. Then import your .player file into each one, and progress for holds not imported will be ignored, saving memory during import. It's yucky, but I don't know what else we can do. I'm looking into it, but if anyone who understands the issue at hand has any tips, I'll look into them.

Edit: After some more empirical evidence and thought, it does seem to be a fragmentation issue. I'm working on a solution.

Briareos wrote:
Which reminds me - how likely are the DAT files to survive a crash while importing?

Honestly, with the current model, I have no idea.

Edit: I've cleaned up import a bit, removing a feature I added during 2.0.15 (rev2) that will be unneeded for (rev4). Now a crash during import should never affect the .dats at all.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 07-28-2006 10:04 PM]

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

mrimer wrote:
Edit: After some more empirical evidence and thought, it does seem to be a fragmentation issue. I'm working on a solution.

Great! If you want, I can take try to figure out the code over the weekend and give it a thinking over as well...

By the way, I made an experiment yesterday: I installed a new copy of DROD, imported all my holds into it and imported my player - it topped out at 1569MB without crashing. Still, it might not work to import the same player file again, now that everything is both in the DAT files and the player export - but it was getting too late yesterday to try that...

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

Pilchard VIII
Level: Smiter
Rank Points: 392
Registered: 01-21-2006
IP: Logged

I've also noticed that ever since I installed DROD 2.0.15 on my computer, every time I minimise it by pressing the Windows symbol on my keyboard, and then bring it up again, all sound is stopped, music, SE's, voices, everything. I know this must be a boug in 2.0.15, because it's never done it inh 2.0.14.1. I'll get that demo set up sharpish.

____________________________
The DROD Dragon

---

[Last edited by Pilchard VIII at 07-28-2006 03:56 PM]

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

Pilchard VIII wrote:
I've also noticed that ever since I installed DROD 2.0.15 on my computer, every time I minimise it by pressing the Windows symbol on my keyboard, and then bring it up again, all sound is stopped, music, SE's, voices, everything. I know this must be a boug in 2.0.15, because it's never done it inh 2.0.14.1. I'll get that demo set up sharpish.

I think I've fixed this. Let me know if (2.0.15.rev4) still has this problem on your computer.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

Good news. I've reworked the import process (performing a lot more code maintenance than I had expected) and now DROD is much better behaved when importing huge .player files like yours. Loading and saving of game data is a bit faster. The initial import parsing (i.e. the first 50% of the progress bar, before demos+saved games begin to be verified) is probably significantly faster than it was before, too. Please redo your imports and let me know if everything seems to be working correctly.

I've also fixed some crashes and assertions that happen when trying to feed the game bad data files.

Added an INI option: LogErrors, which may be disabled (0) to stop the game from outputting error messages to drod.err.

Rev4 is posted at the top.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 07-29-2006 07:19 AM]

mrimer
Level: Legendary Smitemaster

Rank Points: 5149
Registered: 02-04-2003
IP: Logged

Warning: I introduced an import problem (in rev2?) that makes imported CaravelNet holds not match up with published CaravelNet holds, so CaravelNet doesn't work for any of them after they are imported. I've fixed it in rev5, posted above. Thanks to coppro for reporting the problem to me.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 07-29-2006 04:37 AM]

Pilchard VIII
Level: Smiter
Rank Points: 392
Registered: 01-21-2006
IP: Logged

File: Slayer Killing Hold 3 East No Sound Bug.demo (790 bytes)
Downloaded 46 times.
License: Public Domain

Okay, I've got the demo right here showing the no sound effect bug. I've tried it three times on two different computers, and it worked every time. But it seems it's only when the fuse is burning. Listen very carefully and you should catch where the bug is.

____________________________
The DROD Dragon