StuartK
Level: Smitemaster

Rank Points: 564
Registered: 06-10-2003
IP: Logged

I recently decided to reinstall DROD. I exported my player (a few minutes) reinstalled, reimported my player (a few minutes) reimported all the holds (the highscores list was extremely handy for reference here) and reimported my player (a few hours)

Following this procedure reduced the total size of my dats from 110MB to about 80MB. That's probably partly because I was playing with the import image function, on some rather large images though. My guesswork is that though the hold was deleted, the space used was not cleaned up.

I'll also reupload my highscores at some point to see if that fixes the missing ones I reported seperately on the Site forum.

Anyway, the main reason I did this was to see if I saw any improvement in game responsiveness. It's a marginal case, but I do see a subjective improvement while playing. Some of the more frustrating delays remain, however. I decided to replay The Odyssey, and got the following approximate timings;

Start up DROD, go to the restore screen - responsive so far.

Odyssey L2 entrance - view demo screen from restore screen (I have one demo in the local list) - 55 seconds

Download a demo - 25 seconds

After clicking OK - 60 seconds

Navigating between both demos - 30 seconds each

'W' to watch demo - 10 seconds

Exit demo prematurely - 40 seconds

Exit restore screen - instant

Continue playing - 15 seconds

Return to menu - 10 seconds

I'm using a Duron 1300, 768MB of PC2100 RAM. A number of tasks running in the background (a few of them probably unnecessary) but Task Manager was reporting that DROD was taking the lions share of CPU during the above pauses.

I'll add some more timings to this thread later. Of particular interest to me is the increase in delays when switching holds, which I've seen previously, and which is usually resolved by quitting and restarting DROD.

StuartK
Level: Smitemaster

Rank Points: 564
Registered: 06-10-2003
IP: Logged

I've not seen any particular additional delays when switching between holds ingame, but it's difficult to tell at the moment, since the delays do vary in length fairly dramatically.

One thing I wondered was if the delays were Caravelnet related, but switching off the connection made no difference, so IMHO it's something to do with how the data for demos is stored or accessed, and/or the size of the DATs. I've played maybe half of the available holds.

---

[Last edited by StuartK at 10-30-2005 04:41 PM]

Rabscuttle
Level: Smitemaster

Rank Points: 2460
Registered: 09-10-2004
IP: Logged

StuartK wrote:
I recently decided to reinstall DROD. I exported my player (a few minutes) reinstalled, reimported my player (a few minutes) reimported all the holds (the highscores list was extremely handy for reference here) and reimported my player (a few hours)

...

I'll also reupload my highscores at some point to see if that fixes the missing ones I reported seperately on the Site forum.

Hrmm... I think there might be a problem with reinstalling and reuploading - it seems that some demos when uploaded later are overwriting themselves when uploaded earlier.

Most of your scores are now dated 29th October. Which means that a few No.1s are now No.2s Two rooms where I can definitely confirm that this has happened are "Caverns of Animus: The Long Ascent: 2 North, 2 East" and "Journey to Rooted Hold : Twenty-Fourth Level: 1 North"
Your number of downloaded rooms has also dropped dramatically from over 1000 to less than 400.

StuartK
Level: Smitemaster

Rank Points: 564
Registered: 06-10-2003
IP: Logged

My number of #1s has dropped from a high of about 850 down to 700. I'm sure some of that was your doing Rabs, but 100 since yesterday is a little much even for you...

Schik
Level: Legendary Smitemaster

Rank Points: 5381
Registered: 02-04-2003
IP: Logged

StuartK wrote:
My number of #1s has dropped from a high of about 850 down to 700. I'm sure some of that was your doing Rabs, but 100 since yesterday is a little much even for you...

Oh boy.... I think this happened because a bug of mine. I will do everything I can to rectify this. Sorry.

____________________________
The greatness of a nation and its moral progress can be judged by the way it treats its animals.
--Mahatma Gandhi

StuartK
Level: Smitemaster

Rank Points: 564
Registered: 06-10-2003
IP: Logged

Looking good now, thanks I'm back up to 810-odd again. Not that it'll stay that way, what with Rabs playing all those large holds he hadn't quite gotten round to, and me slowing down just recently.

One other thing I've noticed just now is that the total number of moves to complete JtRH has taken a jump from 58k to 61k. I presume this is spider related, and will fix itself shortly?

Even with these minor niggles, this is an impressive thing you've put together Schik. Fixing a live system is a whole different ballgame to just a static piece of code.

Schik
Level: Legendary Smitemaster

Rank Points: 5381
Registered: 02-04-2003
IP: Logged

StuartK wrote:
Looking good now, thanks

Good.

One other thing I've noticed just now is that the total number of moves to complete JtRH has taken a jump from 58k to 61k. I presume this is spider related, and will fix itself shortly?

Whoops - I forgot one thing - to fix the rankings after re-inserting your prior #1s. So there were just a bunch of extra #1s, thus upping the total number of moves in #1 demos. Fixed.

Fixing a live system is a whole different ballgame to just a static piece of code.

Yeah, you're not kidding. I hate messing around with the high score table when it's live. Makes me nervous every time. Thousands and thousands of hours of players' time has gone into those scores... the thought of screwing it up is not a nice thought.

____________________________
The greatness of a nation and its moral progress can be judged by the way it treats its animals.
--Mahatma Gandhi

Alneyan
Level: Smitemaster
Rank Points: 622
Registered: 07-06-2004
IP: Logged

Just switch your invulnerable flag on, Schik, and you will be fine. Well, except for all the insults, miscellaneous items thrown at you, and things of the like.

Stuart, if it is any confort, think about all of us, small players caught in the middle of your fights. I don't think I am the only one with fragile best scores, and even the ones snatched from top players aren't quite *totally* secure (I know I somehow managed to beat a very widespread 15-move demo, though I haven't been able to figure out why I happened to be the first of making it in 14 moves).

Shame you can't sign non-aggression pacts with your fellow delvers.

Rabscuttle
Level: Smitemaster

Rank Points: 2460
Registered: 09-10-2004
IP: Logged

StuartK wrote:
My number of #1s has dropped from a high of about 850 down to 700. I'm sure some of that was your doing Rabs, but 100 since yesterday is a little much even for you...

Well, I do what I can.


There's still a small problem with the Misc pages - it looks like the rooms that got fixed appear multiple times in the longest room list - e.g. the 60000 room at the top. It's messing with the page numbering too - I think the "last page" link assumes that the extra scores aren't there, as it takes you to about 5th last page - the 9s rather than the 2s.

Schik
Level: Legendary Smitemaster

Rank Points: 5381
Registered: 02-04-2003
IP: Logged

Rabscuttle wrote:
There's still a small problem with the Misc pages - it looks like the rooms that got fixed appear multiple times in the longest room list - e.g. the 60000 room at the top. It's messing with the page numbering too - I think the "last page" link assumes that the extra scores aren't there, as it takes you to about 5th last page - the 9s rather than the 2s.

It was actually unrelated, but the problem has been fixed. Thanks for reporting the problem.

____________________________
The greatness of a nation and its moral progress can be judged by the way it treats its animals.
--Mahatma Gandhi

michthro
Level: Smitemaestro
Rank Points: 679
Registered: 05-01-2005
IP: Logged

These delays can become really frustrating when your data files start getting large. I looked into what happens when selecting a different demo on the demos screen, and the biggest reason for the slow response seems to be that Commit() is called 4 times in LoadPrep(), when it shouldn't be necessary at all in this situation. Deleting this line makes a huge difference to the response time, but of course that's not a fix, since it may be necessary when LoadPrep() is called from elsewhere. I think other delays, e.g. when restoring, can also be reduced for similar reasons. Please could it be fixed in the next patch so that the database files are only updated when really necessary?

mrimer
Level: Legendary Smitemaster

Rank Points: 5056
Registered: 02-04-2003
IP: Logged

michthro wrote:
These delays can become really frustrating when your data files start getting large. I looked into what happens when selecting a different demo on the demos screen, and the biggest reason for the slow response seems to be that Commit() is called 4 times in LoadPrep(), when it shouldn't be necessary at all in this situation. Deleting this line makes a huge difference to the response time, but of course that's not a fix, since it may be necessary when LoadPrep() is called from elsewhere. I think other delays, e.g. when restoring, can also be reduced for similar reasons. Please could it be fixed in the next patch so that the database files are only updated when really necessary?

Before 2.0.10, each call to Commit did write the .dats to disk. In 2.0.10, however, it should already be the case that Commit only writes the .dat files to disk if something in them has changed.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

michthro
Level: Smitemaestro
Rank Points: 679
Registered: 05-01-2005
IP: Logged

mrimer wrote:
Before 2.0.10, each call to Commit did write the .dats to disk. In 2.0.10, however, it should already be the case that Commit only writes the .dat files to disk if something in them has changed.

Well, I don't know exactly what all happens when Commit is called, but still, deleting the call to Commit reduces the delay when selecting a different demo enormously. So it's not writing to disk, but something inside Commit takes up a lot of time unnecessarily. Or am I missing something?

trick
Level: Legendary Smitemaster
Rank Points: 2580
Registered: 04-12-2003
IP: Logged

Are you sure you're using 2.0.10 ?

- Gerry

StuartK
Level: Smitemaster

Rank Points: 564
Registered: 06-10-2003
IP: Logged

Well I am (a fresh install with a reimported .player file) and the slowdowns definitely exist for me - the timings from initial post were all using 2.0.10. Whatever the cause, I believe it's a function of either the total size of all the demos in the current hold, or the largest demo in the hold, since Caverns of Animus and Malcolm Brock & Dave: Death by Goblins are both particularly slow for me.

---

[Last edited by StuartK at 11-19-2005 04:15 AM]

eytanz
Level: Smitemaster

Rank Points: 2708
Registered: 02-05-2003
IP: Logged

I get very similar slowdowns as Stuark, in 2.0.10.

____________________________
I got my avatar back! Yay!

michthro
Level: Smitemaestro
Rank Points: 679
Registered: 05-01-2005
IP: Logged

trick wrote:
Are you sure you're using 2.0.10 ?

Definitely. I have slowdowns similar to StuartK's with the official 2.0.10 version, and the source I'm using is also 2.0.10 (it says so on the main menu screen).

michthro
Level: Smitemaestro
Rank Points: 679
Registered: 05-01-2005
IP: Logged

mrimer wrote:
In 2.0.10, however, it should already be the case that Commit only writes the .dat files to disk if something in them has changed.

ok, I see what you mean: the flag bDirty should cause Commit to return without doing anything if nothing has changed. The problem is that every time a demo is loaded to get info/verify it etc., bDirty is set (ultimately in CDbSavedGame::Update).
I added a flag bSetDirty, which is true by default, and in all Update methods I added "if (CDbBase::bSetDirty)" before "CDbBase::bDirty = true". Then I clear bSetDirty during CDemosScreen::SetWidgetsToDemo and CRestoreScreen::SetWidgets. The result is that the delay when selecting a different demo/room/level reduce from at least 10 secs to less than 1 sec.

Then, in CDbXML::VerifySavedGames, I clear bSetDirty during demo and saved game verification, except when deleting or explicitly updating. I then imported a player file, and while importing the data is still very slow (as expected), the time taken to verify demos and saved games reduced from 100 mins to 8 mins.

I think there are a couple of other places where the same bug causes unnecessary delay. This is quite a serious problem, unless you have a very fast computer, I suppose, so please look into this and fix it for the next patch?

mrimer
Level: Legendary Smitemaster

Rank Points: 5056
Registered: 02-04-2003
IP: Logged

michthro wrote:
The problem is that every time a demo is loaded to get info/verify it etc., bDirty is set (ultimately in CDbSavedGame::Update).
...
I think there are a couple of other places where the same bug causes unnecessary delay. This is quite a serious problem, unless you have a very fast computer, I suppose, so please look into this and fix it for the next patch?

Good snooping! Sure, this is important to fix.

It took me a while to find out where the import slowdown was actually coming from (e.g. what is calling Commit so much during import). As you mentioned, I see now that Commit is called every time a new current game is loaded (in CCurrentGame::LoadPrep). I think invoking Commit at this point is improper. Better, I think, to call Commit when an operation completes, not when it starts. If an imporatant transaction completed right before loading a new game, for example, it should have called Commit itself rather than expecting something else to call it. The only proper time for this in DRODLib is when an Import operation successfully completes. So I'm going to yank this Commit call. (I've already tightened up Update calls even more in 3.0.)

This will fix the needless import delay, and it should avoid the delay in other situations where new games are invoked for various reasons.

Another problem I noticed is in CCurrentGame::AddRoomsToPlayerTally(), invoked by all room load calls. This call always calls CDbSavedGame::Update. I'm fixing it as follows:
1) Don't Update here when "GetAutoSaveOptions() <= ASO_CHECKPOINT" (i.e. during a demo load/test or when playtesting).
2) Don't Update if no new rooms are being added to the room tally.

This should fix the first situation you mentioned (needlessly setting the dirty bit). I like this solution better because this tally record shouldn't be updated at all in the above situations and it will fix the Restore and Demo(s) Screen delays.

That's everything I can think of. Any more possible fixes? Send 'em my way!

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 11-22-2005 02:09 AM]

StuartK
Level: Smitemaster

Rank Points: 564
Registered: 06-10-2003
IP: Logged

Really looking forwards to that next release....

Michthro (and Mrimer) *thankyou* for looking into this.

mrimer
Level: Legendary Smitemaster

Rank Points: 5056
Registered: 02-04-2003
IP: Logged

I found another possible source of delay: after watching a demo, there's slowdown when returning to the Demos Screen (especially after watching a long demo, and maybe when connected to CaravelNet). I've removed this delay in the next patch.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.