eytanz
Level: Smitemaster

Rank Points: 2708
Registered: 02-05-2003
IP: Logged

Ever since I downloaded all the Caravelnet holds, switching between rooms is really slow. Considerably slower than in 2.0 (where I also have all the Caravelnet holds) - I timed around 16 seconds for a room transition, while in 2.0 it's maybe 3-4 seconds.

I never had anything approaching this in development, and the only things that I can think of that are different are the fact that I downloaded a lot of holds, and that I have caravelnet activated.

____________________________
I got my avatar back! Yay!

---

[Last edited by eytanz at 04-02-2007 02:25 PM]

piderman
Level: Delver

Rank Points: 51
Registered: 12-06-2005
IP: Logged

I have the same problem. I am thinking that maybe it's something to do with CaravelNet, since a lot of people are querying the server at the same time, so you may have a long time waiting for a response. Haven't tried turning it off yet though.

zex20913
Level: Smitemaster

Rank Points: 1721
Registered: 02-04-2003
IP: Logged

I'm having no issues with room-switch times in TCB, though I did in JtRH. I have no other downloaded holds at the moment, and I have CaravelNet activated.

____________________________

eytanz
Level: Smitemaster

Rank Points: 2708
Registered: 02-05-2003
IP: Logged

I had no problem early on when I had Caravelnet enabled but before I downloaded the holds. While not conclusive, I think it's the holds that are the problem.

____________________________
I got my avatar back! Yay!

piderman
Level: Delver

Rank Points: 51
Registered: 12-06-2005
IP: Logged

Hmm yes some more research shows that my player.dat file, with all holds downloaded is 50MB big. That would take a few seconds to write everytime a demo/highscore is stored.
In JtRH, while not having downloaded all holds, still quite a few, my player.dat is only 18KB.
So, speculation: custom downloaded holds are now stored in player.dat instead of drod3_0.dat or data.dat? Or at least something needing loads of disk space, TCB is over 100MB larger in .dat files than JtRH. This all is causing demo and highscore saves to slow down a lot because of DROD having to write 50MB instead of only 18KB.

MeckMeck GRE
Level: Smitemaster
Rank Points: 594
Registered: 01-03-2005
IP: Logged

zex20913 wrote:
I'm having no issues with room-switch times in TCB, though I did in JtRH. I have no other downloaded holds at the moment, and I have CaravelNet activated.

Seconded. TCB seems to be a lot faster than JtrH at changing rooms/levels/menues.

Doom
Level: Smitemaster

Rank Points: 3226
Registered: 07-05-2004
IP: Logged

I experienced slowness after a mass download of all holds, too.

I just deleted all non-official holds to see if it'd help with making the game more playable again. The delays between room switching dropped from constant 3-4 seconds to less than a half, so the holds were definitely the issue!

Stefan
Level: Smitemaster

Rank Points: 2119
Registered: 05-25-2004
IP: Logged

piderman wrote:
So, speculation: custom downloaded holds are now stored in player.dat instead of drod3_0.dat or data.dat? Or at least something needing loads of disk space, TCB is over 100MB larger in .dat files than JtRH. This all is causing demo and highscore saves to slow down a lot because of DROD having to write 50MB instead of only 18KB.

As I've understood it (in DROD 3.0):
- player.dat contains your demos and savegames.
- data.dat contains imported holds and media.
- drod3_0.dat contains the TCB hold and all media/texts used in the game (no user data).

____________________________
0.099³

---

[Last edited by Stefan at 04-02-2007 07:29 PM]

mrimer
Level: Legendary Smitemaster

Rank Points: 5056
Registered: 02-04-2003
IP: Logged

In 2.0, drod2_0.dat is the file containing saved games, and not player.dat. How big is your drod2_0.dat, piderman?

Stefan wrote:

piderman wrote:
So, speculation: custom downloaded holds are now stored in player.dat instead of drod3_0.dat or data.dat? Or at least something needing loads of disk space, TCB is over 100MB larger in .dat files than JtRH. This all is causing demo and highscore saves to slow down a lot because of DROD having to write 50MB instead of only 18KB.

As I've understood it (in DROD 3.0):
- player.dat contains your demos and savegames.
- data.dat contains imported holds and media.
- drod3_0.dat contains the TCB hold and all media/texts used in the game (no user data).

Mmm...actually, player.dat also contains the holds, but not the media (as you said). Probably, we should split that into a fourth .dat file that just contains the hold data, eh?

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 04-02-2007 07:46 PM]

piderman
Level: Delver

Rank Points: 51
Registered: 12-06-2005
IP: Logged

mrimer wrote:
In 2.0, drod2_0.dat is the file containing saved games, and not player.dat. How big is your drod2_0.dat, piderman?

This is from JtRH:

161.572.743 data.dat
 19.535.677 drod2_0.dat
     18.341 player.dat
  2.815.644 text.dat

This is TCB:

115.700.272 data.dat
120.376.911 drod3_0.dat
 51.489.681 player.dat

Though I'm not sure how big the difference should be, as 2.0 doesn't have all holds downloaded.

eytanz
Level: Smitemaster

Rank Points: 2708
Registered: 02-05-2003
IP: Logged

Mike, my player.dat is ~64MB large.

____________________________
I got my avatar back! Yay!

coppro
Level: Smitemaster
Rank Points: 1308
Registered: 11-24-2005
IP: Logged

Mine is 28 MB, according to ls.

WyattSmee
Level: Goblin
Rank Points: 19
Registered: 11-14-2006
IP: Logged

mrimer wrote:
Mmm...actually, player.dat also contains the holds, but not the media (as you said). Probably, we should split that into a fourth .dat file that just contains the hold data, eh?

If player.dat is being completely rewritten each time to save player progress (still not sure of this based on earlier comments), then splitting out the hold data would be an excellent idea. On the other hand if it's merely updated in place, might still be nice for aesthetical reasons. But AHA! Perfectionist programming can be a nasty trap.

Anyway... according to another thread, I thought the recent CaravelNet slowness was Murphy's fault.

eytanz
Level: Smitemaster

Rank Points: 2708
Registered: 02-05-2003
IP: Logged

WyattSmee wrote:

mrimer wrote:
Mmm...actually, player.dat also contains the holds, but not the media (as you said). Probably, we should split that into a fourth .dat file that just contains the hold data, eh?

If player.dat is being completely rewritten each time to save player progress (still not sure of this based on earlier comments), then splitting out the hold data would be an excellent idea. On the other hand if it's merely updated in place, might still be nice for aesthetical reasons. But AHA! Perfectionist programming can be a nasty trap.

I don't think it's being completely rewritten, but the database DROD uses is considerably more efficient for smaller data files. Mike wouldn't have suggested splitting it if he didn't think it would make an actual improvement.

____________________________
I got my avatar back! Yay!

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

eytanz wrote:
I don't think it's being completely rewritten, but the database DROD uses is considerably more efficient for smaller data files. Mike wouldn't have suggested splitting it if he didn't think it would make an actual improvement.

I've always wondered why there weren't several files for each hold - like, one with the hold data and one for the progress of each player.

After all, during the game itself all you ever need to read or write is data about the current hold and the current player's progress, while more is only needed in the "Change Location" screen and the editor's holds list.

(Have you heard? PCs have those nice filesystems nowadays that can cope with a directory with several hundred files just fine... -- 640th Gossip )

(Oh, and Jade Empire runs just fine despite it's 36 thousand files... )

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

halyavin
Level: Delver
Rank Points: 52
Registered: 02-20-2006
IP: Logged

I heard about clusterized indexes in databases that should do this job better than file systems. I don't know is metakit support such indexes though. But using a file system and variable number of databases/tables is a way to the hell.

jdyer
Level: Smiter
Rank Points: 329
Registered: 03-19-2003
IP: Logged

Just to add a strange data point, I'm getting minute-long delays between rooms ... but only after hitting the Negotiations level.

Playing Beethro-centric levels isn't causing this problem.

coppro
Level: Smitemaster
Rank Points: 1308
Registered: 11-24-2005
IP: Logged

I have an idea. Have an option in drod.ini to have another thread preload room data from adjacent rooms and rooms with stairs/script commands too them. It would probably help a bit. I don't know if it would work well when transversing blank rooms, though.

mrimer
Level: Legendary Smitemaster

Rank Points: 5056
Registered: 02-04-2003
IP: Logged

coppro wrote:
I have an idea. Have an option in drod.ini to have another thread preload room data from adjacent rooms and rooms with stairs/script commands too them. It would probably help a bit. I don't know if it would work well when transversing blank rooms, though.

It's not the load operations that are slow -- it's the save operations. But neither the loads or saves are thread safe -- we'd need to write a new, special layer to modulate thread-safe loads and saves. I'm not too experienced at that, and I'd be worried about doing things that would bust people's data files. Instead, I'm working on reorganizing the .dats to reduce the size of data files, which should reduce write delays to a manageable level.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

coppro
Level: Smitemaster
Rank Points: 1308
Registered: 11-24-2005
IP: Logged

Well then, why not move all data operations to a separate thread? That way DROD can do data operations without violating the thread-unsafety, and it can save in the background as you play.

mrimer
Level: Legendary Smitemaster

Rank Points: 5056
Registered: 02-04-2003
IP: Logged

coppro wrote:
Well then, why not move all data operations to a separate thread? That way DROD can do data operations without violating the thread-unsafety, and it can save in the background as you play.

Duh. I think I just said that, and also said that if I were to code that up, I'd worry about introducing bugs that could corrupt people's data files, so I'm going to pass on that for now.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

coppro
Level: Smitemaster
Rank Points: 1308
Registered: 11-24-2005
IP: Logged

I don't think we understand each other here. Isn't the problem that if two different threads attempt read/write operations at the same time, they can crash and burn? So wouldn't having just one thread to do all saving/loading work?

Anyway, you probably came up with a better solution.

mrimer
Level: Legendary Smitemaster

Rank Points: 5056
Registered: 02-04-2003
IP: Logged

coppro wrote:
I don't think we understand each other here. Isn't the problem that if two different threads attempt read/write operations at the same time, they can crash and burn? So wouldn't having just one thread to do all saving/loading work?

For the last time -- yes, that's right.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

Jatopian
Level: Smitemaster
Rank Points: 1842
Registered: 07-31-2005
IP: Logged

Mike seems angry. Let's all offer him hugs so his morale won't suffer.
*offers mrimer a hug*

____________________________
DROD has some really great music.
Make your pressure plates 3.0 style!
DROD architecture idea generator