TFMurphy
Level: Smitemaster
Rank Points: 3118
Registered: 06-11-2007
IP: Logged

File: MS2 Infinite Enemies 1S Victory.demo (2.2 KB)
Downloaded 38 times.
License: Public Domain

Once again, Infinite Enemies: 1 South in Magic Show 2. This time the problem isn't the recording of demos (or rather, that problem still exists, but it's not today's problem).

This is a fairly long room. But in all three of my recorded demos, something goes wrong if I back up the playback after playing past a certain point. A lot of the demo is simply the use of the movement macro for swinging, simply to kill all the roaches in the current spawn. At about Turn 392 though, if I back up the last two to three moves, one of the swings will 'fail' to kill a roach.

If I immediately play the next couple of turns forward, Beethro swings as normal and then the demo ends on the next turn.

If I undo the last move and then replay it, Beethro moves *up*. In some situations this is fatal: the move that is misplaced depends on exactly when I start backtracking through the move list.

If I undo the last *two* moves, things resume as normal.

The 'bad' move only seems to happen for me around 390-391. I've yet to find any other turn number that fails. And even those turn numbers only fail if I play them back and forth in a certain way.

I don't know if this is related to the demos not being recorded problem either, but I'm suspicious that they both happened in the same room.

I've attached a 889 move victory demo as an example. As I said previously, all three demos have the same problem, in different locations in the room, around the magic Turn number 390-391. In all cases, I was using a swinging movement macro at around that time. If more information or .dat files are needed, I'll be happy to provide them.

EDIT: And after a little bit of idle testing, my Lowest Proper: 2N1E demo also breaks around Turn 370 somewhere, around the time I ask Halph to open the final door. I haven't been able to track down the exact turn number for this one, but the effects are the same: either the demo ends abruptly, or the demo uses the wrong move and it desyncs. And I definitely wasn't using a movement macro here. It's possible that many long demos of mine are affected by this but at different turns - possibly due to the periodic undo savestates? I'm not sure.

EDIT 2: Just tried the Magic Show demo in 3.0.2. The start of the bug still happens (rewinding a move causes a roach to fail to be unkilled when it was supposed to be), although I'm finding this around 375 instead of 390 (I assume this is due to the changes in 3.0.2 adding more CPU time per turn). However, the two outcomes that I was seeing in 3.0.0 after going forward from that point (either ending the demo or Beethro moving incorrectly) aren't occurring. So while the bug is still there, it's not causing quite as bad a consequence.

---

[Last edited by TFMurphy at 08-06-2007 09:21 PM]

mrimer
Level: Legendary Smitemaster

Rank Points: 5058
Registered: 02-04-2003
IP: Logged

I'm suspecting the game state snapshot isn't recording everything that needs to be recorded in order to revert properly to a previous state. Maybe (wild guesses here) a static var somewhere is the culprit, like in builder paths or pathmapping? [Edit: it was discovered that restoring to snapshots were overwriting the setting to save demo on room conquer in 3.1.0.44 or .45; also, briar state wasn't being completely reset on room restart]

Would you try increasing the amount of time before snapshotting to greater than, say, 100000 ms in the INI and see whether this still happens?

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 09-13-2007 02:39 AM]

TFMurphy
Level: Smitemaster
Rank Points: 3118
Registered: 06-11-2007
IP: Logged

Huhm. Interesting. I set MaxDelayForUndo to 100000, and then 10000000, and neither seemed to make a difference - still saw the bug. Set it to 100 (absolute minimum). Still saw the bug. Still in the same place. So maybe the snapshots aren't necessarily the problem.

Not sure what else to suggest, especially given that it seems to almost happen like clockwork if I repeat the same playback moves, and I'm unsure how linked this might be to other demo and undo-state bugs I've seen reported recently. If you need .dats, let me know and I'll e-mail them.

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

TFMurphy wrote:
Huhm. Interesting. I set MaxDelayForUndo to 100000, and then 10000000, and neither seemed to make a difference - still saw the bug. Set it to 100 (absolute minimum). Still saw the bug. Still in the same place. So maybe the snapshots aren't necessarily the problem.

Well, I imported your demo into 3.0.0 and haven't experienced a single bug while watching it back and forth...

What happens if you delete the demo and re-import it?

np: Retina.IT - Comunicazione Postmoderna (Semeion)

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

---

[Last edited by Briareos at 08-06-2007 11:00 PM]

TFMurphy
Level: Smitemaster
Rank Points: 3118
Registered: 06-11-2007
IP: Logged

Briareos wrote:
What happens if you delete the demo and re-import it?

Exactly the same thing. If I go forward to Turn 393, and then undo two moves to 391, Turn 391 no longer kills a roach. And from that, if I go back one more to 390 and then forward to 391, Beethro moves up instead of swinging anticlockwise.

Also, I decided to continue and try another test. This time, I forwarded to the end of the demo, one turn before Beethro left, and then undoed each turn in succession, listening and watching for any roach that appeared without an accompanying death the turn 'before'. This gave me Turns 827 and 421 in addition to 390-391. (I think there was another suspect one around 842, but I couldn't get that one to repeat on a second try)

I really don't see any reasoning or pattern behind this, and it doesn't help that the only time I can tell that the bug has happened is when I don't hear a roach die when one should've. (The Halph one I found completely by accident by rewinding at just the right place)

Tahnan
Level: Smitemaster

Rank Points: 2459
Registered: 11-14-2005
IP: Logged

I was having demo-playback issues that involved mimic placement; is that something else, or another symptom of this?

TFMurphy
Level: Smitemaster
Rank Points: 3118
Registered: 06-11-2007
IP: Logged

Something different: there aren't any mimics or any sort of potions or turn changing things in both demos I found this bug in.

Also, Mike... are you sure MaxDelayForUndo is used for demos? I compiled 3.0.2 while manually setting the max delay to use 100000 as its default value no matter what, and now I'm having a lot of trouble seeing the same bug I saw before in 3.0.2 (even though the 3.0.2 version never ends in death or demo ending). Also, the undo takes far longer in the recompiled version than just using base 3.0.2 with the INI changed...

...so maybe it's snapshots after all?

Syntax
Level: Smitemaster
Rank Points: 1218
Registered: 05-12-2005
IP: Logged

I'm also using 3.0.2 and have started seeing just this issue.

Specifically:
Play through a demo, backtrack, play the demo from there and it causes death when it wasn't a death originally.

The room in question was "A Quiet Place - 4th Level - 1S1E" right after picking up the only mimic potion, moving a few moves, saving the demo and rewinding (turn 440ish).

"Perfect Chaos" also proved similar points but can't remember which exact rooms...

I'll hunt down more info if that helps but guess TFMurphy is well on the case already.

TFMurphy
Level: Smitemaster
Rank Points: 3118
Registered: 06-11-2007
IP: Logged

Ouch. Was trying to edit the reply I made in this topic when the forum died, and looks like it took the entire post with it. Ah well.

Anyways, definitely absolutely snapshots. I recompiled 3.0.2 to spit out interesting information like the number of computation milliseconds it had counted in place of the turn count, and watched the effects. First, I'll definitely note that MaxDelayForUndo in the INI only works when you're playing, not when you're watching demos.

Secondly, the issue I was seeing in 3.0.2 where rewinding a turn caused an 'unkilled roach' is caused by rewinding to the *exact* turn where the last snapshot was made. It's actually the fact that reloading a snapshot exactly instead of running through the series of commands up to the current point results in DROD not replaying the visual events that occured at the end of the last turn. So monster splats, trapdoors falling and so on don't play.

I also looked through the 3.0.0 to 3.0.2 diff that was posted elsewhere, and it looks like there's been a number of changes to demo playback, so it's possible that the *exact* issue I was seeing with undoing and replaying around snapshots is already fixed. Or it could be they're masking an issue that's still there but is now harder to see... I don't know. But the lack of visual cue events is still in 3.0.2. Of course, Syntax is reporting that he's seeing problems of his own there, but it's difficult to be sure exactly what the problem is.

Oh, and it's not limited to demos: if you're lucky enough to hit Undo in gameplay and revert to an exact turn when a snapshot is made, the same thing happens.

Anyways. The easiest fix I can think of would be to never reload directly to a snapshot. In other words, if undoing would take you to the exact turn count a snapshot was made, then you could instead use the snapshot before that and replay the commands that got you to the current snapshot. That would replay the visual cues as usual.

I don't know if there's a better solution though that doesn't involve throwing away a perfectly good snapshot. The problem here is getting DROD to replay the visual events that happened when the snapshot was made, and if that happens to be stored somewhere or can be collected without replaying the turn that created them, then great. Otherwise, it looks like rewinding past the snapshot and replaying is going to be the solution.

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

TFMurphy wrote:
I also looked through the 3.0.0 to 3.0.2 diff that was posted elsewhere, and it looks like there's been a number of changes to demo playback, so it's possible that the *exact* issue I was seeing with undoing and replaying around snapshots is already fixed.

Well, why don't you apply the diff in reverse and compile version 3.0.0 to test this?

np: Hexstatic - Roll Over (When Robots Go Bad!)

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

TFMurphy
Level: Smitemaster
Rank Points: 3118
Registered: 06-11-2007
IP: Logged

Briareos wrote:
Well, why don't you apply the diff in reverse and compile version 3.0.0 to test this?

Mostly because I have problems getting patch to work on my system, and I definitely don't have the patience to apply a 400k file by hand.

I'll be much happier once 3.1's out and stable and we're all looking at the same thing, I think. Just have to live with this until then.

mrimer
Level: Legendary Smitemaster

Rank Points: 5058
Registered: 02-04-2003
IP: Logged

Thanks for looking into this further.

TFMurphy wrote:
Anyways, definitely absolutely snapshots. I recompiled 3.0.2 to spit out interesting information like the number of computation milliseconds it had counted in place of the turn count, and watched the effects. First, I'll definitely note that MaxDelayForUndo in the INI only works when you're playing, not when you're watching demos.

Hmm...I'm confused. Are you saying that when watching a demo, snapshots aren't created at all? If so, then why would a problem with demo playback be caused by snapshots? (Sorry if I'm being obtuse, but something's not linking up in my brain correctly here.)

Anyways. The easiest fix I can think of would be to never reload directly to a snapshot. In other words, if undoing would take you to the exact turn count a snapshot was made, then you could instead use the snapshot before that and replay the commands that got you to the current snapshot. That would replay the visual cues as usual.

I don't know if there's a better solution though that doesn't involve throwing away a perfectly good snapshot. The problem here is getting DROD to replay the visual events that happened when the snapshot was made, and if that happens to be stored somewhere or can be collected without replaying the turn that created them, then great. Otherwise, it looks like rewinding past the snapshot and replaying is going to be the solution.

Yeah, this makes sense. I hadn't thought about the fact that the cue events relied on to drive the front-end display effects aren't generated at all when rewinding directly to a snapshot. Only rewinding to a snapshot when the destination turn is later than the snapshot seems like an easy fix. I wouldn't worry about performance degradation at all.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

TFMurphy
Level: Smitemaster
Rank Points: 3118
Registered: 06-11-2007
IP: Logged

mrimer wrote:
Hmm...I'm confused. Are you saying that when watching a demo, snapshots aren't created at all? If so, then why would a problem with demo playback be caused by snapshots? (Sorry if I'm being obtuse, but something's not linking up in my brain correctly here.)

Default Max Delay is 500 milliseconds. This is set no matter what. In certain conditions (for example, actually playing), DROD reads a replacement from the INI file (at the same time it does things like check if the player wants checkpoints displayed and the like). However, if you watch the demo, the setting in the INI file will be ignored and the default value of 500 is used. Hence when you asked me to change the INI file, I noticed no change in the demos. This would be the reason why.

mrimer
Level: Legendary Smitemaster

Rank Points: 5058
Registered: 02-04-2003
IP: Logged

TFMurphy wrote:

mrimer wrote:
Hmm...I'm confused. Are you saying that when watching a demo, snapshots aren't created at all? If so, then why would a problem with demo playback be caused by snapshots? (Sorry if I'm being obtuse, but something's not linking up in my brain correctly here.)

Default Max Delay is 500 milliseconds. This is set no matter what. In certain conditions (for example, actually playing), DROD reads a replacement from the INI file (at the same time it does things like check if the player wants checkpoints displayed and the like). However, if you watch the demo, the setting in the INI file will be ignored and the default value of 500 is used. Hence when you asked me to change the INI file, I noticed no change in the demos. This would be the reason why.

Ah, gotcha! That seems an oversight bug. How about we have the demo screen use the current value for snapshotting too.

Edit: Now we just need to figure out what is missing from the snapshot data that's causing replay corruption, right?

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 08-08-2007 09:01 PM]

TFMurphy
Level: Smitemaster
Rank Points: 3118
Registered: 06-11-2007
IP: Logged

Pretty much. Syntax seems to be finding something on his end, but my tests in 3.0.2 couldn't repeat the premature demo end and deaths I was personally finding. Like I said, I noticed several changes to demo playback (including a mention of a bug fixed from 2.0) in the diff file, but actually confirming it is kinda difficult for me.

I suspect the problem is related to the commands list, since that's what got out of sync (at all times I was seeing the bug in 3.0.0, Beethro moved in a way he shouldn't have, rather than the roaches jumping into places they weren't). Tricky to pinpoint though.

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

TFMurphy wrote:

Briareos wrote:
Well, why don't you apply the diff in reverse and compile version 3.0.0 to test this?

Mostly because I have problems getting patch to work on my system, and I definitely don't have the patience to apply a 400k file by hand.

I know Syntax will disagree, but might I suggest installing cygwin? It comes with a working patch executable (not to mention a lot of other stuff) right out of the box.

Here's the result of reverse-applying the approximate 3.0.0 -> 3.0.2 patch:

$ patch -l -R -E -F 10000 -t --dry-run -p0 < c:/Temp/diff_3.0.2.txt | grep FAILED
Hunk #12 FAILED at 1625.
Hunk #14 FAILED at 2124.
2 out of 15 hunks FAILED -- saving rejects to file DROD/DrodScreen.cpp.rej
Hunk #32 FAILED at 4891.
1 out of 32 hunks FAILED -- saving rejects to file DROD/GameScreen.cpp.rej
Hunk #2 FAILED at 338.
1 out of 7 hunks FAILED -- saving rejects to file DROD/Main.cpp.rej
Hunk #7 FAILED at 1465.
1 out of 7 hunks FAILED -- saving rejects to file DROD/MapWidget.cpp.rej
Hunk #35 FAILED at 4803.
1 out of 39 hunks FAILED -- saving rejects to file DROD/RoomWidget.cpp.rej
Hunk #21 FAILED at 1631.
Hunk #31 FAILED at 2258.
2 out of 39 hunks FAILED -- saving rejects to file DRODLib/Character.cpp.rej
Hunk #21 FAILED at 3166.
1 out of 45 hunks FAILED -- saving rejects to file DRODLib/CurrentGame.cpp.rej
Hunk #8 FAILED at 340.
Hunk #20 FAILED at 1190.
2 out of 20 hunks FAILED -- saving rejects to file DRODLib/DbBase.cpp.rej
Hunk #3 FAILED at 90.
1 out of 5 hunks FAILED -- saving rejects to file DRODLib/DbBase.h.rej
Hunk #17 FAILED at 6788.
1 out of 25 hunks FAILED -- saving rejects to file DRODLib/DbRooms.cpp.rej
Hunk #10 FAILED at 1565.
1 out of 11 hunks FAILED -- saving rejects to file DRODLib/DbSavedGames.cpp.rej
Hunk #4 FAILED at 179.
Hunk #5 FAILED at 192.
Hunk #9 FAILED at 390.
3 out of 9 hunks FAILED -- saving rejects to file DRODLib/Halph.cpp.rej
Hunk #2 FAILED at 872.
1 out of 6 hunks FAILED -- saving rejects to file DRODLib/Slayer.cpp.rej
Hunk #1 FAILED at 196.
1 out of 3 hunks FAILED -- saving rejects to file DRODLib/Stalwart.cpp.rej
Hunk #3 FAILED at 708.
1 out of 4 hunks FAILED -- saving rejects to file FrontEndLib/EventHandlerWidget.cpp.rej

Doesn't seem too bad to me - it's just that I'm too tired to really clean up the failed hunks right now.

np: Praxis - Machine Gun (Tennessee 2004)

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

coppro
Level: Smitemaster
Rank Points: 1308
Registered: 11-24-2005
IP: Logged

Isn't a reverse diff included anyway?

silver
Level: Smitemaster
Rank Points: 915
Registered: 01-18-2005
IP: Logged

coppro wrote:
Isn't a reverse diff included anyway?

every diff is a reverse diff to patch -R. I'm not sure I understand what you're trying to say here?


____________________________

Briareos
Level: Smitemaster

Rank Points: 3516
Registered: 08-07-2005
IP: Logged

coppro wrote:
Isn't a reverse diff included anyway?

Eh, well... I was wondering myself what that other file was for... (Not. )

____________________________
"I'm not anti-anything, I'm anti-everything, it fits better." - Sole
R.I.P. Robert Feldhoff (1962-2009)

Mazer
Level: Master Delver
Rank Points: 162
Registered: 04-08-2007
IP: Logged

File: VU The First Realm 1S1E Victory.demo (1.2 KB)
Downloaded 38 times.
License: Public Domain

This might the same bug. In Val Unrich, 1 South, 1 East, the player suffers a horrible death, even though I had conquered the room. I think I saw the enemies act weirdly.

I'm attaching the demo here.

schep
Level: Smitemaster

Rank Points: 865
Registered: 03-01-2005
IP: Logged

Oo, the same thing happens with my own demo there. It's not the same bug as earlier in this thread, since that one had to do with stepping backwards during demo playback, and this happens just watching.

Uh oh. A pathmap problem. Right after I kill a monster and all other monsters take their first step, if I step the demo forward one then back one, some of the monsters are in different positions.

---

[Last edited by schep at 11-29-2007 04:34 AM]

schep
Level: Smitemaster

Rank Points: 865
Registered: 03-01-2005
IP: Logged

At this code in CCurrentGame::ProcessCommand() between ProcessPlayer() and ProcessMonsters(), nothing was happening, since the pathmap already existed:

    //When player becomes a monster target, brain pathmap needs to be updated.
    if (!bPlayerIsMonsterTarget && this->swordsman.IsTarget())
        this->pRoom->CreatePathMaps();

Adding SetPathMapsTarget() just after CreatePathMaps() seems to fix the problem.

It's tempting to take out the CreatePathMaps() here, but I'm really confused about when CreatePathMaps() and SetPathMapsTarget() are needed. If the pathmaps exist but aren't updated when the player is a non-target, does the CreatePathMaps() at the end of SetPlayerRole() do anything? (Perhaps it's the SetPathMapsTarget() for non-Beethro near the end of ProcessCommand() which really fixes that situation.)

---

[Last edited by schep at 11-30-2007 02:43 AM]

schep
Level: Smitemaster

Rank Points: 865
Registered: 03-01-2005
IP: Logged

File: PathMapTest.hold (1 KB)
Downloaded 38 times.
License: Public Domain

Aha, doesn't actually work, that's why. Pathmaps for non-ground creatures aren't updated if the player doesn't move on the turn after a script sets him back to Player Role Beethro. In the attached hold, don't move (just wait 10ish turns), and watch the wraithwing act unbrained.

So the options here are:
1. Make sure SetPathMapsTarget() is called every time the player becomes a target. It should not be necessary to call CreatePathMaps() - that seems to be more for when a monster is created. I think the part that calls SetPathMapsTarget() when the player is not Beethro would be unnecessary too.

2. Change CreatePathMaps() to reset the target for all existing pathmaps, like it already does for the GROUND map.

Maybe both would be good.

---

[Last edited by schep at 11-30-2007 03:06 AM]

mrimer
Level: Legendary Smitemaster

Rank Points: 5058
Registered: 02-04-2003
IP: Logged

Yes, these (both) sound like appropriate changes.

____________________________
Gandalf? Yes... That's what they used to call me.
Gandalf the Grey. That was my name.
I am Gandalf the White.
And I come back to you now at the turn of the tide.

---

[Last edited by mrimer at 11-30-2007 07:57 PM]

schep
Level: Smitemaster

Rank Points: 865
Registered: 03-01-2005
IP: Logged

Well, I think I've fixed this. But many demos and saved games which involve (Player Role AND brain AND (wraithwing OR seep OR water skipper)) were invalid to start with. When build 55 or 3.1.1 is out, or maybe when the spider is updated to use 3.1.1, everybody may want to check for rooms to replay in Val Unrich and similar holds.

I didn't touch that instance of SetPathMapsTarget() called after character scripts after all -- I realized that's really for the case when the player is not a target but an NPC Beethro is moving around.