azb
Level: Smiter
Rank Points: 460
Registered: 03-08-2014
IP: Logged

It is a well-known fact that private beta boards allow you to have a serious discussion across an entire board dedicated to your hold without clutter or spam. It also allows you to bug-track your hold.

However, many holds could benefit from the wisdom of the crowd. This philosophy is that the general public may catch an unintended solution or other type of bug, such as script-related bugs, that a handful of only 5 or less testers or an automated bug-tracker would not catch on as easily unless you have a skillful group of testers.

Also, the Architecture board has recently been ignored by some holds ranging from 2014's Trail by Fire to 2016's Beethro Budkin's Big Burrito Butt Blast. These holds are very entertaining, but may have been beta-tested by only hardcore DRODders, making tedious rooms overlookable because the select few do not find them tedious.

The Architecture board has a purpose, and that purpose is to test holds. There is no beta bug report or feature request boards. Therefore, with the exception of official SMS's, there should not be a private Architecture beta-test.

Feel free to discuss your opinions here whether you approve of more utilization of Architecture or more utilization of beta boards. This has not been glazed over much, but I would like to hear the opinions of beta-testers and private architects as to why or why not use the private boards.

____________________________
Please feel free to test the ultimate arena: King Azb's Arena today!

---

[Last edited by azb at 04-24-2016 02:05 PM]

skell
Level: Legendary Smitemaster

Rank Points: 3734
Registered: 12-28-2004
IP: Logged

Speaking from personal experience, the biggest benefit of beta boards is specifically hiding a hold from the hungry eyes of majority of people and making an impact upon the release. There is a big difference between being hit by a hold like Burrito Blast on its release and seeing it grow from nothing.

making tedious rooms overlookable because the select few do not find them tedious.

That's a valid issue but I don't think it's such a big problem either. How many people on average really play and comment on a hold in the architecture board? Is that number greater or smaller than the average number of people that playtest a beta board hold? Again, speaking from my personal experience, even if the number of people who actually play and comment on the hold in a private beta is smaller they generally tend to provide much better feedback because the connection between the architect and the playtester is much more direct and intimate - the need to do the best is higher.

In my opinion it's far more important to encourage people to actually play the holds posted on the architecture board and leave feedback, because this is the part that tends to be problematic.

Finally, if you feel a published hold you're playing has tedious rooms leave that feedback on the hold's thread in the Holds board for the architect and anyone else interested to see.

____________________________
My website | Facebook | Twitter

Xindaris
Level: Smitemaster

Rank Points: 1531
Registered: 06-13-2015
IP: Logged

I've been using the Architecture board every time even after I got a subscription that made it possible to use beta boards, and I echo the sentiment of "please come try stuff in Architecture". It feels like there's maybe two or three people who consistently look at threads in architecture and give those holds-in-progress a real shake. I don't know if that's actually true, I'm reporting an emotional sensation more than a fact.

I'm actually quite curious if the testing in beta boards has more people on average or if it's actually just the same few people who are willing to try unfinished holds populating them.

____________________________
109th Skywatcher

Here are some links to Things!

---

[Last edited by Xindaris at 04-24-2016 03:31 PM]

kieranmillar
Level: Smitemaster
Rank Points: 2670
Registered: 07-11-2014
IP: Logged

I've had holds use public testing (Curiosity, The 'Neatherhood) and private testing (3 in 1, DRO, Beethro Budkin's Big Burrito Butt Blast), but note that those last two involved me making a public post in this thread on Architecture for testers. Neither method has really resulted in getting more testers and in practice the quality of the feedback ends up being solely on how lucky you get in getting someone who's willing to spend the time to provide through feedback. One good tester can easily outshine the sum of everybody else who tests.

Given that DROD is a hobby most people simply haven't got the time. I myself am guilty of this, ideally I would play more holds in Architecture and provide feedback, but I don't, because I haven't got the time (but on the plus side I'm an HA now so I'll likely end up playing it before release anyway, but not with the aim of providing any feedback for the author). The problem with hoping for the wisdom of the crowd to come along and help make your hold better is that there usually is no crowd until the hold is published.

What testing tends to be very useful for in practice is catching the more obvious unintended solutions. You should generally have at bare minimum one other person play through your work before release to catch these because as the author you tend to become blind to alternative solutions because you're too focussed on the way the room is supposed to work. For this, using a private beta board is significantly more useful because of the way it automatically uploads a room demo when someone who has C-Net active solves it, because many times I've spotted some use an unintended solution for part of the room and not report it because they weren't even aware that it was unintended. When using public beta testing, people don't always post demos.

If you're looking to make your holds better and the rooms more enjoyable, then based on my own experience, the best thing to do is to make a variety of stuff, release it, and see how people react, both in terms of the anonymous score ratings and the comments in the threads. If you want to get experimental, make your hold short so you can more easily and quickly publish it to get an idea for how people feel about it. If you really want to maximise feedback, then I recommend intentionally making something on the easy side, as I've seen that people are far more likely to play it and comment on it if they can actually manage to beat it. In general people only have limited time and if they can't get past a room or two after a certain short length of time, the feedback ends there. Rather than ask for hints, they will instead feel that their skill is not enough and they are not the right audience, and just not say anything.

uncopy2002
Level: Smiter
Rank Points: 431
Registered: 07-28-2014
IP: Logged

Sorry for being the straight man here, but I think I have to.

azb wrote:
However, many holds could benefit from the wisdom of the crowd. This philosophy is that the general public may catch an unintended solution or other type of bug, such as script-related bugs, that a handful of only 5 or less testers or an automated bug-tracker would not catch on as easily unless you have a skillful group of testers.

Ahh, the typical open source argument.

The problem with this kind of argument is that being open does not give you more exposure. Sure, your hold is out there for everyone to download, but it only helps if there are actually more people downloading and playing it, and at the same time, these people actually give feedbacks. Which, as you can see from the recent holds in the Architecture board, isn't the case.

azb wrote:
Also, the Architecture board has recently been ignored by some holds ranging from 2014's Trail by Fire to 2016's Beethro Budkin's Big Burrito Butt Blast. These holds are very entertaining, but may have been beta-tested by only hardcore DRODders, making tedious rooms overlookable because the select few do not find them tedious.

Umm... I don't see your point? Long-time or well-known DROD users does not equal to hardcore DRODers, and even hardcore DROD players have their own standard to what is tedious. If you mean the sample size is not enough, then I'd gladly refer you back to point 1.

Again, you can't put every problem in the "problems solved by throwing your hold in the open" bin.

azb wrote:
The Architecture board has a purpose, and that purpose is to test holds. There is no beta bug report or feature request boards. Therefore, with the exception of official SMS's, there should not be a private Architecture beta-test.

I know I'm probably being too frank here, but let's get it straight: it's up to the architect to decide whether he wants to go with private, closed development or in-the-open, open source development. Enforcing either one of them are pointless.

All in all, I'd say it's actually the best that we have both opposite way of development all available for our architecting needs. Nobody's feared of the architecture board - making and managing a beta board is even harder! - and there are 3 holds in architecture that are almost ready for HAs too.

---

[Last edited by uncopy2002 at 04-24-2016 05:35 PM]

Jeff_Ray...
Level: Smitemaster

Rank Points: 962
Registered: 05-16-2005
IP: Logged

My experience with Architecture is that I rarely get any feedback from anyone. Looking at my hold "Like Grains in an Hourglass", after the initial release, I've had to make lots of posts in chat to get people to test the hold.

Another great example is Homeward Adventure. It spent about 4 years without any comments, and when I finally brought it back from the dead, I similarly had trouble getting people to voice their thoughts on the hold. This led to the hold being full of unintended solutions upon release to the Holds board, which I find honestly very frustrating.

(EDIT) And another great example of the public's unwillingness to give feedback was with the Passing of the Hold that I ran in 2014 and 2015. After the initial construction was done, the testing phase was basically an utter failure, since almost none of the architects pitched in to go over the rooms, which led to the unbalanced finished product that got released to the Holds board.

I haven't had much of a chance to try out Private Boards in many years so I can't properly evaluate whether they're that effective, but my experience on the Architecture board has shown me that the crowd doesn't share its wisdom much.

(EDIT 2) Chaco mentioned in chat that we have more architects to make holds than we have people willing to play holds, which is mainly why Passing of the Hold's testing phase didn't go so well. It's easier (and more attractive a prospect) to create things than it is to playtest someone else's. So to really make the Architecture board worth its while, we'd need to have more people who are willing to go through holds and give feedback.

____________________________
I make Let's Plays too!
Click here to find out my LPs' progress, and find out what I plan to tackle next!

Currently playing:


My Holds:
-Completed:

-Work in Progress:

---

[Last edited by Jeff_Ray... at 04-24-2016 06:00 PM]

hyperme
Level: Smitemaster

Rank Points: 1062
Registered: 06-23-2006
IP: Logged

While everyone else is being more general about how this doesn't work, I'm going to pivot this in a more personal direction: Your actions and words really don't match.

You haven't posted in the Architecture board at all in 2016. In 2015 you posted in two architecture threads, one of which was your own. Whether or not a hold is in public or private beta will change nothing about the feedback I get from you, because that feedback was never going to be given. Making holds public does very little to increase feedback, since the number of people playing Architecture Board holds isn't that high.

Private Beta boards help to reduce this problem because they connect architects with people willing to test. Additionally, they collect demos and push updates directly to player's DROD clients, which is pretty useful. Taking away beta boards is only going to harm the amount of feedback gained, since architects have one less tool to gather it with.

The fact that only more experienced DROD people are the ones offering feedback isn't caused by beta boards, it's caused by other type of players not giving feedback. And no amount of digital hippyism is going to solve that.

(also beta boards are good for big projects like Entry Point)

____________________________
[Insert witty comment here]
Qzvlkx?

Chaco
Level: Smitemaster
Rank Points: 3624
Registered: 10-06-2005
IP: Logged

Some more stuff I'll bring over from chat to a permanent post:

I feel like part of this is just a function of how many people we have active in the community at any given time. Testing holds is a really time-consuming amount of work, and there are plenty of people who just want to play already-published holds that are guaranteed to be complete, already have checkpoints installed in good places, have had most of the annoying rooms taken out, etc.

Other people also mentioned that most of the people who test holds in Architecture are part of the "core active users" group; most newcomers either just play the official holds and usermade holds without testing other holds, or are authoring their own holds and are understandably focused on that iterative process instead of testing others.

Since there's so much time and effort required, I hope that we can just encourage more people to do a little bit of work, rather than hoping that people who are already doing a lot of work do even more work.

Also, any time architects can express appreciation for testers' comments, that would certainly help too. (Even if it's just a post in the form of "I'm not going to make this suggested change, and here's why, but thanks for making the suggestion.")

____________________________
Quick links to my stuff (in case you forgot where it was):

Dragon Fogel
Level: Smitemaster
Rank Points: 2434
Registered: 06-21-2014
IP: Logged

I made a suggestion in chat of setting up some kind of informal group focused on testing.

What I imagine is something like this: you "sign up" to the group, and then some people will say "here are some threads that have had fewer than 5 testers, maybe some more people could look at them."

Being in the group isn't an obligation to test every hold, but you're signaling that you're available. And architects who want more feedback can contact the group, rather than trying to bother individual people or just hope for more people to comment.

I don't expect this to solve all the problems, but right now testers are "whoever notices the thread and feels like testing". An actual group with an emphasis on getting holds tested would help at least a little.

Chaco
Level: Smitemaster
Rank Points: 3624
Registered: 10-06-2005
IP: Logged

As for the main "public testing vs. private testing" question, I agree with the points raised earlier that they're intended to accomplish different tasks. I think it's mostly a function of the size of the hold; small or large holds can be posted in either area, but I think large holds are the ones that benefit the most from private testing since that enables you to organize the comment threads, see sets of recorded demos for over 100 rooms, do collaborative script-writing on a neat plot, etc.

The website works wonderfully for what it's intended to accomplish (thanks Schik!) and the quality of the holds themselves are a function of the architects' work themselves, and how many rounds of iteration between architecture and testing the holds undergo. That can happen either in public or private testing.

____________________________
Quick links to my stuff (in case you forgot where it was):

Insoluble
Level: Smitemaster

Rank Points: 1638
Registered: 09-04-2014
IP: Logged

Dragon Fogel wrote:
I made a suggestion in chat of setting up some kind of informal group focused on testing.

Please sign me up

As for the general topic of this thread, I'm somewhat surprised at the perception that feedback on architecture threads is somehow less frequent or less helpful than feedback on private beta boards. I've been included on several beta boards (including one for a hold I published.) and I'd say that the rate and quality of feedback is about the same as it is in the architecture threads. It often takes quite a while for testers to post comments after updates on private boards as well. The fact of the matter is that, while this is an amazing community of folks dedicated to the game, we all have busy lives. Few of us have enough time to thoroughly play through, test, and comment on dozens of rooms every day. Sometimes you just have to be patient and post a reminder that you're looking for testers. I don't think that's necessarily a bad thing, and for what its worth, I never look down upon posts in architecture asking for more participation in testing for the umpteenth time.

____________________________
Links to neat forum tools that I always have trouble finding:

Dischorran
Level: Smitemaster

Rank Points: 3407
Registered: 09-10-2005
IP: Logged

Beta boards come with CaravelNet tracking of demos, which is critical - even if testers don't bother saying much, I can see what they do in-game. Assuming they beat the room, of course.

____________________________

Insoluble
Level: Smitemaster

Rank Points: 1638
Registered: 09-04-2014
IP: Logged

Dischorran wrote:
Beta boards come with CaravelNet tracking of demos, which is critical - even if testers don't bother saying much, I can see what they do in-game. Assuming they beat the room, of course.

This is a good point. I found that extremely helpful when I had a hold in architecture. I've made it a habit when beta testing stuff in architecture to post my demos. I'll even record demos of rooms I don't clear on some occasions. I would encourage other testers to hit the export all button on the demo screen and attach it to your post when posting in architecture.

____________________________
Links to neat forum tools that I always have trouble finding:

LeoS
Level: Master Delver
Rank Points: 276
Registered: 04-02-2015
IP: Logged

For the past year or so, I'm certainly guilty of - as kieranmillar referenced - dl'ing a hold in architecture, fully intending to comment helpfully, and instead hitting the solving wall quite early into it and concluding that maybe I'm just out of my league. That's currently happening to me in Rodents of an Unusual Size, for instance, where I've solved 3 of the 20 rooms I've visited. I'm also uncomfortable turning an architecture thread into a massive H+S spreadsheet, which is probably an advantage to private beta boards, rather than having 80% of a hold's solutions laid bare before it's even published.

Then there's my solving style: leaving the game for a while (or switching to a different hold) if I'm really stumped and letting things percolate, which isn't necessarily conducive to regular or timely updates for an architect. (As an aside, I love the times when I solve a room while not even *playing* DROD, the Hercules Poirot little grey cells effect.) I also suffer from Intend to Comment but Never Actually Do So syndrome, with generalized flakiness to boot, both of which evidently carry over to the Holds board, as well.

But after typing all that, I don't see why in my mind No Feedback is somehow preferable to Irregular or Partial Feedback...

So...after having really sold myself here as an *invaluable* playtester , I am going to head over to DF's volunteer board and see if that helps prod me into commenting/contributing more often, or at least PM'ing architects if I'm playing their hold but don't have much to add since I'm not getting anywhere. I'm sure it's frustrating to see 'downloaded 11 times' and have a comment or two at best.

tl; dr: Hesitant Poster/Slow Solver resolves to be a better CaravelNetizen

Insoluble
Level: Smitemaster

Rank Points: 1638
Registered: 09-04-2014
IP: Logged

LeoS wrote:
...fully intending to comment helpfully, and instead hitting the solving wall quite early into it and concluding that maybe I'm just out of my league...

Actually, this is precisely the kind of testing that we could probably use more of. Having testers of all different skill levels is super helpful for determining the relative difficulty level of a hold. Letting the architect know how difficult you find the hold in general and specific rooms can be invaluable feedback. Even if the architect is going for an intentionally difficult hold, it can be useful to know which rooms are Clearly Impossible, and which are merely brutally hard so that they can be sequenced well in the level map. (Full disclosure, this is the kind of feedback that I probably didn't do a very good job of heading when others commented on the hold I was beta testing.)

____________________________
Links to neat forum tools that I always have trouble finding: