Friday, June 1, 2012

Is This An Effective Survey?


The DND:Next open playtest survey has been posted.  Take a few minutes and take the survey, if you want to make your voice heard.

The first question is not terribly useful.  Instead of asking how long you have played, they should be asking which editions of DnD you have played.
The next question asks which edition of DnD is your favorite.  Not a terribly good question, since the survey does not delve deeper, into WHY that is your favorite edition.  What game feature(s) make it your favorite?  It is unclear what this question will measure: how do they intend to cross-reference this question with data WOTC collects in later questions?  If i've only ever played 4E, can I provide any useful data by responding to this question?
Next, WOTC asks what kind of player you are: power-gamer, explorer, thinker, actor, and so on.  You can choose more than one category.  I don't see a lot of value in this question.  Remember, this is a self-selecting questionnaire.  WOTC is bound to get a preponderance of responses from gamers that enjoy 4E, so that play-style will be over-represented in the results.
There is a set of questions asking for your level of satisfaction with various rules in the open playtest package.  Interestingly, there is a question asking if there is a sufficient amount of simulation in the ruleset.  Thankfully, they provide a comment box where you can give additional feedback.
It's hard for me to evaluate the dangerousness of the monsters, since that all depends on your DM and playstyle.  Make up your own mind on the amount of healing, I certainly have strong views on that subject.
This is really starting to sound like design-by-committee.  Hmmm, so if you are an old-school player, how are you likely to response to the hit point question?  How about the power-gamer?  Just design a good game already, and stop worrying about this stuff, or do what you promised to at the outset, and make the game modular so game-groups can start characters with fewer or more hit points.

More game balance BS.  And battle-grid questions.  I thought DnD:Next was going to provide gridless combat options.  Has that ALREADY gone out the window?
I ask because the above question is perhaps the most provocative of the survey questions.  Why announce that the grid will be optional, and then drop that bomb on us?

Some good questions to determine your capacity to evaluate the new ruleset, from a DMing point of view.  Hopefully the cross-reference the subsequent answers with the DM's stated experience.
The above questions posed to the DM are quite revealing, as they confirm that clarity and ease-of-use are two design goals related to DM resources.

The DnD:Next designers seem hopelessly entangled in the gamist model.  The above player character questions focus on gamist concerns.  It would be refreshing to see questions asking whether the character's special features adequately simulated reality, or allowed the player to get into character.

How do you answer the question about the feel of a class?  Particularly if all the respondent has experience with is 4E, and maybe 3E?  This is why I have a problem with this survey structure.  You are asking for feedback from people, some of whom may have NO IDEA what the Cleric "feels" like, across the entire history of DnD.  You need to cross-reference this, and the other character questions, against the earlier question regarding the number of years a respondent has played DnD (or better, number of editions).

The game designers seem terribly concerned about giving the Fighters "interesting things to do" in combat.  They don't seem concerned that Fighers have interesting things to do outside of combat.

Having recently read Jack Of Shadows, I would love to see a more "magical" Thief class.  What did they call Thieves in 4E, strikers?  Annoyingly MMORPG of them.  How about a Thief class that emulates the swashbuckler style of character instead?
WOTC is asking for feedback on the hit points and cantrips of the Wizard.  Do the right thing here, people.
The non-human classes get special abilities (traits) and i've already read some reactions to those traits.  It appears that humans get higher ability scores to compensate for their lack of special abilities.
Can you really answer the first of two questions, above, if you havn't played ODnD, ADnD or 2nd Edition?  Come on, WOTC, give your heads a shake.

7 comments:

biopunk said...

No, this is not an effective survey.

But a rogue with a sling that does a d8 +3 in damage seemed overly effective. Dare I say, it struck me as very "strikerly"?

ps: This is the survey I was referring to the other day.

Aaron E. Steele said...

This is a terrible survey. I really wish WOTC would hire a proper market research / polling company.

Dennis Laffey said...

Funny. I did NOT get all of those questions when I did the survey. Only about 2/3 of them.

Definitely a crappy survey, if they can't even give all of the questions to every respondent. In addition to the rather pointless array of questions presented, that is.

Unknown said...

The only theme I got out of it was that they seem to be aware that people are having a problem with HP inflation at low levels, too much self healing, general PC's being too hard to kill at low levels, etc.
As to the "filler" questions my best guess is the survey outfits contract calls for x number of questions per poll or something, so they drop in floaters that have a slim connection to the subject.

I just answered them and added comments where I could, managing to shoehorn in a comment to the effect of "Stop fetishizing balance - this isn't a video game. When class variety and fun conflict with balance, cut balance's throat and throw it to the sharks."

Timothy S. Brannan said...

You can't say it's a bad unless you what it is being used for. To my knowledge none of us know.

But I have done survey analysis. Quite a bit of it in fact.

Let's look at this as an base line or even D&D demographic (since we really are not interested in actual demographics yet.)

This is not a great survey, but it is not a terrible one either.

We just don't know what questions they are asking that this survey is supposed to get too.

Nor is this something that exists in a vacuum. This is a snapshot of data collection whose biggest advantage (and frankly disadvantage as well) is that it will have a huge sample size.

Greg G said...

"The DnD:Next designers seem hopelessly entangled in the gamist model."

Pretty accurate to me.

DaveL said...

Unless we're expected to "assume" most of the playtest info is using 4e to fill in the missing bits, I found it difficult to effectively voice an opinion of rules that are yet to be writ. A lot of stuff like advantages and disadvantages were hinted at but not really spelled out, as well as spell regeneration; it seems as though once the spell slot is vacant, it will remain so until the next "long rest," but that was implied, rather than spelled out specifically.