I can’t really speak to difficulties with components, but I have had some play testing experience. There are a few problems:

1.) Limited time. I wonder how many complete games get played during testing. I suspect it would be rare to top 100, with maybe as few as a dozen or so.

2.) An emphasis on playing rather than testing. The latter should include setting up the game at all sorts of unlikely mid-points, trying ahistorical strategies, doing multiple Monte Carlo simulations where you assume average die results every time you roll, then doing it again assuming extreme results, etc.

3.) Preaching to the converted, where the people who volunteer to test are already familiar with the topic, if not the game. This is even more pronounced with series games where everybody plays according to rules they learned long ago ... rules that are not actually in the game!

4.) A lack of simple proof reading, the kind where you literally read it word for word and check for the simple things like typos, spelling errors and missing words. Running the spellchecker is not adequate in any sense of the word.

5.) A lack of detailed proof reading, the kind where you look for internal inconsistencies (e.g., adjacent units must attack; single step units can’t attack); the wording is backwards (e.g., supplied units suffer attrition); rules that don’t seem to make historical sense (e.g., armor is more effective in urban areas); oddball situations that are unlikely to come up in the game but would be allowed by the rules (e.g., the US joins Germany in an attack on England).

6.) Examples of play based on previous drafts of the rules which don’t gibe with the published rules. Again, final proofreading should include actually setting up the units exactly as illustrated and walking through the example. I suppose there are many more, and the examples above may not be the best, but you get the idea. From what I’ve seen, and maybe even more from the final products, I think it is pretty clear that testing is often nothing more than the designer and a few buddies playing several games, often as part of the design process (so the rules are still in flux) then maybe getting a few new players to try the final version (although I suspect that happens less often than desirable).

By the way, none of this is meant to denigrate the efforts of all involved, few if any of whom are actually making a living at this. I understand the constraints involved (time, money, lack of qualified testers in geographic proximity, etc.) But as a consumer, being asked to pay $50 a title or so, I have to say that I wish for better quality. Careful proof reading takes about 10 hours per game (fixing it takes longer) so that wouldn’t be too much to ask. Better testing is hard, but worth the effort in my mind.

The bottom line is that I want to have the sense that I understand the system and am playing it right by at least the second playthrough, and I don’t want to have to continually adapt my understanding to the latest errata. That’s what I do as a playtester, where I at least get the game for free. I expect somebody else to have done their job on the games I purchase.

At times the mistakes I see are downright embarrassing, and yet the wargaming community seems to accept this sort of thing as a given with the industry. Here’s a possible reason: grognards will buy many games anyway, and enjoy them, in spite of their flaws. If I were the average businessman, and knew that sales of one of my products would be virtually the same whether or not I’d spent the time or money or effort for good quality control, what do you think I would do? That’s right -- I’d kick up my feet, relax, put up with the grumbling and know that the next time I issued a product with similar quality control, the same consumers would most likely buy it as well.

See the problem, here? Many people aren’t willing to vote with their wallets when it comes to quality control. Many (if not most) of us would rather pay for, and play, a flawed game in the hope (and likelihood) that the enjoyment will exceed the frustration of the experience. As long as that continues to be the case, everyone should expect to see the status quo continue. Of course, when everyone fears the alternative (that grognards showed more discriminating taste, and more wargame publishers went out of business before they could get their acts together), maybe the status quo is the best we can expect.

A big problem appears to be incestous playtesting, with too small a sample group. Proper playtesting requires fresh playtesters at several stages in the testing, and completely new playtesters at the final stage: people who have never seen the game before.

This story didn’t have legs, as they say in the news business. And more’s the pity, because I have a terrible feeling that we grognards are deliberately skirting the issue. On the whole, we’re people of substance in our community, holding down responsible jobs and making significant decisions on a regular basis. How many of us would accept this kind of behaviour from one of our suppliers or service providers? Not very many. So why are we willing to accept it from our hobby? Because we don’t want to offend the designers and manufacturers and drive them away. Well, denial is not just a river in Egypt, and our behaviour is, to put it bluntly, bullshit.

The most amusing thing about this issue, other than the paradox that our behaviour is not very self-serving at the moment, is that the main reason the manufacturers rush the games out the doors is their concern about competition. If they delay a release too long, they may get scooped by another manufacturer (e.g. GMT’s Battles for North Africa in the face of The Gamers’ DAK?), or they may lose market share. So each additional month of playtesting must be balanced against an additional month of reduced cash flow, but because we consumers roll over and play dead so easily, the answer is a no-brainer. -ed