Lessons Learned in Development

G'day all,

As some of you are no doubt aware, BIS is facing intense criticism over releasing an incomplete product (ArmA 2), the campaign isn't completed and hasn't been played to death by testers, there have been numerous CTD reports and this is quite literally the tip of the iceberg. If you look in the German manual on page 75, you will see the names of the community testers, these are average people with what one would assume a reasonable variation in system profile. Developing for consoles or even a single SOE in the military context doesn't require the level of testing you see for a commercial game like A2, however we can still learn from their mistakes.

First of all a note, it's likely that the beta testers in this instance were unpaid volunteers who signed a NDA, selected for their enthusiasm for the game and their knowledge they would of conducted limited testing in their spare time. Unfortunately a quick check of the manual credits no one from either Morphicon Ltd. or BIS with being a Community Testing Coordinator or similar. Additionally the release in Germany before the rest of the world could be seen as a mistake, given the context it looks like an entire country became beta testers, and a few externals like myself.

In the military context as previously mentioned, we largely develop for our own SOE with little consideration for others that will use our products, although we rely on them to conduct their own procedures for testing and get back to us with issues. This was true for one recent addition by a community organisation within the mil user group. The addon in question overwrote part of the target machine's config via inheritance and changed various UI elements. This may not be noticeable on the developers machine. Take for example the standard customer distribution of VBS2, and JCOVE (British MoD version). JCOVE has a modified front end to suit the UK context as largely in name only it is separate to VBS2, it must be noted that a patch I release as a General Customer (YYMEA) or any other customer for that matter must be tested on systems like JCOVE to ensure it doesn't rework their system.

Mantis bug report #0001945 for instance originated from the NZDF, regarding the wording of the login box bottom right. The original text for this box was "Nickname" however, and this is correct, soldiers training saw nickname as an opportunity to feel the need for speed and Maverick it up for lack of a better term. I responded to this request, changing Nickname to Rank/Name, unfortunately the difference in the text required me to resize and shift this element from its normal position. This is fine for most users using the largely common front end provided by BIA, however other users may have the log-in box in the top left, in which case the text Rank/Name will appear bottom right away from the rest of the log-in box. The NZDF were happy for the contribution, I in fact use the patch PBO on my own personal copy of VBS2 and handed off a copy to another customer organisation for them to consider using.

Early on in the development process it must be decided who forms the target audience and ideally they should be notified of the content in development. The target audience should include people and organisations who you regularly train with via simulation. For example, if I was to create an addon in my own time for the gaming community, I would distribute the addon to them. The reason for this is simple, interoperability, the other customer will see what I'm seeing and be able to benefit from my contribution, without that addon, at the least VBS2 would give errors, and potentially a target indication to a coalition asset would go unseen by those who don't have the addon.

Another benefit of early coordination with other customers is bug reports and feedback guiding the development process. You can from bug reports adjust your methods to suit all target audiences, even put measures in place to switch features on or off based on the user version, and HASP key code. Additionally if another developer has an idea they may share information with you and give an example of how to go about it within your current code base. Take the BIS example, consider each tester as a military organisation on its own SOE, now take the additional customers as other organisations, again on their own SOE. The more variations on the operating environment you encounter, the more problems you will see. Potentially a lot of CTD bugs are from Windows 7, both release candidate and beta users, in this instance it is unfair to expect BIS to build on this shaky foundation. If you're an ArmA 2 fan running W7 and experiencing crashes, I suggest you roll back to XP or Vista. W7 is not final, neither is the RC, if BIS made a patch for the current builds, it's likely in the next iteration they would have to release yet another patch, quickly turning A2 into bloatware.

Just a few points for you to consider, it's for the reasons of interop some of my works have been delayed, pending appropriate testing. Anyone who wants to test on behalf of a customer organisation or as a serious gamer should contact me at their earliest conveniance.

Jamie.