Technical Solutions
Standard industry best-practices were able to help game developers prevent and detect many errors that C/C++ developers commonly encounter. Practices such as coding standard enforcement, unit testing, runtime error detection, and regression testing help the developers - ranging from inexperienced to expert - produce better code. With these practices, many errors are prevented, and the errors that were introduced are rooted out as early as possible - when they are fastest, easiest, and cheapest to fix.

Considering the tight timelines that are characteristic of the gaming industry, it is critical that attempts to improve quality do not impact the gaming developers' already busy schedule or disrupt the creativity that is critical to keeping ahead of competitors. That's why it's critical to automate these practices. For minimal disruption, the practices are configured to run behind the scenes - for instance, overnight, as part of an automated build process - then alert developers only if a problem is found. Much testing can be done without user intervention, by scanning the code and executing it with automatically-generated test cases. Of course, the more effort developers put into the testing, the greater benefit they get out of it. For instance, to improve code coverage, developers could analyze the coverage achieved by automatically-generated unit tests, then extend these tests to cover a larger percentage of the code and/or to verify specific functionality requirements.

Automating the implementation of these key industry practices helps the gaming developers overcome many of the challenges associated with producing top-quality software in a high-pressure environment and working on teams that have many inexperienced developers. However, it does not help them with one of their toughest tasks: correctly interacting with graphics libraries.

As I mentioned earlier, there are two main graphics libraries used by game developers and other developers creating high-resolution graphics: OpenGL and DirectX. OpenGL, introduced by SGI in 1992, is the most widely-used and supported interface for realistic 2-D and 3-D high-resolution graphics. It is an open, vendor-neutral graphics standard that works on a wide variety of platforms, including Linux. DirectX is a group of technologies for running and displaying multimedia applications on Microsoft Windows and Xbox. We found that the best way to prevent errors related to library misuse was to customize the coding standards analysis practice and technologies to automatically check OpenGL and DirectX rules as well as the standard C/C++ rules. The basic set of rules, based on the guidelines set forth in the libraries' specifications, includes over 30 rules for avoiding pitfalls with OpenGL and DirectX API development. DirectX rules checked include:

• Use the macros SUCCEEDED() and FAILED() to check if a DirectX function has failed
• Include stdafx.h at the beginning of the file
• Use D3DVALUE instead of float
• Effects should be closed (End()) in the reverse order in which they were opened (Begin())
• Use Pass() between Begin() and End()
• Don't call other functions in the same line as a Begin call
• Don't call other functions in the same line as an End call
• Don't change Effect ::Technique between Begin() and End(); call End() first
• Don't use #include more than 10 times
• Use an equal number of lock calls and unlock calls
• Don't use exclusive mode
• Don't call BeginScene() twice without first calling EndScene()

OpenGL rules checked include:

• Use the appropriate number and order of "Begin" and "End" functions calls
• Use the appropriate number and order of "NewList" and "EndList" functions calls
• Use GL commands between a Begin/End pair
• Use GL commands between a NewList/EndList pair
• Don't use designated functions in Begin/End blocks
• Don't use designated functions in NewList/EndList blocks
• Don't use designated functions outside Begin/End blocks
• Don't use an End block without a Begin in each NewList/EndList block
• Don't use forbidden bracket commands between a Begin/End pair
• Don't use forbidden bracket commands between a NewList//EndList pair
• Use only GL functions between every Begin/End block
• After every Begin(GL_LINES), use the count of "vertex" function calls divided by 2
• After every Begin(GL_TRIANGLES), use the count of "vertex" function calls divided by 3
• After every Begin(GL_QUADS), use the count of "vertex" function calls divided by 4
• After every Begin(GL_POLYGON), use more than 4 vertexes
• Don't use the function ÔLoadMatrix' to initialize a matrix
• Don't use the function ÔMultMatrix' to change a matrix
• Don't use negative vertex and texture coordinates
• Don't use more than one GL command in a single line
• Don't use more than five levels of functions calls

This set of rules can be extended with rules that check additional guidelines, such as the guidelines for avoiding Open GL pitfalls described in Mark Kligard's "Avoiding 16 OpenGL Pitfalls (www.opengl.org/resources/features/KilgardTechniques/oglpitfall/), as well as guidelines in the growing set of resources for these libraries. Most gaming developers are aware of these guidelines. However, due to the extreme working conditions and time constraints mentioned earlier, they sometimes make mistakes and they simply don't have the time to verify that the guidelines are followed.

In addition, we provided the game development organization with a technology that allowed them to design and check custom rules that verify compliance with additional guidelines. These custom rules could be used to check additional OpenGL or DirectX guidelines that the team decides are helpful, check guidelines for custom graphics libraries, and check additional requirements unique to their organization, technologies, projects, etc.

Process Improvement Solutions
Truly implementing a practice in a team requires not just the appropriate tools, but also the team culture, workflow, and supporting infrastructure required to embed the practice into the team's development process.Teams that attempt to implement practices with tools alone typically do not achieve the expected quality improvement benefits. For instance, assume that a team tries to implement the coding standards enforcement practice by only purchasing a coding standards enforcement tool and asking each developer to use that tool. Over time, it's likely that most of the coding standard violations will remain in the code. Why? Without additional team-wide support for the coding standards enforcement practice, developers typically become overwhelmed by the number of problems reported and do not know how to approach them. The tool helps the team members recognize the faults in their code, but if the developers do not have the necessary support, the faults remain and the code quality does not significantly improve.

The Parasoft AEP Methodology details one strategy for embedding best-practices into a team's development process. In a nutshell, the AEP Methodology is a new methodology for improving software quality and increasing the efficiency of a team's software development lifecycle. It is based on the AEP Concept, which is essentially to learn from your own mistakes and the mistakes of others, and then automatically apply that knowledge in the software lifecycle to make software work. The basic principles of the AEP methodology are:

1. Apply industry best-practices to prevent common errors and establish a foundation for full-lifecycle error prevention.
2. Modify practices as needed to prevent unique errors.
3. Ensure that each group implements AEP correctly and consistently.
   a. Introduce AEP on a group-by-group basis.
   b. Ensure that each group has an appropriate supporting infrastructure.
   c. Implement a group workflow that ensures error prevention practices are performed appropriately.
   
4. Phase in each practice incrementally.
5. Use statistics to stabilize each process, and then make it capable.

For a detailed discussion of how this methodology works, see http://www.parasoft.com.

By applying the previously mentioned best-practices within the AEP methodology, gaming development organizations gain the following benefits:

• Higher quality: Fewer errors are introduced into the code, and introduced errors are identified and fixed early in the cycle (when fixing them is generally less difficult, time-consuming, and costly).
• Fewer schedule slips, faster time to release: Because of the improved error prevention and error detection, less time is required for end-of-cycle debugging, and games are more likely to pass the independent validation in the first attempt.
• Easier, faster software updates: Team code reflects a standard style, not individual preferences, and meets a predetermined quality standard. When an organization wants to enhance or extend popular game, misunderstandings/buried errors don't impede the process.
• More proactive management: Increased visibility into code quality, test scope, project readiness, and team productivity helps management identify problems as they arise and start resolving them as soon as possible.

Learning From the Game Development Industry
Even if you're not one of the relatively few developers working on games for Linux, you can still benefit from the lessons we learned by working with gaming organizations.

There is a growing tendency to use Linux for high-resolution graphic development, which shares many of the same challenges as game development. As many industries working on high-resolution graphics are looking to move from SGI systems, they are finding that porting legacy code to Linux is much faster and cost-effective than porting it to Windows. Linux is already emerging as the platform of choice for developing and running graphical work for animation projects (including major projects at DreamWorks, Pixar, and Sony Pictures), special effects, and film production. In addition, Linux is becoming a popular operating system for other high-resolution graphics applications, such as computer aided design, medical devices, and geographical imaging systems.

Developers in these industries face many of the same challenges as developers in the gaming industry. For instance, developers working on high resolution graphics in any of these industries are all too familiar with crazy deadlines, long hours, low tolerance for graphical errors or other errors, and the need to master standard graphics libraries (for Linux, OpenGL) and/or custom graphics libraries.

Consequently, the same technical solutions and process improvement solutions that help developers in the gaming industry can help developers working on other high-resolution graphic projects - or even in other industries with similar pressures and development environments. The automation of practices such as coding standards analysis, unit testing, runtime error detection, and regression testing helps ensure that common coding problems don't delay intense production schedules. To verify compliance with guidelines for standard graphical libraries, guidelines for other technologies that the application interacts with, and unique organizational or project requirements, the coding standards analysis practice can be extended to check custom rules. And, to ensure that these practices become an enduring and seamless part of the development process, they can be implemented within the AEP framework.