The client has a strong reputation for innovative and ground-breaking work. They have won an abundance of awards, and received global critical acclaim. Being a frontrunner in VR games, the client wanted to launch their VR title at the same time as new VR hardware.

The company places a strong emphasis on functional quality assurance. To give their new products the best chance of success, they required testing for the following features:

• All aspects of game vehicle functionality including, but not limited to, graphical and collision, audio, VFX, mechanics and general playability
• Game environments
• New features
• Menu functionality
• Network matchmaking
• All aspects of virtual reality integration including visuals, head tracking and frame rates

As a VR testing project, this presented a variety of interesting challenges to PTW’s functional testing team, including:

• Ensuring hardware met required specifications.
• Experiencing and dealing with cybersickness, and neck and back problems, which were intensified when using older developer kit versions as part of the testing array of VR head mounted display units.
• Allowing for frequent breaks in immersion.
• The title was initially not feature-complete, which created delays in issue resolution.
• Resource allocation within the team was determined by the device scheduling and functionality.

Working in close partnership with our client, we were able to deliver bespoke VR Quality Assurance services. Our functional testing team worked as an extension of the client’s internal teams, reporting directly to the company’s internal Quality Assurance team.

Key areas of working ensured the success of the projects. These included:

Training
The team members went through self-learning and training programs to give them a full understanding of the products and hardware. This included extensive VR immersion tests to allow the team to become familiarized with prolonged periods of VR testing, and establish the optimum duration for screen breaks, down time and recovery.

Risk Assessment
Risk assessments were carried out prior to assigning staff. These considerations included establishing if any of the testers had a history of migraines, eye conditions, epilepsy, neck or back issues. We also looked at other situational factors, both medical and non-medical.

Testing Environment
We created a specific VR testing environment with a larger space for movement, light blockers for direct sunlight, and well-organized placement of equipment and cables.

Test Design and Execution
Early on in the project, the team adopted an exploratory testing method. Each feature was then tested before it was added to the title specification, focusing on stability and connectivity issues when multiplayer games were accessible.

As the game evolved, the team had to fragment the game into different sections to enable them to complete thorough testing.
Towards the end of the project, the team were able to formalize testing by breaking the different features of the game into test cases. Our team create the test cases, in collaboration with the client. They also began developing methods that had never previously been used on projects involving an outsource team. These included using Visual Studio to open and analyze crash dumps to assist the development team in determining the cause. Once the defects were fixed, they were re-tested and regressed around the fixes to ensure that there were no breaks created during fixes.

Client Reviews and Feedback
Periodic reviews and regular customer feedback were in place to ensure that the program was on track and the client was satisfied with PTW's work.
As our team reported directly to the client's internal Quality Assurance team, and we operated as a fully aligned partnership, the lines of communication were refreshingly open.

A great result
The meticulous testing, supported by robust test processes and continuous improvement, helped in achieving exceptionally high quality deliverables.

The functional testing team discovered more than 2,000 bugs, in addition to bugs discovered by the client's in-house teams.

Some of the technology and tools we utilized included:

• Selenium
• Java
• C#
• Appium Studio
• Jira
• SharePoint
• J-Meter

The success of the project results in a number of positives, including:

• Delivery of a cost-effective solution, with higher Return on Investment (ROI).
• Automation testing that reduced 80% of effort and time spent by the manual testing team.
• Achieving the same level of compliance on all the supported environments.
• Increasing confidence on release to the client’s target consumer markets, thanks to high quality releases and greater test coverage
• Helping the client with performance testing and analysis to optimize the application performance

Outsourcing quality assurance also reduced the client's staffing overhead, including measures to cover shift rotations through overtime, particularly where there were workflow or scheduling bottlenecks. This method of working also freed up the client's in-house teams, allowing them to divert to priority tasks elsewhere on the project.