In the creation of high functioning test environments that work for both VR testers and game developers, there are real physical challenges that testers must overcome. And while virtual reality keeps evolving, the testing environment needs to evolve too.

We take a closer look at the very real challenges of VR testing.

Since 2016, VR has been boldly breaking through into the mainstream. But for those involved in VR hardware, software, and content development, the revolution started a lot earlier.

As more tech and content companies strive to get on board the VR express, the need for well-developed functional and localization VR quality assurance has been steadily rising. Functional quality assurance testers are experts in adapting to the latest technology, and have always been able to apply their skills and experience to the latest innovations. With the launch of VR, games testing teams have been presented with some unique and difficult challenges.

It quickly became clear that effective planning, risk assessment, and creating the right test environment were essential to tackle the difficulties inherent in VR testing, and to protect the safety of testers. The physical challenges of VR have been widely publicized, and developers continue to work to overcome them. Among these challenges are hardware issues, software issues, and the physical susceptibilities of individual users.

For testers, increased exposure to VR can mean that the physical challenges are heightened.

VR games and hardware are not designed for the prolonged exposure that testers endure. Furthermore, as they are often testing initial and early versions of games or software, even the bugs themselves can cause adverse physical reactions. This is also true of the hardware that is used. Early versions of head-mounted displays had not evolved much further than the initial prototypes – the process of design refinement, aimed at comfort and enhanced user experiences, had simply not yet occurred.

Early Discoveries, Insights and Knowledge

Discomfort, calibration errors, and tracking latency hardware issues are intensified in VR. These challenges revealed unique early discoveries in VR quality assurance, and helped create a valuable body of specialist insights and knowledge. We’ll look at each condition in turn, and how it impacts on VR testing.

The Reality of Cybersickness

Cybersickness is caused when the body’s visual system registers movement, but no movement is registered in the vestibular system. (The vestibular system provides sensory information about motion, equilibrium and spatial orientation.) Subsequently, this creates a mismatch in the autonomic nervous system.

In VR, the senses are presented with the illusion of movement and, although the user may not remain still, the movement does not always correlate to what they are seeing. The more realistic the visual content, the increased likelihood of mismatch and the greater the risk of experiencing cybersickness.

In addition, particular bugs can create obvious mismatches between sight and motion. For example, if the player is looking ahead and their avatar is remaining still but the presented visual cues are of a moving landscape, this will create a clear mismatch. A higher field of view is also linked to increased cybersickness through changes and motions of optical flow. The flip side of this is that an increased field of view also connects positively with increased immersion. (Source)

There have been numerous studies into whether demographics contribute to cybersickness susceptibility. These include studies on ethnicity, age, spontaneous postural sway during natural stance (the degree of biomechanical action required to correct balance), flicker fusion frequency threshold (the degree to which an intermittent light stimulus appears steady to the average human observer), and gender. (Source)

There are no definitive conclusions to suggest that people of a particular demographic should not use VR, so consideration should still be on an individual basis. However, it’s important to stay aware of any new and developing studies.

Sometimes the phrase ‘simulator sickness’ is used in relation to VR. Simulator sickness and cybersickness are both strands of motion sickness, but they are not the same. (Source) With simulator sickness, a person will experience symptoms in a different order from cybersickness. It’s also worth noting that the severity of cybersickness was found to be approximately three times greater than that of simulator sickness.

So, what exactly is cybersickness? The intensity may vary from person to person, but the symptoms of cybersickness can include:

• Disorientation
• Dizziness
• Drowsiness
• Stomach awareness
• Headaches
• Nausea
• Vomiting
• Fatigue

Vergence-Accommodation Conflict

VR users can experience pain and discomfort in the eyes due to vergence-accommodation conflict. Testers are at a higher risk of this, due to their increased exposure to VR on a daily basis.

The key to understanding vergence-accommodation conflict is to consider the focal point of the eyes. Our eyes operate by focusing and converging on a point in space. This distance is the same for both eyes, and the brain combines this response into what is known as vergence-accommodation coupling. VR causes the focus and vergence points to be different. The eyes focus on the display screen, but the vergence distance is on the VR environment. Subsequently, this means the brain has to act against its natural coupling instinct and this can cause eye strain. (Source)

PTW has found that eye discomfort is one of the most regularly complained about symptoms of VR testing.

Seizures and Coordination

There have been concerns that VR increases the likelihood of seizures. This could depend on the design of the game being played, or the software being used - although further studies may be required.

Oculus Rift’s health and safety manual advised that about 1 in 4000 VR users may experience severe dizziness, seizures, epileptic seizures or blackouts even if they have no history of it. They also warn that prolonged use could negatively impact hand-eye coordination, balance and multi-tasking ability. (Source)

Upper Body Pain

VR can include a variety of hardware, many of which are commonly used, such as Head Mounted Displays (HMDs). HMDs come in different designs and dimensions, and there is a wide range of styles and designs on the market. The weight of HMDs can affect the user, with heavier HMDs causing greater discomfort in a shorter period of time. The movement involved when playing, combined with the use of HMDs and input devices, can result in neck, back, and even arm or hand aches. This type of Repetitive Stress Injury worsens with repeated, regular use, such as in a testing environment.

The design of HMDs varies widely, but some can cause difficulties for people who wear glasses, as the positioning of the head straps and the pressure on spectacle frames can cause discomfort across the nose and head.

VR-Specific Bugs

Accounting for 3-5% of overall bug volumes, some bugs are specific to VR games and VR experiences and can intensify all of the physical conditions we’ve described already. These could be development choices which work well in 2D environments, but cause specific physical challenges in VR.

As an example, the risk of cybersickness can increase due to specific game features - such as if a game movement occurs that the player hasn’t triggered. This might include an unexpected cutscene using the player’s camera, or if the game movement occurs too quickly, such as catapulting or transferring the player from one point to another without audio or visual cues. This can cause a change in orientation towards a specific plane, which in turn can cause significant disorientation.

Another good example involves games which use different methods of acceleration. These can result in the field of view moving erratically. As this cannot be controlled by the player, they have no focal point and are subsequently at risk of extreme disorientation.

In-game visual effects, particularly those requiring significant machine processing power, can slow down the frame-rate, causing visual ‘stutters’ and distractions. These interrupt the VR immersion experience, and can occur with even the smallest dip in frame-rate. Similarly, any rendering latency can disorientate and disrupt. Bright flashes in a game can be significantly magnified by head mounted displays, resulting in head and eye pain.

The user interface (UI) should always be within comfortable reach of the player, without them having to exert themselves, or twist uncomfortably. UIs which are held in the player’s peripheral vision, or obscured by in-game objects, can cause strain when they’re accessed regularly.

Before testing any new software or equipment, established and reputable testing organizations will have procedures in place and will conduct a variety of workplace assessments.

However, as virtual reality presents greater risks for testers, robust screening, planning, and preparation processes must be in place, and adhered to.

It’s essential to take medical history into account when assigning testers to work on VR. This is because there are some predispositions and conditions which are more likely to cause an adverse reaction. However, it is important to note that the conditions may not necessarily prevent the person from conducting VR testing. As with all medical conditions, testers should be advised to consult their physician for advice.

Motion sickness

As cybersickness is a form of motion sickness, people with a history of motion sickness are more likely to experience similar feelings when using VR.

Inner ear infections

Inner ear infections, like labyrinthitis, can cause dizziness and balance issues. Anyone who has an inner ear infection may be unsuitable for testing VR products.

Eye conditions

Binocular vision abnormalities may be incompatible with VR use. These include conditions where the sight from one eye does not align with the sight of the other eye. The eyes are then under strain to realign to a single focus. The existing strain on the eye combined with the difficulty in achieving a single focus mean that testing VR is unsuitable for people with binocular vision abnormalities.

It’s still important for VR Testers without pre-existing eye conditions to have regular checks on their eyes during testing. Questions such as ‘Do you feel your eyes are heavy or tired?’ ‘Do you feel your eyes are straining?’ ‘How is your vision?’ should be asked regularly during breaks.

These questions need to be incorporated as part of written safety checks, as reminders to self-check, and as timed verbal questions. Regular eye examinations are also recommended.

Medical Devices

Some VR hardware may emit radio waves which could interfere with the normal operations of implanted medical devices, such as pacemakers.

Epilepsy

The increased risk of seizures while using VR equipment may mean that testing is not suitable for people with epilepsy, or symptoms of epilepsy.

Upper Limb Disorders

People with upper limb disorders may find their symptoms become worse from the prolonged VR exposure that testers experience.

By spending increased amounts of time in VR environments, experienced testers can build up a tolerance. Although tolerances are individual to each tester, they are easier to build through frequent short periods of usage rather than prolonged, longer periods of usage.

VR requires greater considering when it comes to time planning, task scheduling, and the variables of project management.

Testers are unable to test for 40 hours per week in a virtual environment, or while wearing VR headsets. This needs to be accounted for in initial project scheduling. When planning testing, aim for 50-75% of productive hours compared with normal test plans, and always start at the lower end of this range. It’s also important to factor in regular, frequent breaks for VR testers - particularly when they are starting out.

Bear in mind when scheduling that most bug regression can be done in 2D environments.

It’s vital for any organization involved in VR testing to assess the physical testing environment first. The key question that needs to be asked is ‘Do the current facilities have the capabilities to create a high standard VR testing environment – one that will provide a safe and secure environment for testing operations?’ Here are some guidelines that we believe it’s important to work to.

With so many aspects to consider, it’s essential to create a dedicated VR environment. This should incorporate a larger desk area than normal, so allow for a 15 -20% increase in size. An expansive, clear space for movement while playing is crucial. This area should be clearly signed and well maintained, so that there are no disturbances or hazards that other people can introduce into the environment.

Cables and equipment should be well organized so there are no dangers in the testing area. As direct sunlight can affect the tester, and also burn out VR lenses, light blockers are vital to provide maximum comfort for testers, and also protect the testing equipment.

It’s important to be aware that when using multiple VR devices, infrared sensors can ‘cross over’ if kits are too close together. This can cause ‘ghosting’, with two images being presented in a single image to the tester. Not only is this disorientating for the tester, it can also create false bugs which can directly impact on the project.

It cannot be over-emphasized that staff must feel comfortable when testing VR. Asking testers to immerse themselves in a virtual world means they are functionally blind in the real world, and therefore vulnerable. There should always be someone to supervise and check that the tester is comfortable, feels well, and that the testing environment is suitable.

The tester needs to be comfortable with the people around them, and feel confident that nobody will try to interfere or cause disruption to the experience. This is especially important as testers have to experience immersion to fully carry out their role.

Full briefing and training for new staff on VR is a priority. This should include the importance of team support, and a warning on the dangers of interfering with infrared sensors.

Health and safety laws and advice vary from country to country. Most guidelines advise employers to identify any hazards, identify the risk from the hazard, and implement measures to control that risk.

Generally, the responsibility for health and safety practices are on both the employer and the employee.

In order to be compliant with health and safety, everyone involved in VR testing needs to:

• keep abreast of any changes in law
• comply with the safety advice given by each platform
• note the risks, and implement measures to reduce those risks
• ensure staff are aware of the implemented measures
• do not put staff into an unknown situation, or ask them to use new devices without carrying out a full risk assessment first

Virtual Reality has undoubtedly been one of the biggest talking points in technology in recent years, and has already left its mark on quality assurance.

There are increasing demands for VR testing, and this looks set to continue. Organizations who want to get involved in testing must be fully prepared to have dedicated VR processes in place. This keeps projects on schedule, protects the safety of testing staff, and guarantees that the results the high standards that are expected.

With over three years of VR testing experience, beginning in early 2014, PTW recommends the following best practices:

• Consider the medical history of staff before assigning them to a VR project. Consult with medical staff for recommendations and best practices.
• Tolerances for virtual environments are on an individual basis. Generally, they will increase with experience.
• A custom-built VR testing environment is essential.
• Close team supervision and staff briefings will help testers to feel safe in VR.

VR content owners who are looking for a VR testing firm should make sure that their potential testing partner has taken all of the above into consideration, and can demonstrate vigilance in each area.

Game developers do not want to work with a company which has not given full and thorough consideration to staff safety, and the potential implications on the outcome of the overall project.

Finally, as VR is continually developing and advancing, we believe that best practices, operational documentation, procedures, physiological considerations and physical environment recommendations for VR testing need to keep pace. This means they must be regularly assessed and refined. New devices may bring new challenges, and, as committed professionals, we must assess and adapt to them in the most timely, safe and skillful way.