Remote usability testing has been around for quite a long time. Early studies were carried out about 25 year ago in the mid-nineties [1] [2] [3] and has since evolved significantly. While the 00’ies witnessed many academic studies and methods, this is particularly evident over the recent decade with the arrival of numerous commercial tools and platforms for automated remote testing. However, the Covid-19 pandemic has put an even greater focus on remote work and the paradigm is more relevant now than ever before. The basic idea of remote usability testing is that a test person need not be physically present at a test facility in order to perform a usability test. Instead, s/he accesses a product, a service or maybe a prototype via a web browser or an app on a computer or a mobile device. In some cases, a test facilitator may be (remotely) available to guide the user though the test scenarios, debriefing, etc. This is called supervised remote testing. A more attractive perspective is to carry out the remote test unsupervised. This will in part save manpower and in part make the logistics of the test much easier as the test participant can perform the test any time and place s/he wishes. Unsupervised tests tend to mostly collect quantitative data, whereas supervised tests can capture a larger degree of qualitative data.
- Supervised Remote Testing
Supervised testing is also referred to as synchronous testing, as it requires the test participants and the facilitator to be present at the same time. Supervised testing was described by e.g. Hartson and colleagues in 1995 [4]. They discussed various possibilities for remote usability testing and made a comparison to a lab-based test. 10 years later, Brush et al. investigated differences in the participants’ and facilitators’ qualitative experience between lab and remote tests [5]. They had assumed that “participants would be more comfortable talking to the facilitator and would find it easier to think aloud and concentrate on tasks in the local condition”. Contrary to the assumption, they found that most participants preferred the convenience of the remote testing approach. The findings suggested the remote test participants contributed to the results as much as in the lab-based test [5]. A special case is remotely supervised testing. This could e.g. be when the participant is a with special needs, such as children, persons with physical or cognitive impairments, the elderly, or those not comfortable with the digital solutions surrounding us? In such cases a local “moderator”, typically from the participant’s household may be recruited to help execute the test.
- Unsupervised Remote Testing
Unsupervised usability testing is also referred to as asynchronous testing, as it does not require the test participants and the facilitator to be present at the same time. Bruun et al [6] compared three different methods in an unsupervised remote testing scenario: user-reported critical incidents; forum-based online reporting and diary-based user reporting in 2009. They found that that the unsupervised testing actually performed significantly worse compared to corresponding lab-based usability tests. However, unsupervised remote testing has gained a large foothold during recent years, due to its ease of deployment and corresponding cost-efficiency.
- Remote Testing Platforms
A number of companies have built platforms to facilitate remote usability testing over the recent decade. One such platform supplier is the Danish company Preely [7]. The platform lets the interaction designer develop prototypes using a variety of tools, such as Sketch, Figma, Invision or Adobe XD [8] [9] [10] [11] and then deploy them for test via the platform. Preely will import and execute the prototype and do all the bookkeeping and logging of user test data etc. for any number of test participants.
Other similar platforms are online services such as: Maze, UserTesting and UseBerry [12] [13] [14]. Similarly, consultancy services, like UserTribe, UserZoom and LookBack offer supervised remote tests [15] [16] [17]. These platforms are insight-based and qualitative and demands more resources post-test to make the analysis.
- Remote testing of physical products
However, the present state-of-the-art of remote usability testing methods and platforms are restricted to software products. At the present time it is relatively straightforward to distribute software products, but if the product being tested is a physical device or a service, a number of new issues arise. Many have to do with the costs and logistics of distributing and later collecting the test products, but others are directly concerned with the test session itself.
References:
- Castillo, J. C., Hartson, H. R., & Hix, D. (1997). Remote Usability Evaluation at a Glance* (Tech. Rep.). doi: https://10.1145/286498.286736
- Hammontree, M. L., Weiler, P., and Nayak, N. P. (1994). Remote usability testing. Interactions, 1, 21-25.
- Hartson,H.R., Castillo, J.C.,Kelso, J., Neale, W.C. and Kamler, J. (1996). Remote evaluation: the network as an extension of the usability laboratory. In Conference on human factors in computing systems – proceedings (pp. 228–235). New York, New York, USA: ACM Press. doi: http://10.1145/238386.238511
- Dray, Susan and Siegel, David.(2004). Remote possibilities? International usability testing at a distance. Interactions. 11. 10-17.
- Bernheim Brush, A.J., Ames, M., and Davis, J. (2004). A comparison of synchronous remote and local usability studies for an expert interface. Conference on Human Factors in Computing Systems – Proceedings (April 2015), 1179–1182. doi: https://10.1145/985921.986018
- Bruun, A., Gull, P., Hofmeister, L., and Stage, J. (2009). Let your users do the testing: A comparison of three remote asynchronous usability testing methods. In Conference on human factors in computing systems – proceedings (pp. 1619–1628). New York: ACM Press. doi: https://10.1145/1518701.1518948
- Preely: https://preely.com
- Sketch: https://www.sketch.com/
- Figma: https://www.figma.com/
- InVision: https://www.invisionapp.com/
- AdobeXD: https://www.adobe.com/products/xd.html
- Maze: https://maze.co/
- UserTesting: https://www.usertesting.com/
- UseBerry: https://www.useberry.com/
- UserTribe: https://usertribe.com/
- UserZoom: https://www.userzoom.com/
- LookBack: https://lookback.io/