Quantity over quality: Do single-item scales reflect what users truly experienced?

Single-item scales are widely used in the field of user experience to report the emotions. However, they are strongly criticized and discouraged by the scientific community. While they have several practical advantages, single-item scales are mainly criticized for their psychometric weakness. Our research explores to what extent single-item scales reflect what a user has experienced during his or her interaction with technology, overall, but also at specific moments such as first and last impressions. This research also explores the sensitivity of these measures across different contexts of use brought by the presence or absence of certain interface features. We conducted a correlational study with 40 users while interacting with financial institution websites. We used two methods to evaluate the experience: lived and measured implicitly using psychophysiological instruments on one hand and self-perceived and measured using single-item scales, on the other hand. Overall, our results suggest limited correlations and in most cases contradictory between lived and perceived experience. We conclude by highlighting the limits to be taken into account when professionals are led to use single-item scales.

FM3 Literature Claustrum, #Paper

Annotations

(21/10/2023, 15:02:59 )

“Our study has a main managerial implication: it highlights the limit of single-item scales” (Cuvillier et al., 2021, p. 10)

“The only measure that seems to be truly related to what a person experienced during the interaction is perceived arousal.” (Cuvillier et al., 2021, p. 10)

“Our research explores to what extent single-item scales reﬂect what a user has experienced during his or her interaction with technology, overall, but also at speciﬁc moments such as ﬁrst and last impressions.” (Cuvillier et al., 2021, p. 1)

Valence background (Basic Emotions Theory)

“Valence: Six emotional states are commonly accepted as universal (Ekman & Friesen, 2003): joy, sadness, surprise, fear, anger and disgust. The research usually uses the construct “valence”, which is the result of” (Cuvillier et al., 2021, p. 2)

Definition of Valence as unidirectional construct

“the difference between positive and negative emotional states. Thus, Valence can be deﬁned as: “… the direction of behavioral activation and the degree of positive (toward) or negative (away from) emotion for a stimulus” (Seo, Lee, Chung, & Park, 2015 – p. 73–74). This construct is thus lived, through our body and manifests itself, in particular through our facial muscles (Lang, Greenwald, Bradley, & Hamm, 1993). Valence is one of the dimensions of the emotions (Herbon, Peter, Markert, & Meer, 2005), and thus can be viewed as a unidimensional construct.” (Cuvillier et al., 2021, p. 2)

Emotional Arousal strongly linked to Valence (see )

“Emotional arousal: This corresponds to the strength of the expression of an emotion. This construct is also lived within us and is expressed, in part, by the activation of our sweat glands. It is very strongly linked to valence. It is common in research to associate them in the same circumplex pattern with, on the one hand, level of pleasure and activation on the other hand (calm or excited) (Russel, 1980)."

"Arousal seems to be a unidimensional construct” often measured with the affective slider

“The arousal is another dimension of the emotions (Herbon & al., 2005), and seems to be a unidimensional construct. It is common to measure these last constructs (valence and arousal) by using the Affective Slider (Betella & Verschure, 2016). In this scale, respondents are asked to indicate their perceived level of enjoyment and perceived level of arousal by dragging a slider. Responses usually range from 1 to 100.” (Cuvillier et al., 2021, p. 2)

!Bild 21-10-2023.png

Perceived Arousal and Cognitive load to be consistent with implicit measures (but correlation is supprisingly low, ~.35)

“At the overall level, we found that the level of perceived arousal and the level of perceived cognitive load appear to be consistent with their direct implicit measures.” (Cuvillier et al., 2021, p. 9)

“Perceived arousal seems to be in relationship with both implicit ways of measuring arousal: EDA and ECG, respectively visible in cells. 5 (0.3771; p ¼ 0.04) and 6 (0.3655; p ¼ 0.0432).”

(Some) correspondances between measuring self reported arousal and arousal measured by Electrodermal Activity (EDA) and Heart Rate

“Ortiz De Guinea et al. (2013) had observed certain correspondences between different ways of measuring their constructs. In particular, they highlighted certain relationships between self-reported arousal and arousal measured by electrodermal activity and heart rate. This relationship exists both considering the overall experience (EDA) and at ﬁrst impression level (EDA and HR). This last observation seems to converge with the observations of Lourties & al. (2018). They had tried to ﬁnd out whether the continuous emotional reporting using a single-item scale was subject to primacy and recency effects. They reported ﬁnding some inﬂuences of these effects: a primacy effect on valence and a recency effect on emotional arousal (Lourties & al., 2018).” (Cuvillier et al., 2021, p. 9) check first paper: correspondance between physiological and other measures

Primacy-Effect on CLA valence and Recency effect on emotional CLA arousal

“Our data behaved in part the same way, for emotional arousal. When taking website features into account, we can see that these relationships change, and a last impression effect appears on the relationship between perceived arousal and lived arousal (EDA) when the experience is lived throughout hedonic features. Regarding valence, there also seems to be a serial position effect in our data, although it is not differentiated between ﬁrst and last impressions. This difference could come from the way we measured perceptions: at the end of the interaction.” (Cuvillier et al., 2021, p. 9)

Issue: psychophysiological inferences (arousal Electrodermal Activity (EDA), arousal ECG)

“In this debate, the main issue remains that psychophysiological inferences (e.g., valence, arousal EDA, arousal ECG and cognitive load) are longitudinal measures that capture a phenomenon over a certain time period at a speciﬁc sampling rate. Therefore, those psychophysiological inferences are signals that ﬂuctuate throughout the measurement period with a certain amplitude and latency.” (Cuvillier et al., 2021, p. 10)

Access Required

Quantity over quality: Do single-item scales reflect what users truly experienced?