Entertain me!

Editor’s note: Stefan
Althoff is team manager, market research at Lufthansa Technik, a
Hamburg, Germany, subsidiary of Lufthansa German Airlines. Bill
MacElroy is president of Socratic Technologies Inc., a San Francisco
research firm.

Why do people surf the
Internet? Many go to hunt for information, to shop or do online
banking. In the end, the search for entertainment – in the widest sense
– is often most important. Online games or sites like YouTube are
popular. To meet other people, even in a cyber sense, is another
motivation. The value of entertainment extends to the ad realm, where
ads that are perceived as funny are often judged to be more effective
than those that don’t elicit laughs.

What motivates people to
participate in a research study? A kind-looking interviewer at the
front door, a well-dressed and polite mall researcher or a telephone
interviewer with a pleasant voice can convince almost anyone to
complete a survey.

Of course, this is different
in the online market research world because of the self-administration
aspect: The e-mail invitation and the questionnaire must motivate the
potential participant. This is the same with registered online panel
members: The registration in the panel and the general interview
willingness are not enough.

Online research is becoming
more and more popular worldwide as a channel for gathering market data
and customer feedback. More than 50 percent of all interview projects
in the U.S. are conducted online. Online research has lost the aura of
novelty; therefore it is no surprise that response rates have dropped
since its introduction in the mid-1990s. In addition, the number of
people who start Web surveys only to abandon them midway has also grown
significantly.

As a consequence, the main
target in the analysis of non-response has shifted. In the past, the
focus was on the interviewers and what they could do to increase
response rates. Now the interviewees themselves and their motivation
for participation are the target of multiple analyses.

Many of the techniques to
increase response rates for Web surveys are not rocket science. The
so-called “Anita effect” (“Does the survey sender’s gender matter?” Quirk’s,
February 2007) is an example for a simple method: The response rate –
especially in a male-dominated environment – is increased by using an
obviously female sender in the invitation e-mail. Furthermore, there
are strategies to make the experience of interviewee as pleasant as
possible, e.g., with the PRD technique (“Using the PRD technique
online” Quirk’s, January 2008) by making the recall of the
previous assessments easier (a finding of Lufthansa Technik Market
Research, proven by a comparison study by Socratic Technologies).

In addition, there is much
known about the most effective strategies for reaching the best
potential participants in online research. The control of the e-mail
timing alone can influence the success of a study. It can be of
importance which day of the week the invitation e-mails are sent out,
and that also makes a difference whether you are inviting respondents
to a business-to-business or a business-to-consumer online survey.

The degree of personalization
within the invitation mail and the survey itself can be of importance:
For a B2B survey it is better to invite on behalf of a well-known
counterpart instead of a CEO who people only know by name. This is
similar to the problem with invitations automatically signed by the
company president. Lufthansa Technik found that the response rates are
significantly higher in more personalized survey designs: The average
completion rate rose from around 30 percent to 40 percent.

Number of options

The online medium offers a
number of design options. Since so many people use the Internet for
entertainment, why not make survey participation fun? An interesting
question was posted in the German XING.com expert forum on online
research in the summer of 2007: Is the use of engaging technology
enough to motivate participants or do these experience-enhancing extras
only serve to frustrate survey participants?

Nowadays nearly all online
surveys have a familiar layout. The times of directly adapting a paper
questionnaire for use on the Web should be long over. An attractive
design, matching the corporate identity, is now standard. Furthermore,
multimedia elements are being increasingly integrated into interviews
(e.g., entering single and multiple answers by clicking on pictures
instead of check boxes). However, does this help with response rates?

Socratic Technologies
analyzed which factors influence participation and dropout rate with
the aim of solving the problems, if possible. One major finding was
that the design of surveys must take into account the interaction
between the variables of the burden and the intrinsic personal returns
one experiences from participating. These tend to be related to three
factors surrounding survey design (together with the extrinsic factors
like interest in the topic and affinity with the sponsor):

1. Length of the questionnaire (both in terms of time to complete and number of questions).

2. Incentive (either total incentive offered as prize package or the approximate value of the incentive on an individual basis).

3. Engagement level (the degree to which the survey is perceived to be entertaining and/or intellectually stimulating).

A combination of these three
factors influences the number and proportion of survey avoiders and
mid-survey abandoners. While a great deal of attention has been paid to
the length of the survey and the level of incentive, less study has
been devoted to the in-survey experience as an intrinsic motivator.

Following are findings and
recommendations based on Web-based studies from 2004 to 2006. All of
the studies were with business and consumer technology-related decision
makers and included American, European and Asian respondents. The total
number of respondents included in these surveys was 28,437, with a
median sample size of 422.

These studies examined the
level of engaging activities, which ranged from low-engagement,
indicative of a standard questionnaire-based survey with little
interactivity beyond simple skip patterns and some calculated
variables, to high-engagement, in which Flash-based animations and
game-like activities were used as data collection vehicles.

The analysis focused on the
level of mid-terminates – an indication of the point at which
respondent fatigue, boredom or lack of perceived value becomes
critical. Socratic set the critical threshold of fatigue at the point
where surveys have a mid-terminate rate of more than 30 percent.

Findings from these studies
indicated that no significant differences can be found across different
geographies, meaning that engagement appears to be a global phenomenon
that influences people of many cultures in a similar way.

Survey abandonment

Socratic found that while
incentives can help get people into a survey, the length of the survey
and the degree to which it is perceived to be boring causes participant
attention to slip and leads to survey abandonment. As might be
anticipated, the degree of overcoming mid-termination by successfully
embedding engaging elements is related to how long the survey is to
begin with: The more screens/questions, the greater the number of
mid-terminates.

•   Specifically, surveys that exceed 30 screens/questions are predicted to exceed the maximum acceptable level of dropouts.

•   Other studies have shown that if the survey requires more than 55 clicks, the threshold level for abandonment is met.

•   This phenomenon is also
related to time. Surveys should ideally last no longer than 17 to 18
minutes to prevent more than 30 percent of people from mid-terminating.

When attempting to offset the
length and duration of surveys, animation and interactivity are only
effective within a certain range. For surveys less than 10 minutes in
duration, animation produces no significant effects on dropout rates.
This is probably due to the fact that most people will self-engage in a
survey activity for the required length of time without additional
stimulation.

The primary range of effect
(Figure 1) appears to be between 17 to 42 minutes, where significant
improvement in completion rates is seen with animation added, compared
to surveys of that length without additional stimulation. For surveys
beyond 45 minutes (which are not recommended), animations, interactive
exercises, etc., have little effect on reducing dropout rates. (As a
point of reference, none of the survey satisfaction factors – interest
in the topic, level of incentive, sponsor affinity, etc. – are
effective for retaining respondents in surveys lasting more than 45
minutes.)

Are boring

This research indicates that
the single greatest factors influencing survey avoidance and
mid-termination are respondent fatigue and the perception that online
surveys are boring. Socratic Technologies has experimented with
game-like or engaging environments (where appropriate to the subject
matter) and has found that certain interactive and animated elements
lead to higher satisfaction with surveys and higher likelihood of
future participation.

This is a starting point for
advanced Web survey design. Examples of animated activities that
increase satisfaction include sorting and arranging tasks, dragging and
dropping elements, videos, demonstrations of product functions, and
selection tools that have the look and feel of arcade-style games. Many
companies now employ some form of animation within their survey
environments.

But this does not necessarily
mean complex Flash programming! Many surveys from Lufthansa Technik
contain picture strips, and even in standardized questionnaires it is
possible to use animated GIF files to replace the pictures or to
randomize their display.

In addition to the
entertainment value, advanced animations can also help track the
experience and actions of the user. Ancillary variables that can be
collected and analyzed include how much time someone spends examining
an object, how many times they change their minds and how many items
they explore.

Techno trap

One word of caution, however,
is that using technology for the sake of technology can lead to poor
results. Not everything that can be done from the technical point of
view must be done. This is the techno trap, which occurs when
technology is applied to a research problem when it is unnecessary.
Elements such as extraneous animated characters, flying spaceships,
blinking lights, annoying sounds, etc., tend to have the opposite of
the desired effect: creating distraction and annoyance instead of focus
and engagement. Even for simple questions the design can influence the
answers. Using random animations and sounds also can increase survey
costs, increase the chances of survey technical problems and tends to
create overly complex or confusing situations for the respondent.

In the end, everything depends on the design and content of the
survey. A good questionnaire is the foundation of good research. It
should be the aim of all online researchers to make the job for the
participants as easy and pleasant as possible.