HCI 314 ACT 3

Evaluation of E-health
Quynh Lê

Learning Objectives
On completion of this topic, students will be able to:
–

Introduce current evaluation frameworks;

–

Explore methods/tools used in E-Health evaluation;

–

Provide understanding of evaluation frameworks applied to E-Health
applications;

–

Identify research methods/tools used in E-Health evaluation; and

–

Develop an evaluation plan for a systemic E-Health initiative.

Introduction
Evaluation is an activity which requires considerate amount of time and resources.
Any products, processes or services have their own aims as they are developed for a
purpose. In other words, they are there because they fulfil a need. For example, a
comprehensive cancer care network is established to improve community access to
cancer care (NSW Health Telehealth Initiative 2003); a web site is developed to
provide the public with information about various services in the hospital and how to
access these services on-line; an E-Health programme to keep teenagers from
smoking and alcohol consumption.
Broadly speaking, evaluation has the following functions and purposes:
–

We need to find out whether the aims and objectives have been achieved or
not.

–

Evaluation helps to identify the strengths and weaknesses of a product or
service. The result of evaluation can be used to improve a product or service.

–

Evaluation is useful to health organisations to ascertain how online health
application is situated in the broad context of health care.

What is evaluation?
Evaluation is assessing and judging the value of a piece of work, an organisation or a
service. It is fundamentally about asking questions, and then designing ways to find
useful answers.
Evaluation is a process which examines the quality of a product or entity (a piece of
work, an organisation or a service) such as a training program, sport project, and
health software.
Evaluation involves a goal-orientated task which takes place in a certain context
during a period of time involving certain methods of evaluation.
Without evaluation, it is hard, if not impossible, to know how a product works. For
example, a Paediatric Oncology Video is designed for clinicians to assist with the
management of patients upon returning home (NSW Telehealth Initiative, 2003). It is
important for stakeholders such as administrators, developers, participating health
professionals to know whether the programme achieved its aims, how the

1 of 7

participants thought and gained from the video, what needs to be improved etc.
On one hand, evaluation may take place individually and informally, particularly in
self-evaluation, such as a teacher evaluates his or her first year of teaching in a rural
school. On the other hand, evaluation can be a huge task involving a team using a
variety of evaluation approaches and tools, e.g. evaluation of a health program for
rural people in isolated regions of Australia.
Please note evaluation is not equivalent to research even it uses research
techniques as a mean of generating information and employs criteria for reliability
and validity to judge the quality of evidence. The main difference between evaluation
and research is that evaluation involved making judgement about the value of what is
being evaluated whereas research determined the questions you want to ask and
how you are going to answer them (Murdoch University, 2004). Evaluation tends to
be broader than research and requires information about products, services and
processes.

E-Health evaluation
With the rapid development of Information Technology (IT) and its permeation in
various aspects of modern society, E-health is a testimony of the impact of IT in the
field of health. The Internet and other electronic media are present in many health
institutions and health services. To some people, E-health is not just an introduction
of modern technology to health but it is a new paradigm which requires adjustment in
thoughts and behaviours. The term E-Health includes use of the internet or other
electronic media by the public, health workers, and others to disseminate health
related information or services (Wyatt & Liu, 2002). We should not take for granted
that everything goes smoothly with E-health. It is important not to welcome the
implementation of IT in health without critical consideration of its effectiveness in a
new context. Thus programs and services which involve E-health need evaluation,
formally or informally, depending on their nature, scope and social context.
Health informatics which is an important aspect of E-Health provides a new way of
informing the public and health workers with information and opportunities for
accessing health services. Many useful web sites and electronic materials have been
produced to make health care readily available to the public. Though this is
considered as a very innovative and productive development in health care, there are
also risks caused by inaccurate information, mismanagement of health data, and ITbased problems, particularly when health services heavily depend on health
informatics. Thus evaluation of E-Health is vital in improving health care.
E-Health evaluation depends heavily on users. The appropriate level of evaluation
depends on the needs of the users. Therefore, important issues that need to be
evaluated in design and developing an online health application are users’ needs,
product’s risks and benefits, its feasibility, and user acceptance to the new adopted
technologies (Gustafson D, Taylor J, Thompson S, & P, 1993; Gustafson & Wyatt,
2004).

What should be evaluated?
One of the first and most important questions in E-Health evaluation is to identify
what are the targets of evaluation. It can be a health informatics program, shared
health database access facility, or a website promoting a health service. Evaluation
can cover the whole product or some specific aspects. Generally an evaluation is
needed to find out what works and what does not work.
User satisfaction is the primary aim of E-health evaluation. It is vital to find out how
users value the product developed for them when other alternatives are also

2 of 7

available. User-friendly evaluation includes cultural sensitivity and user needs. The
quality and quality of information are two important aspects of evaluation of E-health
products. User evaluative feedback helps us to determine which information needs to
be included as it is a primary source and which should be treated as less important or
unnecessary.
Usability and accessibility issues are important when developing an appropriate user
interface for different kinds of target users (Gustafson & Wyatt, 2004; Wyatt & Liu,
2002). For example, the interface of an E-Health programme that provides services
to palliative care, clinical consultation, bereavement support and education will be
different from the one that prevents teenagers from alcohol and other drugs issues.
The web site for teenagers needs to be fun to use whereas easy access to key
information is more important to palliative caregivers. Up-to-date and accurate
information on the web site are essential to avoid confusion and misleading.
As E-health includes the use of the Internet and other electronic media, evaluation
generally covers the screen design, navigation, written information, visual
information, hardware and software support; and HELP facilities.
Cost implications are important in evaluating E-Health programme (Smaglik ,
Hawkins, Pingree, & Gustafson, 1998). We should consider how E-health
programmes affect the use and costs of healthcare services relative to other options.
What change in access to service or support has an E-Health application made for its
end-users? What is the cost effectiveness of different E-Health delivery programme
(Eng, 2002; Glasgow, 2007; Gustafson & Wyatt, 2004). Cost effectiveness and/or
cost benefit is highly valued within the decision making of chosen E-Health
application.

Evaluation framework
There are a wide range of evaluation methods used in health services and the
approach and tools used will depend on the application (Evers, 2006; Hawe,
Degeling, & Hall, 1990; Nguyen, Cuenco, Wolpin, Benditt, & Carrieri-Kohlman, 2007).
All evaluations need to ask a specific question and the type of data collection. Thus,
data analysis and report on results are determined by the question in the scope. The
possible questions to be addressed are:
–

What is to be evaluated?

–

Who is the evaluation for?

–

Who are the relevant stakeholders?

–

Who you will report the results to?

–

What are the benefits and limitations of the projects?

–

What are the cost implications for the project?

–

What are privacy, security and standards issues?

–

How are health services performing compared to other service alternative
available?

–

What resources (hardware/software/human resources) are needed?

The generic evaluation process of E-health can be described in the following
diagram:

3 of 7

Generic E-Health Evaluation Process

Stage 1
Needs Analysis and
Design

Determine the needs of
the target groups and
project feasibility

Produce/Select a
working model for EHealth applications

Stage 2
Development / Selection
of E-Health applications
Find out if the product
meets the aims of the
Stage 1
Yes

Stage 3
Formative
Evaluation

Make changes for
improvements

Questionnaires
Organisation’s data
Observation
Follow-up post test

Select methods of
collecting evidence

Checklist
Questionnaires
Interviews
Focus groups
Expert review
Observation

Stage 6
Long Term
Impacts / Learning
for future service
delivery

Determine the long term
effects of the E-Health
programme on the
organisation and
individuals

No
Yes

Stage 4
Implementation /
Decision for
implementation

Stage 5
Summative
Evaluation

Decide whether to
continue
implementation

Analyse data and
summarise results
with stakeholders

Select methods of
collecting evidence

Checklist
Questionnaires
Interviews
Focus groups
User trialling
Expert review
Observation

Determine adequacy of
the E-Health programme
for the needs of its target
users

Figure 1: Generic E-Health evaluation framework

Source: Adapted from Phillips et al (2004) and LTDI (1998)
Planning an evaluation normally involves the following steps:
1. Identify the purpose of evaluation;
2. Select the questions to be addressed;
3. Select the method of collecting the evidence: see methods of evaluation

4 of 7

section;
4. Prepare the documentation: test the questions to ensure they are clear and
address adequately the purpose of your evaluation.
5. Conduct activities for collecting the evidence;
6. Analyse the results; and
7. Report on the results.

Who should be involved?
Any health product should have its target users. An evaluation of an E-health
program for health workers need to find out how they think and feel about the
product. Their feedback needs to be monitored during the formative evaluation
process. Similarly, an evaluation of a health website for the public needs to identify
who are main target users and potential users. Their views and attitudes need to be
taken into account the evaluation. It is no use to develop a website written in
scientific English for migrants who are semi-illiterate.
The second group of stakeholders is the implementer of the E-health product. What
are some issues which concern them during the implementation? In their views, does
it ‘works’? Why? What needs to be done to improve the product?
Expert advice is important in the evaluation. The experts may include IT specialist,
medical professionals, and allied health workers.

Methods of evaluation
The way information is presented can be essential to key parties perceive the study.
Different types of information convince different people (LTDI, 1998). Quantitative
measurements may offer more use in demonstrating concrete achievement to
funding organisations and top management, but qualitative feedback may be more
useful in establishing improvements necessary for users of a system. Quantitative
evaluations should be supplemented by qualitative studies where users explain what
they feel when they use the system and how it affects their lives (Gustafson & Wyatt,
2004).
E-health evaluation can be conducted by using a single method or a combination of
methods. It may take place in a short or long period of time and involves simple or
complex tools depending on the scale and scope of an evaluation. The information
you choose to gather will affect the tools and techniques you adopt, with
consequences for the resources you require to complete the study successfully.
Formative evaluation and summative evaluation are often used to describe the
process of evaluation. Formative evaluation takes place during the time of
implementation and its information is used to shape the activity as it happens
whereas summative evaluation occurs at the end of the implementation phase and is
used to make judgement about its success or failure. It is common to combine these
two types in E-health evaluation.

5 of 7

The following methods can be used for evaluation in E-health:
Table 1: Methods for collecting evidence

Methods
Checklist

Description
Brief and introductory way of highlighting main items to be
‘checked’. It is useful for follow-up evaluative tasks.
Reading:

ead

Questionnaire

Question items cover different aspects of user evaluation. It may
include users’ feedback choice (closed questions) and openended questions.
Reading:

endhead

Interview

Interview is a direct way of seeking feedback from users. It can
be conducted on a face-to-face basis or on-line, individually or in
small groups. Confidentiality needs to be maintained. There are
different types: structured, semi-structured, and open-ended.
Reading:

ead

Focus group

It involves a selected number of users to be included in a focus
group for a period of time, including a series of interviews,
observations and group discussions.
Reading:

ndhead

Expert review

Expert feedback is important, particularly when a program deals
with highly technical health information.

Tester and user
trialling

It is necessary to test initially whether the product matches the
aims and objectives of an intended product. It may involve a
small group of target users to seek their feedback before the
product is finally introduced to the public.

Observation

The main purpose of the observation is to obtain a detailed
understanding of the ways E-Health applications are used and
the problems they are encountered.
Reading:

index.html#endhead

Source: Adapted from Evaluation Cookbook (LTDI, 1998).

Conclusion
E-Health plays an essential role in the contemporary context of health in general and
health care in particular. It builds innovative bridges of information and interaction for
health services, health workers and the public. However, one should not take for

6 of 7

granted that it is all smooth sailing in E-Health. Like many IT-based development,
there are problems and risks which need to be examined. Evaluation is an important
way in ascertaining whether an E-Health product works.

References
Eng, T. R. (2002). eHealth research and evaluation: challenges and opportunities. J
Health Commun, 7(4), 267-272.
Evers, K. E. (2006). eHealth promotion: the use of the Internet for health promotion.
Am J Health Promot, 20(4), suppl 1-7, iii.
Glasgow, R. E. (2007). eHealth evaluation and dissemination research. Am J Prev
Med, 32(5 Suppl), S119-126.
Gustafson D, Taylor J, Thompson S, & P, C. (1993). Assessing the needs of breast
cancer patients and their families. J Qual Manage Healthcare 1993;2: 1. . J
Qual Manage Healthcare, 2(1).
Gustafson, D. H., & Wyatt, J. C. (2004). Evaluation of ehealth systems and services.
Bmj, 328(7449), 1150.
Hawe, P., Degeling, D., & Hall, J. (1990). Evaluating Health Promotion: A Health
Workers Guide. Sydney: Maclennan & Petty.
LTDI. (1998). Evaluation Cookbook. Retrieved 2 July, 2007, from

Murdoch University. (2004). Handbook for Learning-centred Evaluation of Computerfacilitated Learning Projects in Higher Education. Retrieved 2 July 2007, from

Nguyen, H. Q., Cuenco, D., Wolpin, S., Benditt, J., & Carrieri-Kohlman, V. (2007).
Methodological considerations in evaluating eHealth interventions. Can J
Nurs Res, 39(1), 116-134.
NSW Telehealth Initiative. (2003). Telehealth Projects 2002/2003 – Telehealth
Initiative Newsletter Issue No. 9. Retrieved 2 July, 2007, from

Phillips, R., Bain, J., McNaught, C., Rice, M., & Tripp, D. (2004). Handbook for
Learning-centred Evaluation of Computer-facilitated Learning Projects in
Higher Education. Retrieved 2 July 2007, from

Smaglik , P., Hawkins, R., Pingree, S., & Gustafson, D. (1998). The quality of
interactive computer use among HIV infected individuals. J Health
Communication 3, 53-68.
Wyatt, J., & Liu, J. (2002). Basic concepts in medical informatics: a glossary.
Epidemiol Community Health, 56, 808-812.

7 of 7

Purchase answer to see full
attachment