FindArticles > AORN Journal > Nov, 2000 > Article > Print friendly
Evaluating evidence found on the Internet
Suzanne C. BeyeaSearching the Internet for evidence to support practice decisions has become a common strategy for both practitioners and patients. Numerous data resources on the Internet provide up-to-date, reliable, valid, and relevant information. The challenge is differentiating quality sites from those that post unsubstantiated reports or articles, as well as information not based on research or clinical evidence. As clinicians and health care information consumers, nurses must develop the skills essential for evaluating web sites for trustworthiness and be skilled at helping patients critically examine information they find on the Internet.
LEVELS OF EVIDENCE
Keep in mind that in the published literature, many different types of evidence exist. Not all published material is based on fact, research, or scholarship. One author describes the "publishing wedge for therapy," which consists of seven levels of evidence within the health care literature, as an approach to illustrate different levels of evidence. Level one includes ideas, editorials, letters, and opinion papers. Level two information consists of case reports, case studies, and unusual happenings. Levels three through seven present research findings from laboratory studies (ie, level three), animal studies (ie, level four), and studies involving human subjects (ie, levels five, six, and seven). The various levels of human research reflect increases in complexity, scope, and rigor.(1) Some clinical problems have been studied through level seven or may not be amenable to clinical trials, so regardless of where information is published, always consider the level of the evidence.
When searching for evidence, one might identify systematic research reviews, research-based protocols, and clinical practice guidelines. These types of reports can be helpful because they provide a comprehensive literature review and synthesize research evidence. Regardless, the reader must evaluate the report or guideline critically to determine if the information is accurate and appropriate to the clinical situation. Most guidelines are developed using a framework to rate both the type and strength of the evidence and use a rating scale to help the reader understand the classification.(2)
Keeping these concepts and frameworks in mind, consider how one can approach evaluating evidence found on the Internet. Remember that information found on the Internet has to be appraised critically in a process similar to the approach used when evaluating other published materials. For example, one would expect the trustworthiness of information found in the AORN Journal to be much higher than that found in a tabloid newspaper. Within the AORN Journal, readers find level one articles (ie, editorials and opinion articles), as well as research studies with human subjects (ie, levels five, six, and seven), so the reader must always consider the strength of the evidence within a publication. Never be misled by the belief that, if a research study is published in a reputable journal, it guarantees high-quality research. Some of the most reputable medical journals have been criticized for the inclusion of research studies that lack appropriate attention to scientific rigor. The reader must evaluate critically every article he or she reviews, whether published in scientific journals or on the Internet.
The Internet poses some special challenges to identifying up-to-date, valid, reliable evidence-based references and materials. Consider the vastness of the resources on the Internet. For example, if one performs a search on the term diet using http:// gotonet.google.com/, more than 924,000 web sites are identified. Narrowing that search to the term low fat diet results in more than 11,300 possible hits. Trying to review and evaluate all of those web sites obviously is impossible. Both of these searches identified numerous web sites proposing strange, unusual diets and supplements that might suggest quackery to many professionals and patients. Users must identify and use sites they have evaluated as reliable and trustworthy and carefully critique any unfamiliar sites.
In the aforementioned situation, one might look at the American Heart Association's (AHA) web site (ie, http://www .americanheart.org) and examine its recommendations for a low fat diet. At such a site, one would expect to find the latest evidence-based recommendations about following a low fat diet. On this web site, in fact, there are links to scientific papers and statements for professionals, as well as consumers.
Web resources, such as the AHA, the National Library of Medicine (NLM), the Agency on Health Care Quality and Research, and the National Guideline Clearinghouse, are recognized for the timeliness, quality, and trustworthiness of information; however, one must evaluate all the information found on the Internet and consider the level and strength of the evidence, regardless of the source. Within the NLM databases, users may find references to editorials, opinion articles, and case studies, as well as controlled clinical trials with human subjects.
USING A MNEMONIC DEVICE
One's motto should be "reader beware." A simple mnemonic device that can be used is: "Are you PLEASED with the site?"
* Purpose of the site (ie, clearly explained, appropriate content for the purpose);
* Links (ie, working links, reliable, linking to reputable sources);
* Editorial (ie, accurate and current);
* Author (ie, who is the author? Is the author appropriate for the content, with credentials identified and relevant?);
* Site (ie, easy and fast navigation, pages download quickly, judicious use of graphics);
* Ethical (ie, contact information for author is included, full disclosure of purpose and objectives of the site are outlined); and
* Dates (ie, information is posted on when site was created and last updated, site has been updated within a reasonable period [ie, six months] for time sensitive information).(3)
This approach can be applied readily to many situations by both professionals and patients and is consistent with published recommendations related to quality standards for medical information on the Internet (Table 1).(4)
Table 1 EVALUATING INFORMATION ON THE INTERNET(1)
Core standard Characteristics
Authorship who wrote it, and Authors names, credentials,
who do they work for? and affiliations.
Attribution where did this References and sources clearly
come from? identified, along with copyright
information.
Disclosure who owns this web Ownership, sponsorship, and
site, and who pays to have the advertising support specifically
information posted? stated.
Currency when was this Date content was posted and most
information posted and last recently updated.
updated?
NOTE
(1.) W M Silberg, G D Lundberg, R A Musacchio, Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor Let the reader and viewer beware, JAMA 277 (April 16, 1997) 1244-1245.
THE DISCERN INSTRUMENT
The Discern Instrument, located at http://www.discern.org.uk/ discern_instrument, is the first standardized instrument to evaluate the quality of consumer health information (Table 2). It offers a useful approach specifically designed to help anyone who uses consumer health information judge the quality of written materials about treatment choices. This instrument can help users assess the reliability of a publication and rate the quality of its content. This rigorously developed and tested instrument provides a valid and reliable way of assessing quality and establishes a standard for consumer health information.
Table 2 THE DISCERN INSTRUMENT
Section 1 Is the publication reliable? No Partially Yes
1. Are the aims clear? 1 2 3 4 5
Hint: Look for a clear indication at the beginning of
the publication of
what it is about;
what it is meant to cover (and what topics are
meant to be excluded); and
who might find it useful.
If the answer to question 1 is No, go directly to
question 3.
2. Does it achieve its aims? 1 2 3 4 5
Hint: Consider whether the publication provides the
information it aimed to as outlined in question 1.
3. Is it relevant? 1 2 3 4 5
Hint: Consider whether
the publication addresses the questions that readers
might ask, and
recommendations and suggestions concerning treatment
choices are realistic or appropriate.
4. Is it clear what sources of 1 2 3 4 5
information were used to compile
the publication (other than the
author or producer)?
Hint:
Check whether the main claims or statements made
about treatment choices are accompanied by a
reference to the sources used as evidence
(eg, a research study or expert opinion).
Look for a means of checking the sources used, such
as a bibliography/reference list, addresses of the
experts or organizations quoted, or external links
to the online resources.
5. Is it clear when the information 1 2 3 4 5
used or reported in the
publication was produced?
Hint: Look for
dates of the main sources of information used to
compile the publication,
date of any revisions of the publication
(but not dates of reprinting in the case of
print publications), and
date of publication (copyright date).
Rating note: The hints are placed in order of importance
in order to score a full 5, the dates relating to the
first hint should be found.
6. Is it balanced and unbiased? 1 2 3 4 5
Hint: Look for
a clear indication of whether the publication is
written from a personal or objective point of view;
evidence that a range of sources of information
was used to compile the publication
(eg, more than one research study or expert); and
evidence of external assessment of the publication.
Be wary if
the publication focuses on the advantages or
disadvantages of one particular treatment choice
without reference to other possible choices;
the publication relies primarily on evidence from
single cases (which may not be typical of people
with this condition or of responses to a particular
treatment); and
the information is presented in a sensational,
emotive, or alarmist way.
7. Does it provide details of 1 2 3 4 5
additional sources of support and
information?
Hint: Look for suggestions for further reading or for
details of other organizations providing advice and
information about the condition and treatment choices.
8. Does it refer to areas of 1 2 3 4 5
uncertainty?
Hint:
Look for discussion of the gaps in knowledge or
differences in expert opinion concerning treatment
choices.
Be wary if the publication implies that a treatment
choice affects everyone in the same way (eg, 100%
success rate with a particular treatment).
Section 2 How good is the quality of No Partially Yes
information on treatment choices?
9. Does it describe how each 1 2 3 4 5
treatment works?
Hint: Look for a description of how a treatment acts on
the body to achieve its effect.
10. Does it describe the benefits of 1 2 3 4 5
each treatment?
Hint: Benefits can include controlling or getting rid
of symptoms, preventing recurrence of the condition, and
eliminating the condition, both short-term and long-term.
11. Does it describe the risks of each 1 2 3 4 5
treatment?
Hint: Risks can include side-effects, complications, and
adverse reactions to treatment, both short-term and
long-term.
12. Does it describe what would happen 1 2 3 4 5
if no treatment was used?
Hint: Look for a description of the risks and benefits of
postponing treatment, of watchful waiting (ie, monitoring how
the condition progresses without treatment), or of
permanently forgoing treatment.
13. Does it describe how the treatment 1 2 3 4 5
choices affect overall quality of
life?
Hint: Look for
description of the effects of the treatment choices
on day-to-day activity, and
description of the effects of the treatment choices
on relationships with family members, friends,
and career.
14. Is it clear that there may be more 1 2 3 4 5
than one possible treatment
choice?
Hint: Look for
a description of who is most likely to benefit from
each treatment choice mentioned and under what
circumstances; and
suggestions of alternatives to consider or investigate
further (including choices not fully described in the
publication) before deciding whether to select or
reject a particular treatment choice.
15. Does it provide support for shared 1 2 3 4 5
decision-making?
Hint: Look for suggestions of things to discuss with family
members, friends, physicians, or other health professionals
concerning treatment choices.
Section 3 Overall rating of the
publication
16. Based on the answers to all of the above questions, rate the
overall quality of the publication as a source of
information about treatment choices.
Low serious or Moderate potentially
extensive important but not High minimal
shortcomings serious shortcomings shortcomings
1 2 3 4 5
This information is copyrighted by the British Library and the University of Oxford. Organizations are authorized to reproduce this instrument provided it is used in accordance with the instructions contained at www.discern.org.uk/discern _instrument.htm and that they complete an evaluation form provided on the web site.
The Discern Instrument consists of 16 questions, including one that assesses the overall quality of the publication. Fifteen questions reflect a specific quality criterion or an essential feature or standard. Questions are organized in three sections and address the publication's reliability, relevance, description of treatment, description of treatment benefits and risks, description of treatment's effect on overall quality of life, and the overall rating. For each question, the instrument provides hints to help explain terms or to offer suggestions to assist with evaluating the publication. Users must rate each question on a five-point scale ranging from one (ie, no) to five (ie, yes), and they are provided resources for additional guidance throughout the instrument.
Becoming familiar with such resources as the Discern Instrument, can help any clinician or consumer. It provides a helpful framework to involve patients in shared decision-making about treatment choices and helps them become sophisticated consumers of health care information. Although there are many other resources for evaluating Internet health information, the Discern Instrument provides an easy-to-follow, user-friendly format.
Nurses should practice using the Discern Instrument by evaluating an Internet health information site. When you have a question about a health problem and search the Internet, use the Discern Instrument as you evaluate sites. Share this information with your patients, family members, and colleagues. The amount of information available on the Internet will only increase in scope and quantity. Be a savvy consumer--learn how to effectively evaluate sites and provide your patients with resources and strategies to do the same.
SUZANNE C. BEYEA RN, PHD, CS AORN DIRECTOR OF RESEARCH
NOTES
(1.) A McKibbon, PDQ: Evidence-Based Principles and Practice (Hamilton, Ontario: B. C. Decker, Inc, 1999) 7-12.
(2.) S C Beyea, "Preventing surgical site infections--Guiding practice with evidence," (Research Comer) AORN Journal 72 (August 2000) 305-307.
(3.) L H Nicoll, S C Beyea, "Working with staff around evidence-based practice: The next generation of research utilization," Seminars in Perioperative Nursing 9 (July 2000) 133-142.
(4.) W M Silberg, G D Lundberg, R A Musacchio, "Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware," JAMA 277 (April 16, 1997) 1244-1245.
RESOURCES
"Criteria for evaluation of Internet information resources." Available from http://www.vuw.ac.nz /~agsmith/evaln/index.htm. Accessed 12 Sept 2000.
"Critical evaluation of resources." Available from http://www.lib.berkeley.edu/TeachingLib/Guides /Evaluation.html. Accessed 12 Sept 2000.
"Discern online: Quality criteria for consumer health information." Available from http://www.discern.org.uk. Accessed 12 Sept 2000.
"Evaluating research resources: World Wide Web sites." Available from http://www.lib.waldenu.edu /web_a.html. Accessed 12 Sept 2000.
"Indiana University Bloomington libraries present: Critical thinking and the World Wide Web." Available from http://www.indiana.edu/~librcsd/eval/checklist.html. Accessed 12 Sept 2000.
"Information quality www virtual library: The Internet guide to construction of quality online resources." Available from http://www.ciolek.com/WWWVL-InfoQuality.html. Accessed 12 Sept 2000.
"Mitretek systems: Innovative technology in the public interest." Available from http://hitiweb.mitretek.org /info/projects.asp. Accessed 12 Sept 2000.
Smith, A G. "Testing the surf: Criteria for evaluating Internet information resources." Available from http://info.lib.uh.edu/pr/v8/n3/smit8n3.html. Accessed 12 Sept 2000.
COPYRIGHT 2000 Association of Operating Room Nurses, Inc.
COPYRIGHT 2001 Gale Group