A key element in the design of a high quality research project is use of a pilot study. The pilot study can represent different choices: a feasibility study conducted to prepare for the major research study, or as part of the research plan to develop or refine the methodology. The purposes in the conduct of a pilot study are numerous and inform researchers about many aspects of methodology (see below). Implementing a pilot study will often add to groundwork time for a project, however, it will provide an opportunity to correct defects in the ultimate research design.

Lessons learned from the conduct of a pilot study

Use of pilot studies

Pilot studies are a valuable part of research that informs both process and outcome. Within the research community, pilot studies have several aliases: exploratory study, feasibility study, preliminary work, and stage one. The lessons learned in its conduct can serve as a guide to the study that follows, however seldom are the problems confronted by researchers in a pilot study published. This is unfortunate as researchers can learn from others’ mistakes.

Pilot studies are applicable to both qualitative and quantitative research and are not merely research projects that were unsuccessful in obtaining adequate sample size. As demonstrated in the literature by Dallas et al. (2005a,b), large-scale studies may contain several pilot studies before the launch of the main study.

Because of differences in purpose, pilot studies are not the same as stages or phases of a major study. For example, the research questions asked in a pilot study differ from those of a major study. Sample phrases for pilot study research questions and aims include:

• To identify design and methodological problems…
• To establish feasibility of recruitment and retention procedures…
• To test feasibility of protocols, interventions…
• To estimate effect size…
• To determine time required to administer an instrument…
• To assess preliminary data about reliability, readability, applicability of an instrument…
• To develop calculated costs of an intervention and budget justification…
• To refine analysis procedures…

In some cases, a pilot study and subsequent major study may share similar aims and research questions. If pilot research questions are identical to the larger study and if results from the pilot are published, funding agencies and journals may no longer consider the research as “original.” However, larger funded grants are now expecting preliminary data from pilot work to be present in the application, and the researcher needs to be watchful in phrasing. See below for more information.

Similarities and differences of a pilot studies to major research study

Consideration of pilot study results

Pilot studies inherently have limits. The most noteworthy of these occur when a sample size is either too small or lacks a control group. Small sample size mandates interpretation of results with caution and provides limited insight into the validity and reliability of measures. A pilot study can represent the testing of only a small portion of a larger study, and may not test the feasibility of the whole research design. However, to conserve valuable research dollars the identification of any design flaws in pilot work must be corrected before the larger study begins.

Data from pilot study subjects should not be included in an intervention study if they not randomized. If the design is changed because of the pilot study, the data from these subjects may not be acceptable. If the researcher has unrealistic expectations for design, some pilot studies may never be completed because they are more fitting for a subsequent phase of the program of research. Furthermore, a successful pilot study does not mean that the larger study will also be successful, but even a failed pilot study yields important information.

Relevance to funding opportunities

A pilot study can demonstrate to a funding agency that the research design is feasible and worthwhile, therefore more creditable for financial support. Pilot work in a major grant application demonstrates that the researcher is knowledgeable about the topic to be studied, and is competent in the skills necessary to execute the study. Funded pilot work facilitates the forward momentum of a program of research. Extensive pilot work illustrates the comprehensive effort prepared for the conduct of the larger research study.

Pilot studies can also assist to verify budgetary expenses and avoid cost overrun. With the limited available research funding currently available, the researcher has an ethical and fiscal responsibility to engage in the most rigorous research design that can potentially yield the greatest influence on clinical practice or add to the body of nursing science. To accomplish this goal, large studies that follow pilot studies, must be adequately powered. Pilot work can assist to calculate effect size (measure of the strength of the relationship between two variables independent of sample size) and direct power analysis (how large a sample is needed for accurate and reliable statistical judgments).

Lessons learned from pilot studies

Inclusion criteria can be tested to determine if an adequate sample size for data analysis can be obtained. In sampling, pilots may help determine the feasibility of recruiting and retaining subjects. Researchers who share information regarding the challenges of working with a diverse group in a pilot study can ease the path for those who follow. The research team can demonstrate their ability to maintain confidentiality, which may assist in the development of trust between the researchers and a vulnerable population. Furthermore, wording on advertisement flyers can be evaluated to determine the accuracy of the recruitment message and level of reception. Preliminary pilot work can assess strategies for uncovering the gatekeepers in subject recruitment, as well as, appropriateness and feasibility of research interventions, staff cooperation, and management of subject attrition.

Pilot work can guide appropriate choice of reimbursement for research participation. It provides experience in logistics and institutional politics, accessibility, and procedures. Learning the organizational structure and securing cooperation of key personnel in research sites can help eliminate barriers to success. Training programs for the research team, instituted during pilot work, can lead to modifications in operating procedures and clarification of roles. Oberle and Allen state that pilot work can test the link of the research plan to the conceptual model, and links between operational definitions and conceptual framework.

It is during the pilot phase that researchers develop and evaluate the use of instruments and questionnaires. The clearness, accuracy, readability, and comprehensibility of questionnaire or scale items can be assessed, in addition to presence of translation errors. Many pilot studies can begin the assessment of internal consistency, content validity, discriminate validity, and reliability of instruments used in new populations or in new languages. The appraisal of sensitivity to ethnicity and culture in translated instruments may also take place. Besides questionnaires and surveys, pilot studies can also examine the adequacy of mechanical instruments to obtain the desired data. While not true after the initiation of the larger study, selecting a more appropriate instrument is still possible after the conduct of a pilot study.

Pilot study work can focus on the methods and ethical issues that emerge in research, especially in the discovery of potential threats to internal and external validity. Researchers evaluate interview guides for ease of use and participant test burden. Furthermore, researchers assess methodological concerns for inter-rater and inter-coder reliability.

Experience with management and analysis of data can be achieved during a pilot study. Practice with statistical software, developing a codebook, constructing efficient data entry forms, and determining how to handle missing data can all occur. Regardless of research site or population, pilot work can give the researcher experience with new instruments, new equipment, and new forms of analysis.

Survey of the use of pilot studies in nursing research journals

Prescott and Soeken examined the use of pilot studies in three nursing research journals: Nursing Research (NR), Research in Nursing & Health (RINAH), and the Western Journal of Nursing Research (WJRN). In review of 212 journal studies spanning three years (1985-1987) only 18 (9%) mentioned pilot studies, while only five (2%) were actual reports of pilot studies. The remainders either focused on instrument reliability or were a miniature test of an already designed research plan. The authors found no examples of methodological issues addressed by pilot studies.

The conduct of a systemic review for the current use of pilot studies in these same three nursing research journals for the years 1999-2005 was built on Prescott’s and Soeken’s work. The number of research articles in this period were 873 (NR=282, RINAH=264, WJNR=327). A Medline database keyword “pilot study” was performed. The sum of “hits” was 210 (24%) of the published research articles (NR=90, RINAH=24, WJNR=96). For a comparison analysis, non-qualifying articles were omitted yielding a total of 45 articles or 5% of the total 873 articles that reported pilot study data (NR=14, RINAH =20, WJNR=11). Non-qualifying articles were editorials, conference reports, policy statements, methodology articles that mentioned pilot studies, titles of articles cited in the reference list, or works cited in the review of literature. A general trend visible over the last seven years was an increase in the number of citations with the key words “pilot study”, as well as, an increase in published results of pilot studies. There was no significant difference in chi-square analysis between the journals.

Nursing implications

The American Nurses Association has included the engagement in research in the Standards of Professional Performance. Understanding the purpose and use of pilot studies will facilitate participation by the clinical nurse and enable their contributions to the research process. Pilot studies can yield valuable and useful information to make a larger research study more efficient, more effective, and well constructed. The pilot study builds a strong foundation on which a nurse scientist can build a program of research associated with rigorous science. A pilot is not a label for a poorly designed, poorly conducted, or small sample research study, but rather stands on its own merits. A seven-year review of three current nursing research journals demonstrated that pilot study results are now published in greater frequency (5% increased from 2%) compared to past-published assessments. Citation of the phrase pilot study in research articles was increased from 9% to 24%; indicating greater general use. For the benefit of future researchers, it is suggested journals publish more lessons learned from pilot studies. A pilot study serves like the helmsman on a ship, guiding the investigator to a more rigorous larger research study.

Jill Kilanowski is assistant professor and CTSA KL2 scholar at the Frances Payne Bolton School of Nursing, Case Western Reserve University, Cleveland, Ohio.

References

American Nurses Association. Nursing: Scope and Standard of Practice, 2nd ed. Silver Springs, MD. American Nurses Association: (2010).

Burns N, Grove S. The Practice of Nursing Research. 4th ed. Philadelphia: W.B. Saunders Co.: 2001.

Dallas C, Norr K, Dancy B, Kavanaugh K, Cassata L. An example of a successful research proposal: Part I. West J Nurs Res. 2005a; 27(1): 50-72.

Dallas C, Norr K, Dancy B, Kavanaugh K, Cassata L. An example of a successful research proposal: Part II. West J Nurs Res. 2005b; 27(2): 210-231.

Kilanowski JF. Lessons learned from a pilot study on the health status of children from itinerant populations. J Pediatr Health Care. 2006; 20(4): 253-260.

King K. Editorial: The problem of under-powering in nursing research. West J Nurs Res. 2001; 23(4): 334-335.

Lindquist R. Don’t forget the pilot work! HeartLung. 1991; 20(1): 91-92.

Oberle K, Allen MN. Clinical trials with complementary therapies. Western Journal of Nurs Res. 2005; 27(2): 232-23.

Prescott P, Soeken K. The potential uses of pilot work. Nurs Res. 1989; 38(1): 60-62.

Teijlingen E, Hundley V. The importance of pilot studies. Social Research Update. 2001; 35(Winter):1-4.

Teijlingen E, Rennie A, Hundley V,Graham W. The importance of conducting and reporting pilot studies: The example of the Scottish Birth Survey. J Adv Nurs. 2001; 34(3): 289-295.