Advertisement
Research Article

Reporting Guidelines for Survey Research: An Analysis of Published Guidance and Reporting Practices

  • Carol Bennett mail,

    cbennett@ohri.ca

    Affiliation: Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada

    X
  • Sara Khangura,

    Affiliation: Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada

    X
  • Jamie C. Brehaut,

    Affiliations: Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada, Department of Epidemiology and Community Medicine, University of Ottawa, Ottawa, Canada

    X
  • Ian D. Graham,

    Affiliation: Canadian Institutes of Health Research, Ottawa, Canada

    X
  • David Moher,

    Affiliations: Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada, Department of Epidemiology and Community Medicine, University of Ottawa, Ottawa, Canada

    X
  • Beth K. Potter,

    Affiliation: Department of Epidemiology and Community Medicine, University of Ottawa, Ottawa, Canada

    X
  • Jeremy M. Grimshaw

    Affiliations: Ottawa Hospital Research Institute, Clinical Epidemiology Program, Ottawa, Canada, Department of Medicine, University of Ottawa, Ottawa, Canada

    X
  • Published: August 02, 2011
  • DOI: 10.1371/journal.pmed.1001069
  • Published in PLOS Medicine

Abstract

Background

Research needs to be reported transparently so readers can critically assess the strengths and weaknesses of the design, conduct, and analysis of studies. Reporting guidelines have been developed to inform reporting for a variety of study designs. The objective of this study was to identify whether there is a need to develop a reporting guideline for survey research.

Methods and Findings

We conducted a three-part project: (1) a systematic review of the literature (including “Instructions to Authors” from the top five journals of 33 medical specialties and top 15 general and internal medicine journals) to identify guidance for reporting survey research; (2) a systematic review of evidence on the quality of reporting of surveys; and (3) a review of reporting of key quality criteria for survey research in 117 recently published reports of self-administered surveys. Fewer than 7% of medical journals (n = 165) provided guidance to authors on survey research despite a majority having published survey-based studies in recent years. We identified four published checklists for conducting or reporting survey research, none of which were validated. We identified eight previous reviews of survey reporting quality, which focused on issues of non-response and accessibility of questionnaires. Our own review of 117 published survey studies revealed that many items were poorly reported: few studies provided the survey or core questions (35%), reported the validity or reliability of the instrument (19%), defined the response rate (25%), discussed the representativeness of the sample (11%), or identified how missing data were handled (11%).

Conclusions

There is limited guidance and no consensus regarding the optimal reporting of survey research. The majority of key reporting criteria are poorly reported in peer-reviewed survey research articles. Our findings highlight the need for clear and consistent reporting guidelines specific to survey research.

Please see later in the article for the Editors' Summary

Editors' Summary

Background

Surveys, or questionnaires, are an essential component of many types of research, including health, and usually gather information by asking a sample of people questions on a specific topic and then generalizing the results to a larger population. Surveys are especially important when addressing topics that are difficult to assess using other approaches and usually rely on self reporting, for example self-reported behaviors, such as eating habits, satisfaction, beliefs, knowledge, attitudes, opinions. However, the methods used in conducting survey research can significantly affect the reliability, validity, and generalizability of study results, and without clear reporting of the methods used in surveys, it is difficult or impossible to assess these characteristics and therefore to have confidence in the findings.

Why Was This Study Done?

This uncertainty in other forms of research has given rise to Reporting Guidelines—evidence-based, validated tools that aim to improve the reporting quality of health research. The STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) Statement includes cross-sectional studies, which often involve surveys. But not all surveys are epidemiological, and STROBE does not include methods' and results' reporting characteristics that are unique to surveys. Therefore, the researchers conducted this study to help determine whether there is a need for a reporting guideline for health survey research.

What Did the Researchers Do and Find?

The researchers identified any previous relevant guidance for survey research, and any evidence on the quality of reporting of survey research, by: reviewing current guidance for reporting survey research in the “Instructions to Authors” of leading medical journals and in published literature; conducting a systematic review of evidence on the quality of reporting of surveys; identifying key quality criteria for the conduct of survey research; and finally, reviewing how these criteria are currently reported by conducting a review of recently published reports of self-administered surveys.

The researchers found that 154 of the 165 journals searched (93.3%) did not provide any guidance on survey reporting, even though the majority (81.8%) have published survey research. Only three of the 11 journals that provided some guidance gave more than one directive or statement. Five papers and one Internet site provided guidance on the reporting of survey research, but none used validated measures or explicit methods for development. The researchers identified eight papers that addressed the quality of reporting of some aspect of survey research: the reporting of response rates; the reporting of non-response analyses in survey research; and the degree to which authors make their survey instrument available to readers. In their review of 117 published survey studies, the researchers found that many items were poorly reported: few studies provided the survey or core questions (35%), reported the validity or reliability of the instrument (19%), discussed the representativeness of the sample (11%), or identified how missing data were handled (11%). Furthermore, (88 [75%]) did not include any information on consent procedures for research participants, and one-third (40 [34%]) of papers did not report whether the study had received research ethics board review.

What Do These Findings Mean?

Overall, these results show that guidance is limited and consensus lacking about the optimal reporting of survey research, and they highlight the need for a well-developed reporting guideline specifically for survey research—possibly an extension of the guideline for observational studies in epidemiology (STROBE)—that will provide the structure to ensure more complete reporting and allow clearer review and interpretation of the results from surveys.

Additional Information

Please access these web sites via the online version of this summary at http://dx.doi.org/10.1371/journal.pmed.1​001069.

Introduction

Surveys are a research method by which information is typically gathered by asking a subset of people questions on a specific topic and generalising the results to a larger population [1],[2]. They are an essential component of many types of research including public opinion, politics, health, and others. Surveys are especially important when addressing topics that are difficult to assess using other approaches (e.g., in studies assessing constructs that require individual self-report about beliefs, knowledge, attitudes, opinions, or satisfaction). However, there is substantial literature to show that the methods used in conducting survey research can significantly affect the reliability, validity, and generalisability of study results [3],[4]. Without clear reporting of the methods used in surveys, it is difficult or impossible to assess these characteristics.

Reporting guidelines are evidence-based, validated tools that employ expert consensus to specify minimum criteria for authors to report their research such that readers can critically appraise and interpret study findings [5][7]. More than 100 reporting guidelines covering a broad spectrum of research types are indexed on the EQUATOR Network's website (www.equator-network.org). There is increasing evidence that reporting guidelines are achieving their aim of improving the quality of reporting of health research [8][11].

Given the growth in the number and range of reporting guidelines, the need for guidance on how to develop a guideline has been addressed [7]. A well-structured development process for reporting guidelines includes a review of the literature to determine whether a reporting guideline already exists (i.e., a needs assessment) [7]. The needs assessment should also include a search for evidence on the quality of reporting of published research in the domain of interest [7].

The series of studies reported here was conducted to help determine whether there is a need for survey research reporting guidelines. We sought to identify any previous relevant guidance for survey research, and any evidence on the quality of reporting of survey research. The objectives of our study were:

  1. to identify current guidance for reporting survey research in the “Instructions to Authors” of leading medical journals and in published literature;
  2. to conduct a systematic review of evidence on the quality of reporting of surveys; and
  3. to identify key quality criteria for the conduct of survey research and to review how they are being reported through a review of recently published reports of self-administered surveys.

Methods

Part 1: Identification of Current Guidance for Survey Research

Identifying guidance in “Instructions to Authors” sections in peer reviewed journals.

Using a strategy originally developed by Altman [12] to assess endorsement of CONSORT by top medical journals, we identified the top five journals from each of 33 medical specialties, and the top 15 journals from the general and internal medicine category, using Web of Science citation impact factors (list of journals available on request). The final sample consisted of 165 unique journals (15 appeared in more than one specialty).

We reviewed each journal's “Instructions to Authors” web pages as well as related PDF documents between January 12 and February 9, 2009. We used the “find” features of the Firefox web browser and Adobe Reader software to identify the following search terms: survey, questionnaire, response, response rate, respond, and non-responder. Web pages were hand searched for statements relevant to survey research. We also conducted an electronic search (MEDLINE 1950 – February Week 1, 2009; terms: survey, questionnaire) to identify whether the journals have published survey research.

Any relevant text was summarized by journal into categories: “No guidance” (survey related term found; however, no reporting guidance provided); “One directive” (survey related term(s) found that included one brief statement, directive or reference(s) relevant to reporting survey research); and “Guidance” (survey related term(s) including more than one statement, instruction and/or directive relevant to reporting survey research). Coding was carried out by one coder (SK) and verified by a second coder (CB).

Identifying published survey reporting guidelines.

MEDLINE (1950 – April Week 1, 2011) and PsycINFO (1806 – April Week 1, 2011) electronic databases were searched via Ovid to identify relevant citations. The MEDLINE electronic search strategy (Text S1), developed by an information specialist, was modified as required for the PsycINFO database. For all papers meeting eligibility criteria, we hand-searched the reference lists and used the “Related Articles” feature in PubMed. Additionally, we reviewed relevant textbooks and web sites. Two reviewers (SK, CB) independently screened titles and abstracts of all unique citations to identify English language papers and resources that provided explicit guidance on the reporting of survey research. Full-text reports of all records passing the title/abstract screen were retrieved and independently reviewed by two members of the research team; there were no disagreements regarding study inclusion and all eligible records passing this stage of screening were included in this review. One researcher (CB) undertook a thematic analysis of identified guidance (e.g., sample selection, response rate, background, etc.), which was subsequently reviewed by all members of the research team. Data were summarized as frequencies.

Part 2: Systematic Review of Published Studies on the Quality of Survey Reporting

The results of the above search strategy (Text S1) were also screened by the two reviewers to identify publications providing evidence on the quality of reporting of survey research in the health science literature. We identified the aspects of reporting survey research that were addressed in these evaluative studies and summarized their results descriptively.

Part 3: Assessment of Quality of Survey Reporting

The results from Part 1 and Part 2 identified items critical to reporting survey research and were used to inform the development of a data abstraction tool. Thirty-two items were deemed most critical to the reporting of survey research on that basis. These were compiled and categorized into a draft data abstraction tool that was reviewed and modified by all the authors, who have expertise in research methodology and survey research. The resulting draft data abstraction instrument was piloted by two researchers (CB, SK) on a convenience sample of survey articles identified by the authors. Items were added and removed and the wording was refined and edited through discussion and consensus among the coauthors. The revised final data abstraction tool (Table S1) comprised 33 items.

Aiming for a minimum sample size of 100 studies, we searched the top 15 journals (by impact factor) from each of four broad areas of health research: health science, public health, general/internal medicine, and medical informatics. These categories, identified through Web of Science, were known to publish survey research and covered a broad range of the biomedical literature. An Ovid MEDLINE search of these 57 journals (three were included in more than one topic area) included Medical Subject Heading (MeSH) terms (“Questionnaires,” “Data Collection,” and “Health Surveys”) and keyword terms (“survey” and “questionnaire”). The search was limited to studies published between January 2008 and February 2009.

We defined a survey as a research method by which information is gathered by asking people questions on a specific topic and the data collection procedure is standardized and well defined. The information is gathered from a subset of the population of interest with the intent of generating summary statistics that are generalisable to the larger population [1],[2].

Two reviewers (CB, SK) independently screened all citations (title and abstract) to determine whether the study used a survey instrument consistent with our definition. The same reviewers screened all full-text articles of citations meeting our inclusion criteria, and those whose eligibility remained unclear. We included all primary reports of self-administered surveys, excluding secondary analyses, longitudinal studies, or surveys that were administered openly through the web (i.e., studies that lacked a clearly defined sampling frame). Duplicate data extraction was completed by the two reviewers. Inconsistencies were resolved by discussion and consensus.

Results

Part 1: Identification of Current Guidance for Survey Research – “Instructions to Authors”

Of the 165 journals searched, 154 (93.3%) did not provide any guidance on survey reporting. Of these 154, 126 (81.8%) have published survey research, while 28 have not. Of the 11 journals providing some guidance, eight provided a brief phrase, statement of guidance, or reference; and three provided more substantive guidance, including more than one directive or statement. Examples are provided in Table 1. Although no reporting guidelines for survey research were identified, several journals referenced the EQUATOR Network's web site. The EQUATOR Network includes two papers relevant to reporting survey research [13],[14].

thumbnail

Table 1. Instructions to authors—Examples of relevant text per category.

doi:10.1371/journal.pmed.1001069.t001

The EQUATOR Network also links to the STROBE (STrengthening the Reporting of OBservational studies in Epidemiology) Statement (www.strobe-statement.org). Although the STROBE Statement includes cross-sectional studies, a class of studies that subsumes surveys, not all surveys are epidemiological. Additionally, STROBE does not include Methods' and Results' reporting characteristics that are unique to surveys (Table S1).

Part 1: Identification of Current Guidance for Survey Research - Published Survey Reporting Guidelines

Our search identified 2,353 unique records (Figure 1), which were title-screened. One-hundred sixty-four records were included in the abstract screen, from which 130 were excluded. The remaining 34 records were retrieved for full-text screening to determine eligibility. There was substantial agreement between reviewers across all the screening phases (kappa = 0.73; 95% CI 0.69–0.77).

thumbnail

Figure 1. Flow diagram of records and reports—Guidelines for survey research and evidence on the quality of reporting of surveys.

doi:10.1371/journal.pmed.1001069.g001

We identified five papers [13][17] and one internet site [18] that provided guidance on the reporting of survey research. None of these sources reported using valid measures or explicit methods for development. In all cases, in addition to more descriptive details, the guidance was presented in the form of a numbered or bulleted checklist. One checklist was excluded from our descriptive analysis as it was very specific to the reporting of internet surveys [16]. Two checklists were combined for analysis because one [14] was a slightly modified version of the other [17].

Amongst the four checklists, 38 distinct reporting items were identified and grouped in eight broad themes: background, methods, sample selection, research tool, results, response rates, interpretation and discussion, and ethics and disclosure (Table 2). Only two items appeared in all four checklists: providing a description of the questionnaire instrument and describing the representativeness of the sample to the population of interest. Nine items appear in three checklists, 17 items appear in two checklists, and 10 items appear in only one checklist.

thumbnail

Table 2. Checklist items for reporting survey research.

doi:10.1371/journal.pmed.1001069.t002

Part 2: Systematic Review of Published Studies on the Quality of Survey Reporting

Screening results are presented in Figure 1. Eight papers were identified that addressed the quality of reporting of some aspect of survey research. Five studies [19][23] addressed the reporting of response rates; three evaluated the reporting of non-response analyses in survey research [20],[21],[24]; and two assessed the degree to which authors make their survey instrument available to readers (Table 3) [25],[26].

thumbnail

Table 3. Systematic review – evidence on the of quality of reporting of survey research.

doi:10.1371/journal.pmed.1001069.t003

Part 3: Assessment of Quality of Survey Reporting from the Biomedical Literature

Our search identified 1,719 citations: 1,343 citations were excluded during title/abstract screening because these studies did not use a survey instrument as their primary research tool. Three hundred seventy-six citations were retrieved for full-text review. Of those, 259 did not meet our eligibility criteria; reasons for their exclusion are reported in Figure 2. The remaining 117 articles, reporting results from self-administered surveys, were retained for data abstraction.

thumbnail

Figure 2. Identification process for article selection—Review of published reports of self-reported surveys.

doi:10.1371/journal.pmed.1001069.g002

The 117 articles were published in 34 different journals: 12 journals from health science, seven from medical informatics, 10 from general/internal medicine, and eight from public health (Table S2). The median number of pages per study was 8 (range 3–26). Of the 33 items that were assessed using our data abstraction form, the median number of items reported was 18 (range 11–25).

Reporting Characteristics: Title, Abstract, and Introduction

The majority (113 [97%]) of articles used the term “survey” or “questionnaire” in the title or abstract; four articles did not use a term to indicate that the study was a survey. While all of the articles presented a background to their research, 17 (15%) did not identify a specific purpose, aim, goal, or objective of the study. (Table 4)

thumbnail

Table 4. Items reported by 117 included articles.

doi:10.1371/journal.pmed.1001069.t004

Reporting Characteristics: Methods

Approximately one-third (40 [34%]) of survey research reports did not provide access to the questionnaire items used in the study in either the article, appendices, or an online supplement. Of those studies that reported the use of existing survey questionnaires, the majority (40/52 [77%]) did not report the psychometric properties of the tool (although all but two did reference their sources). The majority of studies that developed a novel questionnaire (91/111 [82%]) failed to clearly describe the development process and/or did not describe the methods used to pre-test the tool; the majority (89/111 [80%]) also failed to report the reliability or validity of a newly developed survey instrument. For those papers which used survey instruments that required scoring (n = 95), 63 (66%) did not provide a description of the scoring procedures.

With respect to a description of sample selection methods, 104 (89%) studies did not describe the sample's representativeness of the population of interest. The majority (110 [94%]) of studies also did not present a sample size calculation or other justification of the sample size.

There were 23 (20%) papers for which we could not determine the mode of survey administration (i.e., in-person, mail, internet, or a combination of these). Forty-one (35%) articles did not provide information on either the type (i.e. phone, e-mail, postal mail) or the number of contact attempts. For 102 (87%) papers, there was no description of who was identified as the organization/group soliciting potential research subjects for their participation in the survey.

Twelve (10%) papers failed to provide a description of the methods used to analyse the data (i.e., a description of the variables that were analysed, how they were manipulated, and the statistical methods used). However, for a further 55 (47%) studies, the data analysis would be a challenge to replicate based on the description provided in the research report. Very few studies provided methods for analysis of non-response error, calculating response rates, or handling missing item data (15 [13%], 5 [4%], and 13 [11%] respectively). The majority (112 [96%]) of the articles did not provide a definition or cut-off limit for partial completion of questionnaires.

Reporting Characteristics: Results

While the majority (89 [76%]) of papers provided a defined response rate, 28 studies (24%) failed to define the reported response rate (i.e., no information was provided on the definition of the rate or how it was calculated), provided only partial information (e.g., response rates were reported for only part of the data, or some information was reported but not a response rate), or provided no quantitative information regarding a response rate. The majority (104 [87%]) of studies did not report the sample disposition (i.e., describing the number of complete and partial returned questionnaires according to the number of potential participants known to be eligible, of unknown eligibility, or known to be ineligible). More than two-thirds (80 [68%]) of the reports provided no information on how non-respondents differed from respondents.

Reporting Characteristics: Discussion and Ethical Quality Indicators

While all of the articles summarized their results with regard to the objectives, and the majority (110 [94%]) described the limitations of their study, most (90 [77%]) did not outline the strengths of their study and 70 (60%) did not include any discussion of the generalisability of their results.

When considering the ethical quality indicators, reporting was varied. While three-quarters (86 [74%]) of the papers reported their source of funding, approximately the same proportion (88 [75%]) did not include any information on consent procedures for research participants. One-third (40 [34%]) of papers did not report whether the study had received research ethics board review.

Discussion

Our comprehensive review, to identify relevant guidance for survey research and evidence on the quality of reporting of surveys, substantiates the need for a reporting guideline for survey research. Overall, our results show that few medical journals provide guidance to authors regarding survey research. Furthermore, no validated guidelines for reporting surveys currently exist. Previous reviews of survey reporting quality and our own review of 117 published studies revealed that many criteria are poorly reported.

Surveys are common in health care research; we identified more than 117 primary reports of self-administered surveys in 34 high-impact factor journals over a one-year period. Despite this, the majority of these journals provided no guidance to authors for reporting survey research. This may stem, at least partly, from the fact that validated guidelines for survey research do not exist and that recommended quality criteria vary considerably. The recommended reporting criteria that we identified in the published literature are not mutually exclusive, and there is perhaps more overlap if one takes into account implicit and explicit considerations. Regardless of these limitations, the lack of clear guidance has contributed to inconsistency in the literature; both this work and that of others [19][26] shows that key survey quality characteristics are often under-reported.

Self-administered sample surveys are a type of observational study and for that reason they can fall within the scope of STROBE. However, there are methodological features relevant to sample surveys that need to be highlighted in greater detail. For example, surveys that use a probability sampling design do so in order to be able to generalise to a specific target population (many other types of observational research may have a more “infinite” target population); this emphasizes the importance of coverage error and non-response error – topics that have received attention in the survey literature. Thus, in our data abstraction tool, we placed emphasis on specific methodological details excluded from STROBE – such as non-response analysis, details of strategies used to increase response rates (e.g., multiple contacts, mode of contact of potential participants), and details of measurement methods (e.g., making the instrument available so that readers can consider questionnaire formatting, question framing, choice of response categories, etc.).

Consistent with previous work [25],[26], fully one-third of our sample failed to provide access to any survey questions used in the study. This poses challenges both for critical analysis of the studies and for future use of the tools, including replication in new settings. These challenges will be particularly apparent as the articles age and study authors become more difficult to contact [25].

Assessing descriptions of the study population and sampling frame posed particular challenges in this study. It was often unclear whom the authors considered to be the population of interest. To standardise our assessment of this item, we used a clearly delineated definition of “survey population” and “sampling frame” [3],[27]. A survey reporting guideline could help this issue by clearly defining the difference between the terms and descriptions of “population” and “sampling frame.”

Our results regarding reporting of response rates and non-response analysis were similar to previously published studies [19][24]. In our sample, 24% of papers assessed did not provide a defined response rate and 68% did not provide results from non-response analysis. The wide variation in how response rates are reported in the literature is perhaps a historical reflection of the limited consensus or explicit journal policy for response rate reporting [22],[28],[29]. However, despite lack of explicit policies regarding acceptable standards for response rates or the reporting of response rates, journal editors are known to have implicit policies for acceptable response rates when considering the publication of surveys [17],[22],[29],[30]. Given the concern regarding declining response rates to surveys [31], there is a need to ensure that aspects of the survey's design and conduct are well reported so that reviewers can adequately assess the degree of bias that may be present and allay concerns over the representativeness of the survey population.

With regard to the ethical quality indicators, sources of study funding were often reported (74%) in this sample of articles. However, the reporting of research ethics board approval and subject consent procedures were reported far less often. In particular, the reporting of informed consent procedures was often absent in studies where physicians, residents, other clinicians or health administrators were the subjects. This finding may suggest that researchers do not perceive doctors and other health-care professionals and administrators to be research subjects in the same way they perceive patients and members of the public to be. It could also reflect a lack of current guidelines that specifically address the ethical use of health services professionals and staff as research subjects.

Our research is not without limitations. With respect to the review of journals' “Instructions to Authors,” the study was cross-sectional in contrast with the dynamic nature of web pages. Since our searches in early 2009, several journals have updated their web pages. It has been noted that at least one has added a brief reference to the reporting of survey research.

A second limitation is that our sample included only the contents of “Instructions to Authors” web pages for higher-impact factor journals. It is possible that journals with lower impact factors contain guidance for reporting survey research. We chose this approach, which replicates previous similar work [12], to provide a defensible sample of journals.

</?twb=.3w?>Third, the problem of identifying non-randomised studies in electronic searches is well known and often related to the inconsistent use of terminology in the original papers. It is possible that our search strategy failed to identify relevant articles. However, it is unlikely that there is an existing guideline for survey research that is in widespread use, given our review of actual surveys, instructions to authors, and reviews of reporting quality.

Fourth, although we restricted our systematic review search strategy to two health science databases, our hand search did identify one checklist that was not specific to the health science literature [18]. The variation in recommended reporting criteria amongst the checklists may, in part, be due to variation in the different domains (i.e., health science research versus public opinion research).

Additionally, we did not critically appraise the quality of evidence for items included in the checklists nor the quality of the studies that addressed the quality of reporting of some aspect of survey research. For our review of current reporting practices for surveys, we were unable to identify validated tools for evaluation of these studies. While we did use a comprehensive and iterative approach to develop our data abstraction tool, we may not have captured information on characteristics deemed important by other researchers. Lastly, our sample was limited to self-administered surveys, and the results may not be generalisable to interviewer-administered surveys.

Recently, Moher and colleagues outlined the importance of a structured approach to the development of reporting guidelines [7]. Given the positive impact that reporting guidelines have had on the quality of reporting of health research [8][11], and the potential for a positive upstream effect on the design and conduct of research [32], there is a fundamental need for well-developed reporting guidelines. This paper provides results from the initial steps in a structured approach to the development of a survey reporting guideline and forms the foundation for our further work in this area.

In conclusion, there is limited guidance and no consensus regarding the optimal reporting of survey research. While some key criteria are consistently reported by authors publishing their survey research in peer-reviewed journals, the majority are under-reported. As in other areas of research, poor reporting compromises both transparency and reproducibility, which are fundamental tenets of research. Our findings highlight the need for a well developed reporting guideline for survey research – possibly an extension of the guideline for observational studies in epidemiology (STROBE) – that will provide the structure to ensure more complete reporting and allow clearer review and interpretation of the results from surveys.

Supporting Information

Table S1.

Data abstraction tool items and overlap with STROBE.

doi:10.1371/journal.pmed.1001069.s001

(DOC)

Table S2.

Journals represented by 117 included articles.

doi:10.1371/journal.pmed.1001069.s002

(DOC)

Text S1.

Ovid MEDLINE search strategy.

doi:10.1371/journal.pmed.1001069.s003

(DOC)

Acknowledgments

We thank Risa Shorr (Librarian, The Ottawa Hospital) for her assistance with designing the electronic search strategy used for this study.

Author Contributions

Conceived and designed the experiments: JG DM CB SK JB IG BP. Analyzed the data: CB SK JB DM JG. Contributed to the writing of the manuscript. CB SK JB IG DM BP JG. ICMJE criteria for authorship read and met. CB SK JB IG DM BP JG. Acqusition of data: CB SK.

References

  1. 1. Groves RM, Fowler FJ, Couper MP, Lepkowski JM, Singer E, et al. (2004) Survey Methodology. Hoboken (New Jersey): John Wiley & Sons, Inc.
  2. 2. Aday LA, Cornelius LJ (2006) Designing and Conducting Health Surveys. Hoboken (New Jersey): John Wiley & Sons, Inc.
  3. 3. McColl E, Jacoby A, Thomas L, Soutter J, Bamford C, et al. (2001) Design and use of questionnaires: a review of best practice applicable to surveys of health service staff and patients. Health Technol Assess 5: 1–256.
  4. 4. Edwards P (2010) Questionnaires in clinical trials: guidelines for optimal design and administration. Trials 11: 2.
  5. 5. Simera I, Moher D, Hoey J, Schulz KF, Altman DG (2009) The EQUATOR Network and reporting guidelines: Helping to achieve high standards in reporting health research studies. Maturitas 63: 4–6.
  6. 6. EQUATOR NetworkIntroduction to Reporting Guidelines. Available: http://www.equator-network.org/resource-​centre/library-of-health-research-report​ing/reporting-guidelines/#what. Accessed 23 November 2009.
  7. 7. Moher D, Schulz KF, Simera I, Altman DG (2010) Guidance for developers of health research reporting guidelines. PLoS Med 7: e1000217. doi:10.1371/journal.pmed.1000217.
  8. 8. Smidt N, Rutjes AW, van der Windt DA, Ostelo RW, Bossuyt PM, et al. (2006) The quality of diagnostic accuracy studies since the STARD statement: has it improved? Neurology 67: 792–797.
  9. 9. Plint AC, Moher D, Morrison A, Schulz K, Altman DG, et al. (2006) Does the CONSORT checklist improve the quality of reports of randomised controlled trials? A systematic review. Med J Aust 185: 263–267.
  10. 10. Smith BA, Lee HJ, Lee JH, Choi M, Jones DE, et al. (2008) Quality of reporting randomized controlled trials (RCTs) in the nursing literature: application of the consolidated standards of reporting trials (CONSORT). Nurs Outlook 56: 31–37.
  11. 11. Prady SL, Richmond SJ, Morton VM, Macpherson H (2008) A systematic evaluation of the impact of STRICTA and CONSORT recommendations on quality of reporting for acupuncture trials. PLoS ONE 3: e1577. doi:10.1371/journal.pone.0001577.
  12. 12. Altman DG (2005) Endorsement of the CONSORT statement by high impact medical journals: survey of instructions for authors. BMJ 330: 1056–1057.
  13. 13. Kelley K, Clark B, Brown V, Sitzia J (2003) Good practice in the conduct and reporting of survey research. Int J Qual Health Care 15: 261–266.
  14. 14. Burns KE, Duffett M, Kho ME, Meade ME, Adhikari NK, et al. (2008) A guide for the design and conduct of self-administered surveys of clinicians. CMAJ 179: 245–252.
  15. 15. Draugalis JR, Coons SJ, Plaza CM (2008) Best practices for survey research reports: a synopsis for authors and reviewers. Am J Pharm Educ 72: 11.
  16. 16. Eysenbach G (2004) Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res 6: e34.
  17. 17. Huston P (1996) Reporting on surveys: information for authors and peer reviewers. CMAJ 154: 1695–1704.
  18. 18. AAPORHome page of the American Association of Public Opinion Research (AAPOR). Available: http://www.aapor.org. Accessed 20 January 2009.
  19. 19. Badger F, Werrett J (2005) Room for improvement? Reporting response rates and recruitment in nursing research in the past decade. J Adv Nurs 51: 502–510.
  20. 20. Asch DA, Jedrziewski MK, Christakis NA (1007) Response rates to mail surveys published in medical journals. J Clin Epidemiol 50: 1129–1136.
  21. 21. Cummings SM, Savitz LA, Konrad TR (2001) Reported response rates to mailed physician questionnaires. Health Serv Res 35: 1347–1355.
  22. 22. Johnson T, Owens L (2003) Survey Response Rate Reporting in the Professional Literature. Available: http://www.amstat.org/sections/srms/proc​eedings/y2003/Files/JSM2003-000638.pdf. Accessed 11 July 2011.
  23. 23. Smith TW (2002) Reporting Survey Nonresponse in Academic Journals. Int J Public Opin Res 14: 469–474.
  24. 24. Werner S, Praxedes M, Kim H (2007) The reporting of nonresponse analyses in survey research. Organizational Research Methods 10: 287–295.
  25. 25. Schilling LM, Kozak K, Lundahl K, Dellavalle RP (2006) Inaccessible novel questionnaires in published medical research: hidden methods, hidden costs. Am J Epidemiol 164: 1141–1144.
  26. 26. Rosen T, Olsen J (2006) Invited commentary: the art of making questionnaires better. Am J Epidemiol 164: 1145–1149.
  27. 27. Dillman DA (2007) Mail and Internet Surveys: The Tailored Design Method. Hoboken (New Jersey): John Wiley & Sons, Inc.
  28. 28. Coons SJ (2007) Responses to survey research: transparency and representativeness are key. Clin ther 29: 466–468.
  29. 29. Fincham JE (2008) Response rates and responsiveness for surveys, standards, and the Journal. Am J Pharm Educ 72: 43.
  30. 30. [No author listed] (1999) Editor's Preface. A word about manuscripts. CMAJ 161: 5.
  31. 31. Cook JV, Dickinson HO, Eccles MP (2009) Response rates in postal surveys of healthcare professionals between 1996 and 2005: an observational study. BMC Health Serv Res 9: 160.
  32. 32. Narahari SR, Ryan TJ, Aggithaya MG, Bose KS, Prasanna KS (2008) Evidence-based approaches for the Ayurvedic traditional herbal formulations: toward an Ayurvedic CONSORT model. J Altern Complement Med 14: 769–776.