Got polls? Disclose, disclose, disclose

See correction/update in post, added 5 hours after initial post.

Maine is a lightly-polled state, at least compared to larger states and the nation at whole.

Therefore it’s not surprising that every poll that’s released gets a fair amount of attention.

Unfortunately, some pollsters and news stories don’t meet the minimum disclosure standards for pollsters, as set out by the American Association of Public Opinion Research. (For these, see bottom of post.)

Those standards don’t ensure that the results will be exactly correct. Rather, as statisticians know, random samples enable pollsters to define the likelihood that a single poll’s result will depart from the actual by the the margin of error.

But not following the standards undermine every reader’s ability to assess the quality of the poll.

This is the case in a recent poll of 403 people from Pan Atlantic SMS. A Bangor Daily News article gives some information about the poll, such as the margin of error and minimal information about the composition of the sample.

Besides disclosing relatively little information about the sample, one clear reporting flaw is that the actual wording of the questions is not included. This is a rather significant problem, since differences in question wording can have a significant impact of the results found.

While everything desired by the AAPOR is a lot to include in a news story, it would be a trivial matter to link to the full information.

As more Maine polls roll in, fuller disclosure should become the normal practice.

Update/correction: When checking the news story again, which has been updated through the day, I discovered there was a link to the full poll report, which includes demographic data and question wording. The link may have been present in earlier versions but it was not obvious to me. I thank Patrick Murphy of Pan Atlantic-SMS for calling this to my attention. 

Regarding journalistic reports of polls, a better practice would have been to explicitly tell the reader that the full report could be found and to point him or her to its location. 

And one more thing: Maine news outlets should stop running those totally non-scientific surveys on their websites. They truly tell you nothing reliable about actual public opinion.

AAPOR standards for poll disclosure:

Disclose all methods of the survey to permit evaluation and replication.

Excellence in survey practice requires that survey methods be fully disclosed and reported in sufficient detail to permit replication by another researcher and that all data (subject to appropriate safeguards to maintain privacy and confidentiality) be fully documented and made available for independent examination. Good professional practice imposes an obligation upon all survey and public opinion researchers to include, in any report of research results, or to make available when that report is released, certain minimal essential information about how the research was conducted to ensure that consumers of survey results have an adequate basis for judging the reliability and validity of the results reported. Exemplary practice in survey research goes beyond such standards for “minimal disclosure,” promulgated by AAPOR and several other professional associations (e.g., CASRO and NCPP) by (a) describing how the research was done in sufficient detail that a skilled researcher could repeat the study, and (b) making data available for independent examination and analysis by other responsible parties (with appropriate safeguards for privacy concerns).

A comprehensive list of the elements proposed for disclosure by one or more sources which in combination, exceed the “standards for minimum disclosure” proposed by any one of the professional organizations includes:

  • who sponsored the survey, and who conducted it;
  • the purpose of the study, including specific objectives;
  • the questionnaire and/or the exact, full wording of all questions asked, including any visual exhibits and the text of any preceding instruction or explanation to the interviewer or respondents that might reasonably be expected to affect the response;
  • a definition of the universe the population under study which the survey is intended to represent, and a description of the sampling frame used to identify this population (including its source and likely bias);
  • a description of the sample design, including cluster size, number of callbacks, information on eligibility criteria and screening procedures, method of selecting sample elements, mode of data collection, and other pertinent information;
  • a description of the sample selection procedure, giving a clear indication of the methods by which respondents were selected by the researcher, or whether the respondents were entirely self-selected, and other details of how the sample was drawn in sufficient detail to permit fairly exact replication;
  • size of samples and sample disposition the results of sample implementation, including a full accounting of the final outcome of all sample cases: e.g., total number of sample elements contacted, those not assigned or reached, refusals, terminations, non-eligibles, and completed interviews or questionnaires;
  • documentation and a full description, if applicable, of any response or completion rates cited (for quota designs, the number of refusals), and (whenever available) information on how non respondents differ from respondents;
  • a description of any special scoring, editing, data adjustment or indexing procedures used;
  • a discussion of the precision of findings, including, if appropriate, estimates of sampling error with references to other possible sources of error so that a misleading impression of accuracy or precision is not conveyed and a description of any weighting or estimating procedures used;
  • a description of all percentages on which conclusions are based;
  • a clear delineation of which results are based on parts of the sample, rather than on the total sample;
  • method(s), location(s), and dates of interviews, fieldwork or data collection;
  • interviewer characteristics;
  • copies of interviewer instructions or manuals, validation results, codebooks, and other important working papers; and
  • any other information that a layperson would need to make a reasonable assessment of the reported findings.
Amy Fried

About Amy Fried

Amy Fried loves Maine's sense of community and the wonderful mix of culture and outdoor recreation. She loves politics in three ways: as an analytical political scientist, a devoted political junkie and a citizen who believes politics matters for people's lives.