Survey Selection: Evaluating the Reliability of Different Types of Surveys

A guide to help journalists and discerning news consumers evaluate the reliability of different types of surveys.

As a data reporter, I’ve seen many examples of misused, misconstrued, or sensationalized survey results. My goal is to help people understand the strengths and weaknesses of survey data, and provide insight into the many factors that go into building, launching, and interpreting surveys.

In this series, we are exploring the use of public opinion polls and surveys in news media. Topics include responsible usage, methodology, and fact-checking with real-world examples.

CNBC survey on-air | CNBC article citing a similar version of the survey

Transparent methodology sections are the best way to tell if a survey is worth citing in a news article; without information about the survey itself, the data is no good. Knowing the type of survey being cited can help determine the reliability of the takeaways. Here we are going to look at three broad categories of surveys, each with different motivations for development and different techniques used to gather data.

Survey types:

• Academic surveys are the most reliable source for insights. While publications have been peer-reviewed for accuracy, the data can be unwieldy to work with.
• Public polling organizations and research institutions are next-best for reliable data as they follow consistent research design and often publish takeaways and data in readily accessible reports.
• Marketing or consumer surveys require the most reviewing as they are not beholden to academic standards and are often used to fill content gaps.

Useful definitions:

• Probability sample: a method of selecting survey participants in such a way that each member of the population has a known and equal chance of being included in the sample. This could include specific methods like simple random sampling or stratified sampling.
• Non-probability sample: a method of selecting survey participants where all members of the population being researched do not have an equal chance of being selected. Internet opt-in surveys are the most common example of this.
• Sample size: the number of individuals participating in a survey. Larger sample sizes generally mean more detailed and reliable analyses can be performed on the data. The sample size for every question and response should be considered, not just the overall sample; more on this later!
• Audience panel: a group of pre-recruited individuals who take part in surveys, typically selected to be demographically representative of a broader population. It is important to consider that any biases in the audience panel could be magnified during statistical analyses.
• Content marketing: a strategic tactic for businesses that focuses on creating and distributing valuable, relevant, and consistent content to attract and engage a target audience.
• Link-building: the business practice of getting authoritative websites (like news outlets) to hyperlink to a business page (featuring a research study) to boost search engine ranking and online visibility.

Academic survey research

Researchers at universities use surveys as a tool to gather data and information on a variety of topics, from political ideology to consumer sentiment. To hold up to the rigors of peer review and publication in academic journals, academic surveys often make use of very large sample sizes, probability sampling, and consistent survey design over time. Adding these benchmarks to a survey study increases costs and often requires funding through research grants and approval by an academic review board. This is important for maintaining independent research standards and producing high-quality, replicable data.

American National Election Study | “About the Dataset”

An example of extremely robust survey documentation can be found on the documentation page for the American National Election Studies, and their downloadable, 70-page PDF: ANES Methodology and Dataset User Guide and Codebook. Not all survey methodologies need to be this robust for journalists, but knowing that a team of researchers has put this effort into the data means it is extremely reliable for reporting.

Public opinion polling

Outside of academia, countless research organizations conduct public opinion research through surveys. These surveys are typically conducted using panel audiences selected using probability sampling methods to ensure that the results are representative of the population being studied. Reports from these organizations are (subjectively) easier to read than academic journals and the data is often readily available for further analysis.

The Associated Press-NORC at the University of Chicago | Methodology example

While many institutions adhere to many of the same benchmarks found in academia, different organizations may have different goals and biases which can affect how survey questions are framed and the results interpreted. It is important to evaluate the methodology and funding sources of any survey before drawing conclusions from the results. Similarly, a survey is considered to be a single snapshot of public opinion, reiterating the reliability of long-term studies typically reserved for academia and well-funded institutions.

Pew Research Center | “How we did this” and methodology example

Public opinion polling is widely used by the news media to gauge public sentiment on a variety of issues. The results of these polls are often reported in news articles and can shape public discourse and policy decisions. While journalists often rely on polls covering political candidates, policy issues, and social trends, it is still important to approach polling data with a critical eye.

Marketing and consumer surveys

Consumer surveys are often used by content marketing agencies and public relations companies as a way to generate unique content for their clients. They are cheap to launch and provide fresh, novel data that can be pitched to journalists for publicity or link-building campaigns. These surveys are often designed with leading questions and small sample sizes, making them nearly impossible to replicate and not representative of the larger population.

Forbes Advisor | PR survey with no additional methodology links

While these surveys can be useful for generating sensational headlines and shocking statistics, they should not be relied upon for making important decisions or drawing conclusions about public sentiment. Occasionally, they can be useful for exploring a topic not typically covered by more authoritative sources. It is important to critically evaluate any survey's methodology, funding sources, and purpose before accepting the results as valid and citing them in a news article.

How to check the reliability of a survey source

There are many reputable sources for survey data from universities, research institutions, and even market research companies. If the organization that sponsored the survey isn’t obvious, explore the methodology section; methodologies lacking detail should get extra scrutiny before earning a citation in a news article.

1. Review the article’s introduction for an executive summary stating the purpose or objective of the study and the publisher.
2. Check the credentials of the author to verify their background as a researcher or data scientist.
3. Assess the methodology section looking for full details on the sample size, survey date, data collection methods, and questionnaire design.
4. Look for sponsoring information to see if the study was developed for academic studies or marketing purposes.
5. Review data analysis techniques, noting any weighting or statistical inferences being made to the sample.
6. Consider the “gut-check” and ask yourself if the takeaways make sense or if the topic seems like something the general public would even have informed opinions of.

Understanding the type of survey being cited greatly helps with determining its reliability. While we focused on academic surveys, public opinion polling, and marketing or consumer surveys, there are many other ways and methods surveys can be launched and data gathered. Reviewing the methodology section and considering the purpose of the survey is a couple of the most important steps in this vetting step.

Stay connected: Upcoming content will delve into responsible usage and methodological considerations for survey data to explore these sources' strengths and weaknesses, featuring more vetting methods and practical walk-throughs of real-world examples. Let me know what topics you would like to explore!