A new paper released this week by a pair of political scientists claims, as The New York Times reports, that, “pastors are even more politically divided than the congregants in their denomination.” As the abstract of the paper states:

Pastors are important civic leaders within their churches munities. Several studies have demonstrated that the cues pastors send from the pulpit affect congregants’ political attitudes. However, we know little about pastors’ own political worldviews, which will shape the content and ideology of the messages transmitted to congregants. In this paper, we employ a novel methodology pile a database of over 130,000 American clergy across forty religious denominations. These data provide us with a sweeping view of the political attitudes of American clergy. Using CCES data, pare pastors’ partisanship to congregants’ political affiliation and policy views. The results demonstrate that pastors’ denominational affiliation is much more informative of their partisanship than for congregants. These results provide a nuanced understanding of the relationship between clergy’s political orientations and those of the individuals they lead.

Are pastors more partisan? My initial intention was to evaluate that claim based on the evidence provided in the paper, “Partisan Pastor: The Politics of 130,000 American Religious Leaders.” Like many of you, I had noticed my friends on social media linking to reports about it in such outlets as The New York Times and The Atlantic.

After reading the paper, though, I came to the conclusion the data wereinsufficient to support the conclusion. How did such a paper get published? As it turns out, it wasn’t.

The paper has not yet been published in a peer-reviewed political science journal, and one of the authors lists it on their curriculum vitae under “Works In Progress & Under Review.” As we will see, there is reason to believe the paper does not meet the standards worthy of publication.

Here are some of the specific concerns about the quality of the research:

• “To our knowledge, this is the pilation of religious leaders ever assembled,”the authors say. pilation is indeed impressive, but the means of collecting the data raise questions about its reliability, especially concerning minority pastors.

For example, to find the names of pastors from several minority denominations the authors “hired Mechanical Turk workers to find the pastors’ names.” Mechanical Turk is a service run by Amazon where anyone can hire people plete online tasks for mere pennies. For this data collection project workers were paid between one to three cents to search on the internet, locate the pastor’s name, and enter it into a form.

• What is the likelihood such data collection is accurate? It’s hard to say. But the fact that the authors are confused about the spelling of the name of one of America’s most (in)famous preachers—Joel Osteen is referred to as “Joel Olsteen”—raises questions about how stringent they were in collecting names of lesser-known ministers.

• Another concern with the data collection is with the way pastors were matched to voter registration. Almost half of the entries (44 percent) were considered a “match” if the name of the pastor matched a name on a voter registration list within muting distance” of the church (the authors do not clarify what they consider muting distance). While this is a creative method, it is bound to lead to numerous false positive results. For example, if there is a pastor named “Joe Carter” who lives outside the muting distance and another voter named Joe Carter who lives closer to the church, the latter person would be presumed to be a “match” even though heisnot only not the pastor but may not even belong to the same denomination or be of the same political persuasion.

• The authors base their claims of “partisanship” mitment to a particular political party or ideology) solely on voter registration. So a pastor or congregant is considered a “partisan” if he isregistered to vote as either a Republican or Democrat. Voter registration, as the authors admit, is a “basic” proxy for partisanship. So when they say a pastor is more “partisan” than their congregants, the authors are merely stating the pastor is more likely than his congregants to be registered for a particular political party.

• However, the fatal flaw of the paper is in its use of voter registration as the sole proxy for pastoral partisanship. As the paper acknowledges, only 29 states ask voters to register with a particular political party. How is the “partisanship” of pastors in the other 21 states identified? The paper doesn’t say because, based on their own criteria, they can’t know. Since the paper is based on pastors in a little more thanhalf of the states, it’s impossible to know how representative the results are for pastors in the entire United States.

This is a long-winded way of saying, “Don’t believe everything you read.” But it also highlights a concerning trend in religion reporting. As we’ve repeatedly seen over the past few years, reporting on religion that confirms the biases of secularists (e.g., Christian pastors are hyper-partisan Republicans) gets reported based on the flimsiest of evidence.

As the paper ironically notes, “Attitudes and behaviors of ordinary Americans are affected by ‘elite influencers.’” This is all too true, which is why the elite influencers in America’s newsrooms and Ivy League political science departments need to hold themselves to a higher standard of trustworthiness.

What reason did The New York Times have for reporting on an unpublished study? And why did they choose to do so a mere day after the unpublished paper was posted online (the paper is dated June 11)? Whatever the newspaper’s motives, they put their credibility behind the flawed study. Despite having never been published, the paper is likely to be cited for years e as credible “evidence” for the conclusion that pastors are more partisan than their congregants.