A NEW DIRECTION FOR SURVEY RESEARCH
Address to the World Association for Public Opinion Research (WAPOR)
on receipt of the Helen Dinerman Award
New York, New York, September 1995

IJPOR No.1, March 1996

Daniel Yankelovich

Receiving this award has a special resonance for me because of my great affection and respect for Helen Dinerman. For me, Helen personified the hopes, ideals and ambitions of the profession. She and her contemporaries -- Paul Lazersfield, Sam Stouffer, Hadley Cantril and others -- saw themselves as practitioners of a powerful instrument of social science. In survey research they had found a way to blend the rigorous standards of science with insightful social theory and their own judgment, wit and wisdom.

They also shared a larger vision of survey research’s potential: Survey research would strengthen democracy by permitting leaders to understand the state of mind of the public in all of its subtlety and complexity and to address the real concerns of the electorate, free of bias. The kind of knowledge that good surveys provide would set the stage for genuine dialogue between leaders and citizens.

Since their inception, AAPOR and WAPOR have embraced this larger vision of survey research. Fifty years ago at the founding conference, Julian Woodward foresaw the expansion of polling to clarify policy issues. So important to the society would this role become that Woodward felt polls must not be "subject only to the laws of the marketplace" but evolve into a "public utility imbued with the public interest." Hence, he concluded, "polls must conduct themselves in such a way as to justify the responsibilities which will increasingly be theirs and to deserve the respect with which the public will regard them."

The Fate of the Vision

What is the current status of this vision?

In the United States ( I hesitate to generalize to other societies), the news is mostly bad. Despite the polling profession’s greater technical sophistication, polls have grown ever more misleading. Far from serving to give leaders insight into the real concerns of the public, they often add to the lack of confidence that disconnects the leadership class in the United States from the mainstream of the citizenry.

There has been an enormous proliferation of media polls in the United States. Almost all, even the very best, have come to reflect the current fashion in polling, which is to provide the public and users of polls with quick snapshots of public opinion. In these snapshots, virtually all complex issues are reported in the form of answers to single questions. For example:

What, then, is the effect on policy makers when they act on these poll findings, only to discover too late that the polls did not mean what they said? The policy maker who urges the five cent tax on gasoline later discovers to his shock and dismay that a majority of voters vehemently reject politicians who support gasoline surtaxes. Based on poll findings, the Clinton Administration assumed that the public supported its program of providing universal health insurance, only to learn later that it had been badly misled.

Let me elaborate this last point. In a postmortem analysis of why the Clinton health care plan failed, I counted no fewer than 17 different national polls which had reported the average level of public support for universal health care insurance -- a key feature of the President’s failed health care plan -- to be an impressive 71 percent. Both President Clinton and Mrs. Clinton (who headed up the President’s task force on health care) relied on these polls and took them literally.

These 17 polls, however, turned out to mean almost the very opposite of what they seemed to mean. What most Americans really mean when they say they support universal insurance coverage can be paraphrased this way: "Ideally, we don’t want anybody to be deprived of health care because of lack of money. We therefore endorse the President’s goal of ‘insurance for everyone that can never be taken away’ -- provided, of course, that the nation can afford it, and provided also that it doesn't limit our choice of doctors or raise our taxes or cause employers to cut jobs."

By taking the poll finding literally, the President was unprepared for the reality that at the time he introduced his health care plan Americans were oblivious to the costs and consequences of universal coverage, and that once they began to grasp these consequences it caused them to fear that they would suffer personally: they were afraid they would end up with poorer quality care for which they would have to pay higher taxes and also be saddled with bureaucratic government programs. In the few private polls that did probe beneath the surface, once people began to consider costs and consequences the 71 percent level of support evaporated in thin air: it sank to between 20 to 30 percent, not enough to support the President’s plan.

I could go on multiplying examples of poll findings that badly mislead policy makers -- and not because of bias or deliberate deception or bad sampling or technical amateurishness. They mislead because the prevailing snapshot conception of polling is radically incomplete: it leaves out of account essential information about the state of the public’s opinion: whether it is stable or volatile; whether it contradicts other opinions that people also hold; and whether people are aware or unaware of the consequences of their own views and how they would respond to these consequences.

I believe that two very different causes are driving the flood of misleading poll results. One is familiar to all of us; the other is new and less familiar.

Media Ownership. The familiar cause is the unintended consequence of media ownership of polls. As public opinion polls have grown in popularity, and as magazines, newspapers and TV stations feel they must control their own proprietary opinion polls, journalistic values inevitably take precedence over values essential to the credibility of survey research.

Many media polls are primarily designed to catch an issue when it is hot and to produce snappy headlines ( "Public Disagrees with High Court on Abortion") or 30-second sound-bites. These polls rocket briefly into public view and disappear as quickly. The values that motivate them are journalistic, not oriented toward public benefit or insight.

Sad to say, the media who sponsor opinion polls on policy issues have little or no stake in the quality of the poll findings they report. The validity of a poll finding that makes a good headline is a subject that editors would just as soon not dwell upon. Because there are no easy ways to prove validity, the editors often opt to avoid the subject. They do not seek deliberately to mislead, but their canons of journalistic validity do not apply to survey research, and they would just as soon not go to extra trouble and expense. A ritualistic bow to sampling error (plus or minus 3 percent) salves their conscience and gives the user the ludicrously false impression that poll findings mean what they say within a plus or minus 3 percent margin.

As the mass media have taken control over the polling profession, Gresham's law applies in full force. The quickie poll becomes the standard. Fewer resources are available for the kinds of searching studies conducted in Helen's day, and when such studies are done, they are lumped together with all the others. The prevailing attitude is, "a poll is a poll is a poll." Poor quality drives out good quality.

So long as the media values polls for their attention-getting abilities we will have more quickie polls, more single item findings, more confusion, more misleadingness and less genuine understanding.

The Other Driver. The less familiar driver of misleading poll findings is not a matter of ownership or a technical defect in polling. It is a qualitative change in the nature of public attitudes.

The dominant public mood in the United States is one of frustration. The American public is frustrated by many aspects of contemporary life. The majority of Americans fear that they are in danger of losing what they have. They fear that their children may not live as well as they themselves do. They fear they might lose their jobs or their health care. They fear that a college education for their kids is growing prohibitively expensive, at exactly the time when a higher education is essential to making a good living. Even successful dual earner households feel they are on a treadmill: they feel that for the sake of maintaining a high standard of living they have sacrificed time with their kids, time with each other, and time for leisure. And yet, even with these sacrifices they are falling further and further behind. They have come to mistrust most of their institutions, especially government.

In this disturbing mood, when pollsters ask them questions, they often answer by ventilating their frustrations, or they say "Yes, Yes" to almost any harebrained idea that seems to offer hope of a solution. Unfortunately, many politicians read these polls as if they were engraved in stone -- the authentic voice of the people. They would not make this mistake if they were talking face to face with their constituents. But when an opinion appears in a poll with a number attached to it, it looks scientific, even though it may simply voice people’s frustrations, not their considered judgments.

For example, when you ask the public how criminals and drug pushers should be treated, the first thing people say in polls is: "lock them up and throw away the key." If you ask a few more questions, you realize that most people don’t mean to be taken literally. They are merely ventilating their feelings. They know that prisons are breeding grounds for training criminals and drug pushers and that they take resources away from badly needed social goals. (You can send three young people to a good college for the amount of money it takes to house one prisoner). But polls that report only what is, in effect, people clearing their throats and venting their anger can seriously mislead.

We are thus confronted with a bitter irony. In recent years, the profession has made great strides in its technical prowess. But just when it has established the technical base it needs to do truly insightful survey research, it is saddled with a misleading conception of polling that threatens its credibility and its claim to be an important field of inquiry.

Never has the vision of the founders of survey research faced greater danger.

A Two Pronged Strategy

What is to become of the vision that Helen Dinerman symbolized? Will it quietly fade away, to be commemorated wistfully on occasions such as this? Will it be replaced by ever more ingenious technology with precious little thought, wisdom or creativity? Unfortunately, this is the direction toward which survey research is headed in the United States.

Does it have to happen?

Of course not. There is nothing inevitable about it. But stopping and reversing its momentum will require vigorous initiative.

Two kinds of leadership are needed to salvage the vision and put it back on the track to fulfillment. The first is short term damage control. The second is giving survey research a new focus.

Let me say a word about each.

Damage Control

Unfortunately, a profession like survey research is poorly organized to take a leadership initiative. This weakness is not unique to our profession; it is characteristic of all professional organizations. Most professionals are so absorbed in their work that they have neither the time nor the inclination to act in a leadership capacity. As a result, they are rarely organized to take initiative when it is needed.

If the profession were to organize itself for proactive leadership, it should probably start with short term "damage control" to slow the flood of misleading poll findings. The leadership should look for modest and practical actions that can be taken immediately to diminish the misleading nature of single item poll answers on complex policy issues.

There are several such actions that might be considered. A number of years ago, my firm attempted to do something about the problem. We developed for Time magazine a simple battery of questions for determining whether public opinion on important policy issues was firmly and thoughtfully held or volatile and likely to change from one form of question wording to another. Making this determination was particularly important on questions for foreign policy where other nations were being misled by American poll findings. We called the innovation a "volatility measure"; the editor of Time referred to it simply as "the mushiness index."

We suggested that Time use a visual devise like an asterisk to signal readers whether a finding was stable or "mushy". Time liked the idea, paid for its further development, willingly shared it with others -- and then never used it. It added just enough extra expense and complexity of explanation to discourage the editors from doing something that in their view did not have much journalistic value.

Related to volatility, but different and a more serious source of misleadingness, is the fact that people often answer poll questions without being mindful of the consequences of their views. Leo Bogart has pointed out that questions such as "Are you for or against a balanced budget amendment?" inevitably generate meaningless answers. This is because respondents have given little thought to "what a Constitution is for" or what sacrifices they will have to make if such an amendment came to pass.

To exercise damage control on this problem, the profession ought to insist that all queries on important policy matters be accompanied by one or more questions that probe whether the public is aware of the consequences and ramifications of its opinions, and how it reacts to these. To be effective, this probing cannot simply ask people vague and bland questions such as whether they would be willing to pay slightly higher taxes for x, y or z policy. The profession is capable of creating realistic probes to which people can react.

I am convinced that if the profession as a whole insisted that innovations such as these were even more important to validity than the calculation of sampling error, they would be adopted. Journalists do not wish to perpetrate error; they just wish to avoid extra expense, bother and complexity of explanation. And why should they not, unless those who uphold the standards of the profession insist upon a minimum standard for avoiding misleading the public?

A Change in Direction

This kind of short term damage control will slow the downward trend, but it will not stop or reverse it. To do that a fundamental change in direction is needed.

The present direction suffers from two devastating flaws: it puts the credibility of the profession at risk because of the proliferation of misleading poll results, and it degrades the vision that inspired the founders of survey research.

The new direction I propose addresses both flaws. It conceives of public opinion differently than the polls-as-snapshots concept. The snapshot concept implies that public opinion is in the process of evolving, but it doesn't say how it is evolving. In some situations, this information is unnecessary, for example, voters’ views of Presidential candidates at various stages in an election campaign. In this context, everyone already knows that a campaign is in progress, what stage it has reached, and when it will end.

But what happens when the context shifts from a campaign to a policy issue? Unlike an election, most issues do not have a definite beginning, middle and end. Public opinion may be evolving, but no one knows how and where it may be heading.

For all policy issues, I propose that the new direction of survey research be to locate what stage of its evolution public opinion has reached on any one issue. More than forty years of survey research experience have persuaded me that all policy issues evolve in roughly the same manner and proceed through the same invariant steps. They all start with ‘raw opinion’-- opinion that is unformed, volatile and often self-contradictory. They then evolve through a series of stages. By journey’s end, if they haven’t bumped into an obstacle that retards their evolution, people’s opinions on the issue have grown firm and stable, more internally consistent, and above all, more conscious of the consequences of their views and more willing to accept responsibility for them. Over the course of the long journey of evolution, raw opinion has been transmuted into thoughtful, deliberative judgment.

In my book, Coming to Public Judgment, I describe seven steps through which public opinion evolves on complex issues:

• becoming aware of the issue (Step 1);
• developing a sense of urgency (Step 2);
• searching for action (Step 3);
• confronting wishful thinking and other forms of resistance (Step 4);
• "choicework": deliberating the pros and cons of hard choices, weighing tradeoffs and clarifying priorities (Step 5);
• achieving cognitive resolution (Step 6), and
• coming to full deliberative judgment (Step 7).
  

This complex series of steps is sometimes referred to as the "deliberative process." But the term conjures up a misleading image of calm rational contemplation. The process of moving from raw opinion to considered judgment is far more stormy and emotional, ridden with contradictions, conflicts and intense feelings. But on critical public policy issues, going through all seven steps is unavoidable. If any are skipped, public support will be thin and unreliable, melting away at the first sign of trouble.

Typically, it takes years to evolve though all seven steps. More often than not, people get stalled at one or another step and can remain stalled for years, even decades.

Let me illustrate the process with an example. Where does health care reform stand today in terms of this 7-step Public Judgment model? Through what steps has it already progressed? What comes next?

• Awareness. This first stage has been reached and passed. Americans started to have their consciousness raised about swelling health care costs way back in the mid 1970s, two decades ago! From the mid-seventies to the early 1990s, awareness grew slowly, always lagging behind an ever worsening situation.
  
• Urgency. This stage has also been reached and passed. The upset senatorial victory of Democrat Harris Wofford in Pennsylvania in November 1991 (Wofford ran and won on health care) gave the issue strong political urgency. Abruptly, it seized the center of the political stage.
  
• Searching for Action. The public has also passed this stage. Starting in the Presidential election of 1992, voters began to demand that political leaders offer specific proposals for reforming the health care system.
  
• Resistance. Here in the stage of resistance is where the American public is currently bogged down. The public is just beginning to confront its own wishful thinking and to acknowledge that attacking waste and greed will not magically improve benefits and maintain quality and access and guarantee good medical care for everyone and at the same time bring rising costs under control. Americans are just beginning to learn that "managed care" is not the magic bullet that will deliver all of these benefits without calling for sacrifice on anybody’s part.
  
• Choicework. Americans have not yet reached the choicework stage. They remain stalled at the resistance stage on all of those aspects of health care that requires citizens to make sacrifices or accept real change. Only when their resistances have been confronted and worked through will they be ready to do the hard choicework of deliberating alternative policy choices and their pros and cons.
  
• Cognitive Resolution. After choicework comes the sixth step when people reach tentative conclusions. For example, at some time in the future people will come to accept in their minds the need for curbs on heroic medicine. Initially, they will not fully realize all of its emotional and moral ramifications as, for example, when they have to forego high tech interventions that could conceivably prolong life for a loved one.
  
• Judgment. In the seventh and final step, people reach emotional and moral conviction. When, for example, people let a loved one die without resorting to heroic, last-ditch measures, and fully accept all of the practical, emotional and moral consequences of their decisions, they would then have reached full deliberative judgment. The public is years away from reaching this stage.
  

The more closely I have studied the journey toward judgment, the more convinced I have become that leaders cannot win real public support without understanding what these stages are, where in the multistage process the public stands at any one time, and how to assist the public, when it is stuck at a particular stage, to move onto the next one.

When public opinion on an issue is located in the early stages of its evolutionary journey, we know that one of its defining characteristics is its volatility. This volatility is not evidence of the shortcomings of polling techniques. It is a substantive finding. It is important that poll users know whether they are dealing with soft, thin, mushy public opinion at an early stage of evolution or with rock solid opinion at a later stage.

When people feel an issue is urgent, are hungry for action and desperate for results (steps 2 and 3), we know that poll results are likely to be maximally misleading. In this state of mind, the public’s support for any and all issues will be inflated and will fall away at the first dawning of realism.

Once leaders have insight into the stage of evolution of public opinion, the opportunity for genuine democratic dialogue opens wide. Leaders and public are able to engage each other on the issue instead of talking past each other in a dialogue of the deaf. The existing snapshot concept tempts leaders to use polls manipulatively, thereby weakening democracy. Locating what stage in the journey from raw opinion to judgment the public has reached lends itself far better to the founders’ vision of survey research as a mainstay of democracy.