The method of public deliberation (PD) provides decision makers with guidance from an informed public that will be impacted by a value-laden decision for which there is no “right” technical answer. In late 2020, the Center for Evaluation and Applied Research (CEAR) at The New York Academy of Medicine, working in collaboration with the NYC Department of Health and Mental Hygiene, convened a series of PDs to understand New Yorkers’ perspectives on the fairest way to prioritize COVID-19 vaccine receipt during an initial period when supplies are limited. Here, CEAR presents the various components of public deliberation and the tools used to implement the COVID-19 vaccine PD.
Developing and conducting a PD is resource intensive. By making these resources available, CEAR hopes that other localities will better understand PD and use the method to gather informed community member recommendations on fair vaccine distribution, or other related questions regarding population health and equity.
The set of resources included on this website can be adapted for use by local and state health departments to support public deliberation (PD), a method for gaining advice from an informed public regarding decisions that are value-laden and not easily decided on a technical basis. Contained here is a case study of public deliberations conducted by the Center for Evaluation and Applied Research (CEAR) at the New York Academy of Medicine (NYAM), and funded by Altman Foundation, the New York State Health Foundation and the United Hospital Fund, on behalf of the New York City Department of Health and Mental Hygiene (NYC DOHMH) in December 2020 and January 2021. The public deliberations were conducted to inform priorities for distribution of the COVID-19 vaccine in an initial period when supplies are limited.
The COVID-19 pandemic and development of the vaccine have required rapid decision making about allocation of vaccine at all levels of government. Some decisions have ultimately been made at the federal level and make the particular questions explored during this work less timely. Nonetheless, we think that the approach, and many of the materials provided here, will prove useful for adaptation by health departments in gaining informed constituent recommendations on population health and fair distribution of the vaccine during the COVID-19 pandemic. In addition, we see this approach as having enduring value for soliciting guidance from members of the public that relates more generally to health equity.
Public deliberation is a method for gathering recommendations from community members who will be affected by a decision when there is no “right” technical solution available. Deliberations are convened by organizations external to a decision-making enterprise to ensure that the event is developed and implemented without any potential preconceptions the sponsoring decision maker might hold. Conveners of PDs recruit a diverse group of local residents, provide them with unbiased background information (presented by experts) to support their decision making, and create discussion groups and exercises that allow individuals from a wide variety of social, cultural, economic, and educational backgrounds to debate which course of action would best serve the wider community. Deliberants are asked to provide the reasoning behind the views they advocate for in order to test one another’s ideas and to provide nuanced information back to the decision makers, who can then better understand the issues as they are perceived by community members. Deliberative questions are typically framed to solicit “yes” or “no” responses, or they ask for orders of priority or preference.
Decision makers who are faced with choosing a value-based course of action use PDs in different ways. Recommendations that emerge from PDs have been used by some institutions to gather a more fine-grained understanding of the perspectives of people who will be affected by a policy change. Other decision makers have made a public commitment to be guided by the results of the PD, thus providing accountability to their constituents. Examples of public deliberations conducted by CEAR within the health care sector and in urban planning can be found here.
The NYC DOHMH Public Deliberations
In late 2020, NYC DOHMH asked CEAR to convene a series of PDs to provide guidance to the Department’s planning for priority allocations of the COVID-19 vaccine when, in the early phases of roll out, vaccine supply would be limited. At that time, recommendations from both the National Academies of Sciences, Engineering, and Medicine (NASEM) and the CDC’s Advisory Committee on Immunization Practice (ACIP) made clear that the 24 million people who were front line health care workers, first responders, and adults in congregant living would gain initial priority. Essential workers, estimated at 60 million nationally, were also a high priority population according to NASEM and ACIP guidance, though specific occupations were left undefined. New York City’s DOHMH was, thus, seeking public input as to how the City’s essential workers might be best ordered considering the different risk profiles of individuals working in particular occupations and their likelihood for COVID-19 infection and transmission. Given the known heightened risk for serious morbidity and mortality conveyed by age, race, ethnicity, neighborhood “hotspots,” and pre-existing conditions, NYC DOHMH also asked that participants in these deliberations provide recommendations on whether the presence of one or more of these risk factors should alter the priority in which individuals were vaccinated.
New York City’s population is large and diverse. While the City has great pockets of wealth, nearly 20% of New Yorkers live in poverty; a similar percentage has not completed high school. With respect to race and ethnicity, non-Hispanic whites constitute a third of NYC’s population, nearly a quarter of the population is Black or African American, 28% are Latinx, and 14% are Asian. Fifteen percent of New Yorkers are over age 65. As in other localities across the nation, the risk for COVID-19 morbidity and mortality falls more heavily on New Yorkers of color, those in poverty, with less education, older residents, and those with underlying health conditions. And risk across neighborhoods varies, as evidenced by widely different infection rates and health outcomes in adjacent communities that differ according to socioeconomic characteristics.
Every locality has a unique set of industries and occupations, and New York City is no different. In seeking counsel from a diverse cross-section of New Yorkers, NYC DOHMH asked CEAR to develop PDs that asked participants to recommend priorities for the order in which a representative set of six essential NYC occupations should be vaccinated together with the reasons for those recommendations. NYC DOHMH posited that understanding the underlying reasons for the recommendations would allow them to apply the reasoning to broader categories of essential workers.
The six occupations that were chosen for the deliberations represented low and middle-income jobs with varying levels of COVID-19 transmission risk and diverse contributions to the safety and functioning of the City. Outside of their jobs, workers in particular occupations might also carry a range of risk. For example, workers in some of the occupations more commonly live in neighborhoods that have been virus hotspots, have a preponderance of individuals of color, or have higher proportions of older workers.
A critical first step in designing a public deliberation is to develop a clear close-ended deliberative question or questions with answers that declare a position. CEAR worked with NYC DOHMH to develop deliberative questions that asked participants to provide specific guidance on how to prioritize different essential worker occupations and the order in which concomitant risk factors – age, race, ethnicity, neighborhood of residence, and existence of underlying conditions – should elevate peoples’ order for vaccination.
The five PDs convened by CEAR on behalf of NYC DOHMH were each held in two half-day sessions in early winter. Three of the deliberations were held in English and two in Spanish. In a departure from the customary in person format for delivery of PDs, all were held virtually because of COVID-19 (89% of New Yorkers have computers; 81% have internet access). The concerns CEAR staff held about the challenges to both recruiting and engaging participants virtually were not borne out. A series of internet-based recruitment strategies yielded over 1400 interested New Yorkers from whom 105 were invited to the five deliberations. These individuals represented diversity across age, race, ethnicity, gender identity, education, neighborhood, occupation, ability to work remotely, health insurance status, experience with COVID-19, and attitudes toward vaccination. These groups of New Yorkers stayed active and engaged through all five of the PDs. Surveys conducted post-PD showed participants liked being part of the process and were likely to share what they learned with others.
Developing and delivering public deliberations are resource intensive. In sharing CEAR’s materials for these deliberations we hope to provide ready-made tools that other localities could easily adapt to answer key questions with respect to how their constituencies view priorities for initial receipt of vaccine. However, vaccine distribution decisions have moved rapidly, and individual states and localities have set in place vaccination policies unique to their settings. Still, debates have been stirred and remain within the public about what is acceptable and what is fair. We hope that in making these resources available other localities can tailor them to current and future difficult questions that generate debate. CEAR staff are available for further information and technical support. Please contact the authors for more detail.
Public deliberations generally have approximately 20 participants and include small and large group discussions. While there are many virtual platforms, our team used Zoom, and the technical instructions included below pertain to that platform, specifically. To conduct a virtual public deliberation on that scale, we suggest you engage team members to fill the following roles:
Technology Point Person / Zoom Meeting Organizer (1 person)
Technology Support (1-3 people)
Lead & Co-lead Facilitators (2 people)
Small Group Facilitators (3 people)
Small Group Co-Facilitators, if capacity allows (3 people)
The Technology Point Person / Zoom Meeting Organizer is the person overseeing the management of the Zoom meeting, including starting and stopping the meeting and recordings, allowing participants to enter the meeting, and displaying slides and polls.
TheTechnology Support team members are there to assist participants who experience any technological difficulty accessing and participating in the meeting, including issues with audio, connecting to Zoom, and completing the pre- and post-surveys and polls. Depending on capacity, there can be 1-3 people in this role.
The Lead and Co-Lead Facilitators provide the framing for the PD, explaining each group activity, introducing all content experts, facilitating dialogue among participants, and clarifying areas of controversy and agreement. One of the aims of the PD facilitators, distinct from facilitators of other types of community engagement, is to create an environment where participants engage with and respond to one another, rather than speaking through the facilitators. The co-lead facilitator supports the lead facilitator, paying particular attention to keeping the group on schedule, and the level of engagement from participants.
Small Group Facilitators are responsible for facilitating each small group activity, ensuring all participants’ viewpoints are heard and ideas are tested against one another in the group.
Small Group Co-Facilitators take notes during each small group activity, focusing on the recommendations and the “reason-giving” aspect of the discussion. If there is limited team member capacity, small group facilitators can be asked to take notes in addition to facilitating the group discussion, thereby removing the small group co-facilitator role.
Public deliberation calls for a broad range of participant backgrounds and perspectives. Accordingly, efforts should be made to disseminate recruitment materials widely. For the COVID-19 vaccine PDs outreach was conducted electronically via local community-based organizations, NYC DOHMH, local universities, and Craigslist. Those receiving outreach materials directly were encouraged to disseminate them within their networks.
A screening survey was used to collect basic demographic information from participants using an online survey platform to manage the responses. Initial screening data were used to identify a diverse group of potential participants who were subsequently invited to participate in a secondary screening call using the virtual platform (i.e., Zoom) to ensure participant facility with the technology.
Participants were required to have access to a quiet space, a computer with a web camera and speakers, and internet. The recruitment process was designed to screen for these resources while also ensuring a range of socioeconomic backgrounds could be represented as much as possible. Not all screening questions used in the NYC COVID-19 vaccine public deliberations are relevant in other localities and changes to the documents listed below may be made to reflect differences in need and relevance.
Recruitment flyer: Provides basic information about the purpose of the event as well as an electronic link to the screening survey.
Screening form: Completed by all individuals interested in participating in the PD. Data collected are used to select a diverse group of participants and to describe the final set of participants following the deliberation.
Secondary screening script: Used during ten-minute secondary screening Zoom calls with project staff. These calls function to confirm participant data provided in the screening form, eligibility, participant facility with the virtual platform technology (key to a successful interactive virtual event), and to acquire consent from all participants. Additionally, participant questions are answered, and they are given additional information about the event (e.g., contact method for meeting invitation and materials).
When conducting a public deliberation, it is useful to have two sets of agendas: (1) a detailed agenda for the internal team that visually displays which staff are responsible for each activity and the associated amount of time, and (2) an abbreviated agenda, which is shown to participants as part of the first day’s introduction and provides general information on how participants will be spending their time during the PD.
Expert presenters are used in public deliberations to provide essential background information that facilitates development of a shared knowledge base for the groups’ discussions and recommendations. They provide short and typically non-technical presentations that are followed by a question-and-answer period. Public deliberations are designed to recruit a diverse cross-section of people who will be affected by a decision and that means that the room will contain people with greater and lesser familiarity with any given topic. Critical to the process is that experts create presentations that are accessible to a broad audience.
The net effect of expert presentations should be to inform participants of issues arising from the content area. In some settings, two experts holding oppositional perspectives are used to balance one another. In others, a single expert provides an unbiased overview of a knowledge base that will be relevant to the deliberation. Members of the facilitation team work closely with experts to assure that presentations are accessible and that the cumulative effect is neutral with respect to the deliberative question(s) posed.
For the NYC COVID-19 vaccine public deliberations we invited experts who could cover three topics (shown below). Links to the presentations used are included here and can be adapted for local use.
Approaches to Fairness, Erika Blacksher, PhD Provides an overview of various approaches to fairness and ways to consider vulnerability within communities and individuals for resource allocation in a setting of scarcity.
Participants are pre-assigned to small groups to ensure each has sufficient diversity. Small group activities provide participants an opportunity to process the information presented by the content experts and engage in discussion that brings participants closer to answering the deliberative questions as well as test their own perspectives and opinions against those of their fellow group members. Small group discussion also offers individuals who are less likely to share their ideas in a large group another avenue for participation, ensuring all voices are heard. Small group co-facilitators take notes during the discussion using a note-taking template. After each small group activity, one representative per group presents the main points from their group’s discussion to the large group.
Small Group Activity: Prioritizing Risk Characteristics The purpose of this activity is for participants to identify which risk characteristics they deem most important to consider for vaccine receipt among essential worker occupations. As part of this conversation, participants are asked to select one of the ethical approaches for vaccine distribution, which were included in a content expert presentation (and video) and explain their reasoning. They were also asked to reflect on a set of essential worker “characters,” each with a distinct set of risk characteristics. Again, one member from each group presents the main points from their group’s discussion to the larger group.
Large group activities occur repeatedly during the deliberation. Their purpose is to explore views across the full deliberative group exposing participants to the range of perspectives in the room and allowing for broader cross-pollination of ideas. Additionally, they provide an opportunity to summarize group recommendations at the end of the deliberation. Below are the large group activities conducted during the NYC COVID-19 vaccine public deliberations, which can be adapted, as needed.
Large Group Activity: How to Account for Risk Characteristics within Essential Worker Occupations
This activity asks participants to consider whether an entire occupation should be prioritized because of the number of people with risk characteristics within that occupation, or if individuals with risk characteristics should simply be prioritized within their occupation group. Participants respond to a brief poll and facilitators probe for reasons and values behind responses.
The following activities take place at the end of the deliberation when participants have heard all background and expert content and have had an opportunity to explore their own and others’ views in small and large group discussions. In this setting, individuals may present reasons or arguments that impact the perspectives of others in the group.
Large Group Activity: Risk Characteristic Prioritization for Essential Workers Participants are asked to fill out a poll selecting the two risk characteristics they think are most important to consider when prioritizing vaccination among essential workers. Results are displayed to the group and facilitators moderate the discussion asking participants to focus on the reasons that some risk characteristics were chosen over others.
Large Group Activity: Ranking Essential Worker Occupations Participants respond to a poll asking them to select the three essential worker occupations they believe are most important to prioritize to receive the COVID-19 vaccine. The group discusses results with facilitators drawing out the “why” behind participant responses, as well as any common themes across the group.
Large Group Activity: Risk Characteristic Prioritization for the General Population This activity revisits the earlier question of which risk characteristics should be prioritized for vaccination focusing on how they should be ranked within the general population rather than the essential worker population. The purpose of this activity is to compare participant perspectives regarding prioritization across the two groups. Like the others, this activity also begins with a poll, the results of which are displayed before the group as a starting point for discussion.
The concept of prioritizing risk characteristics may prove useful for a range of questions and decisions sponsors may consider, which extend beyond the focus of the COVID-19 vaccine public deliberations. In this section we have provided brief descriptions and links to all the material related to prioritizing risk characteristics used during our PDs, including:
Presentation: Approaches to Fairness, Dr. Erika Blacksher, PhD This presentation (and video) provides an overview of various approaches to fairness and ways to consider vulnerability within communities and individuals for resource allocation in a setting of scarcity.
Small Group Activity: Prioritizing Risk Characteristics The purpose of this activity is for participants to identify which risk characteristics they deem most important to consider for vaccine receipt among essential worker occupations. As part of this conversation, participants are asked to select one of the ethical approaches for vaccine distribution, which were included in a content expert presentation (and video) and explain their reasoning. Again, one member from each group presents the main points from their group’s discussion to the larger group.
Large Group Activity: Risk Characteristic Prioritization for Essential Workers Participants are asked to fill out a poll selecting the two risk characteristics they think are most important to consider when prioritizing vaccination among essential workers. Results are displayed to the group and facilitators moderate the discussion asking participants to focus on the reasons that some risk characteristics were chosen over others.
Large Group Activity: Risk Characteristic Prioritization for the General Population This activity revisits the question of which risk characteristics should be prioritized for vaccination focusing on how they should be ranked within the general population rather than the essential worker population. The purpose of this activity is to compare participant perspectives regarding prioritization across the two groups. Like the others, this activity also begins with a poll, the results of which are presented to the group as a starting point for discussion.
Below are the components of the deliberation that deal directly with the essential worker deliberative question. While the specific essential worker occupations identified in the NYC COVID-19 vaccine public deliberations are limited in number, the values and reasons given by participants during the deliberative process provide rich information that can be extrapolated to a broader set of essential worker categories in a variety of geographic locations. Other localities may want to select other occupations of greater relevance to them, but the number should remain limited with the assumption that the selected occupations are examples only.
Facilitator-Provided Background on Essential Workers and COVID-19 At the beginning of the deliberation the lead facilitator provides information about essential workers in the U.S., including how many there are and the prevalence of risk factors that make them more vulnerable to poor outcomes from COVID-19.
Small Group Activity: Prioritizing Essential Worker Occupations The purpose of this activity is for participants to identify the reasons they deem most important to consider when prioritizing essential worker occupations. Participants are also asked to choose their #1 and top 3 essential worker occupations to prioritize for vaccine receipt.
Large Group Activity: How to Account for Risk Characteristics within Essential Worker Occupations This activity asks participants to consider whether an entire occupation should be prioritized because of the number of people with risk characteristics within that occupation, or if individuals with risk characteristics should simply be prioritized within their occupation group. Participants respond to a brief poll and facilitators probe for reasons and values behind responses.
Large Group Activity: Ranking Essential Worker Occupations Participants respond to a poll asking them to select the three essential worker occupations they believe are most important to prioritize to receive the COVID-19 vaccine. The group discusses results with facilitators drawing out the “why” behind participant responses, as well as any trends across the group.
Several data collection methods are used to screen and select potential participants, collect demographic and other background information, assess attitudes related to the deliberative questions (both before and after participating in the deliberation), and to capture qualitative information describing participant reasoning and values related to the deliberative questions. Below are the tools and instruments used for the COVID-19 vaccine public deliberations in New York City, which can be adapted for local use. Data collection methods may be used selectively depending on the goals of the deliberation.
Screening Form: Electronically linked to outreach emails, completed by all potential participants to gather general background and demographic information. Data collected are used to select a diverse group of participants and for descriptive purposes.
Pre- and Post-Survey: Used to formally document participant responses to the deliberative questions before and after the deliberation, and to assess attitudes related to the deliberative questions. The post-survey also includes questions on participant perspectives on the deliberation itself.
Polls: Used to “take the temperature” of the group during deliberations — they can be pre-loaded into Zoom and launched during group discussions. Participants respond and results are displayed immediately for all to see. Results serve as a starting point for group discussion and provide additional opportunities to collect data throughout the deliberation.
Note-Taking Forms (small groups and large groups): The forms used to document the discussions during the deliberation provide an easy way to organize and report on the results of the deliberation, as well as the reasons behind participant decision making. Notes are less detailed than transcripts but do not require outside resources (e.g., a transcription company) and are available immediately following the deliberation.
Transcripts: Audio recording and transcribing large and small group discussions provides a more detailed account of participant perspectives and the values and reasons they give for their responses to the deliberative questions.
