Citizen Surveys
Survey Administration
Following the mailing of a pre-survey notification postcard to a random sample of 3,000 households in 2003, surveys were mailed to the same residences approximately one week later.
A reminder letter and a new survey were sent to the same households after two weeks. Of the mailed postcards, 207 were undeliverable due to vacant or “not found” addresses.
Completed surveys were received from 994 residents, for a response rate of 35%. Typically, the response rates obtained on citizen surveys in other cities range from 25% to 40%.
It is customary to describe the precision of estimates made from surveys by a “level of confidence” (or margin of error). The 95 percent confidence level for this survey of 994 residents is generally no greater than plus or minus 5 percentage points around any given percent reported for the entire sample.
The results were weighted to reflect the demographic profile of all residents in the City of Denver.
Survey Validity
The question of survey validity has two parts: 1) how can we be confident that
the results from our sample are representative of the results we would have gotten had we administered the survey to the entire population? and 2) how closely do the perspectives recorded on the survey reflect what residents really believe or do?
To answer the first question, we use the best survey research practices for the resources spent to assure that the results from the sample reflect the opinions of residents in the entire jurisdiction. These practices include:
1) Using a mail-out/mail-back methodology, which typically gets a higher response rate than phone for the same dollars spent.
2) Selecting households at random within the jurisdiction.
3) Over-sampling attached units to improve response from hard-to-reach,
lower income, or younger apartment dwellers.
4) Selecting the respondent within the household using an unbiased sampling
procedure.
5) Contacting potential respondents three times to encourage response from
people who may have different opinions or habits than those who would
respond with only a single prompt.
6) Soliciting response on jurisdiction letterhead signed by the highest ranking
elected official or staff member.
7) Providing a self-addressed, postage-paid return envelope.
8) Offering the survey in Spanish when appropriate and requested by city
officials.
9) Using the most recent available information about the characteristics of
jurisdiction residents to reweight the data to reflect the demographics of the
population.
The answer to the second question about how closely the perspectives recorded on the survey reflect what residents really believe or do is more complex.
Resident responses to surveys are influenced by a variety of factors. For
questions about service quality, residents’ expectations for service quality play a role as well as the “objective” quality of the service provided, the way the resident perceives the entire community (that is, the context in which the service is provided), the scale on which the resident is asked to record her opinion and, of course, the opinion, itself, that a resident holds about the service.
Similarly a resident’s report of certain behaviors is colored by what he or she believes is the socially desirable response (e.g. reporting tolerant behaviors toward “oppressed groups,” likelihood of voting a tax increase for services to poor people, use of alternative modes of travel to work besides the single occupancy vehicle), her memory of the actual behavior (if it is not a question speculating about future actions, like a vote), her confidence that she can be honest without suffering any negative consequences (thus the need for anonymity) as well as the actual
behavior itself.
How closely survey results come to recording the way a person really feels or behaves often is measured by the coincidence of reported behavior with
observed current behavior (e.g. driving habits), reported intentions to behave with observed future behavior (e.g. voting choices) or reported opinions about current community quality with objective characteristics of the community (e.g. feelings of safety correlated with rates of crime).
There is a body of scientific literature that has investigated the relationship between reported behaviors and actual behaviors. Well-conducted surveys, by and large, do capture true respondent behaviors or intentions to act with great accuracy. Predictions of voting outcomes tend to be quite accurate using survey research, as do reported behaviors that are not about highly sensitive issues (e.g. family abuse or other illegal or morally sanctioned activities). For self-reports about highly sensitive issues, statistical adjustments can be made to correct for the respondents’ tendency to report what they think the “correct” response should be.
Research on the correlation of resident opinion about service quality and “objective” ratings of service quality tend to be ambiguous, some showing stronger relationships than others. NRC’s own research has demonstrated that residents who report the lowest ratings of street repair live in communities with objectively worse street conditions than those who report high ratings of street repair (based on road quality, delay in street repair, number of road repair employees).
Similarly, the lowest rated fire services appear to be “objectively”
worse than the highest rated fire services (expenditures per capita, response time, “professional” status of fire fighters, breadth of services and training provided).
Whether some research confirms or disconfirms that relationship between what residents think about a community and what can be seen “objectively” in a community, we have argued that resident opinion is a perspective that cannot be ignored by government administrators. Elsewhere we have written, “If you collect trash three times a day but residents think that your trash haul is lousy, you still have a problem.”
The birthday method requests that the respondent in the household be the adult (18 years old or older) who most recently had a birthday, irrespective of year of birth.
Use of the “Excellent, Good, Fair, Poor” Response Scale
The scale on which respondents are asked to record their opinions about service and community quality “excellent,” “good,” “fair” or “poor” (EGFP). This scale
has important advantages over other scale possibilities (very good to very bad; very satisfied to very dissatisfied; strongly agree to strongly disagree, as examples). EGFP is used by the plurality of jurisdictions conducting citizen
surveys across the U.S.
The advantage of familiarity is one we did not want to dismiss because elected officials, staff and residents already are acquainted with opinion surveys measured this way. EGFP also has the advantage of offering three positive options, rather than only two, over which a resident can offer an opinion. While symmetrical scales often are the right choice in other measurement tasks, we have found that ratings of almost every local government service in almost every jurisdiction tend, on average, to be positive (that is, above the scale midpoint). Therefore, to permit finer distinctions among positively rated services, EGFP offers three options across which to spread those ratings. EGFP is more neutral because it requires no positive statement of service quality to judge (as agree-disagree scales require) and, finally, EGFP intends to measure absolute quality of service delivery or community quality (unlike satisfaction
scales which ignore residents’ perceptions of quality in favor of their report on the acceptability of the level of service offered).
“Don’t Know” Responses
On many of the questions in the survey respondents may answer “don’t know.”
The proportion of respondents giving this reply is shown in the full set of
responses included in Appendix I. However, these responses have been
removed from the analyses presented in the body of the report. In other words,
the tables and graphs display the responses from respondents who had an
opinion about a specific item.
For two of the items related to crime victimization and crime reporting, “don’t know” responses were not removed. These questions were not evaluative; rather, respondents were asked if they or any member of their household had been a victim of a crime within the last year. If they were, they were then asked whether the crime had been reported to police.
Use of the “Excellent, Good, Fair, Poor” Response Scale
The scale on which respondents are asked to record their opinions about service and community quality is “excellent,” “good,” “fair” or “poor” (EGFP). This scale has important advantages over other scale possibilities (very good to very bad;
very satisfied to very dissatisfied; strongly agree to strongly disagree, as
examples). EGFP is used by the plurality of jurisdictions conducting citizen surveys across the U.S. The advantage of familiarity is one we did not want to dismiss because elected officials, staff and residents already are acquainted with opinion surveys measured this way. EGFP also has the advantage of offering three positive options, rather than only two, over which a resident can offer an opinion. While symmetrical scales often are the right choice in other measurement tasks, we have found that ratings of almost every local government service in almost every jurisdiction tend, on average, to be positive (that is, above
the scale midpoint). Therefore, to permit finer distinctions among positively rated services, EGFP offers three options across which to spread those ratings. EGFP is more neutral because it requires no positive statement of service quality to
judge (as agree-disagree scales require) and, finally, EGFP intends to measure absolute quality of service delivery or community quality (unlike satisfaction scales which ignore residents’ perceptions of quality in favor of their report on the acceptability of the level of service offered).
“Don’t Know” Responses
On many of the questions in the survey respondents may answer “don’t know.”
The proportion of respondents giving this reply is shown in the full set of
responses included in Appendix I. However, these responses have been
removed from the analyses presented in the body of the report. In other words,
the tables and graphs display the responses from respondents who had an
opinion about a specific item.
For two of the items related to crime victimization and crime reporting, “don’t know” responses were not removed. These questions were not evaluative; rather, respondents were asked if they or any member of their household had been a victim of a crime within the last year. If they were, they were then asked whether the crime had been reported to police.
Putting Evaluations Onto a 100-Point Scale
Although responses to many of the evaluative questions were made on a 4 point scale with 4 representing the best rating and 1 the worst, many of the results in this summary are reported on a common scale where 0 is the worst possible rating and 100 is the best possible rating. If everyone reported “excellent,” then the result would be 100 on the 100-point scale. Likewise, if all respondents gave
a “poor” rating, the result would be 0 on the 100-point scale. If the average rating for quality of life was “good,” then the result would be 67 on a 100-point scale; “fair” would be 33 on the 100-point scale. The 95 percent confidence interval around an average score on the 100-point scale is no greater than plus or minus 5 points based on all respondents.
Interpreting Comparisons to Previous Years
This report contains comparisons with prior years’ results. In this report, we are comparing 2003 data with 2002 data in the graphs. In the graphs, there are two separate representations labeled by year. The table following a graph contains 2003 data only, and is labeled accordingly.