Read Paul Born's latest book:
Read Paul Born's latest book:
On Thursday November 26, 2015 Tamarack hosted a webinar with guest Sarah Wakefield, Associate Professor in the Department of Geography and Planning at the University of Toronto. Sarah shared lessons learned while evaluating large-scale, multi-stakeholder interventions that bundle together multiple projects, working with a variety of diverse stakeholders and communicating the findings to decision-makers. Using the work with Hamilton, Ontario’s Neighbourhood Action Strategy as a case example, this webinar discussed ways of evaluating the planning, implementation, and ultimate outcomes of these kinds of interventions.
During the webinar, participants posed a variety of thought-provoking questions to Sarah, however, due to time restrictions not all of the questions were addressed. Fortunately for us, Sarah was able to take the time to address a few of the questions left unanswered. Please find the responses below:
1. Could you please explain in more detail how you have balanced the dynamic process (changing of plans and actions) in the evaluation process?
In terms of the interview and focus group components, that has been very fluid, which has been consistent with the methodology. We change interview questions and follow up on “leads” as needed to reflect the evolution of the project, while maintaining a general focus on facilitators and barriers.
For the action tracking, this process is still evolving but the plan is to add new actions and initiatives as they arise, and also try to capture when actions are seen as no longer relevant. The key there is to keep everything in the database (i.e., actions are never just deleted). We want to be able to sort the database so that we can only use the subset of “active” actions when we need to do certain kinds of reporting. We have found that rolling the evaluation into strategic planning (e.g., presenting evaluation results from the year to the planning teams, and having them reflect on what actions are completed, what is no longer relevant, and what they would like to add) is a helpful process for everyone.
2. Have you included the residents in the evaluation process as evaluators?
In some cases, yes, although we have learned some lessons about that along the way. Initially we had recruited “action trackers” from each neighbourhood – they were paid a small honorarium for their work. However, we found that it was very difficult to get consistency in the way the data was reported, and also that figuring out the status of all the action items was actually very challenging, as it required connecting with a variety of different people and navigating complex (and often different) understandings of what was going on in each case. We also found that there was a lot of turnover in “trackers”, which created gaps in the data and was burdensome in terms of recruiting and training.
Our plan is now to do focus groups with the resident planning teams on an annual basis – we have done one round of focus groups and found that they allowed planning team members to come to consensus about the status of an action and related facilitators and barriers (important in terms of their internal process as well as for the robustness of the data). The focus groups also helped to facilitate strategic planning (i.e., these focus groups can be used to not only reflect on the activities of the past year, but also to reassess their relevance and to add new actions to the list). In this context, residents are seen as expert assessors, interpreting the data at the same time as they generate it.
3. As someone who attends various hub meetings I have noticed that as neighbourhoods strengthen they become territorial. What have you learned about that?
This is an interesting point, and a real challenge. The social science literature suggests that this is a fairly common phenomenon - the development of “in-group” feelings, including a sense of belonging, attachment to place and a sense of ownership/control over local initiatives. "In-group" feelings are important for a sense of collective identity and engagement, however they can lead to people positioning themselves in relation to (and sometimes in opposition to) an “out-group”. This then can lead to territoriality, essentializing (of both the in-group and the out-group), and reduced ability to collaborate.
It is important to recognize, however, that this is happening in groups and neighbourhoods all the time, and indeed, that the neighbourhoods in question have often been at the receiving end of this dynamic from other parts of the city (i.e., being stigmatized and excluded). So, the challenge is to try and emphasize the positive elements of this (i.e., the building of social ties and a positive sense of identity) while trying to minimize the negative elements.
While this is not completely within the control of the Neighbourhood Action Strategy staff – and in many ways, shouldn’t be – the NAS has tried to emphasize anti-oppressive practice and asset-based community development (both of which try to combat this dynamic in different ways) and also to make sure neighbourhoods don’t see themselves in competition with each other. What we have found is that ongoing community development work (particularly that emphasizes bringing in new and diverse community members to the team), the development of group dynamics (and/or administrative processes) that prioritize inclusion and collaboration, and funding structures that don’t pit individuals or groups against each other are all important. We have also found that this work requires TRAINING and RESOURCES on an ongoing basis, and that expecting groups to miraculously avoid this dynamic on their own without support is not realistic (in any environment).
4. Residents living in the neighbourhoods are feeling satisfied but I am curious to know if "outsiders" opinions are changing?
I would say yes, in some of the neighbourhoods, but also that this is often for what I personally would consider the “wrong” reasons -- for example the external trappings of gentrification, such as redevelopment of what used to be relatively affordable housing into housing for more affluent people or the increasing presence of stores and restaurants that cater to a more affluent crowd, rather than the improvement in health and happiness of the original residents.
This is only anecdotal evidence, as our focus in the evaluation is to assess the impact of the initiative on residents rather than on those who live in other parts of the city. I raise this issue, however, because it is important that those involved in these kinds of initiatives recognize that some urban changes that are widely seen as positive by those living outside the neighbourhood may be experienced negatively by some of those who live there (e.g., the loss of businesses that cater to their needs at prices they can afford, or being forced out of rental accommodation). In this context, a focus on equity can help to sharpen and refine the goals of both the initiative itself and the evaluation of the initiative.