When we monitor, evaluate and assess our work, we use what we learn to continuously improve the quality and effectiveness of our programs, our project planning and design, and our policies and ways of working. We also share our experiences throughout the organisation, with other Oxfam affiliates and our stakeholders. This helps us to learn from what we do, so we can do it better and reach more people in need.
Some of our learning processes and outcomes for 2010–2011 covered the following areas:
- Savings and credit groups in Timor-Leste
- Advocacy for communities forced to resettle in Cambodia
- Engaging young people in the Pacific
- Building monitoring, evaluation and learning skills in South Africa
- Building global youth active citizenship and accountability
- Real-time evaluation of the emergency response to the Laos floods
- Reviewing our Aboriginal and Torres Strait Islander Peoples’ Program
In Timor-Leste we are running a livelihood improvement and food security improvement program in the Covalima and Oecusse districts.
During focus group discussions with beneficiary groups as part of the program’s monitoring and evaluation process, participants identified the desire for more support for these groups particularly in capacity development. This support was provided by contracting two Indonesian non-government organisations (to allow training in a shared language and context) to provide training in various organisational aspects such as managing funds, book- keeping, decision-making.
A case study was then commissioned to examine in more depth the experience of some of these groups as part of the preparation of material for Economic Justice Week and preparation of the annual East Asia impact report.
There are now 54 Oxfam-supported groups with 10–15 members each. Participation of women is high and some groups have exclusive women membership. Combined savings are approximately AUD $20,000.
The case study indicates that an estimated two-thirds of loans are used for welfare purposes — such as food, clothes, school fees and funeral ceremonies — rather than productive purposes such as small businesses, livestock and bulk-buying of petrol or rice for resale. Through monthly contributions and interest from loans, the group can build a savings asset that can be used for larger investments and collective income-generating activities.
The study recognises that savings and credit can bring tangible benefits to the wellbeing, resilience and financial security of members and their families, although up-scaling and replication requires due support and mentoring.
The savings and credit groups represent a community social asset that can support livelihood activities through access to credit (development), provide a mechanism for accumulation of savings, as well as provide a source of cash for immediate needs (welfare). It is therefore a strategy that can be used in a wide range of projects to achieve a wide variety of objectives from women’s economic leadership, to reduced vulnerability, to disasters. The approach does however require substantial investment in organisational capacity. It is now been considered as a “good practice” option for a wide range of programs and projects in East Asia.
Over a number of years, Oxfam Australia has provided strategic support to communities forced to resettle from an Asian Development Bank (ADB) funded highway in Cambodia.
This support has included supporting Cambodian and international non-government organisations to assist the communities to file and manage a formal policy complaint under the ADB’s Accountability Mechanism, as well as supporting them with high-level lobbying in Manila and Canberra.
A learning project was conducted to document the impact of the works in a major published case study, to include learnings in an Oxfam submission to the review of the ADB’s Accountability Mechanism policy, and to conduct an independent evaluation from the point of view of community outcomes and beliefs.
The key learnings include:
- Oxfam and partners must consider livelihood needs during extended advocacy for longer term compensation outcomes
- Oxfam needs to invest in and monitor advocacy capacity in national civil society organisations supporting communities, given that Oxfam can only take on a finite number of direct community-focused campaigns
There is a need to document experience and proactively share strategy with communities and local non-government organisations facing similar project impacts as a result of resettlement. Oxfam is publishing these outputs and findings into information packs to share with our partner networks. Together with key partners in Cambodia, we are also helping shape upcoming campaigns based on these learnings. Our submissions to the ADB review process have also made a major impact on the policy negotiations at the ADB board level.
The Regional Youth Programs Reflection, held in Vanuatu, emerged from country and unit discussions about engaging young Pacific people. The aim was to examine the value of youth engagement through the eyes of partners and beneficiaries, as well as the effectiveness and impact of Oxfam’s various approaches to youth programming.
Input into the reflection design was sought from Australian and country teams responsible for youth programs, and learning was applied from the most recent Kaleidoscope and Oxfam International Youth Partnerships Impact Assessment.
Workshop participants reflected on the nature of youth activism and identified key areas where Oxfam can add value. They also undertook learning exchange visits to local organisations to discuss alternative youth programming models.
Key learning includes:
- The intrinsic value of youth development officers supporting the linking of knowledge to action
- The necessity of incorporating localised intra-generational power analysis into program design
- The need for understanding partnership models with informal groups and networks that allow young people to be in control of decision making
Reflection analysis and recommendations will be drawn into multiple processes, including:
- Individual review and redesigns of country youth programs
- Joint country analysis and strategic planning
- Cooperation between the youth engagement team and the Pacific Unit about monitoring, evaluation and learning, leadership and active citizenship research
A study was concluded in January 2011 aimed at establishing a baseline of Oxfam’s HIV and AIDS related programming in South Africa. The baseline study report highlighted that partner organisations identified the need to build their program monitoring and evaluation skills.
In June 2011, the South African Oxfam team engaged partners in a learning event — Measures: Understanding Our Impact 2011 — aimed at sharing learning on improving our program monitoring, evaluation and learning processes. This was done by providing an opportunity to reflect on the recent Oxfam HIV and AIDS Program baseline process and survey results and sharing a variety of monitoring methods and tools currently being used by partners.
The event was also useful in sharing current learning and experiences of some of the partner capacity building processes focused around monitoring, evaluation and learning, and allowing partners to share their own practices and learnings.
In 2010, the Oxfam International Youth Partnerships (OIYP) program commissioned Social Compass to do a 10-year impact assessment of the program, with a particular focus on OIYP’s approach to building global youth active citizenship and accountability.
Three key recommendations have been accepted from this impact assessment report and were immediately incorporated into OIYP Kaleidoscope 2010 and the subsequent two-year operational plan. OIYP should:
- Continue to develop intergenerational contact between Action Partners
- Invest in a monitoring, evaluation and learning framework, which focuses on leadership processes involving networks for social change, to effectively measure the impact of OIYP
- Focus on developing strategic regional program and funding partnerships
Immediate responses to programming based on these recommendations have been:
- Strengthening the intergenerational learning between current and past Action Partners through the formal mentoring processes that took place at OIYP Kaleidoscope 2010 and also through the implementation of ongoing program activities including e-workshops, the grants program and face-to-face workshops as part of the gender and food justice movements
- OIYP has entered into a new partnership with the AusAID Development Leadership Program (DLP) with the aim of developing a deeper understanding of the role of political dynamics and leadership in shaping development outcomes. Central to the OIYP and DLP partnership is developing high quality theoretical and action research that explores the critical role of leadership in enabling development to happen, as well as informing appropriate international aid policies, programs and operations
OIYP is investing in ongoing learning in order to inform program design under the new strategic direction. A five-year outline has been developed which maps out the path of the program in the following areas:
- Research which has a particular focus on women’s leadership and the functioning of social networks. This work is supported by the partnership with the DLP
- Monitoring, evaluation and learning: development of a monitoring, evaluation and learning framework for the OIYP program, informed by the “theory of change”, and the application of this into systems and tools. Also investment in a retrospective study on the impact of Action Partners in policy and practice change over the last 10 years
- Program implementation: information gathered through research and monitoring and evaluation will guide the OIYP’s movement building work, with an initial focus on gender and food justice. It will also guide how OIYP will structure future cycles and the placing of future Kaleidoscope events and the intergenerational networking
- Partnerships and fundraising: evidence of change gathered through monitoring, evaluation and learning and research will provide an evidence base for fundraising for future cycles and also will guide the development of strategic global partnerships
A real-time evaluation (RTE) was conducted for Oxfam’s response to the Laos floods, which was different to a regular RTE.
Generally, an RTE takes place within the first month of a humanitarian response and is quick, with integrated feedback and action. It is designed to bring maximum benefits for the program with a minimum impact on the local staff team. The three phases of the Laos RTE were partly developed in response to external circumstances (such as the South East Asian Games held in Laos, Christmas holidays and staff turnover), coupled with a desire to look more deeply into the technical elements of the response.
In hindsight, this meant that the RTE was long and drawn out and it had a large impact on the Laos team whilst not having the advantage of being a more rigorous and retrospective impact evaluation. However, the process has still been very useful with some key findings and recommendations.
While it was an extensive evaluation, with many recommendations, a number of key issues stood out — some of them specific to Laos and others which have been raised consistently during previous humanitarian responses. These include:
- Some Oxfam Australia finance, human resources and information technology support systems need to become less convoluted and complicated to prevent delays and staff frustration
- Oxfam Australia needs more clarity about how to address the resettlement issue in Laos and have a clear mandate about how we work with the government, what our policies are and what our role is with regards to supporting partners. Oxfam needs to know when to step back and encourage others, such as the government, United Nations and other international non-government organizations, to take the lead
- The next emergency response training should be based on the needs of the Laos staff and build on what they have learnt from this response. Contingency planning should be written in Lao, and involve all the Laos staff and affiliates
- Language and communication issues are vital and have wide reaching impacts on crosscutting issues such as gender, quality, participation, accountability, as well as the day to day running of the program. This needs to be recognised in our preparedness and planning
Outcomes of the real-time evaluation include:
- A “gender review” took place in Melbourne in May 2010
- A second round of emergency response training was conducted in Laos in early 2011
A workshop was held in Melbourne in April 2010 that looked at the learnings that came out of the Laos RTE around Oxfam’s support systems. The Humanitarian Services Unit Workplan was created as a result of this exercise and many positive changes have been implemented in our international human relations area.
In late 2010 we began a mid-term review of the Aboriginal and Torres Strait Islander Peoples’ Program’s Strategic Plan 2007–2013, which has guided the work of the ATSIP Program.
This review reflected on the work and performance of ATSIPP over the previous five years and was informed by staff members, by our Aboriginal and Torres Strait Islander partner organisations and by analysis of the political climate during what has been a turbulent time in Aboriginal and Torres Strait Islander affairs. It also looked at internal issues which impact on our ways of working with Indigenous partners.
The mid-term review process involved:
- Reviewing project evaluations, reports, case studies and other key documents
- ATSIP program staff and partner workshops and rapid thematic assessments on each area
- Interviews with key external Aboriginal and Torres Strait Islander leaders
- Discussions with the Oxfam Australia Aboriginal and Torres Strait Islander Reference Group and other key stakeholders
The review found that although the program had made some good progress in meeting its overall objectives under its strategic plan, it experienced many challenges. Oxfam is generally seen as a strong and useful ally in making positive changes towards greater Aboriginal and Torres Strait Islander self-determination and community control of Indigenous affairs. We are seen as a well-respected leader among international development organisations and, over the period of the plan, we have strengthened our reputation for our positive contribution and ways of working and have built trusted relationships with many Indigenous organisations. We have successfully developed our national program and we now work in more states.
However, Oxfam is a non-Indigenous agency working in Aboriginal and Torres Strait Islander affairs and this provides some challenges. Over the next three years, we will:
- Seek to gain increased levels of credibility and respect within Aboriginal and Torres Strait Islander communities. This will take time and we will need to make a renewed commitment to cultural awareness, sensitivity and flexibility
- Remain mindful that Aboriginal and Torres Strait Islander staff members have particular cultural responsibilities and loyalties which impact on their work. We aim to support our staff and partners to avoid excessive demands and stresses and will strive to provide appropriate support and culturally-safe ways of working
- Build on the successes of the health and wellbeing, youth and self-determination areas, while being mindful not to over-commit or take on too many new challenges
- Endeavour to address the important areas of gender, incarceration and climate change through the established thematic areas
Based on the revised strategic plan 2011–2014, annual operational plans will be developed and reported on for each thematic area, as well as against the ATSIP Program’s over-arching strategies. Also reflections, monitoring and evaluations are conducted regularly at the project and program levels.