Evaluator self-assessment tool part1

The Pathways Committee have spent the last few months converting the AES Professional Learning Competency Framework into a free online self-assessment tool in collaboration with learnevaluation.org. The tool has been developed for aspiring, emerging and established evaluators to assess themselves against the competencies and be/er understand and manage their strengths and gaps.

In this first part of two-part session, we will debut the tool for you to test. In the second part on Friday, we present the aggregated results, and invite reflection and feedback from the group to refine the tool for wider release.

Big picture demographic trends of the 2020s

Facilitated by Charlie Tulloch and presented by Simon Keustenmacher

Charlie and Simon will have a discussion as a follow-up to Simon’s provocation regarding the major demographic drivers and social trends affecting Australia broadly with flow-on implications for the evaluation sector.

About the facilitator

Charlie Tulloch is an evaluation and public policy consultant who uses evidence to enhance performance. He founded Policy Performance in 2018 and previously worked at KPMG, ACIL Allen Consulting and in the Victorian Department of Premier and Cabinet.

Evidence and Evaluation – why bother?

Presented by members of AES Relationships Committee (Kara Scally-Irvine, Jade Maloney, Tadgh McMahon, Luke Condon) with AES President John Stoney

We intrinsically understand why evidence and evaluation is worthwhile, but we also know that it can be hard to clearly articulate the case for building an evidence base to better understand and achieve impact.

The AES relationships committee recognises the value of advocating and collaborating with our counterpart disciplines for better use of evidence in decision-making. To do this we want to engage you in a discussion to see if we can build a common language or set of ideas that we can all use to advocate together and create opportunities for collaboration and relationship building.

Please join us for this one-hour co-creation session. We will brief you on what the AES has developed – drawing on things like previous submissions, and resources – and ask that you bring your ideas or examples of what you’ve seen works elsewhere to raise the profile of concepts like evidence and impact.

At the end of all this we hope to have a set of clear messages we can all use to champion the use of evidence and how evaluative practice contributes to achieving impact.

Evaluation with and for First Nation’s communities

Facilitated by  Don Bemrose 

This session will provide a space to reflect on and respond to Nicole Tujague’s provocation from the FestEVAL opening.

Don Bemrose is a Gunggari man with family ties to the Cherbourg Aboriginal Community in Queensland and since 2008, has called Narrm (Melbourne) home. He is a primary school teacher and classically trained opera singer and has spent 20 years advocating and working in the arts, youth development, philanthropic, and vocational training sectors. Don provided expert advice through associate consultancy

Agile connections

Facilitated by Susan Garner and presented by Julie Elliott and Judy Oakden

This session responds to a collective sense of expectation to leverage the FestEval’s virtual relational and emotional space.

Driven by accelerating connectivity and new cognitive tools, the nature of evaluation is rapidly changing. Within a new working environment, when so much has shifted, we find ourselves in contexts characterised by many unknowns. Evaluators must exercise agility.

There is an opportunity for us, as evaluators, to leverage different types of connections and convene conversations that would not otherwise happen. Drawing on Arena’s (2018) social capital strategies that may help drive adaptation and innovation, this session considers how we can adapt the connectivity strategies we might use for face to face evaluation conferencing for FestEval.

Susan is an experienced policy analyst and program evaluator. Her career includes consultancy, policy, managerial and research roles. Susan has worked in the public, private and university sectors with over 30 years of policy and program evaluation experience across a range of government portfolios responsible for health and ageing, welfare, social and human services. Susan is an AES Board member.

Julie is an experienced evaluator with extensive background in public sector performance management, evaluation policy and capacity building. She is a member of the AES Canberra Committee and the AES Pathways Committee. Currently she is a PhD candidate exploring evaluation with complexity in mind.

Judy is an experienced research and evaluation specialist. She helps clients make sense of complexity in challenging and dynamic situations. She has an interest in how we can respond to the new challenges that have emerged out of COVID-19.

FestEVAL Club

The FestEVAL Club, open each night from 6pm, will be a relaxed place to drop in and chat with colleagues about evaluation issues.

The Club will operate according to Open Space Technology principles – you choose the topics you wish to discuss and you can move freely between breakout rooms to join different conversations.

Some of the discussions will no doubt be prompted by the earlier proceedings each day, including the provocations on day one.

There will be a chill-out room where the resident DJ will provide a musical backdrop to informal discussions.

The Club is strictly BYO – grog and bean bag – so settle in for as long or little as you like!

Facilitators: Lewe Atkinson (QLD), Charlie Tulloch (VIC), Marie Nissanka (NZ), Donna Holden (SA), Greg Masters (NSW)

Lewis’s systems thinking super-powers, enables him to see patterns, pathways, and solutions that others don’t…his mission is to help you unleash yours.

Donna was once an intrepid global wanderer, puppeteer and jester who is now revelling in unlearning and relearning the way we do business as she basks on the South Australian coast waiting for the first flight out!

Marie is a cat groomer, activist and an unsuccessful musician turned mixed-methods evaluator living in Wellywood.

Charlie has the mind of Roger Federer, the grunt of Rafa Nadal and the passion of Novak Djokovic, without any of the success. He uses his tennis as fuel for his evaluation habit.

Greg, a jazz (or dub) wannabe icon, has been conducting evaluations for decades and still wonders whether they are art, craft, science or folly.

Our facilitators and MCs will be there to manage the process, set up topics for discussion and facilitate the movement of people through breakout rooms. And supply the background music. You’ll be providing the content – we’ll look after the vibe ….. and the technology!

Australian Public Sector Evaluation Network Special Interest Group

Presented by members of the APSEN SIG

Are you interested in evaluation in the public sector? Come along to the launch of Australian Public Sector Evaluation (APSEN) Special Interest Group at AES FestEVAL. Hear about the vision for the new network to support public sector evaluators and evaluation in Australia. You’ll hear about how members of the APSEN steering committee are evaluating within the public sector, and how you can access contemporary best practice tools and templates from across the country through our new website.

Facilitated by members of the Design and Evaluation Special Interest Group

The Design and Evaluation SIG explores the interface between the fields of design (i.e. design thinking, co-design, systems thinking) and evaluation. This session will provide an overview of the SIG's purpose, outline some of what the group has done or explored in the past and identify what is of interest to SIG members going forward. 

Presentation of the COVID-19 evaluation community survey

Duncan Rintoul and members of the AES Vitality Committee present the top-line findings of the AES Engagement Survey 2020 which surveyed members and the wider evaluation community – pre, during, and post-pandemic.

2020 Australian Evaluation Society AGM

Chaired by John Stoney, President 

Join the Australian Evaluation Society (AES) Board as we celebrate another year’s achievements by members of the AES, introduce the 2020-2021 Board, and preview the new AES website.

FestEVAL Club

The FestEVAL Club, open each night from 6pm, will be a relaxed place to drop in and chat with colleagues about evaluation issues.

The Club will operate according to Open Space Technology principles – you choose the topics you wish to discuss and you can move freely between breakout rooms to join different conversations.

Some of the discussions will no doubt be prompted by the earlier proceedings each day, including the provocations on day one.

There will be a chill-out room where the resident DJ will provide a musical backdrop to informal discussions.

The Club is strictly BYO – grog and bean bag – so settle in for as long or little as you like!

Facilitators: Lewe Atkinson (QLD), Charlie Tulloch (VIC), Marie Nissanka (NZ), Donna Holden (SA), Greg Masters (NSW)

Lewis’s systems thinking super-powers, enables him to see patterns, pathways, and solutions that others don’t…his mission is to help you unleash yours.

Donna was once an intrepid global wanderer, puppeteer and jester who is now revelling in unlearning and relearning the way we do business as she basks on the South Australian coast waiting for the first flight out!

Marie is a cat groomer, activist and an unsuccessful musician turned mixed-methods evaluator living in Wellywood.

Charlie has the mind of Roger Federer, the grunt of Rafa Nadal and the passion of Novak Djokovic, without any of the success. He uses his tennis as fuel for his evaluation habit.

Greg, a jazz (or dub) wannabe icon, has been conducting evaluations for decades and still wonders whether they are art, craft, science or folly.

Our facilitators and MCs will be there to manage the process, set up topics for discussion and facilitate the movement of people through breakout rooms. And supply the background music. You’ll be providing the content – we’ll look after the vibe ….. and the technology!

Emerging Evaluators for Social Justice SIG

Join members of the Emerging Evaluators for Social Justice Special Interest Group, Yonas Dare and Kaisha Crupi to explore:

1) What is the definition of an emerging evaluator?

How has this changed from when this group originally started?

At what point do emerging evaluators ‘graduate’ from this classification?

2) What kind of support do emerging evaluators need from the AES/experienced evaluators to further strengthen their evaluation skills/knowledge/career prospects?

Anyone is welcome to join the session, emerging evaluator or not, and we will use the session to hold a briefing on our initial definition of EESJ and current observations followed by a discussion and write down notes to take away and develop a proposal to present to the AES about any assistance this group may require moving forward.

Yonas Dare is an internal evaluator within the Asylum Seeker Resource Centre that is located in Melbourne. Mr. Dare has extensive experience in program management, evaluation, consulting and research in the areas of peace building, capacity building, social accountability, Sexual and Reproductive health and HIV/AIDS with particular emphasis on Most at Risk Population in Ethiopia. He worked as head of programs at the Ethiopian Interfaith Forum for Development Dialogue and Action and taught and supervised students on Program Design and Management at the Ethiopian Graduate School of Theology.

Kaisha Crupi is an emerging evaluator, consultant and learning experience coordinator at Clear Horizon. A current Master of Evaluation student at the University of Melbourne, Kaisha has developed her evaluation skills through the course as well as through her work experiences over the last 2 years. She is interested in evaluation capacity building and taking the client on an evaluation journey to provide the best experience for the context.

Evaluating fast and slow

Facilitated by Charlie Tulloch and presented by Eleanor Williams and Sean Chung

Evaluations can take many shapes and sizes. This session will explore the critical dimension of time - and whether some evaluations fail to strike while the policy iron is hot while others need longer to allow programs to be genuinely assessed. It will ask - how can evaluators have maximum impact while retaining rigor and reliability over different time horizons? In what circumstances could evaluations be expedited?

Charlie Tulloch is an evaluation and public policy consultant who uses evidence to enhance performance. He founded Policy Performance in 2018 and previously worked at KPMG, ACIL Allen Consulting and in the Victorian Department of Premier and Cabinet.

Sean Chung is a principal Director of Paxton Partners and has over ten years of consulting and evaluation experience, including six years with the Healthcare and Human Services practice of KPMG, across both Australia and Canada. Prior to entering consulting, Sean spent four years in academic health research.

Sean has particular interests in the formation and evaluation of collaborative networks to improve health service delivery and the use of technology to facilitate better understanding and management of health and human services clients and system performance.

Sean is also currently assisting the Victorian Government’s Covid policy response.

Evaluation with and for First Nation’s communities

Facilitated by Don Bemrose 

This session will provide a space to reflect on and respond to Nicole Tujague’s provocation from the FestEVAL opening.

Don Bemrose is a Gunggari man with family ties to the Cherbourg Aboriginal Community in Queensland and since 2008, has called Narrm (Melbourne) home. He is a primary school teacher and classically trained opera singer and has spent 20 years advocating and working in the arts, youth development, philanthropic, and vocational training sectors. Don provided expert advice through associate consultancy

AES mentoring initiative

Facilitated by Sue Leahy and David Turner

Like the metaphorical black swan, an AES online group mentoring initiative lies outside the realm of expectations. Reports that it has not happened are of extreme interest to AES members because opportunities we aren’t expecting tend to comes as a surprise and collectively play vastly larger roles than regular occurrences.

This session reports on the development of the AES mentoring initiative. This innovation is being co-designed by a working group of the Pathways committee in conjunction with four AES Fellows: Penny Hawkins, Rick Cummings, Alan Woodward and Scott Bayley and AES Board member Doyen Radcliffe. Through initiating mentee-mentor relationships between less experienced evaluators with these more ‘seasoned’ evaluation luminaries, the new initiave seeks to help less experienced evaluators navigate their professional journey. The initiative has a holistic focus on mentee’s professional development as evaluation practitioners. It recognises that mentoring can provide professional socialisation that helps build confidence and encourage reflective practice.

During this session we will outline the proposed pilot design including its objectives, expectations and evaluation. Also, we will announce the expression of interest process for AES members to become mentees. The mentors will describe the areas of interest they intend to focus on in their mentoring role for the pilot. Participants at this session will be invited to provide feedback on the proposed design.

The theory of black swan events depend on the observer. For example, what may be a black swan surprise for a turkey is not a black swan surprise to its butcher. By identifying vulnerabilities we can all avoid ‘being the turkey’. Therefore, this session will prompt AES members to override any presumption they may hold that an AES mentoring initiative is impossible, lies outside the realm of expectations or is at least non-existent.

Sue Leahy is an independent consultant and the Chair of the AES Pathways Committee.

David Turner is Principal Evaluator, New Zealand Ministry of Education.

Using Systems Theory in Evaluation

Brian Keogh interviewed by Ruth McCausland

What is systems theory and what relevance does it have to evaluation? How does complexity relate to systems thinking? How does a systems style evaluation replace a program logic? What does it look like in practice?

Bring your systems-related questions to this provocative and informative Q&A session with Brian Keogh, who has decades of experience in the complexity of water…a simple little substance that everyone and everything needs for life. This has spawned social, economic and environmental systems of mind-boggling complexity.

This session aims to challenge and broaden participants’ understanding of the value of systems thinking via the reflections of an experienced evaluator who has been facilitating a LinkedIn group on the topic since the last AES conference.

Brian Keogh has worked across community development, performing arts and the environment. After graduating from UNSW, he started working on First Nations issues under Glenda Humes, a proud Gunditjmara woman. He then set up Legs on the Wall, an international touring physical theatre company. When his legs slowed down, he managed the Millennium Celebrations, did his MBA and started working in water. He met the enigmatic Professor Ralph Renger and the amazing Andrew Hawkins (ARTD), and rode ambulances with them in the mid-west of USA discussing Ralph’s ideas around Systems Evaluation Theory. Along came Dr Lewe Atkinson (Haines in Brisbane), Julie Elliott (PhD candidate exploring evaluation with complexity in mind) and the LinkedIn Systems Evaluation Group….the murky waters of complex problems became clearer.

Dr Ruth McCausland’s work focuses on systems-oriented and community-led responses to incarceration. Ruth is Research and Evaluation Director for the Yuwaya Ngarra-li partnership between UNSW and the Dharriwaa Elders Group, and Senior Research Fellow in the School of Social Sciences at UNSW. Her PhD was on evaluation and improving the diversion of Aboriginal women from prison, and she has a Masters in International Social Development. Ruth has previously worked as an evaluation consultant for government and non-government agencies and a policy advisor at the NSW Anti-Discrimination Board and Australian Human Rights Commission.

FestEVAL Club

The FestEVAL Club, open each night from 6pm, will be a relaxed place to drop in and chat with colleagues about evaluation issues.

The Club will operate according to Open Space Technology principles – you choose the topics you wish to discuss and you can move freely between breakout rooms to join different conversations.

Some of the discussions will no doubt be prompted by the earlier proceedings each day, including the provocations on day one.

There will be a chill-out room where the resident DJ will provide a musical backdrop to informal discussions.

The Club is strictly BYO – grog and bean bag – so settle in for as long or little as you like!

Facilitators: Lewe Atkinson (QLD), Charlie Tulloch (VIC), Marie Nissanka (NZ), Donna Holden (SA), Greg Masters (NSW)

Lewis’s systems thinking super-powers, enables him to see patterns, pathways, and solutions that others don’t…his mission is to help you unleash yours.

Donna was once an intrepid global wanderer, puppeteer and jester who is now revelling in unlearning and relearning the way we do business as she basks on the South Australian coast waiting for the first flight out!

Marie is a cat groomer, activist and an unsuccessful musician turned mixed-methods evaluator living in Wellywood.

Charlie has the mind of Roger Federer, the grunt of Rafa Nadal and the passion of Novak Djokovic, without any of the success. He uses his tennis as fuel for his evaluation habit.

Greg, a jazz (or dub) wannabe icon, has been conducting evaluations for decades and still wonders whether they are art, craft, science or folly.

Our facilitators and MCs will be there to manage the process, set up topics for discussion and facilitate the movement of people through breakout rooms. And supply the background music. You’ll be providing the content – we’ll look after the vibe ….. and the technology!

Bringing Multicultural Voices and Values to Evaluations

Presented by members of the Multicultural Evaluation Special Interest Group, Eva Sarr and David Roberts

Join us for an interactive workshop showcasing one process for bringing minority voices and values to evaluations. We will workshop a process for preparing for an evaluation of a program from the viewpoint of a community. The facilitators and participants will work together to elicit key historical, social and political contexts. The workshop will explore how these contexts impacts on community members, on the way programs are implemented and on the program outcomes.

Eva Sarr is the Founder and Director of Centre for Multicultural Program Evaluation. Eva has worked as a monitoring and evaluation specialist for 12 years. Her work included large-scale complex system wide evaluations where change moves non-linearly and at a rapid pace; there are multiple stakeholders; outcome measures are developed as programs are designed and implemented; and evaluation literacy is variable. Eva is one of the founders and the interim chair of the AES's first Multicultural Evaluation SIG.

David Roberts is a self-employed consultant with over 35 years experience in research and evaluations. He has held evaluation leadership positions in Australia and internationally. David’s experience encompasses a wide range of approaches and methods across diverse fields. He has particular expertise in Community Engagement, Participatory Research, Elicitation Techniques, Ethnography and Qualitative Methods.

Roundtable discussion on the AES response to the Indigenous Evaluation Strategy

Members of the AES Indigenous Culture and Diversity Committee and Board members discuss the AES recommendations to the Productivity Commission draft Indigenous Evaluation Strategy. The AES submission is available here: https://www.pc.gov.au/__data/assets/pdf_file/0005/255650/subdr174-indigenous-evaluation.pdf

Evaluator self-assessment tool part 2

The Pathways Committee have spent the last few months converting the AES Professional Learning Competency Framework into a free online self-assessment tool in collaboration with learnevaluation.org. The tool has been developed for aspiring, emerging and established evaluators to assess themselves against the competencies and be/er understand and manage their strengths and gaps. 

In the first part on Monday, we debuted the tool for participants to test. In this part, we present the aggregated results, and invite reflection and feedback from the group to refine the tool for wider release.

FestEVAL Closing Session

Facilitated by Jade Moloney and Charlie Tulloch

This will be an interactive session to tie together the various threads and themes from the conversations held during FestEVAL and to strengthen our ongoing connections as a society of evaluators. The closing will also relaunch the Brisbane 2021 Conference.

FestEVAL Club

The FestEVAL Club, open each night from 6pm, will be a relaxed place to drop in and chat with colleagues about evaluation issues.

The Club will operate according to Open Space Technology principles – you choose the topics you wish to discuss and you can move freely between breakout rooms to join different conversations.

Some of the discussions will no doubt be prompted by the earlier proceedings each day, including the provocations on day one.

There will be a chill-out room where the resident DJ will provide a musical backdrop to informal discussions.

The Club is strictly BYO – grog and bean bag – so settle in for as long or little as you like!

Facilitators: Lewe Atkinson (QLD), Charlie Tulloch (VIC), Marie Nissanka (NZ), Donna Holden (SA), Greg Masters (NSW)

Lewis’s systems thinking super-powers, enables him to see patterns, pathways, and solutions that others don’t…his mission is to help you unleash yours.

Donna was once an intrepid global wanderer, puppeteer and jester who is now revelling in unlearning and relearning the way we do business as she basks on the South Australian coast waiting for the first flight out!

Marie is a cat groomer, activist and an unsuccessful musician turned mixed-methods evaluator living in Wellywood.

Charlie has the mind of Roger Federer, the grunt of Rafa Nadal and the passion of Novak Djokovic, without any of the success. He uses his tennis as fuel for his evaluation habit.

Greg, a jazz (or dub) wannabe icon, has been conducting evaluations for decades and still wonders whether they are art, craft, science or folly.

Our facilitators and MCs will be there to manage the process, set up topics for discussion and facilitate the movement of people through breakout rooms. And supply the background music. You’ll be providing the content – we’ll look after the vibe ….. and the technology!