Introduction

Indigenous-led approaches to use evidence from research to inform on-ground decision-making are increasingly advocated in a range of Indigenous contexts to ensure that research does more good than harm (Coombes et al. 2014; Johnson et al. 2016). Indigenous leaders from around the world have called for research outcomes to be usable and useful (Austin et al. 2019; Kealiikanakaoleohaililani and Giardina 2016; Kwaymullina 2016). As a result, the relationship between research practice and evidence and the concerns of Indigenous peoples is of growing interest for sustainability scientists (Robinson et al. 2016; Lyver and Tylianakis 2017).

Incorporating Indigenous-led evaluations of sustainability science enables researchers to monitor their methodology and adjust where necessary to collaborate effectively and deliver the desired impacts. Inter- and trans-disciplinary sustainability science methods now exist and offer innovative and decolonising methodologies to engage in respectful knowledge sharing and co-production (Barbour et al. 2012; Hill et al. 2012; Winter et al. 2021). Yet surprisingly, there are few published examples of how collaborative knowledge work can be evaluated by Indigenous partners to ensure research is addressing priorities and problems that Indigenous people wish to be solved on their estates. This paper focuses on the characteristics and evaluation of effective Indigenous-led research.

Using research for Indigenous-led sustainable decision-making

Sustainability scientists have highlighted that collaborative processes for sharing established knowledge practices and co-producing new knowledge are the best pathway to guide sustainable action (Berkes 2009; Díaz-Reviriego et al. 2019; Sterling et al. 2017). Collaborative processes involve interactive and adaptive approaches to problem solving that provide ways of integrating different values, priorities and knowledge practices for environmental planning and management (Tengö et al. 2014; Trimble and Plummer 2019). There are many benefits from collaborative practices, including the capacity to share and manage sources of knowledge, build trust, foster social learning, develop mutually agreeable solutions, and lead to desired environmental conditions (Smedstad and Gosnell 2013).

There is also a growing interest in the evaluation of participatory research (e.g. Blackstock et al. 2007) and more inclusive approaches to the evaluation of research impact (Morton 2015). This has prompted growing interest in how sustainability science can accommodate different knowledge contributions and embrace knowledge co-production through research practice (Norstrom et al. 2020). This paper focuses on a small but growing subset of collaborative sustainability science, particularly the additional care and consideration surrounding Indigenous-led research practice and evaluation. Indigenous-led research recognises that Indigenous governance systems underpin how knowledge is shared, used and translated, as well as how science is developed, tested and used (Robinson et al. 2016; Johnson et al. 2016).

Finding ways to enable sustainability science to be useful for and used by Indigenous peoples is critical. Indigenous lands now cover at least one-quarter of terrestrial Earth and Indigenous rights to govern the ways in which sustainable solutions are implemented are increasingly recognised in domestic and international policy (Garnett et al. 2018; IPBES 2019; Mistry and Berardi 2016). Indigenous groups around the world are calling for their intellectual and cultural property rights to underpin research ethics and data curation practice to ensure sustainability science research is negotiated with and for Indigenous people’s benefit (Hudson et al. 2016; Robinson et al. 2021; Walter et al. 2020). Yet finding the balance between Indigenous rights and sustainable solutions for a planet under pressure is challenging. Indigenous leaders make the vital argument that Indigenous rights need to be recognised and Indigenous-supported incentives implemented, so that local Indigenous communities do not unfairly wear the burden of dwindling biodiversity and global solutions for sustainability (Duncan et al. 2018; Lyver and Tylianakis 2017; Robinson et al. 2016).

Knowledge generated from research is an important aspect of the knowledge that can be used to inform Indigenous decisions for their estates and futures (Austin et al. 2019; Woodward et al. 2020). Yet a variety of influential factors exist that limit Indigenous efforts to guide the collaborative sharing and production of knowledge via decolonised research practices and approaches. These include: the long troubling and enduring colonising power dynamics of research on Indigenous peoples (Kwaymullina 2016); the reward systems of research institutions that dis-incentivise Indigenous-led approaches for knowledge sharing and collaboration (Katz et al. 2016); language and cultural differences (Davies et al. 2013); and divergent standards of knowledge credibility and legitimacy (Agrawal 2002). Such barriers are amplified when research is transferred through the traditional pipeline mode in which scientists set the research agenda, do the research, and then transfer the results to potential Indigenous users, assuming the research will be useful and will diffuse automatically through the Indigenous community.

This paper responds to this challenge by outlining an Indigenous-led approach by which the useability and usefulness of sustainability research was evaluated. We begin by outlining the research setting in which research was evaluated as part of a project that involved Indigenous and non-Indigenous collaborators negotiating, sharing and creating knowledge to guide adaptive co-management decisions in Australia’s Kakadu National Park (NESP 2019a). The methods by which Indigenous collaborators developed indicators to evaluate the project’s knowledge sharing and co-production practices are described, followed by the results of the evaluations, including how feedback was used to inform any necessary adjustments to research activities, and to guide action going forward.

The paper contributes to sustainability science literature by showing that Indigenous evaluations of research emphasise the specific qualities of knowledge sharing, co-creation and translation that Indigenous people value. We highlight the benefits research needs to deliver for it to be judged as useful to Indigenous people. It also extends the literature on research co-design and evaluation practice to ensure Indigenous perspectives inform research program design and implementation, that opportunities for reflection and sensitivity to community aspirations are incorporated, and that research outcomes important to Indigenous peoples are included.

Research setting

Australia’s Kakadu National Park (Kakadu) is an internationally significant World Heritage-listed landscape, jointly managed with Bininj Traditional Owners in the north of the Park and Mungguy Traditional Owners (including the Jawoyn people) in the south (Press et al. 1995). Bininj/Mungguy custodianship of Kakadu has been practised for more than 50,000 years and continues today through a living culture committed to ‘caring for country’; through rights and stewardship responsibilities for land, sea and resources, and through a decision-making governance system between clans and kin. The area is leased by its Bininj/Mungguy owners to the Australian Government and is under formal joint management based on a legal framework set in place by Australia’s Environmental Protection and Biodiversity Conservation Act 1999 (Cth) and the Aboriginal Land Rights (Northern Territory) Act 1976 (Cth) (Kakadu Board of Management 2016a). The management plan for the National Park is implemented under the direction of the Board of Management, which consists of 15 members, 10 of whom are Bininj/Mungguy. The chair of the board is appointed from among Bininj/Mungguy members, who are nominated by Traditional Owners and represent the geographic areas and language groups in Kakadu.

This paper draws on the ‘Kakadu NESP project’, an Indigenous-led environmental management project conducted in Kakadu, funded by the National Environmental Science Program (NESP), partnering with research institutions and with in-kind support from Kakadu through Ranger and Staff involvement (NESP 2019a). The project is guided by a Bininj/Mungguy Research Steering Committee (RSC) made up of Traditional Owners from all the clans in the Kakadu region. The aim of the project was to develop Bininj/Mungguy healthy country indicators to guide adaptive and collaborative decision-making in Kakadu, and to support Kakadu’s monitoring program in the longer term (NESP 2019b). We evaluated the useability and usefulness of the research through an Indigenous-led approach throughout the Kakadu NESP project, including in the two priority areas identified by the RSC that are included in this paper: the floodplain site at Nardab (the East Alligator floodplain) in the north of the Park, where research focused on weed control; and the stone country site at Jarrangbarnmi (Koolpin Gorge) in the south of the Park, where research focused on fine-scale cultural fire management. The research collaboration consists of Bininj/Mungguy RSC members, Traditional Owners of each case study site, five non-Indigenous research scientists, Bininj/Mungguy co-researchers, Indigenous and non-Indigenous Kakadu Rangers and Staff, and the neighbouring Njanjma Rangers.

Methods

The Indigenous-led approach to assessing research activities and impacts outlined in this paper was developed based on several important factors. We were mindful that the process of making decisions and modes of engagement are often considered an important measure of effective collaboration by Indigenous community members (cf. Corrigan et al. 2018; Izurieta et al. 2011; Kakadu Board of Management 2016b). We also needed to ensure the evaluation was rigorous and empowered all Indigenous collaborators to have a voice in the research (cf. Blackstock et al. 2007). The evaluation needed to recognise the resilience and assets of Bininj/Mungguy, including knowledge, skills, networks, extended families and cultural identity, while also creating opportunities for ongoing learning and sustainability (cf. Thomas et al. 2016; Thompson et al. 2019). The research collaboration therefore built Indigenous-led evaluation processes to assess research activities and impacts into the entire life cycle of the Kakadu NESP project for agreed environmental, socio-economic and cultural outcomes. In this approach, we paid attention to understanding key insights from what was and what was not working well, to ensure these learnings could direct tangible and achievable research activity adaptations, while acknowledging the more transformative adaptations of the research that may have been needed (cf. Baylis et al. 2016).

Our research evaluation purpose and approach were developed by Bininj/Mungguy RSC members and the non-Indigenous science team, who are all co-authors on this paper. Three distinct components of the Indigenous-led research process were identified and became the focus of the monitoring and evaluation effort: (1) negotiating the context; (2) enabling adaptive and collaborative decision-making; and (3) undertaking Indigenous-led research activities. Each component of the research process is described in more detail below:

  • (1) Negotiating the context

    Indigenous collaborators led the research agenda and approach, which responded to location-based socio-economic contexts and included what to research, where to do it, and who needed to be involved. Indigenous collaborators and the non-Indigenous science team co-designed the evaluation approach and criteria for each research phase.

  • (2) Enabling adaptive and collaborative decision-making

    Adaptive and collaborative research was resourced and supported at multiple scales. This is important when working in a place like Kakadu because while Indigenous peoples’ knowledge systems and decision-making practices are country-based (local), with particular people and clan groups having connections and governance responsibilities to particular land and sea areas, co-management is governed and nested between multiple-scales, including local, district, and the region (whole-of-park). For the research to be useful and useable for Bininj/Mungguy and co-managers, decision-making mechanisms needed to support monitoring and management that was locally grounded and validated, and applicable at the district scale and across the Park (cf. Austin et al. 2019).

  • (3) Undertaking research activities

    Adaptive research activities were undertaken and monitored using Indigenous knowledge practices and non-Indigenous science-based methods and technologies. Science based monitoring methods and technologies can be technical and/or novel to Indigenous people, so training needed to be incorporated into fieldwork, to enable Rangers to use acquired skills in current and future Park-based and scientific projects.

While the first research component—negotiating the research context—occurred at the commencement of the project and at other pivotal moments (e.g. when an Elder for a site died); the second and third component were iterative and occurred throughout the research process (Fig. 1). Conducting evaluations throughout this iterative process allowed for agile testing of usable knowledge and used evaluation processes to move knowledge into collaborative, on-ground action and decision-making.

Fig. 1
figure 1

Ongoing Indigenous-led evaluation to monitor and adapt collaborative knowledge practices through the negotiation of context (1), enabling adaptive and collaborative decision-making (2), and undertaking research activities (3) components of the research process

Full size image

The RSC worked with the non-Indigenous team members to ensure that the appropriate people were being consulted and were given the chance to collaborate in the research project. It was agreed that that the RSC had three main purposes to facilitate the Kakadu NESP research effort:

  1. (1)

    to negotiate and select case-study sites to focus research activities and collaborations;

  2. (2)

    to provide a regional-scale partnership for Indigenous clan leaders to ensure that knowledge generated from the research could be shared, used and co-developed to care for priority areas and issues across the Park in an ethical and equitable manner; and,

  3. (3)

    to evaluate the usefulness of the research and the usability of knowledge generated from the project for Traditional Owner’s efforts to jointly manage their estates.

Care was taken to ensure the evaluation process was culturally safe so that Indigenous collaborators could lead, understand, and learn about research activities and impact, and the Kakadu NESP team could learn and respond to Indigenous perspectives and aspirations. The RSC suggested doing the evaluations in the absence of the non-Indigenous team members, so that people would feel as comfortable as possible to discuss both positive and challenging aspects of the research project and approach. The non-Indigenous team members would then return to hear feedback. The RSC also decided that the evaluation of the research would be open to Bininj/Mungguy collaborators and Indigenous and non-Indigenous Kakadu Rangers and Staff. This decision was based on the recognition that, as one RSC member put it “we are all here to care for Kakadu” (RSC Meeting 13/8/2018); it was this joint aspiration for the Park that offered common ground for co-managers to share and build knowledge to manage the unique biocultural landscapes (cf. Johnson and Larson 2013).

The RSC also decided that we would do the research evaluations throughout each phase of the research process, after each monitoring effort at each of the case study sites, after each local-district workshop, and after ever regional-scale RSC meeting. The research team would then adapt project activities, outcomes and methods based on feedback from the evaluations.

The Kakadu NESP project was funded for three years, although through collaborative efforts further support is being sought for the project to continue. The project is now in its final stages and the Bininj/Mungguy RSC members and the research scientists met together to workshop and agree on key findings and the process for finalising the research project. It was agreed that the science team and Bininj/Mungguy RSC members would co-author this paper to, as an RSC member described, “explain to other researchers how to work effectively with Indigenous people to help care for country” (RSC Meeting 7/10/2020). It was also agreed that the team would hold a forum (scheduled for mid-2021) to enable Traditional Owners and RSC members to share key findings, impacts and lessons from the Kakadu NESP project so that the knowledge and learning collaboratively built together can be translated into co-management decision-making and practice.

The RSC identified three research performance indicators to be monitored to ensure healthy knowledge sharing, co-production, and translation practices.

  • Research engagement: Have the right people been consulted and given the chance to direct research and collaborate?

  • Knowledge sharing and co-production: Is knowledge from the research being shared, learned and used by scientists, National Park staff and Bininj/Mungguy collaborators?

  • Bininj/Mungguy employment and training opportunities: Have training and employment opportunities been provided to interested Bininj/Mungguy co-researchers?

These criteria were evaluated using a survey with a Likert scale scoring system from 1 to 5 (where 1 = unhealthy and 5 = healthy collaborative knowledge work and research practices), with space for qualitative feedback on the reverse of the survey (see Supplementary Material 1). In the absence of the non-Indigenous team members, Traditional Owners, Kakadu Rangers and Staff, and Bininj/Mungguy RSC members completed the evaluation and discussed their feedback and any adaptions to improve research practice and outcomes. Bininj/Mungguy and Kakadu Rangers and Staff either wrote their feedback anonymously on the survey, or, more often, delivered their feedback orally to the non-Indigenous researchers when they returned to the discussion, who would ask for permission to record feedback in notebooks.

The non-Indigenous researchers would then collect the anonymous surveys and enter and analyse the quantitative and qualitative data using excel. The results of these evaluations were reported back to and checked by Traditional Owners, RSC members, Rangers and Park Staff through RSC meetings and project updates, which detailed any actions that all Bininj/Mungguy and non-Indigenous team members had committed to implement.

While there were undoubtedly some topics discussed during evaluations that were not communicated to the non-Indigenous research team members, the agreed purpose was to improve knowledge sharing, co-production, and translation practices and this was the focus of feedback discussions. As with any Indigenous-led research project on biocultural landscapes, sensitive topics were sometimes discussed that included information about sacred stories or places. Discussions sometimes also turned to the practices of other researchers working in Kakadu. This data has not been included based on feedback from the RSC that the purpose of this paper should communicate the process we used to improve our research practices in the Kakadu NESP project. Most importantly, the evaluation process provided a space that empowered Bininj/Mungguy and Kakadu Rangers and Staff to discuss and adapt the research project and practices.

In the next section, the results of evaluating the research team’s knowledge-sharing work and practices are described, with illustrative examples from two case sites — Jarrangbarnmi and Nardab — and those offered by the Bininj/Mungguy RSC. We present the results of research evaluations and discuss any adaptations to the research that occurred based on the evaluations.

Results

Results are presented for each of the three components of the Kakadu NESP project (negotiating the context, enabling decision-making, and undertaking research activities) and highlight examples of evaluations and research responses that occurred as part of this Indigenous-led research project. This process was not without healthy disagreements and required careful consideration of Indigenous-led research ethics and evaluation principles (e.g. Kealiikanakaoleohaililani and Giardina 2016; Kwaymullina 2016). In practice, this meant that the process required careful mediation, recognised Indigenous collaborators were not homogenous in their views, and the research team had to agree on what research priorities and allocation of resources the team could realistically provide. This ongoing collaborative and adaptive process is outlined below.

Negotiating the context

Knowledge work and practices: the first 12 months of the project focussed on team engagement and included sharing all relevant background information required to plan and implement the research and negotiate the focus of the evaluation to respond to Indigenous priorities for research practice, outputs and impact. This required the research team to engage with Traditional Owners to: design research governance arrangements at local, district, and regional scales; determine research priorities and activities within Kakadu; and pay attention to how research questions and underlying research assumptions might impact on Bininj/Mungguy and their aspirations for their traditional estates. Clear communication about the parameters for timeframes, resources and funding, requirements from funding bodies occurred during this phase and continued to be communicated and negotiated throughout the life of the research project. On practical level, the RSC chose the case study sites where the research team would work and directed that the research should focus on monitoring culturally important food resources—termed ‘bush tucker’—available at each site, which are a management priority for Traditional Owners across Kakadu.

Evaluation of this knowledge work: the first evaluation was undertaken after a RSC meeting in May 2019 by 10 Bininj/Mungguy RSC members (see Table 1). During this evaluation, the holistic nature of Bininj/Mungguy indicators were discussed and were reclassified from ‘bush tucker indicators’ to ‘healthy country indicators’. The importance of enabling Elders and young people to be involved in on-ground management and monitoring activities was emphasised, with a desire for this involvement to occur through employment and training opportunities.

Table 1 Results of Bininj/Mungguy Research Steering Committee evaluations of the Kakadu NESP team’s collaborative knowledge practices during the research context negotiation phase of the research
Full size table

Adaptations based on feedback: the research was broadened to focus on healthy country indicators, as opposed to just bush tucker indicators, and a case study site in the south of the park was added to the project. A local Indigenous Research Coordinator was hired to facilitate Traditional Owner consultations for NESP projects in Kakadu. These adaptations were unanimously agreed to by the whole team, including the RSC members and the non-Indigenous researchers.

Enabling adaptive and collaborative decision-making

Knowledge work and practice: once the context of the research was negotiated, the research team worked to enable adaptive and collaborative decision-making at multiple scales. Regional-scale guidance for the research was enabled through the RSC. Additionally, at each site, local-district decision-making enabled on-ground research practice to reflect local priorities and stewardship actions. Indigenous peoples’ in situ knowledge systems were resourced and then non-Indigenous scientific experts and Park Rangers and Staff were included to guide monitoring of healthy country indicators before and after agreed on-ground management activities.

At Jarrangbarnmi, a two-day on-country workshop was held to listen to local Traditional Owners, to understand their immediate social, cultural and environmental contexts and aspirations; agree on ways in which the team and district Rangers and Park Staff could support highly adaptive and flexible people–place interactions; and develop collaborative research ethics and approaches that were Indigenous-led and inclusive of local rights and responsibilities for country.

Non-Indigenous research team members participated in a gendered welcome to country ceremony to ensure that they would be culturally safe while working in the area, and to ensure they understood the knowledge and human–non-human practices that were required to ensure their activities did not cause any harm to country. Non-Indigenous experts then shared their knowledge and expertise of weed and fire management and demonstrated the use of monitoring technologies like drones and motion sensor cameras. It was agreed that drones would be flown to monitor before and after Indigenous-led landscape burning activities in the area. Cameras were installed to take photos every 12 h, enabling Traditional Owners to “watch changes to country while we are away” (RSC Meeting 13/8/2018). The ways in which local Indigenous knowledge and its assessment could be reconciled with non-Indigenous science’s production, verification and validation were discussed, in pursuit of co-developing applied sustainability solutions to achieve Traditional Owner-supported impact.

Evaluation of the research: evaluations were done at the end of the two-day on-country workshop at Jarrangbarnmi with 13 Jawoyn Traditional Owners and Kakadu Rangers and Staff (see Table 2). The results of the evaluations revealed concerns that hot fires focused on protecting the nearby camping grounds for tourists were scorching important bush tucker plants in the area. Traditional Owner concerns echo those within the scientific community that late-season landscape burning activities are damaging threatened species and habitats (Woinarski and Winderlich 2014). In addition, Traditional Owners emphasised the need to enable Elders and young people to be involved in on-ground burning and monitoring activities.

Table 2 Results of evaluating collaborative knowledge practices after decision-making moments at the local-district scale at Jarrangbarnmi
Full size table

Adaptations based on feedback: at Jarrangbarnmi, agreement was reached to experiment with Indigenous-led landscape burning and monitoring using Indigenous-Ranger-scientist knowledge-sharing and monitoring approaches that would connect young people, Elders and district Rangers to share and learn knowledge and practice. As one Traditional Owner articulated: “We need to work together with the scientists and the Traditional Owners and the new generation and the Park people, we’ll work with the three clan groups to fix our area … they’ll [scientists will] give us they skill to our new generation” (Jarrangbarnmi 25/06/2019). Another participant emphasised the importance of appropriate authority over the burning, with Traditional Owners deciding “what area to burn, what not area to burn. Sacred site, not allowed to go there or burn there.” (Jarrangbarnmi 25/06/2019).

These discussions were sensitive to the context of Jarrangbarnmi which is associated with the creator figure Bula and other creation ancestors including Bolung (rainbow serpent) in a region generally classified as Buladjang (sickness country). Bula and Bolung are powerful creation ancestors and if disturbed, can prove fatal to Jawoyn people and others alike (Jawoyn Association 2020). Traditional Owners were anxious about the potential impact of drones flying over sensitive cultural sites so a map of Jarrangbarnmi was collaboratively drawn to identify the boundaries of where the research team could and could not work (Fig. 2). It was agreed that monitoring approaches would include flying drones and installing cameras, with Indigenous protocols guiding the use of technology on-country and the data they procured (Macdonald et al. in press). As one Elder explained: “It’s alright you bring all them technology. But if there’s another way of all Traditional Owner getting out here and watching what you, how you put it on” (Jarrangbarnmi 25/06/2019). In addition to monitoring with drones and cameras, the team agreed that young Indigenous co-researchers should be supported to interview and video Elders to share their knowledge, which was then documented in a short film (NESP 2019c).

Fig. 2
figure 2

Negotiating where and what research activities were required to adaptively monitor the health of landscapes at Jarrangbarnmi. Photo credit: Michael Douglas

Full size image

Undertaking conservation research activities

Knowledge work and practice: to undertake the Indigenous-led research activities, the research team worked with district Rangers and Traditional Owners at each case study location, with funding made available to pay senior Traditional Owners and Bininj/Mungguy co-researchers. Employment was on a casual basis, with shifting teams for those available and interested. This local employment was managed by the Kakadu Indigenous Research Coordinator.

This was a field-intensive research project and healthy country indicators were monitored during each of the six seasons used by Bininj/Mungguy and before and after each agreed management activity. Research activities varied by case study location, depending on the monitoring and management actions being performed. At each case study site, the research team and Bininj/Mungguy co-researchers ran workshops, site visits, interviews, mapping activities and face-to-face surveys with Traditional Owners and Rangers. Through these activities, we agreed to the geographic scope of each case study activity. We then identified Bininj/Mungguy values and uses for each site and articulated healthy country indicators and appropriate methods to track the health of habitats and their care. These indicators guided a holistic approach to accommodate Bininj/Mungguy relationships to each site and took into consideration the ecological and socio-cultural assets and relationships associated with caring for Kakadu. Finally, we worked together to find ways to bridge Bininj/Mungguy and non-Indigenous scientific knowledge for the monitoring of each pilot site and the broader Kakadu region, to effectively and respectfully weave Bininj/Mungguy healthy country indicators and methods of monitoring into performance reporting frameworks and on-ground monitoring activities.

At Nardab, Bininj Traditional Owners identified priority areas and indicators to monitor the success of weed management at important sites used for a range of cultural, hunting, and family purposes. Weeds like para grass (Urochloa mutica) have formed monocultures on parts of the Nardab floodplain, displacing the diversity of native vegetation including those used for breeding and feeding by magpie goose (Anseranas semipalmata) and long-necked turtle (Chelodina rugosa)—both important bush tucker species for Bininj (Adams et al. 2018). Traditional Owners, Kakadu Rangers and Staff, and neighbouring Njanjma Rangers worked together to undertake targeted control of para grass in the identified priority areas including helicopter spraying of para grass and follow up ground spraying. The Kakadu NESP team then returned to monitor the floodplain, with monitoring including interviewing Traditional Owners for ground assessments of the floodplain and aerial assessments using drones (Fig. 3).

Fig. 3
figure 3

Kakadu NESP research team monitoring adaptive co-management efforts to improve the health of the Nardab floodplain. Photo credits: Cathy Robinson and Michael Douglas. a Kakadu National Park Ranger Annie Taylor monitoring the health of the Nardab floodplain with senior Traditional Owner Anita Nayinggul. b discussing the health of the Nardab floodplain with senior Traditional Owner Johnathan Nadji

Full size image

Evaluation of the research: an evaluation was undertaken following research activities to monitor the impacts of para grass spraying at Nardab by 12 Bininj Traditional Owners, and Kakadu and Njanjma Rangers (see Table 3). The results of the evaluation revealed the importance of having appropriate Traditional Owners present to guide all management and monitoring activities on the floodplain. The role of women Rangers in undertaking the on-ground spraying was highlighted, with both Bininj and non-Bininj women Rangers registering their interest in learning to use drones and other monitoring technologies to improve their ability to access employment opportunities and care for women’s sites.

Table 3 Results of evaluating collaborative knowledge practices after research activities at Nardab
Full size table

Adaptations based on feedback: based on the feedback obtained from the evaluations, the importance of combining ground-based monitoring with Traditional Owners and aerial monitoring with drones was highlighted so that the authority of Traditional Owners over their floodplain would be maintained. Resources have been directed to support Traditional Owner authority of the floodplain and the research project, including always paying Traditional Owners as senior authorities during monitoring trips and incorporating the day-to-day monitoring activities of Traditional Owners in survey results. Collaborative research activities have showed tangible impacts, including an increase in the number of magpie geese from 50 in 2018 to more than 1800 by 2020. This has inspired Traditional Owners to engage in the project and has catalysed on-ground management in the region. As one Traditional Owner explained, “Every morning I drive to the front there and have a look. A lot of goose coming in hey, and landing. And in the evenings, you can see them flying back here. So they’re moving around now. And that didn’t happen a few years ago” (Nardab 13/08/2019).

After each field trip, the research team have prioritised the reporting of research activities back to Traditional Owners and Rangers. Mechanisms for reporting were developed throughout the project based on evaluation feedback from case sites and RSC meetings. After each fieldtrip or workshop, the research team produced a ‘project update’ providing photos and details of what was achieved, results from research evaluations and agreed actions for the research team and for joint managers. These were sent via email to Kakadu Staff and Rangers, who distributed them to Traditional Owners. The updates were also printed for distribution during the next fieldtrip, to assist Traditional Owners to recap project activities and achievements. Fourteen project updates have been produced and distributed at the time of writing. The direct benefits of the quality of this knowledge work was eloquently described by co-author and senior Traditional Owner for Nardab, Na-gangila Bangalang:

“Kunkare birrikarrmi… nakka paragrass, bolkkime namekke scientist mob birrimwam birridurrkmirri, birrimarnbom kamak rowk nawu ngadberre kunred. And bolkkime, mayh everywhere now.

“In the past there was … para grass. Now those scientists have come and worked with us to make sure our country is healthy. And now there are animals everywhere now” (Jabiru 30/102019).

Discussion: stronger knowledge work and meaningful collaboration produces healthy country outcomes

This paper builds on the broader field of participatory research and evaluation of knowledge sharing and co-production practice (e.g. Blackstock et al. 2007; Robinson et al. 2014) and highlights the special consideration and care that is needed for scientists to successfully produce science for sustainability in Indigenous-led cross-cultural settings. We outline a method to enable evaluations of research effectiveness that acknowledges that Indigenous-led research is not easy or without tensions. The research project in Kakadu has taken time to negotiate and required long established and trusted relationships to mediate. Collaborators need to pay attention to the relational and procedural aspects of research practice so that the diversity of Indigenous and science perspectives can be heard and resolved. Indigenous-led monitoring of research activities and impact ensures informed consent is maintained and enables ongoing dialogue to ensure science questions and activities can also achieve the impacts Indigenous people seek. This approach is now being implemented in other research projects in Australia (e.g. https://www.csiro.au/en/News/News-releases/2020/Herding-wild-buffalo-and-cattle-from-space); attention is being paid to if and how different institutional, cultural and science contexts affect the ethics, ethos and practice of Indigenous-led research evaluation.

In Kakadu, strong knowledge work and meaningful collaboration developed through iterative evaluations of this NESP research project has led to several tangible outcomes. Research deemed useful and responsive to local community concerns for their estates has improved Bininj/Mungguy engagement in the research project: at the start of the project, seven Bininj/Mungguy enlisted as co-researcher employees; at the time of writing, 40 Bininj/Mungguy have worked as co-researchers. Lessons have been learnt from the Kakadu NESP project for improved collaborative knowledge work in other research and management engagement efforts in this World Heritage Area, with a senior Kakadu staff member reflecting that: “the way we work together in this project will be the standard for all scientists working with Bininj in Kakadu” (RSC Meeting 31/10/19). This is a significant achievement, given the challenges to translate the ideal of joint management of protected areas into operational policies and day-to-day practice (Haynes 2017).

Stronger collaborative knowledge work has also created trusted evidence of impact being delivered from on-ground action that aligns with the priorities of Traditional Owners. Importantly, this included empowering Traditional Owners to guide research directions and on-ground activities, therefore ensuring the knowledge shared and co-created was useful. Additionally, team members remained culturally safe, and research findings were shared with Bininj/Mungguy across the Park through frequent field visits, research reports and videos that explained how the research was being used to co-manage this World Heritage Area.

Indeed, this research project has shown that significant outcomes can be achieved when Indigenous people are not only leading research activities but are also using this knowledge to support on-ground and adaptive actions. At Nardab for example, the adaptive co-management program informed by the Kakadu NESP project led to a sharp reduction in the target weed species, para grass, from 67% coverage in 2018 to just 17% in late 2019. This effort exposed significant areas of previously infested wetland, which led to a dramatic increase in its use by magpie goose, from less than 50 individuals in 2018 to more than 1800 by 2020. Importantly, this evaluation enabled Bininj to report nuanced, ground-based assessments during each of the six seasons with Traditional Owners monitoring the health of their country and reporting an improvement in the abundance of long-necked turtles, hunting access to important sites, and knowledge transmission to young people.

The RSC played a critical role in regularly reviewing at the regional scale, the useability of the knowledge being generated from the Kakadu NESP project and the usefulness of the research for priority issues facing Traditional Owners and their estates across the Park. Lessons learned from each of the case-study sites have been regularly shared to inform RSC members so they can evaluate the efficacy of research engagement, knowledge sharing and co-production, and the benefits of the research for Bininj/Mungguy employment and training. The value of this inclusive and adaptive approach to research has been consistently referenced in the committee’s independent evaluations.

Now that the focus is on finalising the project so that benefits can endure for Bininj/Mungguy and Kakadu Rangers and Staff, research resources are focused on communication. A collaborative impact video has been created to enable other co-managers around the world to learn from this effort (NESP 2019d) and a film has been co-created in English and Kunwinjku, to share key research findings and lessons with Bininj/Mungguy and other Indigenous groups who might be interested in this work (Kunwinjku, Jawoyn and Kundjeyhmi are the three most commonly spoken languages in Kakadu) (NESP 2019e). As co-author and chair of the Kakadu Board of Management, Maria Lee, has stated, the emphasis on ensuring Bininj/Mungguy are driving the research and its impact maintains the strong endorsement for the collaboration and the desire to continue working together:

“We’ve worked hard to build this relationship and we want to keep working with you. This NESP team brings Bininj to life here, they feel good about themselves because [the collaborative research effort] motivates them” (RSC Meeting 31/10/2019).

Conclusion

This paper outlines an approach for evaluating Indigenous-led collaborative knowledge work that seeks to negotiate, create, collect, and analyse evidence to guide adaptive management decisions. This approach was adopted by an Indigenous-led research project to evaluate cross-cultural and interdisciplinary research in Australia’s Kakadu National Park. The evaluations recognised the collaborative cross-cultural conditions that enable knowing sharing, testing and translation, and the need to respect the agency of Indigenous people to assess the quality of working knowledge used to guide research on their country. This evaluative effort resulted in strengthened knowledge-sharing practices and more effective collaboration, enabling the project to achieve its desired impacts. This demonstrates the value of empowering Indigenous perspectives on how knowledge can and should be used to guide adaptive decision-making and learning.

The need to ensure research is useful and useable is of global academic and practical concern and this cannot just be judged by scientists alone. Indigenous people’s cultural and sustainability circumstances have long been the subject of academic research, the outcomes of which are often not particularly useful to local Indigenous communities (Tobias et al. 2013; Chilsa 2019). There are increasing calls for Indigenous-driven research that identifies the knowledge priorities of Indigenous communities, and for Indigenous-research partnerships to generate knowledge that Indigenous people can use to make informed sustainable decisions (Harfield et al. 2020; Zurba et al. 2019). The devolution of control over conservation decisions and resources to local and Indigenous communities has increased the need and demand for assessments of ‘conservation success’ to be broadened to include local assessments (Corrigan et al. 2018).

Theoretically, this research extends mainstream sustainability science efforts to improve the craft of creating and translating useable knowledge for sustainability that focuses on active and inclusive collaboration between experts and decision-makers, so as to develop mutual understanding of language, experience and presumptions (Johnson 2012; Clark et al. 2016). Relationships between scientific experts and Indigenous knowledge holders are complex and dynamic; they are rarely unproblematic or free from tension and dispute. Indigenous-led sustainability science requires knowledge created from research to be worked through on-going evaluation by Indigenous people. These evaluations ensure the research process and impact empowers Indigenous rights to self-governance and autonomy; recognises Indigenous knowledge as intimately connected to the governance practices that guide rights, responsibilities and relationships on Indigenous estates; and creates a culturally safe space for the inclusion or debate of new knowledge offered by science (Barelli 2012). Incorporating Indigenous-led evaluations of knowledge-sharing practices enables research teams to adjust where necessary to collaborate effectively and deliver the desired impacts. As a result, collaborative knowledge co-created between Indigenous and research partners is worked and re-worked so that knowledge sharing and development can be facilitated across Indigenous, scientific and management domains (Robinson and Wallington 2012). This is not an apolitical process of sharing information; instead, the acquisition of knowledge must involve processes of learning, re-framing and understanding that empower Indigenous people and research partners (Zanotti and Palomino-Schalscha 2016).

A consistent theme in the sub-field of Indigenous-led sustainability science is the need to be adaptive and responsive to unforeseen changes, stagnation and shifting community sentiment (Austin et al. 2019). Researchers working in Indigenous sustainability contexts need to be attentive to the scientific and other outcomes (economic, social, and ecological) of their activities as well as potential synergies and trade-offs among them. For Indigenous-led research, this process needs to be part of adaptive, collaborative research management to enable information from a range of perspectives to feed into and improve the way research is conducted in the future (Somerville and Turner 2020). This requires sustainability science to undergo significant community input and scrutiny, to be adaptive, requiring iterative articulation and testing, and be purpose built for actions that reflect the knowledge priorities of local Indigenous communities.

This paper has provided insights into how Indigenous-led evaluation of research can be done. Insights from the evaluation process in Kakadu highlight the value of a collaborative and adaptive approach to evaluation that opens up critical analysis but focuses on identifying ‘what works’. Conducting evaluations throughout the research process, not just at the end of the project, creates meaningful opportunities for iterative and ongoing learning and sustainability. The approach used is particularly valuable for evaluating research projects, as most funding involves short timeframes and discrete activities. Evaluation of research in such contexts can enable Indigenous collaborators to consider what should and can be done in a project’s timeframe and drive the legacy impacts and benefits for Indigenous people once researchers have left.

The results of the Kakadu NESP project have shown that place-based evaluations that support knowledge sharing and co-creation between research partners also helps ensure research outcomes can be translated into on-ground action (cf. Austin et al. 2018; Robinson and Wallington, 2012). Further, sharing ideas and knowledge across case studies and clan groups has been highlighted as valuable for facilitating social learning and can lead to more successful outcomes. Strong Indigenous leadership was a key success factor for collaborative knowledge learning and sharing and underpinned this evaluation effort (cf. Chilsa 2019). The cross-cultural context of this approach to research evaluation also required the non-Indigenous science team to experiment, learn and adapt throughout the research process, particularly in contexts where expectations and actions diverged.

What is clear from the Kakadu experience is that Indigenous-led approaches to assess useable knowledge is not just about using the best, most accessible or most immediately relevant information, as much as the quality of the process of sharing and co-creating knowledge to inform decision-making (Duncan et al. 2018; Robinson and Wallington 2012). These insights can inform other collaborations between sustainability scientists, Indigenous leaders and environmental managers to develop better research practices and partnerships that contribute to more beneficial outcomes for all partners.