Menu
Evidence Standards are research tools that are used to assess the quality and effectiveness of quantitative impact evaluations of interventions.
ANROWS has developed two tools: a risk of bias tool and an effectiveness estimate tool. These are applied to studies in each of the Intervention Reviews and allow comment on whether an intervention “works”. Both tools provide a framework within which to organise the evidence base and comment on effectiveness.
All studies included in assessments of intervention effectiveness within the Evidence Portal will have a risk of bias rating attached to them.
When making funding or policy decisions, policymakers and practitioners may consider the effectiveness findings alongside other types of evidence and contextual information. This may include the concerns and culture of the community, the insights of practitioners and the interventions already in place. Adding the findings of impact evaluations to the decision-making process is an essential step to increase policy effectiveness and make efficient use of limited resources
What is risk of bias? A key component of evidence-based research and practice in this space is appraising the credibility of conclusions drawn from studies of intervention effectiveness.
This requires a critical appraisal of quantitative evaluation studies across multiple domains that collectively examine whether the design, reporting and implementation of an evaluation study can support the conclusion that the intervention ‘works’.
ANROWS developed a bespoke risk of bias tool, named ANROWS-IRIS (ANROWS Instrument for assessing Risk of bias in quantitative Impact Studies). Risk of bias assessment typically requires an assessor to answer a series of signalling questions to identify and rate the impact of several potential sources of bias. These signalling questions are often grouped into domains or categories of potential threats to study credibility.
The ANROWS-IRIS has six domains:
Each domain is scored individually on a 3-point low-medium-high scale, and then scores for all six domains are combined into an overall 6-point score from very low through to very high.
Studies that meet the threshold of very low, low, or medium risk of bias on the ANROWS-IRIS are eligible for inclusion in estimations of whether an intervention is effective.
This approach is taken because it ensures that recommendations for evidence-informed policy are drawn from studies where we are more confident that the design, reporting and implementation of the evaluation can support the conclusion that the intervention “works”.
Studies are grouped together to create an effectiveness rating for each outcome.
The ANROWS Effectiveness Estimate tool uses a clearly set out framework to classify interventions and their outcomes across the following categories:
✅ = Most systematic reviews and studies show effectiveness
🟢 = Most studies show effectiveness
🟨= Most systematic reviews and studies show no effect
🟡 = Most studies show no effect
⛔ = Potentially harmful
🔵 = Mixed evidence
✖ = Insufficient causal evidence
🕓 = Not yet rated
ANROWS acknowledges the Traditional Owners of the land across Australia on which we live and work. We pay our respects to Aboriginal and Torres Strait Islander Elders past and present. We value Aboriginal and Torres Strait Islander histories, cultures and knowledge. We are committed to standing and working with First Nations peoples, honouring the truths set out in the Warawarni-gu Guma Statement.
This material was produced with funding from the Australian Government Department of Social Services. ANROWS gratefully acknowledges the financial and other support it has received from the government, without which this work would not have been possible. The findings and views included in the Evidence Portal are those of the authors and cannot be attributed to the Australian Government Department of Social Services.
