Andrew Leigh (Fraser, Australian Labor Party) Share this | Hansard source

I move:

That this House:

(1)

reaffirms this Government’s commitment to evidence-based policy making;

(2)

notes that:

(a)

the Productivity Commission has highlighted the importance of rigorous evaluation in assessing the impact of social, educational, employment and economic programs; and

(b)

randomised policy trials are increasingly being used as an evaluation tool in developed and developing nations; and

(3)

supports measures to increase the quality of evaluations, and calls on the Government to consider whether randomised policy trials may be implemented to evaluate future Government policies.

No government has been more committed to evidence driven policy than ours. In areas from water reform to climate change, foreign aid to schools reform, and activity based health funding to fiscal stimulus, Labor has drawn on the best knowledge of experts in the field. What drives us on this side of the House is not a love of particular programs but a hope that our time in public life will help leave Australia more prosperous and more tolerant, with a cleaner environment and jobs for the future.

To achieve these goals we need to keep finding better ways to evaluate our policies. As a former economics professor, I can assure the House this is particularly hard in the case of social policies. Unlike scientific experiments, evaluations of social policies are particularly tricky. We do not always get the right answer from simple before-and-after evaluations, nor from comparisons of those who opted in with those who opted out. A great advantage of randomised trials is that participants are allocated to the treatment or control groups by the toss of a coin. The beauty of randomisation is with a sufficiently large sample the two groups are very likely to be identical, both on observable characteristics and unobservable characteristics. The only difference between the treatment group and the control group is the intervention itself, so if we observe statistically significant differences between the two groups we can be sure that they are due to the treatment and not some confounding factor.

In Australia our farmers have used randomised evaluations for over a century and our medical researchers have used randomised evaluations for over half a century, yet social policy randomised evaluations are much rarer. One exception is the New South Wales Drug Court trial conducted from 1999 to 2000. Offenders were referred to the Drug Court by local or district courts, underwent a detoxification program and were then dealt with by the Drug Court instead of a traditional judicial process. At the time it was established the number of places in detoxification was limited, so participants in the evaluation were randomly assigned either to the treatment group or the control group. They were then matched to court records in order to compare re-offending rates over the next year or more. The evaluation found that the Drug Court was effective in reducing the rate of recidivism and that while it was more expensive than the traditional judicial process it more than paid for itself.

In the case of the Drug Court, many of us probably had an expectation that the policy would reduce crime, but high-quality evaluations do not always produce the expected result. Staying for a minute with criminal justice interventions, take the example of Scared Straight, a program in which delinquent youth visit jails to be taught by prison staff and prisoners about life behind bars. The idea of the program, originally inspired from a 1978 Academy Award winning documentary of the same name, is to use exposure to prisons to frighten young people away from a life of crime. In the 1980s and 1990s several US states adopted Scared Straight programs. Low-quality evaluations of Scared Straight, which simply compared participants from a non-random control group, had concluded in the past that such programs worked, reducing crime by up to 50 per cent. Yet after a while some US states began carrying out a rigorous, randomised evaluations of Scared Straight. The startling finding was that Scared Straight actually increased crime, perhaps because youths discovered jail was not as bad as they thought. It was not until policymakers moved from second-rate evidence to first-rate evidence that they learned the program was harming the very people it was intended to help.

Being surprised by policy findings is perfectly healthy. Indeed, we should be deeply suspicious of anyone who claims that they know what works based only on theory or small-scale observation. As economist John Maynard Keynes once put it when asked why he had changed his position on monetary policy during the Great Depression: ‘When the facts change, I change my mind. What do you do, sir?’

One common argument made against randomised trials is that they are unethical. Critics ask: when you have a program that you think is effective how can you toss a coin to decide who receives it? The simplest answer to this is that the reason we are doing the trial is precisely that we do not know whether the program works. The great benefit of a randomised trial is that it gives us solid evidence on effectiveness and allows us to shift resources from less effective to more effective social programs. We should not lightly dismiss ethical concerns about randomised trials but they are often over played.

Medical researchers, having now used randomised trials for several decades longer than social scientists, have grown relatively comfortable with the ethics of randomised trials. Certain medical protocols could be adapted in social policy—such as the principle that a trial should be stopped early if there is clear evidence of harm, or the common practice of testing new treatments against the best available alternative.

One example, again from New South Wales, helps to illustrate this. Since 2005 an NRMA CareFlight team led by Alan Garner has been running the head injury retrieval trial—HIRT—which aims to answer two important questions: are victims of serious head injuries more likely to recover if we can get a trauma physician onto the scene instead of a paramedic; and can society justify the extra expense of sending out a physician or would the money be better spent in other parts of the health system? To answer these questions Garner’s team is running a randomised trial. In effect, when a Sydney 000 operator receives a report of a serious head injury a coin is tossed—heads you get an ambulance and a paramedic; tails you get a helicopter and a trauma physician. Once 500 head-injury patients have gone through the study the experiment will cease and the results will be analysed.

When writing a newspaper article about the trial I spoke with Alan Garner who told me that although he had spent a decade working on it even he does not know what to expect from the results. In a phone conversation, he told me:

We think this will work but so far we only have data from cohort studies.

Indeed he even said:

Like any medical intervention, there is even a possibility that sending a doctor will make things worse. I don’t think that’s the case, but—

until HIRT ends—

I don’t have good evidence either way.

What is striking about Garner is his willingness to run a rigorous randomised trial and listen to the evidence. Underlying HIRT is a passionate desire to help head injury patients, a firm commitment to the data and a modesty about the extent of our current knowledge. High-quality evaluations help drive out dogma As US judge Learned Hand famously said:

The spirit of liberty is the spirit which is not too sure that it is right.

Naturally, randomised trials have their limitations. Not all questions are amenable to randomisation. Like the kinds of pilot programs that we run all the time, randomised trials do not necessarily tell us how the program will work when it is scaled up, and they are not very good at measuring spill-over and displacement effects. Because of these limitations, it is unlikely that we would ever want 100 per cent of government evaluations to be randomised trials. Most likely, the marginal benefit of each new randomised trial is a little lower than that of the previous one. At some point, it is indeed theoretically possible that we could end up doing more randomised trials than is socially optimal.

However, this is unlikely to ever occur, at least in my lifetime. My best estimate is that less than one per cent of all government evaluations are randomised trials and that, excluding health and traffic evaluations, the proportion is probably less than 0.1 per cent. Another way to put this is that, to a first approximation, Australia currently does no randomised policy trials. Governments throughout Australia could safely embark on a massive expansion of randomised policy trials in Australia before we come close to the point where the costs exceed the benefits.

Finally, one way that we might expand randomised policy trials is to learn from the US, where federal legislation sometimes sets aside funding for states to conduct randomised evaluations. The Second Chance Act for rehabilitating prisoners, the No Child Left Behind school reform law and legislation to improve child development via home visits are just some of the US laws in which the federal government explicitly puts aside a portion of program funds for states to run random assignment evaluations.

What we need in Australian policy today is not more ideologues, convinced that their prescriptions are the answer, but modest reformers willing to try new solutions and discover whether they actually deliver results.

See this speech in context