I publish on 1. Contemporary
Vietnam 2. ‘Method’ and 3. Servicisation (other details at the end of
the Blog). I learnt about the DFAT/Asian Foundation experiences when a friend
contacted me after he saw Blog # 4. So please let me know about other
interesting experiences in this area.
What is new?
Until recently I could not point to an example of a DAC
member bilateral aid agency that disbursed into interventions which formally assumed
that change was unpredictable. Generally, this would be contrary to DAC
doctrine, as my recent book explains (here). However, I have
recently found an example and this blog points to this and tries to explain why
I think it is important, what its significance is and how things are likely
moving forward now.
Type I and II - what does this mean?
My book Reinventing development – the sceptical change agent frames
development work as operating either
in environments suited to assuming we can predict what will happen, or not. Clearly, whether an environment
is deemed predictable or not is a matter of choice (eg How much of the budget
should we spend on research?). I call environments thus deemed predictable as ‘Type
I’ and those deemed unpredictable ‘Type II’. It follows that intervening in
ways that assume ‘Type I’ contexts are ‘Type I’ organisations, and vice versa. Thus,
if you get it wrong and organise as though an environment is predictable, and
it is not, that is a ‘II/I’ situation, and so on. This language has its
problems but is useful.
At present, the DAC/OECD requires
as doctrine that aid paid for by their members assume ‘Type I’ environments and then intervene using suitable ‘Type
I’ organisational forms, called the Log Frame, Theories of Change etc. That is,
they assert that we live and work in ‘I/I’ ways. But as there is lots of
evidence that there are many unpredictable environments, we also live with
‘II/I’. This can be tricky.
Clearly, it makes no sense to assume that all change is
predictable. If it makes sense to believe on the contrary that change - in a given
environment - is ‘Type II’, but DAC doctrine requires ‘Type I’ organisation,
there is a problem. Conversely, I think that many aid professionals have good
grounds for believing, based often on trial and error, that some environments
are ‘Type I’, and if they were forced to organise as though they were not (Type
II) this would lead ‘I/II’ and so tensions.
I think of ‘war of attrition’ tactics used to push through,
for example, female participation in primary schooling, where I think many
would reasonably believe that a couple of decades pushing the Ministry etc
would lead to measurably better results, though analysis of just how and why
would likely lead to argument amongst evaluators. Experience shows often that
it is reasonable to assume predictive power even if there is no good theory
that explains it (see Chapter 5 of my book).
Making a distinction between ‘Type I’ and ‘Type II’ in this
way helps think through various tensions.
As a country expert (perhaps the most cited writer on
contemporary Vietnam) it is clear to me that country expertise usually has less
prestige than those of generalists (eg experts in general on topics such as ‘rural
development institutions’) amongst aid donors. However, it is also clear that
the positions taken by generalists are fragile in straightforward debate, and
so country expertise is often managed by ignoring it. Catherine Earl has an
edited collection coming out next year that looks at aspects of this for
Vietnam (Mythbusting Vietnam). I gave a seminar
on aspects of this at the University of Auckland a few days ago (PowerPoints
available on request).[1] My paper
in Earl’s book shows scholars as senior as Dani Rodrik and Jeffrey Sachs making
clear and embarrassing mistakes in articles about Vietnam published in good
journals and which any country expert can spot.
In both cases they are trying
to generalise.
This suggests that such generalisations are often very
fragile. This suggests in turn that part of what aid workers experience is
indeed a rather extreme instability in what ‘Type I’ knowledge that they are
meant to believe in: individual donors’ views change over time, views about
what causes what vary between donors, and views are fragile when confronting
country expertise. This suggests again that a better control over belief is to
be welcomed, and what my book calls the ‘right to scepticism’ – the right to
assert that a context is ‘Type II’ – is itself valuable.
The Reinventing development book tries to think-through why
it is the case that the DAC position exists, and how it works, and why formally
asserting a context is unpredictable and organising accordingly (‘II/II’) is so
hard. When I started writing the book I could not find any such examples, but I
have found one – DFAT money, spent by The Asia Foundation. I thank Foundation
people for talking to me about their experiences (see here
for a clear exposition).
What are the problems formalisation of Type II (ie a ‘II/II’) must solve?
What is the problem?
It is not entirely clear to practitioners what problems
formally working in II/II ways must solve. This is mainly I think due to lack
of formal experience. That is why DFAT/The Asia Foundation work is so valuable.
Watching practitioners and those they report to work this
out is fascinating. We can learn something from other situations (see Reinventing development), and these
often report that working this out requires very big shifts in cultures of
power, responsibility and hierarchy. This does not yet seem to be happening,
which helps explain why pressures to redefine ‘II/II’ as some form of ‘I/I’ can
be strong. For example, discussing evaluations with a bilateral official
recently, it was clear that evaluations from a ‘II/II’ perspective, that did
not assert ‘what had happened’ but reported different accounts from
stakeholders), could see senior levels require a ‘synthesis’ that came down to
‘yes, but state which account is correct’ – thus asserting that the context was
Type I not Type II.
‘Narrative’ evaluations seem rather straightforward
If a context is deemed unpredictable then so-called
‘narrative’ evaluations make sense in that they report different accounts and
different valuations, without asserting that, for example, one Theory of Change
is correct. These have been around for a while but contradict DAC doctrine –
see this JICA-funded exercise (Beyond Logframe; Using Systems Concepts in Evaluation).
See also my own evaluation of the Sida Chia Se program in Vietnam much-quoted here.
Canny aid officials have I think long liked narrative evaluations that report
various stakeholder accounts of what they think happened, and what they thought
of it. These in effect, if people think like that, report multiple Theories of
Change and multiple valuations of subsequent situations. The authors can then
say what they think, but the official is aware of the views of relevant actors
and is therefore (of the reports are reliable) protected against unpleasant
surprises. There is of course evidence that repeated evaluations of
interventions that assume ‘I’I’ do not usually agree with each other (for
example – here).
This suggests that learning from evaluations in Type II situations is not about
learning how to create a better more predictively powerful Theory of Change.
But very little thinking about how to decide whether a context is deemed to be ‘Type I’ or ‘Type II’
The guts of all this, I think, is that thinking about Type I
vs Type II, both in terms of context and organisation, comes down to securing
and managing far better control over belief. After all, a Theory of Change may
be based on a view that a context is predictable 60% of the time, or not,
depending on what people think is important – such as, it is to do with views
of the costs of being wrong, and the costs of some learning that would hopefully
get the 60% up to 80%. It is quite reasonable to argue that, depending on such
considerations, one group decides that it thinks a context is Type I, and
another that it is Type II. Thinking like this is itself I think empowering.
All realities are complex in some sense: whether a complex or a simplifying
model is used depends on many factors, not least the costs of developing and
using the model.
What is striking is that there is so far extremely little
discussion about this issue. Stating as the DAC does that contexts must be
defined as Type I is often attacked on the grounds that ‘development is
unpredictable’ (The Asia Foundation’s position – see here).
I think that aid professionals often have very good grounds in their
accumulated trial and error experiences (and the need to focus on areas where
they believe they can work reliably in ‘I/I’ ways) for thinking that it makes
sense to assume predictability. And why not?
But the dam has broken
The Asia Foundation, paid for by DFAT, has experience in
‘II/II’ that so far as I know is unique. A key document is here.
So far as I know, this project itself was managed outside the LFA, using
Australian tax-payers’ money and so audited. It is therefore a ‘II/II’, an
example of formal organisation that assumes unpredictability. They want to
‘win’:
… efforts to
operationalize a ‘thinking and working politically’ approach have prompted a
broader reconsideration of development practice because the structures and
requirements of standard
development projects do not facilitate innovative, politically-informed ways of
working. While there are multiple articulations of what a more effective
approach to development assistance might look like, one core principle they generally share is
the need for greater flexibility. This emphasis on flexibility stems from an
understanding that development is a complex, unpredictable, and dynamic process
that depends on the changing interests, incentives, ideas, and relationships
among multiple groups and individuals. In this context, an iterative approach
that closely links learning and action is most likely to bring about effective
solutions
and
bringing about
change requires navigating an unpredictable and complex landscape of interests,
[2, stress added]
The position here
is that change is always unpredictable.
Therefore, at any point in time an asserted belief in what is happening, what
interests are, etc, is asserted to be provisional, changeable and so flexible.
This gives more control over beliefs – what is said to be the change pattern is
deemed unstable, changeable, ‘true-for-somebody but not necessarily everybody’
and so on.
This is reflected
in changes in learning processes, which shift training and reflection down
towards activities (and so towards local expertise and away from generalists):
ST {strategy testing – AF} is a time and labor intensive
process for program staff. Unlike some of the more conventional monitoring
techniques, ST cannot be delegated to monitoring and evaluation (M&E)
officers. It requires program teams to dedicate significant blocks of time away
from their day-to-day work and make a serious commitment to critical
reflection, discussion, and documentation. [15]
This
means that INGOs and others can now argue that if DFAT can disburse to The Asia
Foundation on the assumption of ‘Type II’ contexts, then, if they can make the
argument that their context is important and ‘Type II’, they should be allowed
to proceed accordingly.
Are they clearly solved, and if so how?
Thus, I think that the process of learning how formally to
organise in ‘II/II’ ways has now started, and people like me can ask people
like The Asia Foundation, and the DFAT officials who managed them, how they did
it. Since this means (as the JICA evaluation did – here)
that DAC members in effect walk around DAC doctrine, formal experience is now accumulating. For me the key issue is
deeming a context unpredictable – ‘Type II’.
Evaluations of such activities will then assume that one is managing
beliefs and reporting them in relevant ways; this is not complex. Officials
sign-off on projects based upon their acceptance of narrative evaluations.
Reflective learning associated with evaluations seem, in
‘II/II’, to move towards the local and practitioners, devaluing generalists’
knowledge. As in other contexts, practitioners train practitioners more than
bringing in outside experts.
What seems to be lacking, I think in part as experienced aid
workers’ beliefs based upon trial and error are often under-valued, is a
thinking-through of the issues involved in deeming a context to be Type I or
Type II. I think that the argument that ‘we only have $10 mn so why spend lots
on research? Let us assume Type II and then organise as though we are simply
believing this or that now, others disagree but we can move forward’, makes
very interesting sense. Similar arguments suggest avoiding complex models and
explanations that are costly to produce and, more importantly, to understand
and work with. For me, much comes down to the value of asserting a ‘right to
scepticism’ in using beliefs and getting them under control.
As a result, in part as so far there is only a limited
understanding of the major changes in culture and the meaning of leadership
implied by ‘II/II’, the natural tendency is to ‘revert to ‘I/I’ – as was put to
me senior officials would say ‘Yes you have a narrative evaluation but which
account is true? Who is right?’. And that wants to go to a ‘I/I’, but actually
pushes far too readily to a ‘II/I’, where everybody knows (but cannot formally
say so) that the asserted predictive knowledge is very fragile, varies over
time and between agencies, avoids country expertise as it is too destructive,
and often ends up, as it devalues or ignores those who disagree with the
central position that is ‘deemed true’, ignoring voices that may just hit the
donor hard through a complaints procedure with local NGOs getting into
bed with
INGOs who brief a visiting TV crew, or simply the Ambassador asking why when the
projects were deemed successful nobody seems very happy, least of all the
security organisations.
The point is not be right, but to be effective.
Comments, extra information, very welcome.
Prof. Adam Fforde
Victoria
Institute of Strategic Economic Studies, Victoria University www.vu.edu.au/vises (Professorial Fellow, part-time)
Member, Editorial Boards: Journal of Current Southeast Asian Affairs and
Canadian Journal of Development Studies
Consultancy Services (ABN 27412011042)
Website: www.vietnampolitics.net
BOOK JUST OUT - Reinventing development - the sceptical
change agent (Palgrave March 2017) http://www.springer.com/us/book/97833195022670
For details- http://vietnampolitics.net/?page_id=121
And for how to get a review copy -
https://www.springer.com/gp/reviewers/online-review-copies---all-in-a-nutshell/4798
[1]
“‘All good theologians end up as heretics’ – country expertise vs development
experts – Vietnam and Adam Fforde as a case study” - Joint Seminar: Asian
Studies and Development Studies, University of Auckland, August 4th2017.
