Dynamic Evaluation

Is evaluation research or not? This is a complex question, the answer to which is that it can be but need not be. Let us inquire into what might turn an evaluation into research.

The simplest form of evaluation is the feedback questionnaire. They are given out at the end of staff-development courses, as a client satisfaction form. They might give an indication of quality, and ideas for future improvement, if they are seriously filled in. In reality some are and some aren't, some give comment and others ticks in multi-choice boxes. If a quality session is recognised as such by one perceptive person, whilst ten press-ganged attendees offer low scores, the result will be too low a score. Thus it is fundamentally not reliable. These questionnaires tell us more about the attendee than about the session. To be sure of reasonable feedback, a session needs to ensure that people are not taken out of their comfort zone; this also means that they will not move on. I remember race awareness training in the 1980s. This took people so far out of their comfort zones that anger inhibited rational learning. A session needs to push people out of the comfort zone but not too far - staged knowledge construction rather than revolution.

An evaluation might try to find out whether pupils or students have progressed during an intervention. Thus a pre- and post- questionnaire on the topic under consideration, establishing baseline and summative data, can chart progress. Knowledge and skills are simpler to handle than softer aspects such as emotions, behaviour, attitudes and values. Broadening the questioning to take in different stakeholder perspectives (e.g. pupils, teachers, parents, therapists) produces more broadly based feedback data, and spreading that data over time strengths it further. Studies like this are routinely published as research. However, their ability to generalise may vary, since there are many factors which affect the success or failure of a programme. Which processes are effective and which not may be very hard to discover.

A research might look elsewhere for ideas about such processes and seek confirmation in the data. These ideas are contained within the notion of 'theory'. For example the processes of learning draw on the work of Lev Vygotski, on how adult teaching and mentoring help to boost and structure knowledge. The word 'scaffolding' is now commonly used. Social process might draw on the work of Victor Turner, on how social performance or ritual smooths the way for social progress. The processes within individual well-being might use Abraham Maslow's work. New lines of theorising rarely come from a single piece of work but are associated with the life-work of particular individuals. Typically, a piece of research will gather its unique data, and theorise about it using appropriate models from others.

Dynamic evaluation takes us one stage further. Often evaluation simply records what has happened and seeks to grade its effectiveness, using instruments which range from crude to over complex. A crude feedback questionnaire may get many returns but contain no useful information. A 20 page questionnaire may get very useful information but have few returns.
I call this a static evaluation. After the event there are recommendations on how to do it better next time. Over the time of a long project there might be annual recommendations to allow for some formative development, if the project structure allows it. The relationship between the project and the evaluation is mechanistic - recommendations are ticked off during the following year, when the next phase of the evaluation makes further recommendations.

Dynamic evaluation seeks to improve and develop processes day by day. The project needs to allow for growth in the flexibility of its targets. The growth is produced not by the recommendations of the evaluator but by the quality discussions that the evaluator facilitates. The project team, by responding to these, find more effective ways of achieving their goals. The evaluation instrument is likely to be the policy discussion group; if questionnaires are used it will be to feed into this discussion. The facilitator will seek to open up perspectives in the group, and look for inhibitions and negative practices to counter. The group will be encouraged to focus on the real objective of the programme (e.g to help pupils progress in specified ways) rather than on numbers and mechanistic box ticking. The evaluation should therefore get to the heart of the issue and seek ways in which the team from an early stage establish positive strategies to meet their real objectives.

Because of the nature of the relationship, the evaluator is intimately involved with all aspects of the work in hand, attending and actively participating in every significant meeting. This therefore cannot be done by skimming over the surface of the project. There is a value for the project in the quality of internal debate and strategy setting; there is a benefit for knowledge generally (that is by turning this relationship into research) if the change processes are made explicit and investigated more theoretically.

Interpreting Myth

This post begins with Sheila Moon's A Magic Dwells: A poetic and psychological study of the Navaho emergence myth (1970). My purpose is to explore how myths can be approached, and to transfer that is appropriate to other kinds of writings.

The myth cycle describes the emergence of living things and eventually humans. There is no creator God, just the concept of emergence. The interplay between good and evil is central, with the first man being evil and powerful; to be balanced and replaced eventually by Changing Woman with benificent creative power. On causation the myth declares it was 'just because' - no apparent cause, just "the symbol for the ancient curtain of mystery hanging between us and all that we do not yet comprehend" (p.20). Choices are made, without clear reasons. A creator-god provides a feeling of rational choice. In the psyche, the creator-god is an explanation and celebration of life itself. "For the creator-god also is at bottom a symbol of man's will to live" (p.21). It is a mystery, but a mystery under control, deliberate, and beneficial.

Life of course is none of these things. Life can be snuffed out in seconds by disease, accident or disaster. No explanation, no apology. Life is generated by blind impulse to mate, but that is no guarantee of fertility. And the world into which mankind or beast are born is hostile, so we all struggle to survive viruses and predators. That is a hard place psychologically, especially for humans with long memories and deep anxieties.

Things and people, the Navaho observed, developed from somewhere and something. On that mysterious route, choices were made that made humans what we are and not snakes. The process of development balanced the power (authority, dictatorship) of men against the creativity of women. That which is beneficial comes from that creativity. Men have the choice to follow instinct to power or soften it with his creative feminine side (Jung's anima).

In 1989 I wrote a chapter on religious signs and symbols in Creating the Old Testament (ed. S. Bigger). This explored what signs and symbols in artefacts and language might mean psychologically, that we should not be taking them literally. Psychologically, God the father-figure provides a sense of direction and protection - balanced in religions such as Hinduism with parallel ideas of God the mother. Bible faiths were in denial of the feminine, which slipped back into national psyche in a host of embarrassing ways, such as through Astarte and her fertility cult. Also concepts such as faith are rooted in our good faith, our personal authenticity. Our belief in afterline is an affirmation that life has been worth living. Of course, such assumptions are not real, and are a bar to insight. We will only understand life here and now if we cease to view it as a punishment for the past or a preparation for bliss. It is here and now. Our quality of living it now is the whole point.

We express our ideas about the quality of our existence, our hopes, aspirations and anxieties differently, although some of the old myths still have power. Star Trek drew part of its charm from the myth that there will still be a world centuries to come, that it will be united, and that we will still be exploring the galaxy. The American mission of policing the world is extended absolutely everywhere. Where we came from is also included, with planets being modified for life, and superbeings seeding dna into a myriad of cosmic soups in beginning universes. This is deeply settling and comforting.

Meeting deep human need (psychology) is balanced by expressing profound insight (poetry) exploring the cases of 'just because', the mysteries, the wonders. We can therefore use these twin lasers to help us both understand literature and ourselves.

Shirin Ebadi

Shirin Ebadi is an Iranian human rights lawyer fighting for liberty and social justice, especially for the marginalised. She was Nobel Peace Prize winner for her work in 2003. Here is the BBC estimate of her work. She was the first female judge in Iran, and teaches law in the University of Tehran. Her offices from which she complained for the marginal was closed down by the authorities and staff arrested.

Much of her work has been on women's liberty and rights. Since the Iranian revolution, freedoms for women have been considerably eroded.

Her recent work has been to support the Iranian Bahai community. Bahais have been persecuted in Iran ever fince their founding in the middle of the 19th century. Since the Iranian revolution, Bahais have been executed for teaching children, and for being Bahais (regarded by some Muslims as a heresy of Islam, but believing themselves to be a separate world religion). Family wealth and property has been confiscated, and children have been denied an education and entry to university. In January, arrests of six Bahais hit the news, including a member of her own staff. More details linked here. On Channel 4 News today, she showed desecrated Bahai gravestones and expressed great concern for both the treatment of the Bahai minority, and civil rights generally.

The Bahai Faith emerged as a peaceful religion in Iran from the 1850s. Their first teacher, called The Bab, was executed, their first 'Manifestation of God', Bahaullah, was imprisoned until his death in 1892. Both left a considerable body of teaching that encouraged world development to an ideal in which men and women are considered equal in status, that all races and nations should be deemed equal, that extortion and impoverishing must be stopped, that the world should develop a democratic politics in which everyone are valued. They teach that all religions should be valued as expressions of God's revelation. The encourage the empowerment of both families and communities. What is there in this philosophy that must be silenced?

Power in this world is held by vested interests who demand obedience and deny freedom of thought, religion and action. Those in power have mechanisms, such as police and army, to assert their dominence. Individuals therefore have to decide whether to allow this to happen uncontested, or be counted as ethical opponents, and thereby as workers towards peace, equity, respect and human well-being. The opposite is to be supporters of aggression, injustice, disrespect and human degradation. Whatever our beliefs, we have to stand up and be counted as ethical workers towards global and local peace.


See further: http://en.wikipedia.org/wiki/Shirin_Ebadi.

Paulo Freire

Paulo Freire was born in Brazil and worked as teacher in the shanty-town slums. He criticised the tradition approach of the teacher giving knowledge to passive students, comparing it to putting money in the bank - the students accumulating given knowledge. His approach required dialogue, discussion and debate, with students being active in their learning. He affirmed that such an education process should be empowering for the learner, helping them raise their consciousness about the world and about political issues. Education should, he affirmed, help people see their world differently. He was firmly opposed to workbooks, for example for adult literacy, which dominated most of his work. People learn best if the knowledge and skills they are building up are useful to their lives, even life changing. The power between teacher and learner is narrowed, removing if possible the status gulf between the two so that the learning experience is two-way, built on relationship and joint effort towards a single goal. This dynamic approach to teaching and learning he described as praxis.

More detail can be found in the on-line Encyclopedia of Informal Education .

There are lessons for teaching today.
First that teaching means establishing a positive dialogue with learners which is rooted in a good relationship.
Second that there is more to teaching than administering workbooks or worksheets. It is about creating an interest in the topic which is shown to be relevant to life.
Thirdly, that knowledge is not given and received; rather understanding is constructed together.
Fourthly, education should change the way pupils and learners think about their lives, making them more aware, more ethically concerned, and politicised (i.e. inspired to change their world).

The effectiveness of provision in schools, colleges and universities could profitably be measured against these criteria. Where structures make such dynamic learning impossible,, they should be dismantled.

A Good Childhood: Searching for Values in a Competitive Age

The report, A Good Childhood: Searching for Values in a Competitive Age for the Children's Society by Richard Layard has just been published. Details are on The Children's Society website

Recommendations include:

Teachers should:

Help children to develop happy, likable social personalities.

Base discipline on mutual respect.

Eliminate physical and psychological violence from school.

Make Personal, Social and Health Education statutory.

Present sex and relationships education not as biology but part of social and emotional learning.

New tests on emotional and behavioural well being should be carefully piloted.

There are various recommendations for parents, governments, advertises, media and others but these are summarised by the final recommendation:

All Society should:

• Take a more positive attitude to children. Welcome them into society and help them.

Recommendations include abandoning SATs in favour of teacher assessment, investing more heavily in child and youth services, and providing youth facilities and centres with educative as well as recreational aims.

The report, now arrived, is a slim Penguin paperback. It is written in simple language, so simple that it seems like obvious common sense and hardly worth writing about. Yet, if it is common sense, why have opposite policies and strategies been in place for 20 years? - a curriculum more enamoured with testing than with enjoyment; a mechanistic demand for literacy and numeracy than with their development within a relevant and appropriate context which excited the children; an ethos of 'managing' behaviour with punishments rather than developing positive attitudes and self-control? The alternative answers are out there, and were in common use before these new 'reforms' in 1988. Further, the critique of this report was being shouted out loud across these 20 desert years - what about relevance such as work on the environment and citizenship? So these were tacked on to the margins, to be done in stolen time. What about enjoyment and achievement as well as good scores? So new guideline were given to require teachers to get children to enjoy the unenjoyable curriculum. This is non-political. The Tories started it, New Labour continued it equally enthusiastically. Someone has to change it. I hope that the Children's Society report gives a nudge in the right direction.

Facts and figures:
Percentage of children 5-16 with
Anxiety disorders 3.3%
Depression 0.9%
Conduct disorders 5.8%
ADHD 1.5%
Autism/ASD 0.9%
Eating disorders 0.3%
Total, any disorder 9.6%

Treatments: family talking therapy is highlighted, but the quality of the interaction is crucial. The best can improve the situation by up to 28%; this goes down as quality of the therapist decreases (based on family ratings). A very poor therapist can make things worse. Pages 121-3.
Costs of social care, remedial help and crime, per child (averages, 1998 prices)

• with conduct disorder £70,000
• with conduct difficulties £24,000
• without disorders or difficulties £7,000 (source: p. 126 and note 260).

Looked after children:
About half of the children held in custody were 'looked after' in care.

Experience is Pedagogical - John Dewey

John Dewey was a psychologist and philosopher when has had over the decades a profound effect on education. e championed informal learning within the formal education system. These points are taken from 'Experience is Pedagogical', first published in 1897. I present them here as not of historical interest only but helping us to reconstruct what education could become. Here, highly simplified:

Article 1, I believe Education is ...
the individual is a social being, society is an organity unity of individuals. Education must begin with individual capacities, interests and habits. The purpose of education is to improve society.

Article 2. What the school is...
a social institution, involved in a social process, a process of living today and not preparation for living in the future. Schools must represent real life, and grow out of family life. It must encourage children to participate in the process. School should be about personal growth. Examinations are only of use if they determine the potential role the child can play in community life.

Article 3. The curriculum. The child's own social experiences need to be the start of the curriculum. Subjects need to begin with ways in which they are experienced in everyday life.

Article 4. Methods. Only activity can deliver the various aspects of education and learning without dullness or sentimental emotionalism. For activities, interests can be observed.

Article 5. The School and Social Progress. Education needs to be a mechanism to achieve social progress. It should merge arts with sciences. The teachers need to recognise the dignity of their calling as helping to form a better society.

It will be interesting to map this against current educational provision.

Sample quotations from Experience & Education (1938), a lecture series which offers a mature synthesis of his ideas on education:

"basing education on personal experience may mean more multiplied and more intimate contacts between the mature and immature than ever existed in the traditional school, and consequently more, rather than less, guidance by others." [p.21]

"How shall the young become acquainted with the past in such a way that the acquaintance is a potent agent in the appreciation of the living present?" [p.23]

"Experience and education cannot be directly equated to each other. For some experiences are mis-educative. Any experience is mis-educative that has the effect of arresting or distorting further experience. An experience may be such as to engender callousness; it may produce a lack of sensitivity and of responsiveness...
It is not enough to insist on the necessity of experience, nor even of activity within experience. Everything depends on the quality of the experience to be had... [It is the task of the educator] to select the kind of present experiences that live fruitfully and creatively in subsequent experiences." [pp.27-8]

"Every experience is a moving force. Its value can be judged only on the ground of what it moves towards and into" [p.38]

"When education is based upon experience and educative experience is seen as a social process, the situation changes radically. The teacher loses the position of external boss or dictator but takes on that of leader of group activities." [p.59].

Round the World In 80 Faiths

At the point of writing, Peter Owen Jones is finishing his BBC series on marginal faith groups. In general I have welcomed his open-mindedness, an Anglican priest thrust into first hand experiences within the anthropology of religion. Above all he has been quick to recognise the imperialist nature of Christianity engaged on world domination, believing itself to be the only true faith. He apologises for this arrogance on several occasions.

Currently he is visiting El Tio, a devil-god deep in South American silver mines, a relic of the indigenous gods submerged by Catholicism. He handles poisonous snakes with American fundamentalist Christian snake-handles, following a quaint verse in Mark 16. He greatly disliked Voodoo blood sacrifice for its cruel waste of life, although most of the sacrificed flesh was apparently eaten (except for the kitten and puppy). He rejoices when a colonised indigenous faith fights back to form a rich syncretism. He tries to show such living rituals as culturally enriching and empowering.

Too many faiths are shown to cover here, so I am content to explore principles. The anthropologist loves diversity and is saddened when the traditional dies out and is replaced by a global impoverished culture, be it in literature, music or religion. Yet not all traditional beliefs have been helpful and can imprison a people in the past. British beliefs in ghosts, black demon dogs, and witches are not things I would wish preserved or restored, however children's novels such as Whitby Witches or The Ragwitch would wish. There is a balance between people valuing the best of the past (but not all of the past) and moving forward. I worry therefore that faiths such as those presented have become a museum of human delusion if taken literally. I understand that cultural identity is important, but underlying many customs are irrational beliefs, in spirits, demons, deities, fate and the afterlife. On the programme as I write, Catholics are blessing automobiles to bring good fortune. In many cases, what we see is the triumph of superstition over education. Religious people have no problem in believing the impossible or unlikely, and there is no sign globally of a rational coming of age. Owen Jones rejoices in the 'museum' but supports the rational. He said, 'No minister can get rid of my demons. I have to sort out my issues for myself'. Maybe the prisoners 're-directed' by Pentecostal needed to be shaken out of their destructive path and given a new direction. The issue is whether that dependent direction is best for them in the long run.

Do people need bizarre myths? Some are ancient and traditional, like the two former cannibal tribes putting aside enmity by exchanging children - fortunately now a symbolic exchange. Some are however very modern and very strange. All have in common the importance of ritual and performance. Ritual gives some social cohesion, the feeling of togetherness. That togetherness has a purpose. The ceremonials of Nuremberg in 1935-6 aimed to bind together the German people behind an all-powerful leader. Owen Jones shows how rituals can become personally intrusive, inviting psychic distress. Others brought him peace, the residence in a smoke-filled tent, and listening to the river with a shaman. Each of these cases showed that these apparently positive practices were condemned as diabolical, and their celebrants persecuted and killed. He ends in Turin with a ground living in religious harmony, drawing equal insight from all faiths - a marginal cult, but a spiritual high-spot.

So what is diabolical? The insistence that only we are right and others must be forced to agree with us? Or a treasuring of the voices of many people who live, believe and act in ways which enhance the well-being of the world as a community. I know my answer.