information products research distributors copyright bibliography links orders training contact

 

JCRL'S RESEARCH PROGRAMME

MEASUREMENT-DEVELOPMENT PROGRAMME

Raven's Progressive Matrices and Vocabulary tests were based on observations and theories which Spearman formulated at the turn of the century.

Spearman himself noted two crucial defects in these theories.

First, his "general factor" had emerged from studies of the intercorrelations between tests of "educational" abilities language, science, mathematics, etc. Yet, as John Raven has shown in more detail in his book The Tragic Illusion: Educational Testing, these tests themselves lack construct and predictive validity.

Second, the way in which psychologists were trying to tackle the problem of describing and summarising individual differences was basically off-beam. Thus he wrote: "Every normal man, woman, and child is a genius at something É the problem is to identify at what." The absurdity of trying to summarise and report individual differences in terms of two, five, or sixteen variables can be easily exposed by asking "Where would chemists have got to if they had sought to describe all the variation between chemical substances in terms of 2, 5, or 16 variables?" Clearly, if we are to improve our ability to describe individual differences and the emergent properties of groups it will be necessary to develop the equivalent of atomic theory in chemistry or the Darwinian evolutionary map of species in biology.

Another important topic has to do with the conceptualisation of "the environment" and the way it interacts with personal characteristics to determine both individual characteristics and behaviour. Just imagine trying to advance understanding in biology thinking in terms of variables between animals interacting in a linear fashion with variables in the environment. Just imagine trying to advance ecology without thinking in terms of multiple interactions and feedback loops.

John Raven Senior and Junior both devoted much of their lives to work on this collection of topics.

In addition to a range of books, this work has resulted in a range of preliminary assessment tools based on an alternative way of thinking about individual differences and behaviour.

Unfortunately, these tools cannot be immediately applied within our current workplaces, educational systems, and public management systems because the operation of these systems is determined, not by personal developmental needs or by societal need, but by a range of latent, rarely discussed, and hard to influence sociological forces. Nevertheless, if these hidden sociological forces are to be influenced, it will be necessary to utilise the very psychological tools that have been mentioned.

Despite the absence of a research tradition to which this work can be linked, and despite the difficulty of applying it in current organisational context, J.C.Raven Ltd. would be extremely interested to hear from researchers tentatively interested in working in this area.

top

MEASUREMENT - DEVELOPMENT PROGRAMME

JCRL'S RESEARCH PROGRAMME

JCRL's measurement-development activities are grounded in a 40-year programme of research which shows that:

    • our education system,
    • our organisations,
    • our public management system, and
    • our economic system

are not working well, they are, collectively, heading our species towards extinction.

New societal learning and management arrangements are urgently required.

Many surprising findings have emerged.

The most important is that finding a way forward is quintessentially dependent on psychologists contributing to the evolution of:

    • new societal learning and management arrangements;
    • new concepts for thinking about the nature, development, assessment, release, and deployment of human resource, and
    • new tools for assessing these things.

To appreciate the developments that are required, it is necessary to understand the deep-seated nature of the failures in each of the areas mentioned.

Their causes and their cures are far removed from the symptoms.

To understand the developments that are required it is necessary also to understand the new theoretical (psychometric, organisational) frameworks that have emerged from the work.

Please click a link below:

The Failures of the Educational System

Organisational Failure

The Failures of our Societal Learning and Management Arrangements

The Theoretical Bases for the Work

The Conceptualisation and Assessment of Competence

The Conceptualisation and Measurement of Developmental Environments

top

The Failures of the Educational System

The failures of the educational system arise neither, as is commonly supposed, from (1) an inability to convey requisite information from teacher to student, nor (2) from a failure to develop appropriate skills among pupils or students.

Its failures lie in:

1. Its inability to help pupils to identify, develop, and gain recognition for, their idiosyncratic talents. These talents include such things as the ability to put people at ease, the ability to entertain others, the ability to get people to work together, and the ability to generate social chaos...ie the 1001 things one sees people doing if one looks around.

2. Its neglect of opportunities to enable people to learn to do things which will be important in their later lives: to lead, to invent, to influence decisions, to put people at ease, to negotiate, to influence social and political systems...

The reasons for these failures are multiple and deep seated.

But first among the obvious reasons, there is little understanding of the nature of these qualities or how they are to be nurtured or assessed.

As a result, there are:

(i) no concepts or tools to help teachers or lecturers to think about student's motives and how to harness them in such a way as to create, for each pupil or student, an individualised developmental programme which would lead them to practice, and thereby develop, high-level competencies.

(ii) no tools to help those concerned to focus on what they need to do to create more development environments, take stock of what is going on, and identify the steps needed to improve those environments.

However, more basic reasons include the fact that, for the effective education to be widely available, it will be necessary to:

a) create a wide variety of distinctly different types of educational or programme;

b) document the personal and social, short and long term, consequences of each option; and

c) feed this information to the public so that they can make informed choices between them.

Not only are there no concepts and tools to assess the quality of the differential educational processes which are required or to document the outcomes in a comprehensive way, the very idea of generating a choice between well-documented options conflicts with (i) the belief that public provision should be equal and (ii) beliefs about the roles to be performed by public servants.

Furthermore, information on the options available and their consequences needs to flow outwards from public servants to the public and not upward in a bureaucratic hierarchy to elected representatives to take decisions binding on all. New beliefs about public decision taking (democracy) and government are therefore entailed.

How is the necessary ferment of experimentation, monitoring, and learning to be created? What are the institutional arrangements, job descriptions, and staff appraisal systems that are required?

Our research shows that:

1. the requisite institutional arrangements are non-hierarchical;

2. performance appraisal must focus, not on the correctness of decisions, but on the adoption of procedures which have been shown to lead to innovation and learning;

3. the supervisory arrangements to be employed to ensure that public servants (a) initiate the collection of a wide range of information, (b) sift it for god ideas, and (c) act on it in an innovative way in the long term public interest hinge on exposing the behaviour of public servants (including teachers) to the public gaze via professionally developed performance appraisal systems.

To move forward it will be necessary to develop tools which will make it possible to:


focus attention on the aspects of organisational climates which promote innovation, take stock of the current situation, and see what to do next.
recognise public servant's high-level contributions to the process, and
focus attention on key features of the new arrangements - the forms of democracy - required for public surveillance of a public-service-based public management process.

top

Organisational Failure

While the greatest organisational failures may lie in public management, Kanter's work shows that very many organisations fail to set aside time for the vital, non-hierarchical, "parallel organisations activity" which is required for innovation, and Hogan's work shows that some 50% of American managers drive their organisations into the grounds in their quest for personal advancement.

"Human Resource Management" is largely restricted to personnel selection despite the following:

1. The job descriptions used to guide selection are typically grossly deficient in that they ignore the kinds of activity which, as the work of Kanter and others has shown, are the most important from the point of view of organisational survival;

2. People typically move on to different kinds of job shortly after having been hired.

3. One of the most important abilities distinguishing more from less effective managers is their ability to build up their own understanding of, and subsequently intervene in, economic, political and social systems outside their organisations to benefit their organisations.

4. Another is the tendency to think about place, and develop the idiosyncratic talents of subordinates. This activity extends to redeploying part of the time of personnel hired for one purpose in the other activities involved in "parallel organisation activity" in such a way as to utilise all of everyone's talents for at least some of the time.

The main reason for the neglect of these activities are:

1) there are no concepts to help think about the talents required to understand, intervene in, and monitor the effectiveness of interventions in, hidden societal systems processes and political and economic systems:

2) there are no concepts and tools to think about and assess high-level talents like the ability to notice problems, make them explicit, harness the activities of others to the task of doing something about them, conceptualising the activity, generating publicity for it, and so on. As a result, it is very difficult to recognise such latent talents and take steps to ensure that they will be nurtured and deployed

3) there are few concepts and tools to deploy to design individualised developmental programmes - i.e. to (i) surface latent motives and incipient talents of individuals (ii) think about the nature of, help plan, and monitor the effectiveness of, the individualised developmental programmes which are required to nurture those talents. (Such programmes involve placement with others who share the individual's motives so that, by working with someone who shares his or her enthusiasms, the subordinates will learn how to do such things as take initiative and develop more effective strategies for selecting and achieving goals. They also include opportunities to work at tasks which are inherently engaging and thus offer their own reward for determination and persistence.)

4) there are no staff appraisal procedures which enable one to find out about what people are really doing (they are not usually doing what other people think they are doing, still less what others think they have hired them to do) and give credit for such contributions to organisational effectiveness as an inventing way of intervening in political systems, monitoring results and changing behaviour accordingly.

To overcome these problems there is a need for tools to:

assist in guidance, placement, and development relating to high-level competencies - i.e. tools to help identify the incipient motives and talents of individuals and monitor their development. Also to focus attention on the key features to be included in IDPs;

draw attention to the key features to be provided in developmental environments, assess the effectiveness of attempts to move forward, and identify what needs to be done next.

draw attention to features of the arrangements required for innovation and learning, assess the quality of what is currently being provided, and identify what to do next.

give people credit for exercising high level competencies - i.e. for making contributions which usually elude observation and fail to get recorded.

top

The Failures of Our Societal Learning and Management

Following our realisation that behaviour - and therefore competence - is primarily determined by the way people think about society and their place in it, we moved on to study the effectiveness of the arrangements which are currently in place for societal learning and management.

Two of the most striking findings were:

1. That the spending of about 75% of GDP is, in some sense, under government control: we live in managed, not market economies.

2. That our current societal learning and management arrangements not only do not work very well they are actively heading us towards the extermination of the species - indeed the destruction of the planet - at an ever increasing rate. There are numerous indices of these exponential changes and their currently inevitable consequences, but most striking is that it would require 5 back-up planets engaged in nothing but agriculture to enable everyone in the world to live as we do in the West: yet billions in China and Asia are hell bent on doing so. Hardin's "tragedy of the commons" is endemic and pervasive.

The Society we need if the species and the planet are to survive must be as different from ours as industrial society was from agricultural society.

Yet no-one knows, or can know, what such a society would look like.

Pervasive innovation in every nook and cranny of society is required.

Most important is study of, and invention of ways of intervening in, the hidden sociological systems processes which deflect even well-intentioned public reforms (as in education) from their goals and lead us all to do things which we know to be wrong.

What then are the public management arrangements required to move forward?

These have been spelt out in The New Wealth of Nations: A New Enquiry into the Nature and Origins of the Wealth of Nations and the Societal Learning Arrangements required for a Sustainable Society.

Chief among them are:

New institutional arrangements for public management: new forms of bureaucracy and democracy. Central here are:

Recognition of the crucial role of the public servant

The main task of the public servant is: (i) to create a ferment of innovation, to arrange for widespread experimentation, to generate variety and choice, to arrange for the comprehensive evaluation of options - i.e. for the compilation of information on all their personal and social, short and long term, consequences - and to feed that information to the public so that they can make informed choices between them; (ii) to initiate the collection of information, to sift it for good ideas and to act on it in an innovative way in the long term public interest.

If we are to get them to do these things it will be necessary to develop new job descriptions and new staff appraisal procedures.

Recognition that the function of democracy is, in Mill's words, "Not to govern, a task for which it is eminently unsuited, but to make visible to everyone who did everything". Thus flat, non-hierarchical, structures are required to supervise the work of the public service (and others).

Climates conducive to innovation and learning based on the work of Donald Schon and Rosabeth Kanter.

It is the task of the psychologist to develop:


Means of taking stock of climates conducive to innovation and learning.
Means of finding out whether public servants are performing their key role of creating a ferment of experimentation and learning, sifting information for good ideas, and acting on it in an innovative way in the long term public interest.
Means of recognising the wide variety of contributions people make to societal evolution, noting that evolution has never been "efficient" in bureaucratic sense.
A better understanding of forms of public management which do not assume that there are somewhere some all-knowing, wise, and benevolent authorities who will act on the insights gained from systems thinking about the hidden processes which determine the course of history and assessingprogress towards them.

top

The Theoretical Bases for the Work

Besides clarifying the, somewhat surprising, developments that are needed to move forward, our work has indicated the need to rethinking, amounting to a paradigm shift, in the way we think about, not only human resources and institutional arrangements, but also the way we assess them.

Among the developments required are:


To refocus our basic way of thinking away from concepts involving "abilities" and "motivation" to thinking in terms of generic high-level, self motivated, competencies.
To shift from attempting to describe, or "assess", people in terms of a profile of scores on small numbers of internally-consistent factors, or variables, to making statements (in a form analogous to the descriptions of compounds in Chemistry) about the motives and components of competence that individuals display in specified environments.
To develop ways of thinking about emergent properties displayed by groups made up of different kinds of people (in a manner analogous to the way one thinks about the emergent properties of compounds in Chemistry).
To better conceptualise, and find ways of assessing, the nature of the Developmental Environments which engage the motives and interests of a wide variety of individuals and lead them to develop and display a range of high-level competencies.
To better conceptualise and assess Institutional Arrangements Conducive to Innovation, based on the work of Rosabeth Kanter and Donald Schon.
To further extend that work to thinking about and assessing the Societal Arrangements Required for Innovation and Survival.

top

The Conceptualisation and Assessment of Competence

A century of work has demonstrated that, so long as we try to work within the mainstream psychometric tradition, it is impossible to get very far. "g and not much else" works.

However, in parallel with this fruitless activity, a growing number (now amounting to some 700) studies of the competencies which distinguish more from less effective performance in a wide range of occupational roles have documented the importance of a range of high-level generic competencies, or motivational dispositions.

The problem is to develop a suitable framework for summarising , thinking about, and assessing such competencies.

Two possible frameworks have been published: One (in Competence in Modern Society) by Dr. John Raven and, the other by Lyle and Signe Spencer (in Competence at Work).

Both are grounded in the framework for thinking about, and assessing, "motives" developed by David McClelland and his co-workers.

Correctly understood, McClelland's measures are not measures of motivation, but measures of the competence to carry out valued activities...defined in such terms as inventing a new scientific theories or putting people at ease.

They measure the respondent's competence to carry out selected tasks by finding out how many of a number of identifiable, cumulative and sustainable, components of competence he or she displays spontaneously while carrying out specific kinds of activity.

What this procedure makes clear is that the usual, internal consistency-based, measures of such things as "creativity", "self-confidence", etc. are off-beam: Someone who displays a great deal of creativity when, for example, putting people at ease is unlikely to do so when asked to think of as many uses as possible for a brick.

Pursuits of these insights leads to the realisation that psychologists have probably been misguided in their attempt to classify people in terms of scores on variables (as in physics). They should rather have been seeking to describe people in terms of their motives and the components of competence they display when carrying out valued tasks, in a manner analogous to the descriptive statements Chemists make about substances.

Research in which this alternative way of thinking about competence and its assessment has been translated into practice and has led to some extraordinary reversals of some of the most "well established" findings in educational and occupational psychology.

The task now is to refine the concepts and tools which have been developed to assist in:

i. the design, implementation, and evaluation of educational programmes;

ii. the evaluation of the pupils and students who pass through those programmes;

iii. the evaluation of the teachers and lecturers who administer them;

iv. staff guidance, placement, and development in the workplace, and

v. performance appraisal systems in the workplace.

top

The Conceptualisation and Measurement of Developmental Environments

In the course of our research we have studied the nature of development environments as they occur in homes, schools, universities, and workplaces. To our surprise, a common pattern has emerged.

In the developmental environments, parents, mentors, and managers tend to create the following for their children, pupils, students, or subordinates:


Opportunities to practice and develop important components of competence while undertaking activities which the tutee is intrinsically strongly motivated to carry out. These components of competence include making observations, developing better ways of thinking about things, using feelings to initiate action, monitoring the results of that action, and taking corrective action when necessary, persuading others to help, intervening in wider social and political processes outside the institution concerned, and persisting over a long period of time.
Opportunities to experience satisfactions which come from the completion of a difficult and demanding activity . It is the experience of these satisfactions which will lead the individual to put up with frustration in order to do similar things in the future.
Opportunities for the tutee to work with others who, importantly, share his or her basic concerns, values, or motives and (i) share their (normally private) psychological components of competence - e.g. use of feelings to initiate, learn from, and adjust action- while engaged in those activities and in such a way that the tutee can learn from and copy them, and (ii) enable the tutee to see them gaining the very satisfactions the tutee most importantly wants from carrying out those activities.
Opportunities to gain insights along the lines just mentioned from literature and research-based case studies.
Opportunities to "try on for fit", or experiment with, alternative ways of behaving in a non-threatening situations in which a mistake does not bring dire consequences.
Placements with mentors who think in terms of multiple talents and try to create working groups made up of people with very different, but complementary, talents in order to create "teams" with dynamic, emergent, properties.

Note how developmental environments engage with the motives or values of individuals: it is as irrelevant to record features of the environment which do not engage with those values as it is to record features in the "environment" of chemical substances which do not engage with the elements of the substance being studied.

What is needed, then, are measures of developmental environments which draw people's attention to these findings, enable them to take stock of the current situation, see what needs to be done to improve it, and monitor progress. The measures need to include questions to find out if there has been a serious attempt to identify individual's motives and talents and redeploy personnel in such a way as to use all available high-level talents for at least part of the time. And to complement these, there is a need for tools to enable managers and others to recognise, develop and utilise the idiosyncratic talents of individuals.

Note that these tools are of crucial importance for the evaluation of teachers and managers: Have they been able to create developmental environments and climates of dedication and enthusiasm?

The Creation of Climates of Enthusiasm, Innovation, and Action in the Interests of Clients and the Long-term Public Interest.

We have seen that delivery of effective education and health and welfare services, and, still more importantly the introduction and administration of a sustainable society, demands new forms of bureaucracy and democracy.

We have seen that these new ways of thinking about public management must incorporate new ways of thinking about the role of the public servant.

We have need to charge our public servants with responsibility for creating a ferment of innovation and learning and hold them responsible for initiating collecting information, sifting it for good ideas, and acting on it in an innovative way in the long term public interest.

We therefore need tools which can be used to take stock of the extent to which the requisite climates of innovation and learning have been created and, by so doing, direct people's attention to the developments that are required.

Many of the features which should be incorporated into such measures have been discussed in connection with innovation in organisations. At a societal level, more attention will have to be paid to:

(i) the investigation of, and development of strategies to influence, the hidden sociological systems processes which deflect most public-improvement action from the achievement of its goals, and

(ii) the formal arrangements for advancing understanding and especially, for obtaining comprehensive, systems-oriented, evaluations. [Current beliefs about the appropriate procedures required to advance scientific understanding are wide of the mark. The only way to move toward a focus on comprehensive, systems-oriented, thinking (instead of on accuracy in relation to one or to isolated specifics) is to provide resources to a wide variety of people who have very different - currently unsubstantiated - perspectives.]

The features to be incorporated into tools designed to direct attention to the key features of new forms of public surveillance of public servants as managers charged with the duties which have been mentioned are less clear. But there is no mistaking the need to move from concepts of government and supervision based in hierarchy to concepts of democracy grounded in fluid, network-based, and issue-oriented arrangements.

top

 

home contact us