Other formats

    Adobe Portable Document Format file (facsimile images)   TEI XML file   ePub eBook file  

Connect

    mail icontwitter iconBlogspot iconrss icon

Salient. Victoria University of Wellington Student's Newspaper. Volume 31, Number 3. March 19 1968

Exams - Cloud describes the alternatives

page 4

Exams - Cloud describes the alternatives

"Whole intellects disgorged in total recall for seven days and nights with clear eyes ... "

—Allen Ginsberg, Howl.

Of all the students who enter our university only 50 per cent ever gain any academic qualifications. In the previous articles in this series I have tried to show that the traditional essay-type examination is responsible for at least one-half of this enormous wastage of talent. I have quoted evidence from. many sources of the unreliability of examinations and examiners, of fluctuation in standards, of unjust and unnecessary failures. I have argued that examinations do not provide a valid measure of academic achievement, and that their consequences are detrimental to students, staff, and indeed the whole process of education. As they are now conducted, examinations are a parody of the assessment procedures suited to the modern university.

Given all the faults of examinations, should we then simply abandon them? No. The correct solution, I believe, lies in the opposite direction. We should simply multiply examinations, tests, and essays of many kinds, using them frequently, but always informally, casually and sceptically. The results should be recorded, correlated and studied intensively, in conjunction with personal impressions, tutors' estimates, and all available facts in regard to the student's background and achievement.

too important

Administrative decisions regarding graduation, promotion, classification and guidance could then be based on a cumulative record of the student's abilities, aptitudes, attainments, interests, difficulties and potentialities. Such decisions are far too important to be made on the basis of two or three final papers.

This conclusion — that we need more rather than less examinations — is dictated by the circumstances of this particular university. It would not necessarily be appropriate for other institutions, even within New Zealand. An ideal, "spontaneous" university, such as I described in the last article, could conceivably do without any examinations at all. But the ideology and traditions of such a university would have to differ radically from those we know at Victoria. Our university serves a community whose needs are not merely for "educated persons" but also for highly trained specialists whose knowledge of substantive areas and methodology is more precisely measurable.

It follows that as well as giving more examinations, we must devise better ones. Having decided what we want to measure (and this is no small problem), we must set about measuring it in a valid, reliable, scientific way.

Let us consider the instruments at our disposal. There are at least five major ones: the essay, the short-answer or multiple-choice test, the observation of behavioural performance, the oral examination, and the traditional examination.

The first, the essay, combines all the advantages (or supposed advantages) of the traditional examination with few of its disadvantages. Writing an essay requires discrimination, analysis, synthesis, and coherence; it requires a thorough understanding of the subject and reveals the student's ability to organise his thoughts into a logical argument; it allows the student to refine his presentation until he is satisfied that it represents his best, and its depth is limited only by his own resources. As against all these good qualities, the type of essay usually set at Victoria has two major drawbacks: (1) it takes as long as the traditional examination to mark; and (2) it is an invitation to lazy or dishonest students to cheat.

These problems can never be fully eliminated, but I think certain precautions can minimise their effects.

page 4

The difficulties of marking an essay can be considerably reduced by specifying clearly what is desired. Thus an essay question such as "Describe the origins of World War I" is both more difficult to answer pointed'y and to mark fairly than the following:

"Explain why each of the great European powers — Austria, Germany, France and Great Britain — must bear some of the responsibility for the events which led immediately to the outbreak of World War I."

Similarly, the more advanced essay topic, "Discuss the dichotomy between conflict and consensus" is absurdly vague by comparison with:

"How can a society incorporate continuous conflict among its members and yet maintain social cohesion and the legitimacy of state authority? Consider this question in relation to the views of Marx and de Tocqueville."

As well as a clear specification of the topic, students need to know what qualities they should incorporate into their essays. Explicit, operational instructions with regard to requirements should be given with the questions. A vague indication that grading will be partly on the basis of "good organisation" is insufficient. Specifically, this may mean that students should (a) determine early in their discussion what point or points they wish to make; (b) select information relative to making these points; (c) use this information in such a way that it contributes to making these points; (d) offer conclusions when they have presented enough evidence to warrant them; (e) avoid using irrelevant and extraneous information; etc.

If teachers do want these things, then they should say so, and not take them for granted. Markers should also have an explicit marking guide, covering general aspects such as knowledge of subject, critical assessment of theoretical claims, correct use of specialist terminology, good organisation, grammar and spelling, etc. (with a definite number of marks allocated for each aspect), so that they look for the elements they want and comment on any insufficiencies. These guides should be known to the student as well as to the marker.

The problem of dishonesty also yields to concerted effort. The tendency to use long unacknowledged quotations is rarely serious where students know that they will not be penalised for using acknowledged ones.

Criticism of the examination system has often gone unheeded because the critics failed to offer viable alternatives. In this article, the last of a series, Jonathan Cloud gives constructive suggestions for a new system.

cheating

More difficult is the claim, made by a member of staff, that students can buy moderately good essays at Weir House. This can only be checked effectively by supplementing essays with other methods of assessment. And, as Paul Dressel writes:

"The most important step in prevention of cheating ... is to provide challenging courses and fair examinations. Insistence on memorisation of a large number of dates, locations, formulas, and other such isolated bits of information may be resented; many students consider this kind of learning a waste of time. Once convinced that the course requirements are unrealistic or unreasonable, even the able student may find justification for circumventing them."

The essay is widely used at Victoria — perhaps too widely, since it demands a great deal of time, and by itself constitutes only an inadequate and unreliable sample of the student's knowledge. The short-answer or multiple-choice tests, commonly known as "objective tests," are an essential supplement, and deserve to be used much more frequently.

Objective tests are often criticised on the grounds that they require only superficial knowledge of simple facts; but this is in fact true only of those which are poorly constructed. Writing adequate items requires a specialised skill, a great deal of time, and a willingness to experiment. This is offset by the fact that objective tests are easy to administer and to score. The time that is saved in marking, however, must be spent in test-construction if these tests are to have any validity.

Objective Tests

The major drawback of objective tests is that bad items are so easy to construct. By "bad items" I mean items which can be answered on the basis of factual knowledge or common sense alone, or else are so ambiguous that no correct answer is possible. Items of the type—

"Freud was a(n)

1. Experimental psychologist.

2. Psychoanalyst.

3. Ethologist.

4. Marine biologist."

are worse than useless: they are an insult to the student's intelligence and imagination.

Nevertheless, attention to the rules of test-construction can produce items which sample the quality of thought as well as a wide range of types of knowledge. To illustrate this point I have taken several examples from Paul Dressel's Evaluation in Higher Education, a book which should be compulsory reading for all lecturers and examiners. The first example is intended to test knowledge of methodology:

"Of the following, which represents the most important difference between the scientific method as used in the social sciences and as used in the physical sciences?

1.Study of the developmental aspects of various problems.
2.Necessity of regarding hypotheses as tentative in nature.
3.Importance of understanding the problem of causation.
4.Necessity of recognising the value judgements of the investigator."

The correct answer is (4).

It is obvious that the specific facts which are required in "thought" items should not be spoon fed to the student. This is illustrated by the following item, intended to elicit the extent to which facts support a generalisation:

"The prices of practically all consumer goods and services in 1950 were 50 to 100 per cent higher than they were in 1939. This fact most clearly supports which of the following generalisations?

1.Factory workers were better off in 1950 than in 1939.
2.Persons with fixed incomes were economically worse off in 1950 than in 1939.
3.Profits were a higher percentage of national income in 1950 than in 1939.
4.People bought fewer consumer goods in 1950 than in 1939."

Here the correct answer can be determined by logically eliminating each of the alternatives for which the fact does not provide sufficient evidence; it is (2).

Both content knowledge and reasoning are required to answer the following item:

"Assume that over a ten-year period consumer purchases had dropped from 75% to 65% of the G.N.P., and that investment and net exports had remained the same, proportionately. On the basis of this information alone, we can be certain that—

1.Total expenditures had declined, percentage-wise.
2.The size of the G.N.P. had declined.
3.Government spending had increased, percentage-wise.
4.The size of disposable income had declined.
5.Personal savings had increased, percentage-wise."

The correct answer is (3).

It will be obvious from these examples that it is possible to do a great deal more with objective tests to supplement other forms of assessment, and Dressells book includes sample items for the social sciences, the natural sciences, and the humanities.

Half-way between the essay and the objective test is the so-called "take-home test." This instrument may perhaps seem more useful as a teaching aid than as a method of assessment; it is most popular in mathematics, where problems may be set each page 5week. Its use in other disciplines is probably limited, but an ingenious teacher could, for example, devise problems in the social sciences, requiring short answers, and emphasising a particular point. Consider the following:

"People who are prejudiced against one ethnic group tend to be prejudiced against others.

Find one study which attempts to provide evidence relevant to this hypothesis and summarise it in 200 words or less."

The actual quality of this type of test is generally less important than the mere performance of the task: however, lecturers should try to avoid overloading their students with relatively trivial assignments.

The third major instrument, observation of behavioural performance, involves the student in actual participation in activities relevant to his discipline. In some subjects this may be only a matter of engaging in tutorials or debates, but practical work plays an increasing part in many subjects —field trips, social surveys, experiments, and so on. Often these are thrust upon the student with minimal supervision, and no attempt is made to assess anything but the final report.

tutorials

Participation in these activities is, however, an integral part of the professional life of the specialist, and practical work should be guided and assessed at all stages.

Participation in tutorials and seminars is rarely assessed; consequently many students pay little attention to what is potentially the most effective source of instruction at university. Being lectured at is no substitute for personal involvement in the major issues and the opportunity to ask questions. Successful tutorial work depends, however, on adequate preparation and willingness to participate, both of which would be enhanced if students were more or less informally assessed by their tutors. Naturally such evaluation must not be such as to discourage participation: students who ask questions should be praised for doing so, not penalised for their ignorance.

A special case of the assessment of such performance is the oral examination. Oral examinations have long been popular in Europe, although they are subject to the same kind of abuse as traditional examinations. They can. like traditional examinations, provoke anxiety and nervous breakdowns; and the validity and depth of the evaluation they provide rcmains severely limited. Nevertheless, they have their uses if they are conducted casually and sceptically. They are, or should be, discussions and learning situations as well as examinations, and not just third-degree grilling sessions.

The final instrument, the traditional examination, has already been discussed at length, and I can only reiterate here that it should play a very much reduced role in the total assessment process. If time-limited examinations must be given, they should consist primarily of multiple-choice questions like those mentioned earlier, and only one or two essay topics.

formal ritual

One of the first things that should go, of course, is the formal ritual with which final examinations are surrounded. There is no reason why students should not sit their examinations under the same conditions as any other test, supervised by their own lecturers or tutors instead of by old women of doubtful intelligence. There is no need for elaborate time-tables, secrecy about rooms and questions, pink booklets and code numbers. There is no need for any of this — unless we make such a fetish of final examinations that every student can be automatically expected to cheat.

But this pointless ritual, which only aggravates the student's natural anxiety, will probably be the last thing to go: since we are always more willing to rob our institutions of their substance than to dismantel them. Already some departments have begun to assign a specific proportion of the final mark on the basis of the year's work, but the examination papers remain virtually the same. The paraphernalia of examinations will probably remain as a symbol of the inglorious, irrational past, until some far-sighted Vice-Chancellor wakes up to the fact that it wastes time and costs money.

Much of the hard work which examinations involve for the staff could be eliminated very simply, if adequate evaluation of the students could be made during the year. One lecturer at the Christchurch Teachers' College confronts his class at the end of the third term and posts a list of final grades for all his students. Only those who are dissatisfied with the marks assigned to them need sit the final examination. This lecturer — obviously rather more astute than most — usually marks only two or three finals papers every year. The experiences of other teachers prove that much the same could be done at Victoria. I was told by one professor, for example, that it seemed a little pointless to "average out" essay marks with finals marks in one class since they were exactly the same in all cases. This suggests that he need not have marked any of the examinations at all.

Such changes as I have been advocating would do much to improve the evaluation progress at Victoria. It should be clear, however, that the defects of the examination system do not account for all the wastage of talent at the university; consequently, reforming the assessment procedure will not halt this wastage overnight. In broad terms, each university needs its own educational research committee, attached to the administration, to carry out a continuous analysis of the whole of university life.

Such a committee would include at least one expert on educational evaluation, who would be able to advise departments on the construction of tests and examinations. The committee would need, however. a much broader mandate, since the examination system is only one of several factors which influence the failure rate (albeit one of the most important). It would need, for example, to look at the differences between the university and the schools which supply it. These differences — in terms of impersonality. indifference, and occasional hostility of lecturers, the system of note-taking, the anonymity of students and staff, the immensely increased probability of failing — are sufficient to disorient even the best student for months. Considering all these factors, Noel Harrison writes.

"any protestations of concern by the university over the failure rate among first-year students are difficult to understand. The university decides its own failure rate. Why then should there be surprises when this policy leads to the inevitable failure of so many students?"

We know that many factors make it very difficult for students to match up to the standards which the university sets, and they will no doubt continue to have this difficulty. But when these standards are themselves unknown, erratic, or merely capricious, the difficulty becomes an impossibility.

I should like to conclude by referring to two arguments which are often used to oppose reform: (1) that staff shortages prevent any improvements in teaching methods or evaluation; and (2) that any change in assessment procedures or pass rates would lower the standard.

top 10 p.c.

In the first case it is undeniably true that Victoria has a chronic shortage of staff, but it is equally true that we are not using the available staff efficiently.

The second argument is utterly vacuous; but I have even heard it used to justify the high attrition rate of students. "Of course," I was told by one professor, "our standards are too high for 40 per cent of the class — but then they shouldn't be here, they're not intelligent enough." Considering that university students generally come from the top 10 per cent of their age group, this argument proves only that the speaker is intellectually arrogant or a poor teacher or both.

There is only one way to raise the standard of the students academic performance, and that is to improve the quality of his education. The raw material available to the universities is much the same everywhere; the worth of the finished product depends therefore on the "value added" by his teachers. As T. H. Huxley once said, "Students work to pass, not to know; and outraged Science takes her revenge. They do pass, and they don't know."

References:

Dressel, Paul L. (ed.) Evalution in Higher Education (Boston: Houghton Mifflin, 1961).

Harrison, Noel. In The Transition from School to University. edited by P. J. Blizard (N.Z.U.S.A., 1965).

Parkyn, G. W. Success and Failure at the University, Vol. II. "The Problem of Failure" (Wellington: N.Z.C.E.R., 1967).

Listener Correspondence on Exams

Listener Correspondence on Exams