Evaluating course books – checklists

By Pete on October 11, 2016 • ( 3 )

I’m currently studying a module in Materials Development through NILE online. It’s a really worthwhile course so far!

Unit two talked about evaluating materials, specifically course books. We were introduced to a range of checklists that could be used for evaluating a course book, and discussed the pros and cons of each. I can’t imagine everyone would find this topic interesting, but it was really topical for me – in the same week I was given a checklist to evaluate our new course book for teen classes. It gave me the opportunity to reflect on our own evaluation process and suggest some changes if necessary…

What makes a good checklist?

We looked at about six different checklists that were listed in McGrath (2002). In some of my jobs I’ve completed evaluations like this one from Harmer (1991:281)

They’re alright. I mean, the fact that a school is bothering to evaluate a course book in the first place is a positive thing. I’ve worked in other jobs where this wasn’t standard practice and the course books used seemed unsuitable.

Reflecting on pitfalls/considerations when making a checklist, McGrath mentioned these points:

Selection of criteria – if you use an existing checklist rather than make your own can you guarantee that all the criteria are relevant to your context? Do you need to adapt it?
Item format – how do you want teachers to evaluate? Will you use open ended questions, tick boxes, a mixture, etc.
Weighting of criteria – are all items equally important? Should some be given greater importance than others?
Value-laden responses – this comes down to subjectivity from teachers based on their own beliefs. It’s tough to control/address I think
Are the learners involved in the evaluation process?

The last point is one of the most important, but the checklists I’ve looked at are meant for teachers.

In addition to McGrath’s list, I think wording is really important. I noticed in the checklists we looked at that the criteria used can be ambiguous or a bit leading, plus they can be too time consuming for teachers. For example, in Harmer’s example above:

‘Is there are sufficient amount of communicative output in the materials under consideration?’

What constitutes a ‘sufficient amount’? Harmer does provide a description, but that involves more investment as a teacher to read up on this. I don’t often have time, so I’d probably just use my own judgement on what a sufficient amount of communicative output might be. But I’d likely supplement a course book anyway to provide more communicative tasks – so is the question actually whether I have to supplement too much?

Weighting and rating

I’d never come across a ‘weighting and rating’ checklist before. Here’s an example from McGrath (2002):

Different items are given a ‘weighting’ depending on how important they are in the evaluation. Then they are given a ‘rating’ depending on how effective the course book is in this area. Weighting multiplied by rating is the evaluation score for that particular item. I prefer this to YES/NO questions, it gives a bit more information and prioritises the most important features of the materials.

Our checklist – is it any good?

It turns out that our checklist at the BC is pretty much like Harmer’s:

Pros:

It’s comprehensive. It covers methodology, language, skill, practice, variety, ease of use, exploitability, suitability, assessment and more
Most items in the list are clear and measurable
It’s not that time consuming to complete and comments are optional

Cons:

Some items are still a bit vague or open to interpretation
You’re forced to choose between YES/NO, although you can elaborate on viewpoint with comments

Possible improvements:

We could use a rating system, it might tell us more about how effective the material is
Teachers could devise the checklist together rather than using the existing one, but we don’t get the time to do this
The checklist could be accompanied by an evaluation for learners to complete – this might be coming soon though!

Anyway, this course is certainly making me think more about our approaches. I hope I can put some of what I learn to good use.

Do you evaluate the materials you use? Do you use a check list to do so? Do you think it’s effective?

All comments welcome 🙂

All examples taken from McGrath, I. (2002). Materials evaluation and design for language teaching. Oxford: OUP

Feature image: pmmajik.com