Guidelines for Markers

Tim Kovacs. Last updated Dec. 2009.

This was originally written for markers on the Advanced Topics in Machine Learning unit who were marking either essays or reports on a data mining challenge. There are different kinds of assignments and feedback, and this discussion may be less relevant to other kinds.

Summative and Educative Assessment

There are two main objectives in marking coursework:

To obtain a mark. (This is called summative assessment.)
To help students learn by giving them useful, constructive feedback (called educative or formative assessment).

The difference is emphasised by thinking of them as assessment of learning and assessment for learning respectively.

Students typically really appreciate good educative assessment. As a very rough rule a credit requires about 10 hours of work. An assignment worth 1/3 of a 10-credit unit might therefore take about 33 hours. Imagine you have spent 33 hours on an assignment. You would probably want to know why you got the mark you did and how you could do better in the future. This is particularly true if you did your best and could not easily think of ways to improve your work. In short, getting useful feedback is very important!

Educative assessment is particularly important for advanced work (such as final year projects) which requires many skills (e.g. an ability to write well, to judge what is relevant, critical evaluation), a high level of understanding, and ideally shows some creativity and a synthesis of material rather than just repetition of other sources. Note the use of the word 'skills'. Skills, unlike e.g. simple facts, are hard to teach and hard to learn. The only way to develop better skills is to use them, and to get feedback on them.

What to look for

Markers should of course read the assignment specification, and also the hints for written work which contains a list of things to watch out for. Here are some additions.

A report should:

Synthesise existing material: make new connections between existing areas, present new theories, present a new point of view on existing material (e.g. a new taxonomy of algorithms). This tends to appear in only the best work, and there is little scope for it in short reports.
Present only relevant and significant background material.
Criticise shortcomings in other work or in the field generally.
Identify directions for future work (either as a continuation of this report or for the field in general).
Use appropriate citations (e.g. whenever an algorithm or result is mentioned, or a claim is made).
Be clear.
Use appropriate examples.
Develop intuitions.
Present material graphically when appropriate.
Justify claims either with citations, logic, or sufficient empirical evidence.
Consider statistical significance and computational complexity when appropriate.
Have clear objectives.
Have clear conclusions.
Have a complete and clear structure: introduction, body, conclusions, references.
Be nicely formatted.
Longer reports (including dissertations) should also reflect on how the work went and what was learned by doing it.

A report should not:

Plagiarise either text or figures!
Present irrelevant or insignificant background material. In particular it should not be a 'brain dump' of everything you know on the subject. This shows lack of judgement.
Spend much time on well-known material (e.g. what a decision tree is, or a tree, or a computer...).
Follow the structure of other sources too closely when there are sensible alternatives. This does not show any judgement.
Use excessive notation. This complicates things unnecessarily.
Use excessive quotation. This is lazy and shows a lack of conviction in your understanding and ability to express things. It usually suggests to the marker that you don't really understand the material.
Use low-quality screen shots of equations and figures. This is sloppy and does not give the marker a good impression of your abilities and efforts.

Summative feedback

Provide a mark as a percentage using the online system, but do not publish it. For reports it is best to read the reports, make notes on them, and then rank them. Then you can map them to the UK's nonlinear scale. For MSc students a pass is 50% and marks of 70% or more are reserved for excellent work.

Educative feedback

You must provide individual comments on each report. You may also write group feedback in which you cover points which apply to many reports.

Educative feedback should indicate what is good and bad about a report and ways in which it could be improved. The amount of feedback should be in proportion to the value of the assignment. The standard approach is to upload a text file for each student. Here are some examples of good individual feedback for short reports in plain text. Another approach is to write comments on the report, scan it and upload it. Here is an example scanned report. Finally, you can also return the hardcopies to the students but we want to keep feedback on the online system for our records, so only do this if there is little important feedback which appears only in hardcopy.

Educative assessment is most useful when it is returned quickly, while the work is fresh in the students' minds. Since the students do a series of short reports it's important that they get feedback before doing the next in the series. Therefore we will set a deadline for each assignment to be marked. Please meet it or let us know as soon as possible if there is a problem.

Some final words for markers

If they disagree with your comments or marks students may reply to them. (Usually this is in the form of an email from the student which one of the lecturers forwards.) Often these students have worked hard and simply want to know whether their work really is correct or not - they want feedback. It can also be a matter of principle; when you're right you want the marker to acknowledge it. Often they are more interested in this than in changing their mark. (In fact it is the students who care more about whether they are right or wrong than about their marks who tend to be the best students.)

Remember that marking is an opportunity for the marker to develop some of the skills needed to produce reports, especially the ability to evaluate work critically.

If you have any questions on how to mark don't hesitate to ask one of the lecturers.

Some words for students

If you disagree with a marker's opinion you can debate it. Markers aren't always right, and don't always agree with each other.

Feedback on feedback is useful for markers. Some marking is done by PhD students who need to learn how to do it, and even very experienced lecturers have things to learn. So, if you feel you aren't getting the feedback you need, or if you find the feedback particularly useful, let us know.

More generally, if there are things you think are good or bad about a unit or degree, let us know. It's easier to calibrate the pace of a unit, or the way material is presented, or how useful the coursework is, with feedback from students.