CourseMarker

CourseMarker is a system developed in the School of Computer Science and Information Technology for the automated assessment of programming tasks (Foxley et al., 2001, Higgins et al., 2002a). It is based on an older system, Ceilidh, but is a complete reimplementation. In addition to the assessment of programming (primarily in the Java language) CourseMarker has recently been extended to mark diagramatic work (Higgins et al., 2002b).

CourseMarker (and Ceilidh) are different to most other CAA packages in that they assess programs which perform tasks rather than answers to questions. This is highlighted by the comments on CourseMarker in a recent survey of CAA methods:

CourseMaster is a client-server based system for delivering courses based around programming or diagramming exercises. It provides a solution for delivering coursework specifications, allowing students to develop programs and submit them, and the automated marking and monitoring of students' responses. Unlike much of the other software reviewed on this site, CourseMaster is tailored to the needs of teaching programming, and has been developed and used over several years.

CourseMaster is an updated version of the tried-and-tested (10 years) server/client-based Ceilidh system. It has a number of unique capabilities, for example the assessment of diagram-based work. It is robust and easy-to-use, particularly for the student. The lecturer is given a wide variety of statistics, though through a slightly dated interface. Marking is very flexible, though the built in tools do not achieve anything particularly innovative, but are still useful. Installation may be a problem area, depending on the platforms and configurations used at your institution. All aspects of implementation seem to have been thought through, and security is unusually good.

Rawles et al., 2002

Assessment in CourseMarker

CourseMarker offers a number of tools for assessing student submissions (Higgins et al., 2002a). For marking programs these are:

In addition there are other marking tools for non-programmatic assignments which are discussed below.

The three marking tools are used to construct a marking scheme for an assignment. Different tools, and different aspects of each tool, may be combined to return a final grade. Tools are also in place for scaling marks, where required, to fit expected distributions.

In addition, CourseMarker can be configured to accept multiple submissions. This allows students to submit initial attempts, get feedback (see below), and then try again. For summative assessments the number of submissions permitted for each student would normally be limited, to prevent trial-and-error programming, but this can be relaxed for formative assessment.

Feedback in CourseMarker

Systems are also provided for giving feedback to the students on the basis of their submissions. The experience with CourseMarker suggests that "[students] prefer the system to pinpoint the exact weaknesses of their solution", but "too much feedback can be detrimental to the students learning process" (Higgins et al., 2002a). A variety of forms of feedback can be supplied, including suggestions for improvements to the student's current solution and pointers to relevant course material.

Plagiarism Detection in CourseMarker

CourseMarker uses fairly straightforward methods for detecting plagiarism. Student's submissions are compared pair-wise for similarity. This can take some time for large classes, as if there are n students then there are (n(n-1))/2 paris which need to be checked. The plagiarism detection compares student's work for similarity. This process also takes into account the fact that some aspects of programs, such as variable names, comments, and whitespace, can be changed without altering the program in any real manner. For example, the following two pieces of Java code, although they look quite different, are exactly the same as far as a compiler is concerned:

public void printHello(){ public void sayHi() {
  // Prints Hello World to standard out         /* Doesn't look like the other one */
  String str =  
    "Hello World"; // A string to print         String myString="Hello World";
  System.out.println(str); // Print it         System.out.println(myString);
} }

CourseMarker compares student's programs and reports how similar each was, and the type of difference between them. The level of difference indicates what sort of changes (removing whitespace, changing variable names, etc.) needs to be made to make the programs the same. The results are ranked by similarity so that the most similar (ie most suspect) pairs of students are brought to the marker's attention.

Diagram-Based Assessment

A recent extension to CourseMarker is the ability to assess diagramatic courseworks (Higgins et al., 2002b).