Technique – Peer Instruction Question Classification for Computing Science

Summary of Technique

Quintin Cutts and others analysed well over one hundred Peer Instruction questions used in a highly successful introductory programming course.  The questions fell into categories matching key skills involved in programming.  Some of these skills appear to be rarely covered by traditional assessment formats.  Hence, the value of this work is that if we, as teachers, can ensure that we are covering all the categories identified, our learners stand a good chance of picking up the key programming skills.

There are not numerous example questions involved with this post.  As much as anything, the aim is to encourage teachers to consider different question formats that help learners practice a range of programming skills.

Note that although this categorisation came from a programming course, it applies equally well to any computing system where problems are to be solved – in particular in the Scottish context, in database and web systems.

The Question Classification

The categories work across three levels, each of which involves a certain kind of text:

1. The Problem level – this is a programming problem to be solved expressed in English.
2. The CSSpeak level – this could be a programming plan expressed in pseudocode or a description of program behaviour expressed using the language of the discipline (loops, variables, sequence, statements, expressions etc.)
3. The Code level – this is a program expressed using a programming language.

These levels are obviously important to us as programmers.  We develop a problem statement.  To start solving the problem, we consider various approaches initially at a high level, talking about what control structures we need – we are thinking in CSSpeak.  Once we have a plan in our head or on paper, we proceed to code this up in a programming language.  When debugging, we are considering code, comparing it against the problem statement and the plan we formulated, attempting to determine any inconsistencies.  All the time, we are working across these levels.

The categories identified in the analysis were based on (a) which of the three levels the question itself was primarily expressed in, and (b) which level the answer options were mainly expressed in.

For example, consider the following question:

Which of the following Reference Language fragments does not loop 10 times?

The question is expressed in CSSpeak – it’s the kind of language that we as computer scientists understand.  Consider “looping”, “Reference Language fragments” – someone outside our domain would not know the specific meanings we attach to these terms.  The answers are expressed in code.  So this is a CSSpeak -> Code question.

A question which presents a problem and asks for the correct plan to solve the problem would be a Problem -> CSSpeak question.  If it asked for the code to solve the problem instead, it would be a Problem -> Code question.  A question which presents some code designed to solve a given and asks for an explanation of why the code isn’t working correctly is a Code -> CSSpeak question.  A question presenting code and asking for the problem it solves is a so-called Explain-In-Plain-English question, a Code -> Problem question.

Some questions don’t cross levels.  For example, a typical tracing question where a student is asked to hand execute some code and identify the correct output, for example, is a Code -> Code question.

Using the classification in other contexts

The same classification using Problem, CSSpeak and Code levels can be applied to other computing contexts.  Web systems have languages such as HTML, CSS and PHP, they have concepts and constructs that must be described using CSSpeak, such as header, form, tag, style sheet and so on, and problems are specified and solved in these systems.  Database systems have languages (SQL and ER diagrams), CSSpeak (tables, rows, entities, relations, records, trigger, etc.) and again, they are used to solve problems.

Those programming these systems must be able to work across the levels between the problem statement, plans and other descriptions of the solution in CSSpeak, and solution written using the relevant computer language.  Hence questions that similarly span the levels will be valuable in assessing students understanding and skills.

Why? Questions

In the analysis, questions were also distinguished according to a How? or Why? categorisations.  The question above is a How? question in that it asks about what something does which requires the responder to know how it works.

By comparison, a Why? question requires the responder to identify the correct explanation for what something is doing.  An example of a Why? question is as follows:

From the explanations given below, pick the one that fully explains why the while loop in the following code does not cause the <some activity> fragment to be executed 10 times.

These Why? questions are quite hard to construct.  However, two observations:

1. The discussion taking place in the Peer Instruction groups, and class wide, should be getting at this explanation – otherwise how are students convincing one another of the correctness of their answers.  In this sense, every PI question is a Why? question.
2. You can take any How? question and ask students to write down an explanation of Why? their chosen answer is correct.  This is excellent practice in ensuring that they have mastered CSSpeak, elements of the programming language and how they work, and can link all this together with particular problem contexts.

Ways of Using the Technique in the Classroom

Questions developed using this classification are simply PI questions and so would be used in the same way as any other PI questions.  Be sure to write questions that assess both going down the levels (the more normal approach) as well as going up the way (a less common assessment but every bit as important).

Benefits for Learners

The key value in following this categorisation when developing questions is that a broad range of skills are assessed – which will inevitably aid learners in developing a stronger understanding and skill set in programming.

Further reading

Cutts, Esper, Fecho, Foster, Simon “The Abstraction Transition Taxonomy: Developing Desired Learning Outcomes through the Lens of Situated Cognition”.  In: Clear, A., Sanders, K. and Simon, B. (eds.) Proceedings of the Ninth Annual International Conference on International Computing Education Research, Auckland, New Zealand, 10-12 Sept 2012. ACM: New York, NY, USA, pp. 63-70. ISBN 9781450316040 (doi:10.1145/2361276.2361290)