In writing about case study research in educational settings, Bassey (1999) identifies several types: theory building, theory testing, storytelling, picture drawing, and evaluative. This reflective case study, in which test item design is the case, includes elements of the storytelling and picture drawing types and is to some extent also evaluative, given that reflection is at its heart. It should be noted that, from its inception, the PrimTEd Assessment project has combined research and development, with lecturers across South African universities participating in face-to-face and online workshops and in email exchanges. In this article, we reflect on a selection of the items designed by one or both of us as members of a larger English test item team in which the number of participants varied across workshops. Team members designed items individually or in pairs, shared them with the team for comment and, if necessary, adjusted them in response to comments made. Subsequently, all items were reviewed by members of the PrimTEd Assessment project for conformity to the requirements of quality MCQs and for their ‘match’ to one of the knowledge and practice standards or sub-standards. All the items discussed in this article were designed to be used in the pilot phase of PrimTEd, but were excluded from the administered pilot test.

We chose the items presented and discussed to illustrate the concerns we have about the affordances of MCQs for assessing teachers’ knowledge of visual and critical visual literacy and of how to teach learners to be visually and critically visually literate. We agree with Attia and Edge (2017) and Arnold et al. (2022) that there is a need to move away from (Atti & Edge 2017):

However, we brfiefly outline three aspects of a conventional research methods description.

The authors as test item designers, together with other members of PrimTEd test item development team, are the participants.

The test items included in the article are the data and were selected firstly, because they were designed by the authors; and secondly, because they illustrate the challenges we experienced in our attempts to use multiple choice items to assess elements of student teachers’ content and pedagogic content knowledge of visual and critical visual literacy. Our reflections on the items, in terms of their conformity to the requirements of high-quality MCQs and in terms of their affordances and limitations for assessing elements of visual and critical visual literacy, constitute our analysis.

Images selected for test items need to be generative for the assessment of student teachers’ visual and critical visual literacy content and pedagogic content knowledge. Our search for culturally accessible multimodal materials appropriate for use in primary schools was made more challenging by the need for these to be either open access, or materials for which permission to use for non-commercial purposes is easily obtained.

There are also technical challenges associated with using images. Numerous examples of cartoons or cartoon strips or advertisements could not be copied and pasted into Jumptrak, the digital assessment platform being used for testing in PrimTEd. Redrawing these images would be too time consuming and expensive to be viable.

In selecting examples of visual texts, item designers need to consider the likely background knowledge of both the testees and the learners whom they are preparing to teach. Many student teachers’ visual literacy experiences in secondary school classrooms are likely to have been constrained by the washback effect of the National Senior Certificate examination papers in which, for many years, cartoon strips and advertisements have been dominant (Moodley 2015). However, while cartoons and advertisements have a role to play in visual and critical visual literacy education in the later years of primary school, young learners need to learn how to ‘think about looking’ (Ferreira & Newfield 2014) at the images in the stories and information texts they encounter in the classroom.

Thus, student teachers who will work in primary schools need content and pedagogic content knowledge that will enable them to facilitate such thinking.

In the next section we present and reflect on a selection of items designed to assess elements of student teachers’ content and pedagogic content knowledge of visual and critical visual literacy.

Ethical clearance to conduct this study was obtained from the University of South Africa College of Education Ethics Review Committee. The ethical clearance number is 2023/05/10/90502744/36/AM. Permission to use the illustrations on which the designed test items are based was obtained by PrimTEd co-ordinators. Test items selected for this article were excluded from those used in the tests administered to students.

The open access materials included in the storybooks and teacher resources developed for the African Storybook Initiative¹ have proved excellent for devising questions pertaining to visual literacy, but the difficulty of formulating such questions in multiple-choice format remains, as we illustrate in the example below.

The example comes from a South African Institute for Distance Education (Saide)¹ resource designed for teacher professional development work in Uganda. The story used in the resource is originally from Mozambique but was versioned into one of the languages used in Uganda. Neither the Ugandan language nor the Mozambiquan origin of the story is centrally important to the items devised about the illustration for the PrimTEd test, although we acknowledge the value of using a story from Africa in an African language.

The South African Institute for Distance Education presents an illustration (see Figure 1) by Hélder de Paz Alexandre titled Why Frog is so ugly (Mozambican folktale 2014) as a resource for teachers to utilise for conversing with children. They advise teachers to look at the picture, then think about questions to ask children, and finally make some suggestions to teachers. The next section offers more insight into this Saide resource

Look at the pictures in the Mozambican folktale, Why Frog is so ugly.

We had the following ideas for questions:

In the Saide resource, each of the bulleted questions designed to be asked by teachers could be used to promote lively discussion in a primary school classroom and to address aspects of visual literacy. For the PrimTEd test, we attempted to use the illustration and the accompanying questions to design MCQs to assess aspects of student teachers’ visual literacy knowledge together with their understanding of the importance of working with learners’ linguistic repertoires (McKinney 2017).

Box 1 presents the first example of four visual literacy MCQs followed by a brief commentary.

Similar MCQs could be designed to assess student teachers’ knowledge of the metalanguage of visual literacy or critical visual literacy as in the examples presented in Box 2.

For Question 4 in Box 2, in recognition of the limited teaching of visual literacy that some first-year student teachers are likely to have experienced (see Mbelani & Murray 2009), we decided to provide support. In our view, this is an example of an MCQ in which all the distractors are visually similar and plausible with the cognitive level appropriate for entering BEd students, although we hope that graduating teachers would find it easy to identify the correct answer.

While each of the MCQ items above attempts to assess an aspect of student teachers’ visual literacy, none of them attempts to assess knowledge of critical visual literacy. In our view designing MCQs to assess the latter is even more of a challenge. Ferreira and Newfield’s ‘What or Who? How? Where?’ questions provide a framework for critical visual literacy questions that could be asked about the frog illustration:

Teachers could ask primary school learners each of these questions to assist them to think about an illustrator’s choices and the likely influence of these choices on readers of this page from a story. Very similar questions could also be asked of student teachers. However, designing MCQs based on this illustration to assess aspects of student teachers’ critical visual literacy knowledge is a much greater challenge. The MCQ item in Box 3 was designed as an attempt to assess whether student teachers understand that a designer’s choices of image and layout on a page attempt to position viewers or readers to respond in particular ways.

In addition to using the African Storybook Teachers’ Resource to design visual literacy test items, this resource was used to design several items for assessing student teachers’ understanding of the importance of valuing learners’ home languages, and of drawing on their full linguistic repertoires in the classroom. Question 6 is one example of such an MCQ item (see Box 4).

As noted by Callow (2018:5), ‘picture books continue to be cornerstone resources in primary classrooms, enduring in their appeal, but at the same time evolving, with authors and illustrators exploring new forms and concepts’. Given that images and their positioning on the page or screen are central to the meanings available in fiction and non-fiction books for primary school learners, the PrimTEd item writers have attempted to design MCQs to assess student teachers’ visual literacy knowledge and practices in relation to picture books. Question 7 exemplifies such an item (Box 5). Drawn from the American children’s story Charlotte’s web, which features prominently on South African educational websites alongside classroom resources, the item uses a scene from the 2006 edition.

Due to copyright constraints, the book cover is not reproduced here, although it is widely available online. The image depicts Wilbur the pig, with Charlotte the spider perched on a web atop his snout, set against a farm sunset backdrop.

To conclude this article, we reflect briefly on what we have learned thus far from our experiences of designing MCQs for the assessment of visual and critical visual literacy content and pedagogic content knowledge.

Firstly, we suggest that MCQs can be designed successfully to assess student teachers’ subject or content knowledge of the metalanguage of both visual and critical visual literacy because correct and incorrect options can be offered as single words or short phrases. Knowledge of this metalanguage should enable teachers to comprehend and discuss articles and books that focus on visual literacy and critical visual literacy and to use what they have learned in planning lessons. In teaching visual and critical visual literacy in primary schools, teachers also need to know how to ‘version’ this metalanguage to make it comprehensible to primary school learners so that these learners can extend the ways in which they ‘read’ and discuss images and layout on a page or screen.

This is an aspect of pedagogic content knowledge for which we have not yet attempted to design MCQs, although it should be feasible to do so as ‘simpler’ words, phrases, or examples could be offered as options in MCQ format.

Secondly, we have found it challenging to design MCQs that do not position testees to take up the item designers’ position on an image – a position that may not be equally fair to all of them. As some of those interviewed for Schilder et al.’s (2016:43) study noted, ‘[i]t is difficult to take your own philosophical views out of assessment’.

We suggest that it is particularly difficult to do so when offering multiple options, only one of which is ‘correct’, given that an obvious question to ask is, ‘Correct according to whom?’ In our view, ability to read and respond critically to a visual text is more appropriately assessed through questions that enable a range of responses and rationales for these responses on the part of the test taker.

Thirdly, as indicated in Question 5 (Box 3), we suggest that it is slightly easier to design MCQ items to assess student teachers’ knowledge of how to teach critical visual literacy than to assess their critical visual awareness because a range of plausible teaching strategy options or teacher questions can be offered.

Fourthly, while we think it is feasible to design MCQs to assess some aspects of knowledge of pedagogic practices associated with teaching visual literacy, the items are likely to be quite wordy, and thus time-consuming for testees to read, because of the descriptions needed in both the stem and the four options.

While MCQs may be the best option available for large-scale, computer-marked national assessments, perhaps it should be accepted that these are not well suited to assessing all aspects of student teachers’ subject and pedagogic knowledge of visual and critical visual literacy. We suggest that teacher educators teach ITE students how to use a range of more open-ended and multimodal strategies to assess learners’ developing visual literacy and critical visual literacy. In addition to theorising visual literacy and critical visual literacy, the work of scholars such as Janks (2010), Janks et al. (2014), and Ferreira and Newfield (2014) referred to above, together with others such as Vasquez (2014) and Vasquez, Janks and Comber (2019), provides examples of how to engage primary and secondary school learners in activities that can be assessed both formatively and summatively. When teacher educators use such open-ended and multimodal strategies in assessing their students’ content and pedagogic knowledge of visual and critical visual literacy, and provide feedback to their students, they are modelling how graduate teachers could use open-ended, multimodal assessment tools in their classrooms.