Diversifying assessment through multimedia creation in a non‐technical module: reflections on the MAIK project

Creation of multimedia (MM) could be a valuable diversification of assessment methods within non‐technical modules. The apparent popularity of sites based on user‐generated video content such as YouTube and also of podcasting suggests that relevant skills and interest are becoming more mainstream. Translating book learned knowledge into visual forms involves a specific type of intellectual challenge. It seems possible that generating short MM presentations will increasingly come to be part of organisational communication, making it an increasingly authentic form of assessment. It could simply be a fun and creative variant of the group presentation. However, there is an entrenched cultural suspicion of the visual as superficial. The ‘technical skills’ involved may give unfair advantage to some students. Any change process is likely to meet resistance and raise novel and unexpected obstacles to its perception as a fair form of assessment. The paper explores these issues and outlines the investigation of them in the MAIK (Multimedia Assessment In Knowledge) action research project. It discusses in detail justifications for the design of the assessment task in this context, evaluates the success of this structure and reflects on what was learned from the project about the issues.


Introduction
This paper explores the issues with making multimedia (MM) object creation a fair part of assessment for non-technical modules. It is based on findings of an action research project, MAIK (Multimedia Assessment In Knowledge management), funded by the Higher Education Academy for Information and Computer Science (HEA-ICS), at the Department of Information Studies, University of Sheffield, to introduce MM creation as part of assessment on a second level module on Information and Knowledge Management. The paper summarises the case for MM assessment and reflects on some insights and decisions from the project.

Multimedia assessment
Assessment is at the heart of the undergraduate experience. Assessment defines what students regard as important, how they spend their time, and how they come to see themselves as students and then as graduates. (Brown and Knight 1994, 12) *Corresponding author. Email: a.m.cox@sheffield.ac.uk This quotation encapsulates the centrality of assessment in higher education (HE), for students and, indirectly, for lecturers and the institution as a whole. Assessment is key to motivating students and focusing them on what is to be learned. It is an opportunity for feedback. It provides indicators of student and institutional performance and also provides a record of student achievement. Yet Boud (1995) sees it as an area where historically HE has done badly. It can be a 'nightmare' for students (Brown 1997;Race 1995). Several major forces seem to be operating on assessment. On the one hand, is the pressure of numbers, of massification (Gibbs 2006). On the other hand, there is growing disillusion with exams as artificial and unfair forms of assessment (McDowell 1995, 303). Recognition of the need to measure professional competencies, such as group working skills, which cannot be measured by exams, reflects the growing accountability of HE to professional bodies and employers. Yet there is a danger of simply turning HE into a factory for willing employees.
In this context, there are a number of arguments in favour of introducing MM creation for assessment. One is simply that diversifying assessment potentially makes it fairer, by rewarding forms of ability not fully assessed by the usual essay and exam. Ivers and Barron make a strong case for the value of assessment through the creation of MM (2006, 3, 12-15). It is good for assessing collaborative skills, they argue. It represents a new and interesting project management challenge. It tests ability to choose and make effective use of communication mediums. It poses problem-solving and skills in information use. It could also be seen as a valid method of assessing creativity. In the Sheffield context, these outcomes tie quite closely to existing degree programme learning outcomes. For students on an Information Management (IM) programme, it produces a valuable counter-balance to technically orientated approaches to MM. Specific issues around Intellectual Property Rights (IPR) are highly relevant content.
A second argument is around the appearance on the web of a much wider range of user-generated MM content. YouTube is probably the most prominent example, but there have also emerged a range of MM creation and sharing sites such as slide.com (http://www.slide.com), as well as applications within Facebook. Podcasting and videocasting are seeing increasing take-up (Rüdel 2006). Arguably, young people, especially students are at the forefront of this wave of user content creation. Much of the material being shared on such sites is of relatively low technical quality, yet it is now seen as acceptable for public sharing. This suggests there might have been a shift in standards of what is acceptable, further permitting everyday MM creation, where the focus is on the idea not technical quality. According to Pew Internet Survey in 2007, 34% of American males between 18 and 29 actually preferred amateur over professional content (Madden 2007). Given the trend for technologies tried on the web to be quickly adopted in the corporate world, there is reason to think that student equipped with MM creation skills may have an advantage in seeking employment. For the world of practice it may test more authentically useful skills than are evaluated by the ability to write an essay or report.
Thirdly, there is an argument about the validity of assessment. If one tests book knowledge of a topic with an essay, there is a suspicion that even the best students are merely rewriting abstract ideas in their own words, reproducing them at an abstract level, without really understanding them deeply or concretely. Assessment methods which require the student to rethink the concepts in concrete ways and in a different form could increase the validity of the assessment and tend to produce an assessment geared to 'engender appropriate, engaged and productive learning activity' (Gibbs 2006, 23-36). Thus challenging students to rethink a model of knowledge sharing as an imaginary role play seems to be a better test of their ability to creatively, deeply understand the concepts they are dealing with, than an essay. Specifically in the MAIK context, as it is related to a module on knowledge management, these processes themselves could be seen as relevant to the subject. Through reflecting on the translation process between book knowledge and MM, students could more fully understand ideas about different types of knowledge and its management: the theoretical and practical, the kinaesthetic, the experiential and knowledge sharing and transformation processes. Given that such processes of handling the diversity of knowledge in corporate contexts are one focus of the module, then asking students to use a Knowledge Management (KM) model to reflect on the process of producing the MM itself adds a useful reflective loop into the activity. Furthermore, it could be argued in the KM context that if workplace knowledge is diverse, a Knowledge Manager needs to communicate in diverse ways that speak to knowledge at those different levels.
Representing knowledge in MM form as well as abstract essay type writing is an opportunity to think about this.
A final subsidiary argument for the idea of students creating MM was that such content would be reusable for future teaching. It would perhaps articulate ideas in forms more immediately understandable to students. Students like MM used in teaching, by getting them to create the content, they are empowered. It also potentially produces rich material for future class room debate.
There are also some reasons to think such novel assessment might not be fair. At the most general level cultural critics of MM would see it as reducing learning to a spectacle, which tends to naturalise knowledge and reduce criticality. As Gabriel has recently argued in relation to teachers' use of PowerPoint evaluating and responding to visual presentations fits the new range of attitudes and skills current in our society, 'which emphasize multi-tasking, discontinuity, visual alertness and semiotic sensitivity as against patient and deep thinking, long periods of concentration and deference to the authority of the text' (2008,268). We may feel that the latter styles of thinking are precisely what we wish to foster in HE. Yet such social change has its positive side, as Gabriel argues: As consumers in a society of spectacle, we are frequently seduced by image. But we also learn to mistrust image, to question and probe it. We develop skills to read and decode, question and ignore, frame and unframe, combine, dismiss and ignore images. (2008,269) The question then becomes how to align these critical practices with those of academia. Arguably, when we are critical about the visual, it is often as a sceptical consumer, wary of being sold something we do not need. How does this relate to academic scepticism? Certainly it could be argued that some ideas are simply better presented visually (2008,265). Further, putting onto the students the role of creating the MM, rather than being just an audience is likely to sidestep some of these issues.
'Changing assessment procedures is often more difficult than the process of assessment itself' warns Brown (1997, 222). Brown's focus is the resistance to change coming from colleagues, but innovation in assessment can be experienced negatively by students (McDowell and Sambell 1999). This can be because of the extra demands made by unfamiliar forms of work. Students are likely to be resistant to change especially if the requirements are not as clear as those applied to traditional assessment tasks, such as essays. Students may suspect that the change has been made to cut costs or save lecturer time. Students may resist the need to be more independent and responsible implied by many forms of novel assessment. McDowell and Sambell identify central concerns that may be critical to success in innovation in assessment, such as carefully managing students workload, providing clear guidelines and paying attention to organisational procedures which may work less well with novel assessment types or rapid turnaround of formative feedback (1999,80). Assessment methods on the degree programme at Sheffield are already quite varied, further diversifying it could potentially undermine the coherence of assessment (Mutch 2002) making it difficult to understand what is being required. But we certainly need to understand what students actually do and think directly, for they may behave differently from how we imagine (Boud 1995, 39).
Equally, some forms of resistance to change might reflect attitudes which are in themselves also obstacles to learning, and so one of the benefits of the MM creation is to challenge them. Students who are operating strategically will be hostile to a change if it alters the rules of the game. It is likely to be more difficult to 'fake good' (Gibbs 2006, 25) in a novel assessment form. Meeting these fears head-on could be beneficial if it leads students to engage with learning rather than trying to play a system (which may or may not exist), particularly if the process allows an open dialogue in which hidden assumptions and rules behind assessment criteria are articulated and evaluated (Haggis 2006). We often talk of 'being critical' without fully unpacking what this criteria really means or teaching how to do it directly. Assessment change and discussion of potential rubrics could help this by initiating a process in which we articulate such hidden aspects of the curriculum. Again, it is easy to claim that assessment by MM is a challenge to be 'creative', but unpacking what creativity is, is itself very complex (Jackson 2005). Is MM creative just because anything to do with the media is creative? Is it creative simply because it involves the assembly of disparate elements (sound, image) -if so how does this relate to academic synthesis? If creativity is about novelty, for whom is the novelty: the individual, the discipline, the world? If what is being assessed cannot be clearly defined and then supported/taught, the danger is that primarily aesthetic or simply arbitrary preferences are smuggled into the assessment process. The involvement of students in the development of assessment criteria and in peer or self-assessment would increase their understanding of existing criteria and help them develop skills of self-reflection and self-evaluation that are in themselves desirable learning outcomes.

The MAIK project
The MAIK project at the Department of Information Studies, University of Sheffield, explores the issues that are discussed above. Funded by HEA-ICS, the project will build the new assessment into an existing module for Level 2 students on Information and Knowledge Management. The research question for the project could be stated thus: How can we introduce and support assessment by MM production in a non-technical module, in such a way as it will be felt to be fair by stakeholders?
The primary stakeholders we were concerned with, in fact, were students and staff. Of course the concept of 'fairness' is itself ambiguous, and exploring different perceptions of this and articulating the issues were a central aspect of the project.
The literature suggests a number of competing principles which assessment should observe (e.g. 2, 4, 15, 16). It should be: (1) valid, i.e. it should assess what is to be learned (the learning outcomes, both at programme and module level); (2) realistic, i.e. reflect real-world activities and challenges; (3) reliable, i.e. produce consistent results; (4) equitable, e.g. in relation to race, gender, disability, etc.; (5) designed to promote quality learning (e.g. deep learning) and sound learning practices; (6) timely; (7) provide good (timely, useful) feedback; (8) motivating; (9) formative; (10) understandable, i.e. have explicit, public procedures and criteria; (11) lead students to active engagement with the assessment criteria, perhaps through self or peer assessment; (12) integrated into the whole learning experience; (13) appropriately diverse, to meet different learning styles and demands of different topics and to avoid testing the same thing repeatedly; (14) practical and manageable for students and staff, e.g. not excessive in quantity or overly complex; (15) productive of a record of competencies; and (16) able to limit plagiarism.
Since there are competing criteria of assessment and since assessment is for the benefit of multiple stakeholders, no assessment can be ideal. Student choice, for example, although it may increase motivation and learning appropriate to the individual, reduces comparability. We devised a form based on the 16 dimensions listed above, and used it as a way to capture our changing view of how well the module was going in a structured and systematic way.
From McDowell's work a central concern was the change management process. We saw the project revolving around opening up a dialogue with students around the notion of what is fair assessment, and therefore it seemed appropriate to adopt an action research model, in which iterative stages of intervention are observed and reflected upon. The following data collection process is planned: • December and January: Cycle 1. Literature review of use of MM in assessment.
The development of models and templates. • February to Easter: Cycle 2. Delivery of models in class. • April to mid May: Cycle 3. Students develop their own MM. • Second half of May: Cycle 4. Use of student material in classroom discussion. • June: Cycle 5. Summative feedback by students and staff. • July: Report writing.

The assessment tasks
One critical question was how to scope the assessment task such that it should be clear that an intellectual response, not simply visual rhetoric, was being required. We decided that this was effectively conveyed by setting up the assessment clearly in academic language. Thus the group assignment asked them to create: One multimedia object presentation (for example, an YouTube style video presentation OR a PhotoStory OR a PowerPoint animation) with the duration of 3 minutes on the theme: 'According to many KM authors, managing the issues involved in the conversion of tacit knowledge into explicit knowledge lies at the core of Knowledge Management practices (Schultze and Stabell, 2004, 551). Discuss this claim'. (30%) The explicit construction of the question in familiar academic rhetoric helped to clarify that the focus was on intellectual content, rather than technical difficulty. The message was reinforced by reference to an existing standard marking rubric. The assessment guidelines stated: This project is set as an open activity and ultimately, it is up to you to decide how you would want to organise, develop and present your argument, as we welcome different perspectives and viewpoints. You can accept or refute the statement above.
Given that the focus was intellectual not technical, we decided that there needed to be a balance between assessment of the plans and initial ideas with the final product. So students were assessed on four elements: (1) group MM (30%); (2) group plans and storyboards (10%); (3) individual reflection (10%): a reflection on the process of developing the argument presented in the multimedia object, from a knowledge sharing perspective; you can use any of the knowledge sharing models covered in the module or another framework (for example, Kolb's learning cycle) (300 words); and (4) individual discussion (20%): a discussion of the argument you have presented in the multimedia project against both the literature and the case studies (800 words).
This made the assessment quite complex, but the structure clear. There was a careful balance between the created object as realised, original ideas in the plan, individual reflection and a more academic discourse in justification/debate with the MM as a statement. In the context, the balance of individual and group work, of knowledge arising from reflection and book knowledge, mirrors itself the complexity of knowledge, that is inevitably salient in a KM module, and central to the very nature of the topic. Asking the students to frame their reflection within an existing model adds a layer of learning. Assessing the group plans introduced an area where there could be formative assessment. The reflective piece could be written in the period between the date of presentation of the MM which was Week 10 and the final hand in date, Week 12, so allowing a genuine period for the student to digest the experience and respond to anything that arose at the point at which the MM were presented.
We offered support to students to follow three possible technical paths: (1) a video; (2) MS PhotoStory -a free download from Microsoft PhotoStory allows users to produce a movie from photos with sound files; and (3) MS PowerPoint animated model.
Video is quite difficult to handle because it generates very large files and is difficult to edit; we thought the controlled creation process of MS PhotoStory might be easier for novices to handle. PowerPoint offered the opportunity to produce a nonlinear MM, if the topic invited it. As a familiar tool, students might feel happier with PowerPoint, though we were asking them to do more than simply produce the standard presentation. At the beginning of the module we asked students to complete a questionnaire to audit their relevant skills and attitudes. Part of this concerned forms of assessment and how fair they were perceived to be. The survey showed that all forms of assessment listed (exams, essays, business reports, portfolios, group presentations, IT projects, posters and reflective writing) were perceived to be fair or very fair. Interestingly, group presentations were perceived to be least fair; less fair than exams. Essays, business reports and portfolios were seen as most fair. The sample is small (19 returns, representing about half of the class and probably mostly the more successful students) but it perhaps suggests that there is less dissatisfaction with assessment than is sometimes claimed. The bad experiences with assessment that were elicited tended to revolve around unreliable group members. As an alternative to group presentations, MM might potentially address a general concern students had with assessment by this means, since there was a more structured process of collaboration. It also addresses a concern some students expressed through the questionnaire with producing 'usable products' i.e. realistic assessment.
The questionnaire also reviewed students' relevant skills and knowledge. Rating themselves on a six-point scale derived from TFPL's (2005) levels of knowledge, from novice to excellent (also used in our Personal Development Planning (PDP) process) students acknowledged relatively little pre-existing knowledge of MM creation. They scored themselves an average of between 1.5 and 2.5 for storyboarding, video production, recording sound, editing sound/video, MM web pages, flash, animations and immersive environments. Students rated themselves as learners for photography, and at, on average, 'practitioner' level (beginning to demonstrate competence, but requires support) for PowerPoint. Only two of the 19 students said they had ever uploaded content to YouTube, though 13 of the 19 acknowledged using it weekly. Thus the group did not seem to have a high level of MM skill for the module, perhaps contrary to the claims often made for 'The Millennial Generation', especially given the relatively IT orientated character of the students' backgrounds.
Examples of the type of output we expected were offered in the assessment guidelines: • a role play of an example of this conversion or a refutation that it can occur; • a PhotoStory which shows key KM processes; • an animated KM model or contrasting models (e.g. Nonaka and Takeuchi (1995) and Brown and Duguid (2001)) and implications for management; • an attempt to explore different views of what tacit and explicit knowledge are and therefore what the notion of conversion would involve and implications for management; and • a discussion of tacit and explicit knowledge against some other model of different types of knowledge and implications for management.
A key part of the support process was to produce three examples, one for each technical option. These were reused through the course. We designed three models for students to emulate or improve on: a PowerPoint animation illustrated the well-known Socialisation-Externalisation-Combination-Internalisation (SECI) model of Nonaka and Takeuchi (1995). A PhotoStory explored the role of physical space in knowledge sharing. A video based on interviews with colleagues in the department explored different technology preferences to support informal knowledge sharing. The lead in creating the MM was taken by one of the two non-technical staff on the project (Cox), putting the tutors in the role that the students were to take. The process allowed us to directly experience some of the processes involved in content creation. Certainly it drove home the importance of the pacing of presentation and sound to MM as key elements. Thus it was at this point we realised the value of adding sound effects to some photos in PhotoStory to reinforce points and choosing appropriate music, to set the tenor of the presentation or signal shifts in pace. In the creation process there were often unexpected synergies where images and sounds reinforced each other or where qualities of found images or sound added an unexpected useful layer of meaning to the MM. Also immediately relevant, this creation process by the staff surfaced differences of interpretation, visual preference and evaluation criteria which point to the complexities around the evaluation of visual material.
The students' creation of the MM and its evaluation was supported through: • Week 1 -a workshop in which they are introduced to the whole module assessment and reflect on criteria and what they think their support needs are; • (After spending Weeks 2 to 5 on other work) Week 6 -a workshop at which they brainstorm ideas and review some marking rubrics. In part of this session we invited students to imagine they were directors of a Hollywood movie about Information Management, with hints about choice of genre, possible settings, models in famous films. They were to imagine a key scene in the movie. The idea of the exercise was to prompt students to think about how to convert their ideas into MM form, perhaps by embodying the academic dilemma in the question into a specific case or example. It was evident that students really struggled to make the adjustment in thinking. This was a key moment at which we set them thinking in a different way; • Week 7 -a session introducing the use of the three software packages in the project; • Week 8 -storyboarding and planning activities; • Week 9 -a session assembling MM; and • Week 10 -two sessions at which all the MM produced was shown.
We adopted an adapted version of Ivers and Barron's (2006) model Decide-Design-Develop-Evaluate as a framework for development.
Intellectual Property Rights issues were a central concern. On the one hand, we would consider a layer of discussion about IPR as highly relevant to the discipline, so an appropriate focus for learning in itself. Reuse of material created by the students would be vastly easier if students were restricted to the use of copyright-free material (e.g. student authored, with appropriate releases for photographs including human subjects or use of content under creative commons attribution licences). On the other hand, the time-consuming character of authoring all content, such as music or certain images, might radically restrict the creativity of the project. We considered that for educational uses and a one-time showing, we could allow students simply to use material that came to hand. This would significantly reduce the possibilities of reuse of material; however, that was obviously secondary to their using the tools to learn. A compromise position would be to give some added marks for projects that had avoided use of copyrighted material. We certainly encouraged students to use copyright-free material and made available a release form if they wanted to film or photograph human subjects. We did offer a session on copyright-free material that was available from sites such as Flickr (http://www.flickr.com/creativecommons/) and Freesound (http://freesound.iua.upf.edu/). Students had to give full sources for any material they used.
We decided not to involve students directly in the actual marking process, but we consulted them iteratively about what they thought fair criteria should be. The thinking behind this was that the key aspect of acceptance of the assessment as fair would be clarity and transparency. Thus, in the first week of the module there was a workshop for students to discuss the assessment criteria. At this we asked students to think about what criteria they thought should be applied to their MM work, and to pick out the most important criteria. The five student groups picked out the following key criteria: 'clarity of answer', 'present a clear argument', 'clarity to the point', 'quality of content' and 'content'. This showed an immediate understanding that we were looking for content and traditional academic argumentation, not glossy show or technical wizardry. Other criteria identified were: 'enthusiasm for topic', 'references', 'appropriate media'.
At the beginning of the MM creation process we used a second workshop for students to discuss assessment criteria again. We provided various materials for them to use to consider what they thought were appropriate criteria, including MM rubrics from University of Wisconsin (http://www.uwstout.edu/soe/profdev/rubrics.shtml) and a pre-existing departmental evaluation rubric for standard group presentations. We asked them to adapt the latter by deleting irrelevant criteria and adding new ones. What emerged from this process was agreement that content criteria such as Objectives/Introduction, Structure, Arguments, Focus on the question, Knowledge of underlying concepts and practice, Ending remained highly relevant -although some students were unclear about the need for structure in MM. It was the areas previously defined as 'Delivery' and 'Visual aids' that needed to be changed. Thus Timing was retained, but Voice, Eye contact and Audience engagement and Position/stance were dropped. We added instead a section 'Multimedia content', including Images, Audio, Text, Pacing, Timing, MM sources cited as separate headings. From this we developed the rubric that we used for feedback. Students were subsequently given the chance to evaluate one of the examples we had created using the rubric. We were careful to emphasise that the MM element of the assessment was only part of the whole package of assessment and discussed what criteria should be applied to the other elements.

Student work and module evaluation
The quality of student work was very high and produced well-argued and creative responses to the set question. Several groups engaged deeply with the topic generating well thought through illustrations of KM processes using their own examples, such as of football and pulling a pint. These showed more understanding than simply rehearsing the usual examples found in text books. Humour was used very effectively to engage the audience. One group chose some amazingly powerful imagery. 'Found' pictures, identified through Flickr or Google images, often have extraneous elements, but this group had either found just perfect images or cropped what they found very thoughtfully. A central image of a spiral was used repetitively to great effect implying the iterative working through processes in a particular KM model. Panning across the images in the same direction created a sense of forward motion. Text was brief and punchy, so that it could be quickly read on screen and the engaging pace in the presentation was not lost. Music was well used to reinforce the pace and mood and engage the viewer, but was also combined with sound effects. This was a powerful piece of work, but we hesitated over marking it and wondered whether it was too descriptive, that its persuasiveness was more rhetoric than substance. Other groups managed to develop a quite complex piece of argumentation within the three minutes, comparing several interpretations of the same examples, in just as powerful a way. Another group contrasted many detailed examples that confirm the centrality of externalisation in KM, to six or seven contrasting opinions from the literature, specifically drawing on Brown and Duguid's (2001) alternative conception of the 'generative dance between knowledge and knowing'. This illustrates that there were a range of possible good answers within the framework we offered. The MM were quite diverse. One relied heavily on a lucid spoken commentary that explained what was being seen on screen. Others tended to have a strong driving musical theme. One (which used flash) relied heavily on iconic representations, most others used photographs of real people and spaces. Certainly MM, even within the constraints of three minutes, could be used to demonstrate critical understanding of the substantive intellectual issues. Gabriel (2008) emphasises the importance of students' empowerment through the skills to analyse the visual world critically. We did have a session on visual analysis, but like Gabriel consider that students already have highly developed critical visual skills. Our focus was empowering students by encouraging their abilities in creating MM. This would have been a dubious undertaking if the objective had been solely to manufacture more glossy imagery, selling KM. Rather, because the assignment was to create a critical discussion, we were challenging students to create rationally persuasive MM, itself undercutting the thrust of most mass media productions.
Of the other elements of assessment the more traditional short individual essays were generally done well, showing students ability to complement the MM with a more developed argument and detailed references. The reflective pieces were perhaps less successful. Most students either used the SECI model or Kolb's (1984) Learning Cycle to explore the character of the group work. They tended to try and present the whole process of MM creation as neatly fitting a single iteration of the model process, rather than reflect on what actually happened using the model as a lens to better understand both the process and the model. Perhaps this suggests that critical reflection is a harder skill to understand than MM creation. We had given relatively little support in understanding this element of assessment; indeed had not provided any explicit criteria of evaluation.
In the sessions at which the MM was presented, in addition to the standard departmental module evaluation questionnaires, we distributed evaluation forms specifically about the MM and also discussed with students how they felt this part of the assessment had been (Table 1). Students agreed that the assessment exercise was interesting, that it had helped them to understand KM better and that it was not excessively demanding. Despite the exercises with the assessment criteria there was least agreement that the criteria had been clear. So although students were prepared to agree that the assessment was fair in abstract (Questions 1,9), there was more doubt about whether the criteria had been clear, in fact about a quarter of students felt that they had not been clear (Question 7). The evaluation form offered students four suggestions of what had been the hardest part of the MM creation, namely, structure, sound, images and responding to the quotation. Interestingly, structure was generally chosen as the hardest aspect. Sound which had not been done very well in many MM, was second.
There were two recurrent themes in answers to the question: 'state up to three things in the assessment that worked for you'. Group work and the non-traditional assessment task were both mentioned in about half the responses. Students wrote: • working in groups helped idea transfer a lot more. Can build on others' point of view; • working within a group; having a practical, creative task; • using images as we took our own photos; creating an animated project; was something different from normal assessments; Overall assessment (12) The assessment covers most aspects of the module 19% 52% 22% 7% 0% (13) All elements of the assessment fit well together 11% 52% 26% 11% 0% (14) The assessment is too demanding 0% 7% 44% 37% 11% Note: There was one missing return for each of Questions (2), (5) and (11b) and figures are rounded so do not add up to 100%.
• the use of a presentation rather than just writing an essay; the presentation being MM rather than simply PowerPoint; and • working with a good group; using pictures to express a viewpoint; making the structure flow.
This suggests that the value of the tightly focused group work was appreciated. Comments reflected certain group work problems too, such as the fact that only one person could edit the MM file at once. Inevitably where groups had not gelled there continued to be some complaints about the fairness of group work.
Other students commented on pleasure in being creative and expressing themselves by visual means. As regards elements that worked less well, there were far fewer comments, but the main one related to aspects of the clarity of assessment criteria, reinforcing the results in Table 1. For example, it was pointed out, rightly, that we had not provided criteria of assessment for the storyboard. Several students suggested reducing the number of technical options, to make it easier to focus and others asked for it to be made easier to insert music into their work.
Without comparative data, we get a sense of a number of important small problems that need to be corrected, but that overall the assessment exercises were enjoyed and seen as fair.

Discussion
From the formal evaluation and from informal discussions and observations it was apparent that the students enjoyed the work. The quality of the best work was very striking. The exercise had made them think hard about how to apply a key KM model in practice. Furthermore, looking at what students produced, the fear that an MM assessment might produce merely glossy superficial work was unjustified, probably for four reasons. Firstly, the question posed in the assignment immediately put the work in an academic framework, indeed the poorer work was all overly textual rather than overly 'spectacular'. Secondly, we did not rely on the MM as assessment, the other elements required other forms of engagement: on the substantial intellectual question, planning and reflection on process. Thirdly, rather than a superficial engagement, the act of creating MM, especially a short piece, tends to involve iterative and minute viewing of the object. This is perhaps akin more to the 'long periods of concentration' which Gabriel identifies with academic values, than the multi-tasking and discontinuity he suggests are typical of modern life. Whereas a group presentation would probably be a loose assemblage of individual members' work, the concentration of group effort into three minutes of MM increased the need to analyse, evaluate and synthesise ideas and make choices. Achievement in these academically important criteria may not be as easily recognisable as they are in the familiar form of an essay, but this itself is relevant in that information studies is not only an academic subject but also a practical one, where skill in thinking needs to be demonstrated in action, not just in essay or even report writing. Fourthly, it could be argued that there is nothing inherently superficial about 'the visual', especially when the students are active in themselves creating it and the visual is integrated with aural and textual and even kinaesthetic processes. Creating MM requires a multimodal engagement which is itself quite challenging (http://www.vark-learn.com/english/index.asp).
Although we made great efforts to clarify and discuss the assessment criteria with students, especially for the MM itself, inevitably there was a degree of emergence in what the standard was. Ironically it was some of the best students who complained most that the assessment had not been fair in evaluation forms and informal feedback. One group felt that the sound element was irrelevant. The gist of their comments in the evaluation forms was that 'KM is nothing to do with pop music'. Another group were annoyed by seeing that peers had gone significantly over the allotted time, whereas they had limited themselves to exactly three minutes. They felt this had restricted what they had been able to say, though in reality they had been rather successful in putting over a complex argument. Another group felt that it had not been clear that two-sided argumentation was more important than a flow of persuasive imagery. In reality, all these groups did excellent work and were given high marks. Yet they all may have come away feeling that the assessment had not been entirely fair. Again, groups that started the work early (the better students generally) may have been frustrated by the way that we fed further information about what we were expecting and fuller explanation over time, with much of the material being offered late on, primarily to satisfy our perception that students leave work to the last moment. With more familiar assessment tasks students would have had more control over when to start the work. To a certain extent it may be that the complaints surfaced only because, unusually, students saw each other's work (in itself generally advantageous) and because on this occasion we were explicitly asking them to say whether it felt fair. If it had been an essay, we would probably not even have canvassed their opinions, so any dissatisfaction would have been latent. In this sense we were much more engaged with the students in their assessment task. Nevertheless, it has to be conceded that with a new assessment mode, everyone involved may be significantly less clear how the criteria will play out, what will feel like good work. Next year we can show and talk about what was produced this year as a way to establish the criteria much more clearly, for ourselves and the students, but in the first iteration change introduces a level of uncertainty which is unfair.
It is important to ask if good students had some complaints, did the activities engage the less committed, less academically successful students? For if the value of diversifying assessment is to allow students with a different, non-academic skill set to shine, one would hope that students with poor marks for essay writing might do particularly well in an MM exercise. If the point is to develop practical skills and reduce the stress on arguably less relevant essay writing skills, again one would expect the weaker students to be more motivated than by other work. Broadly, however, it seemed that the students with better academic track records continued to do better on this assignment as well.
There remains a danger of assessment being too diverse and not managed strategically within a degree programme. We need to work on spreading related challenges through the curriculum, e.g. we are considering adding a digital storytelling exercise into a Level 1 module.
Another difficulty for us was to fully anticipate the detailed way the requirements of the new assessment task might reconfigure study and group practices. More specifically we perhaps did not fully anticipate the way that the choice of MM tool did subtly influence the group working and learning process. More complex use of sound was possible with PhotoStory and especially Windows Movie Maker, whereas students mostly struggled to effectively embed sound files in PowerPoint (an issue reflected both in evaluation forms and from our direct observation). Those who used PowerPoint tended to have much more text, usually too much text, whereas the restricted text options in the other tools helped students produce readable MM. There is a case for offering only one technical path to students; again an idea suggested by one or two of the students themselves. One group used Flash, and we speculate that this made group work more difficult because it put a lot of power into the hands of the one group member who could use it. Yet from our observations in practical classes, in most cases it seemed that one group member at a time would be assembling the MM, often with others gathered round putting in comments or searching for material. We suspect this subtly altered group dynamics, compared to practices associated with the traditional joint presentation, perhaps leading to more discussion and reflection. Students certainly acknowledged that the assessment had set new group work challenges (see Question 5 in Table 1). Investigating this may be a focus for future research. On a technical level, PowerPoint files being smaller were easier to share for distributed work, whereas the massive project files for the other tools were difficult to share.
One of the explicit intentions in the project was to reconfigure the assessment on the module with the minimum expenditure of effort, to attempt to demonstrate the ease with which MM creation could be brought into the curriculum. The original idea was merely to replace an existing group exercise of producing a diagrammatic representation of a case study, with the creation of an MM object. In reality, looking back, the module was fundamentally changed through the project. Certainly the assessment was radically altered: so that it had become a series of shorter, quite small pieces of work, staged at three points in the semester. It was a diverse mixture of MM and planning documents, reflection and short essays. New also was that students were able to see some of each other's assessed work, both in a workshop and online, before the final assessment elements were complete. Students were involved in developing assessment criteria. We also found ourselves doing group marking of the final MM because we felt that as individuals we did not have a full grasp of the mix of technical and academic criteria to be applied. This was interesting and fun but time-consuming. So the assessment regime had changed radically. The module itself had changed, with a reordering of material in the semester and a shift in the balance of lectures to workshops. This illustrates how thinking hard about assessment tightens up the alignment of planned learning outcomes, activities and assessment. This was certainly in itself a useful exercise in making the module more coherent, but it was time-consuming. Introducing MM is not a time saver.
Our understanding also of what 'technical' meant in this context altered. At the beginning of the project we felt that the point was not to focus on the IT skills needed to create the MM but to focus more on ideas and plans even if they were not executed perfectly. In reality, the tools are mostly fairly easy to use, although there was some sort of learning curve with new tools like PhotoStory, even if their use trades on familiar Windows conventions. The main difficulty, however, lay in other technical areas: more in the area of media skills. For example, the effective use of images implies selection of imagery which is simple and has appropriate connotations (in semiotic terms), iterative use of the same images or visual features and the choice of a relatively limited colour palette. In MM we recognised that pacing was key and had it as an explicit criteria, but on reflection what this meant was not necessarily clearly articulated, beyond the simple need to put text on the screen long enough for it to be read. We could have talked about the need to establish a rhythm (e.g. through repetition, perhaps reinforced with music) but also to make changes of pace. We did not teach these skills, so was it fair to evaluate the final work against such criteria?

Conclusion
Overall, although the visual presentations produced by the students might be criticised for being linear, relying heavily on visual metaphor and connotation and are in some sense 'superficial', in practice most of the work did demonstrate traditional academic values of analysis, synthesis and criticality. The worst assignments were not very sound academically, but they worked even less well as MM. In the better work, there was good evidence of critical evaluation of pre-existing intellectual models and synthesis of ideas. Fundamentally the exercise was aligned with academic values because here the students are the creators of MM not consumers. In addition, the overall package of assessment contained substantial elements couched in traditional academic forms, e.g. short essays and reflective pieces. In this sense introducing MM seems to be able to be a valid form of assessment. Given the increasing prominence of MM arising from the arrival of pervasive broadband internet this seems also to be authentic assessment challenge. Students appeared to be highly engaged in MM. There was little evidence that the assessment was not equitable. Copyright remains a large stumbling block. To oblige students to rely solely on self-created imagery and sounds or even copyright-free material would vastly increase the effort required and probably restrict the creativity of work. Allowing them to use any found material creates a copyright problem.
More challenging are the impacts of simply managing change towards an unfamiliar form of assessment. The precise scope and character of changes arising from a seemingly simple alteration are difficult to predict and manage. Assessment criteria are inevitably less clear, and there was evidence of anxiety around this, especially among students who usually got good marks; certainly as teachers we heard more of the anxiety expressed directly to us. Subtle reconfigurations occur in learning and teaching processes. The learning process is vulnerable to small unexpected technical issues, such as the difficulty of sharing work because of file sizes. It is timeconsuming to reconfigure other teaching material. Nevertheless, the change process had its own value, of increasing engagement with students and each other as teachers, and increasing transparency of assessment criteria. We were rewarded by the quality and interest of material produced, much of which could potentially be reused with future students, and the increased depth of engagement. Although McDowell and Sambell (1999) rightly identify the change process itself as a key obstacle to innovation in assessment, if the issue is addressed directly, it can also be an enabler. We are convinced that the dialogue about assessment criteria led to a greater insight by students into the whole assessment process in general.

Notes on contributors
Andrew M. Cox is a lecturer in information management. His main areas of research interest are internet studies, especially online communities, communities of practice and networks.
Ana Cristina Vasconcelos is a lecturer in corporate information management and programme coordinator for the MSc in information management. Her main research interests are in the area of the interface between information systems development and the management of knowledge and information, with a focus on discourse and identity.
Peter Holdridge is responsible for the creation, implementation and management of all departmental online teaching resources and teaching-related technical facilities, including the provision of teaching, support and training to staff/students in the use of specialist taught course and VLE-related hardware and software. His research interests are in educational informatics.