1. Introduction

1.1 The Scope of this Report

1.2 Motto

1.3 The Runestone Team

1.4 The Runestone Project

1.5 Further Aims with the Runestone Project

1.6 Peer-Teaching

1.7 The Pilot Study

2. The Group Project

3. Data Collection

5. Supervison and assessment

6. Researching the Learning/Teaching Experience

1. INTRODUCTION

The international collaboration within the university environment has in the past more or less been reserved for a few researchers. The reduced cost for communication has however made international collaboration a feasible activity for a broader category. This is likely to change even more in the future.

The international collaboration on the student level is in its infancy today, but there are examples. This report will focus on one such experiment; the Runestone project. This project is developing and evaluating the notion of incorporating international group projects into the undergraduate Computer Science curriculum. Runestone adds new dimensions to student teamwork, requiring students to handle collaboration that is remote, cross-cultural, and linguistically challenging. Runestone is a three year project, with the prototype version running in winter 1998 with students at Uppsala University, Sweden, and Grand Valley State University, Michigan, USA. The 1998 pilot study will be followed by a full-scale implementation in 1999 and another in 2000.

The focus of this report is on the student collaboration, but this collaboration is not possible without an international collaboration on the teacher level. This aspect will also be touched on as well as the issue of looking at the project from the perspective of Computer Science Education Research.

Our students will eventually work in a global market; what better preparation can we provide for international collaboration than … international collaboration?

This report will mainly be on the student aspects of the collaboration, but nothing would have happened if it were not for the teachers and researchers involved. The Runestone team collaboration is in some ways functioning as a role model for the whole project. The Runestone team consists of the following group:

• Mats Daniels, project leader, Department of Computer Systems [DoCS], Uppsala University, Uppsala, Sweden.

• Sally Fincher, Computing Laboratory, University of Kent at Canterbury, U.K.. Sally is experienced in many types of project work and is looking at the teacher aspects of the running of the class.
• Mary Last, Computer Science & Information Systems, Grand Valley State University, Michigan, USA. Mary investigates how virtual groups work in her research and is doing that also within the Runestone project.
• Marian Petre, The Open University, Milton Keynes, U.K.. Marian is very experienced concerning usage of technology in education, as well as in evaluation and collaboration issues, and she has a central role in the group that looks at the collaboration between the students.

The project’s primary aim is to introduce real international experience into undergraduate Computer Science education in a way that has value for all participants. Group projects (typically 5-10 students per team, 5-10 weeks per project) are incorporated into courses at Uppsala University and Grand Valley State University. The international project students will collaborate closely with their foreign counterparts using appropriate communications and computing technology to solve a given problem. Because the students come from different specializations within CS, they have different knowledge to contribute to the project. Problems will be designed to cover the spectrum of backgrounds. Runestone’s secondary aim is to identify effective support structures for remote international collaboration, encompassing strategies for communication, management, and technology use. Runestone will evaluate pedagogical and technical solutions for collaboration, will examine the costs, both in time and money, and will investigate how students learn in such a setting and what they learn. This report introduces the Runestone project, describes the support and pedagogic mechanisms used, and presents preliminary observations from the first year.

The Runestone project aims to:

• Give students international contacts and experience with teamwork with people from a foreign culture.
• Give students experience of collaboration with a group having a different educational back-ground.
• Encourage learning through peer-teaching.
• Give students experience with the use of Information Technology in problem solving.
• Prepare students for the possibility of working in a foreign culture.
• Use the foreign experience to aid students in producing a superior product locally.
• Benefit staff by close collaboration with other universities, giving insights to other departments and ideas for new teaching methods.
• Gain experience with use of new techniques in the running of a course.

Another goal is to create a well-organized setting with courses that, after the initially higher start-up costs, run at normal or lower costs. One example of cutting costs without compromising quality is the use of student peer-teaching, which can reduce the demand for staff hours. Another example of cost cutting is that the costs for renewing the course can be distributed across the departments involved.

In carrying out the Runestone project, we will establish results that address the issue of transferability to other departments and institutions. For this reason, the evaluation will aim to distinguish between domain-specific and general lessons, particularly with respect to the impact of international collaboration on group interaction and personal development, the extent of peer-teaching, and the costs of using this form of education. For example, the project shall examine questions such as how much time is spent on becoming acquainted with new techniques for communication and in what ways (if any) using non-native language impairs learning.

Based on anecdotal evidence from our own experience as teachers, we believe that having students explain concepts and solutions to one another is a powerful learning technique. Our conjecture is that there will be plenty of occasions for the students involved with the Runestone project to help each other with activities such as explanations, clarification, sharing knowledge or rehearsal of ideas. Occasions for peer-teaching can be formal or informal. Formal occasions arise when students at site X present information for the students at site Y. Informal occasions include questions that arise during day-to-day e-mail or simple study sessions.

The Runestone project will systematically examine peer-teaching by considering which settings tend to encourage or discourage peer-teaching as well as factors that contribute to the effectiveness of peer-teaching in these situations. One of our hypotheses is that the rather different educational backgrounds of the two sets of students involved in the project will encourage peer-teaching. The differences in backgrounds should motivate the students to articulate their reasoning, rather than assuming that there is mutual tacit understanding between them and their foreign counterparts.

From early January through late March 1998, the Runestone project ran a pilot study, which involved a group of eight students: four in Uppsala and four in Michigan. These students were all in their third or fourth year of university studies. For the Swedish students, the group project was part of a course that started in September, whereas for the Americans it was the major part of a course that started in early January. The problem that was specified for the group project was fairly advanced, involving study areas such as real-time systems, networking, and distributed systems. A major goal of the Runestone project is to examine the influence of the group project and factors in how the project is set up upon what the students learn and how they learn it.

The actual project was to navigate a steel ball through a maze by tilting the maze in two dimensions with stepper motors. The user submits a navigation algorithm, defines a path for the ball to follow, requests the server to execute the algorithm, then waits for access to the game. When the user gains access, the game server resets the ball in the maze, executes the user's navigation algorithm, then provides feedback to the user on the result of the run. Feedback includes information on how the navigation code executed, and a graphical display of the path which the ball traced through the maze. The input to the navigation algorithm is the position of the ball. The output is the rotational positions of the motors as a function of time. Video images of the maze and ball are available from a black and white digital video camera.

This is how the group project [the Brio project] was presented to the students:

• Connectix camera [inputs, outputs & ports]
• Brio game [stepper motors, controlling electronics & inputs]
• Desktop computer [attached mounted camera & Linux and Win95 installed]
• Laptop computer [serial port & Ethernet]
• Client computer [net connection, web browser & Java compiler]

• Linux device driver for Connectix camera
• Apache HTTP server for Linux
• Linux JDK 1.1
• image processing example code

The hardware components available for the projects were located in Uppsala. The central piece consisted of a desktop computer, with a black and white digital video camera attached to its parallel port and an Ethernet connection to a laptop computer (the Swedish students each had access to a personal laptop). One of the laptops was used to communicate with the two rotational stepping motors via its serial port. The camera was permanently mounted over the Brio maze game as was a light source. A second laptop, connected via Ethernet, was used to run as a client computer with a web browser for playing the game.

The software components included a C library of code to read video signals from the parallel port, control camera settings, Motif app (Ximprov) for viewing camera data, experimenting with camera settings, an example C program using camera data (Ximprov), an Apache HTTP server for Linux, and Linux JDK 1.1 with RMI support

The starting point for playing the game (running the maze) was a website. This website had to:

• Display the currently installed maze board.
• Allow user to define a path for the ball to follow.
• Accept user's navigation algorithm to execute.
• Give feedback to user during the run of the maze.

Optional extras were to:

• Provide information to a user on developing navigation algorithms.
• Notify a user about game queue, estimated waiting time, etc..
• Use RMI technology appropriately.
• Display graphical representation of a run, superimposed on selected path.
• Display a full video image of the maze.

The game server needed to be a concurrent system, either multiple processes or multiple threads, and had to:

• Maintain a queue of users who wish to play, insuring mutually exclusive game semantics.
• Accept navigation algorithms and selected paths from clients.
• Provide feedback to client on the success/failure of navigation code.
• Provide data to the client on the ball's movement during a run of the maze.
• Provide a framework in which navigation code executes predictably and safely.
• Be able to reliably reset the ball to the documented starting position.
• Drive the stepper motors via a serial port interface.
• Use priority to schedule the concurrent entities properly.
• Fetch ball position information from the video server and make available to navigation code.

Optional extras were to:

• Provide information to clients about game queue, estimated waiting time, etc..
• Provide a documented framework in which navigation code executes.

The video server had to consist of one or more processes which must:

• Read video frames from the camera as fast as possible.
• Reduce video data to an x,y location of the ball on the maze.
• Make ball position information available via a network connection.

Optional extras were to:

• Provide a grayscale video image of the maze via a network connection.
• Provide as many positions updates per second as possible.

Data in a variety of forms was collected during the pilot study. This section present the different forms used and some of the results. Data collection was carried out throughout the group project and covered all types of interaction between the students except their informal face-to-face meetings (which were covered by the project logs kept by the students). For detailed exposition of these and other techniques see, for example, [Isaac, and Michael, 1989, Gall, Borg, and Gall, 1996, Denzin and Lincoln, 1994)]. This data will be used to make some preliminary observations about what occurred and how to run the collaboration in the coming year.

All students in the Swedish class were asked to complete the Honey and Mumford Learning Styles Questionnaire. This allowed us to link up with research at the Open University which is investigating individual factors affecting success in using technology in learning CS.

The learning style questionnaire was followed up with a questionnaire, also to the whole class, more focused on group and project work. One aim was to better understand how to run group projects for students. It was pointed out that the replies would be treated with strict confidentiality and would have no impact on the assessment for the course. The questionnaire contained roughly four different areas:

• Media The students were asked to state their experience with use of different media for collaboration.
• Collaboration We wanted to know the attitude the students had towards working in groups.
• Language and Culture The students were asked to state how important they thought it was to be able to speak a foreign language well, and how important they thought it was to be able to understand a foreign culture well.
• Personal profile The students were asked to choose attributes that best characterized themselves.

The actual form used and a summary of the answers are given in the appendix

The first meeting between the students on both sides of the Atlantic was via video conference, with both ends recorded on videotape. While we had planned to hold a second video conference with all of the students after the project was over, this meeting was abandoned due to problems in synchronizing schedules.

The following is the instructions we gave the Swedish students before the video conference in order to prepare them for the video conference [the American students got similar instructions]:

• summary of courses studied, number of years of study
• size of department
• history, size, scope of university
• location with respect to cities, dorms, etc
• pictures of campus
• cost of tuition, cost of housing
• how classes typically meet (when and where)
• how many classes are taken at one time

• place of birth and growing up
• age and year in school
• favorite non-school activities (be specific)
• how many classes you are taking, how much you work per week
• what you'd like to be doing in five years time
• your favorite nerdy joke (Carl's partial to MS/Bill Gates jokes ;-) )

The following is the notes taken by Marian Petre at, and after, the video conference:

Each week, the teachers at both sites held a meeting with their local group of students, where they reflected on how the project had gone during that week. The debriefing followed a standard script, see below, but was sufficiently flexible to allow the teachers to immediately explore the students' observations and any new developments. The meetings were audio-recorded. At the end of these meetings, each student filled out a quick, one-page questionnaire about the meeting. The questionnaire asked about the meeting organization and the outcomes (decisions, learning, conflict resolution, clarifications, etc.), as well as about the respondent's satisfaction with the proceedings, both overall and in terms of their own role in the meeting. This material has not been analysed yet.

The whole trans-Atlantic project group held weekly meetings. One was done with the use of speaker phones, but most were done using Internet Relay Chat (IRC). All these meetings were audio recorded and logs of the IRC meetings were collected, and all meetings were concluded with completing a complete short, individual questionnaires. These one-page questionnaires focused on that meeting, asking about the organization of the meeting, the outcomes (decisions, learning, conflict resolution, clarification, etc), and the students’ satisfaction with the proceedings, both overall and in terms of their own role in the meeting. The students were assured that no one involved in the assessment of the project would have access to the tapes.

The questionnaire for the telephone meeting is found below. This meeting did not function well and was cut short. The students felt uneasy, mainly be feeling pressed to say something and at the same time not talk while someone else was.

Much of the student interaction about the project was via electronic mail and documents shared on the Web. All student mail relating to the Runestone project was collected, and the Web site was monitored. All student mail relating to the Runestone project was copied to Marian, who kept an eye on it and will ultimately analyze it for things like decision strands, student roles, evidence of peer-teaching, and culture issues.

The homepage for the Swedish part of the Runestone-project was used for synchronization. It contained information about the project at several levels as well as some useful information about relevant things, e.g. Java. There were several pointer, e.g. to the American homepage, the individual students homepages, and their work diaries [see 3.6] as well as the IRC interaction logs.

The students completed weekly project logs where they kept a daily log of their time on the project, their activities and interactions during that time, and the outcomes. Other students in the Swedish project course, i.e. those in groups that consisted of only Swedish students, were also asked to keep project logs, in order to provide a basis for comparison. (This was not feasible for the rest of the American contingent.)

The following was handed out to set a standard for how to keep the diary. Log blanks were made available both electronically and on paper and the students could use either or both media.

Each teacher kept a journal of their observations, particularly with respect to peer-teaching, culture clashes or developing sensitivities, collaboration, effective or ineffective procedures, and technology issues.

The fundamental question for Runestone is whether – and in what respects – this is a good education form, meaning that:

• The syllabus is covered at least as well as through ‘conventional’ methods.
• The actual time the students spend on the course is related to the ‘allotted’ time.
• The time staff spend on the course is related to the size of the course and is comparable to other ways of delivering the course.
• The cost of running a class is not higher than other forms.
• The course contributes to the personal development of the students.
• The form is motivating to students.

We address each of these points below. Because the pilot study is just that – a pilot study – any observations we make are necessarily limited and preliminary. Moreover, detailed analysis of the data is not yet complete; the comments given below are based on on-going examination of the data and a first-pass, topic-based review of the material, as well as on a more extensive examination of the data generated on the Swedish side.

The coverage of the syllabus is a special case here, because the primary aim of this part of the course in Sweden is to provide experience in the use of concepts covered in the earlier, more theoretical parts. Hence, the completion of the project task is perhaps a better measure. Based on their performance on previous projects, the Swedish students involved in Runestone are strong students. Under normal circumstances, their project would have been predicted to have been among the first completed and best produced by the class. This was not the case here and was, in our opinion, due to difficulties in coordination and synchronization among the students involved.

The project logs of the Swedish students in the international group show that they spent roughly the expected number of hours on the course: the equivalent of three weeks of full time studies, i.e. 120 hours. The American students spent on average somewhat less, i.e. roughly 100 hours, but this is in line with the expectations for the course the American students followed. It is interesting to see how these hours were actually spent, especially compared to each student's individual estimates from the background questionnaire. One question had asked the student to estimate, for courses taken prior to the pilot study, the percent of their total course time they generally spent studying alone and in groups.

For these summary figures, emailing is considered as working alone. Recategorizing emailing as a group activity would make the focus on group work even stronger. The, on the average, lower percentage spent working in a group among the Americans is due to a higher rate of local group work among the Swedes. This is not surprising, because the Swedes knew each other very well before this course. The fourth Swedish student has not yet filled in the report for the last month and is thus not reported here. One of the American students also has an incomplete time log. Swede 2’s reported time included considerable time searching the web for useful information, which is both time consuming and solitary.

Because this course has required new development, staff time spent on the course cannot be considered typical. The greatest development cost was in setting up the project, which is standard overhead for any project course. This offering certainly involved fewer lectures than usual and less involvement from teaching assistants. There were some special costs, for example running the video-conferences and obtaining special hardware for the project. None of the costs was discouraging.

It is too early to say much about the effect of recent project work on students' personal development. It is likely that the project outcomes for the students were not what they would have been had the project been individual or purely local. In either case, the students would have expected to complete the project, and some of them to excel. Hence, we speculate that the outcomes in personal development are likely to be different in kind from those of a ‘conventional’ project. Our experience as teachers suggests that the experience and frustration of working in a relatively large group with unknown persons is likely to be counted as a key lesson in the long term. The students have had to deal with problems that were different, and in many cases more inter-personal, than usual. Each student appeared to reflect on his or her individual responsibility for communication or other problems with the project. For some, insurmountable frustration and failure to complete a project were new experiences.

After the course, both the American and Swedish students talked about lessons in project and time management, ideas for improving the experience included alternative group structures, having more milestones, and better indicators of progress. The American students described a ‘lack of closure’: they knew some parts of theproject worked, but they hadn’t seen it working and didn’t know if it worked. (The Swedish students, on the other hand, were certain that it didn’t.) The students all realized the value of communication skills (including how to conduct a meeting and set an agenda); perhaps the clearest lesson for the students was the need to acknowledge all email and to answer promptly. All of the students rated the project as being more successful in terms of acquiring knowledge and experience than in terms of producing a product.

Three factors enhanced the initial motivation of students in this international group:

• There was a project to do.
• The team was international.
• The international project was part of an ‘experiment’.

In the initial meetings, some students stated that the real challenge was to make the group work as a team, and to demonstrate the viability of the experiment; others cited both the teamwork and the challenge of the project itself. During the project, motivation was neither constant nor evenly distributed; students cited differences in expectations and motivation within the groups as one of the main problems. At times the awkwardness of physical separation and different time zones impaired student motivation and enthusiasm. Nevertheless, seven of the eight students report that they would be willing to participate in such a project again.

Much of the observed frustration can be attributed to discrepancies between the two groups of students, in terms of expectations, sense of urgency, time available, local cohesion (and hence local group dynamics), technical skill, and access to a key, charismatic lecturer (an American working for the year in Uppsala). The American students felt that they were "a step behind all the way". The Swedish students felt that the American students lacked "passion". One American student expressed regret at not being able to contribute to the extent wanted, for the reason that there was too much else (i.e., job and family commitments) going on. The American students perceived the Runestone project as bigger than those they normally undertake; they felt that future international projects should make clear that all students must participate fully in order for the project to succeed.

• Motivation, commitment: The Swedish students perceived the discrepancies of commitment and motivation as the biggest problem; everything else, they believed, would have been surmountable had all the students been working with the same "passion".
• Technical issues: The American students thought that technical issues were the biggest problem, especially writing code for hardware that was located on the other side of the Atlantic. Testing was difficult when the local platforms differed from the target platform.
• Communication: The American students didn’t perceive communication as a problem, while the Swedish students identified communication as one of the biggest problems. All the students were frustrated by slow or lacking responses to email messages and IRC questions. The students cited multiple missed deadlines as a major problem, although they argued that this might not have happened had the communication been really effective.
• Programming language knowledge: All students mentioned that some extra programming language competence (C or Java) would have been desirable.
• Problem definition: All students said that understanding the problem to be solved in the project was something that caused extra work. The American students rarely used opportunities to ask the Swedish-based teachers for clarification; the Swedish students were unable to quickly diagnose the misunderstandings of the American students.
• Single physical version of equipment: There was only one physical version of the equipment, which was located in Sweden. This, and the fact that the teachers most involved in the course were in Sweden, put the American students at a disadvantage. (This will be changed next year.)

None of the students considered the communication media as problematic. We tried a number of different forms of collaboration; IRC and email were the preferred modes of communication. [Guzdial, M., and Turns, J. (1997) and Kehoe, C., Guzdial, M., and Turns, J. (1997)] report experiences with tools that support project-based learning.

• IRC: All of the students felt that IRC was the best way of communicating with their counterparts. It contributed a ‘liveness’ to the communication and often left the students feeling that they had made progress. They appreciated being able to have different conversations going simultaneously and review the log. Other benefits of the IRC were that it allowed time to think before answering and that silence wasn’t awkward as compared to more direct forms of collaboration, e.g. video and audio.
• Email: As one of the students observed: "Email is great – it’s as good as you make it." The American students observed that email would have been their main medium even if the project were American-only.
• Web pages: The students used Web pages to share documents and to keep records. However, they rejected other possible Internet-based communication tools; electronic whiteboards and ‘CUSeeMe’ were discussed but not tried.
• Video-conference: The initial video-conference was not particularly useful, largely because the connection was quite poor. The fact that the students requested a project-end video-conference shows that they found some value in this medium. Experience reported elsewhere [6] indicates that an initial ‘social kick-off’ helped subsequent communication.
• Audio-conference: Provision was made for weekly audio-conferences between the students (using speaker phones). The first try was not useful and no further attempts were made.

Language per se was not a barrier for these students. The Swedish students are highly competent English speakers (with 8-9 years of study and English usage required in many university courses), although they are not necessarily fully confident. The students’ email and IRC logs are full of jokes – but the students expressed low confidence that their jokes were understood. Everyone was fiercely polite.

The students noticed a few cultural differences between the two groups, specifically in these areas:

• Educational background (e.g., lack of knowledge of C and use of functional programming)
• Age (the Swedes being older: 23-24 vs. 20).
• External obligations; the Americans perceiving that they had more job and family obligations, although some of the Swedish students work as consultants, i.e. the groups actually worked under similar conditions.

Nevertheless, the students were emphatic that culture was a non-problem; each group described their counterparts as being "just like" or "pretty much like" them.

The American cohort was a collection of individuals, whereas the Swedish cohort worked in concert as a team. The American students described a sense of working on an individual basis.

While roles were assigned rather late, there was a good international distribution of responsibilities. The groups have recommended that in the future we appoint a student at each site as local project leaders. Designated responsibilities for these two students would include acting as principal liaison and watching for problems within the local cohort or the overall interaction. This monitoring function might catch problems earlier and help to defuse them; for example, this year the Swedish students helped one another with programming and technical difficulties, preventing these factors from becoming problems.

There was relatively little social interaction between the cohorts; the students felt that they didn’t know their counterparts very well, and the project didn’t help them to get to know each other. Some interactions would probably have been more efficient if the participants had known each other better. Social interaction – jokes and talk about personal topics – increased toward the end, during the hectic efforts to make the project fly. Yet, for each of the students, some part of the process or of their counterparts’ actions or interpretations remained mysterious.

Peer-teaching between the cohorts was limited; it was largely related to craftsman skills, e.g., better technical solutions. This may be accounted for by the lack of familiarity between the students and possibly by the nature of the project, which could be sub-divided in a way that avoided the need to learn about what the others were doing. Some of the Swedes reported peer-teaching within the Swedish cohort, but this occurred largely in face-to-face interactions about which no data was collected.

There are a few issues that arose concerning the supervision and assessment in the pilot version. These were the "normal" problems of assessment of groups, for example the difficulties inherent in assessing individuals within a group and the problems of nationally and culturally heterogeneous groups, but there were also some uniquely interesting assessment features.

Firstly, a great deal and increasingly so in the full scale version to run in 1999, most of the interaction between supervisor and students are not face-to-face, so the problems of formative feedback and assessment are enhanced. Secondly, the same piece of work, produced by a single trans-national group, is assessed under two separate assessment systems, one in Sweden and one in the US. It is therefore possible that the same piece of work will get different grades for different students, calling into question the parity and equity of assessment schemes.

There are several ways of running projects of this type. A useful conceptual model of categorisation which we have developed is based on consideration of how similar the local implementations are. The more similar (and therefore the more constrained) the two ends are, the more tightly coupled: the more dissimilar the two ends can be allowed to be (and still work on the same project) the more loosely coupled. This concept of coupling can be applied to many aspects of the work: the necessary equipment, supervision methods, course content, assessment mechanisms, requirements from students (in terms of individual contribution, knowledge background etc.) and even scheduling — whether the component may extend over different time periods at each institution.

Some of the benefits of loose-coupling are:

• Scale. A loosely-coupled project can be used with diverse student populations — for instance a course of 60 could be interacting with a course of 20. Equally, teams could be formed from many institutions — not every team has to be equally matched in terms of nationality.

• Extensibility. A loosely-coupled project can consist of different student requirements, and even tasks. It may be possible to extend a project over several disciplines — perhaps Computing students and those from Business Studies or Management Science.

• Location Independence. With loose-coupling it is not necessary to provide the same level of equipment at each location. Also it is sufficient to provide a local interface to supervision, without the institutional burden of having to furnish the necessary experts and expertise.

The aim of the Runestone project is to be loosely-coupled wherever possible in order to facilitate transferability of the project. However, from the pilot study, experience suggests that some aspects can be more tightly coupled than others. A probable evolution is that once an instance has been trialled, successive years will tend towards tighter coupling, in the interests of efficiency.

The Runestone project component is designed to be fitted into distinct courses. There is no requirement that the courses at separate institutions should "match" in terms of size, assessment practice or technical content, as the linking feature is the project work itself, which is the same for all participating institutions. Because this is common work, students will have to have sufficient background to be able to contribute, but not every student needs to have equal knowledge or expertise in all areas. The intention is that the Runestone model should be easy to implement as a context-independent component into any combination of institutions and local conditions.

Within the pilot project, the institutional contexts were different in a number of respects.

For the second iteration, the project will be run with the full cohort (taking the specified course) from both sites. This will involve circa 25 students from each institution divided into 6-person teams, three from each continent. Neither the task nor the educational settings will change. However, building on findings from the pilot project, the supervisory mechanisms will be modified and this will have an influence on the assessment process. Another improvement will be to duplicate the equipment, so that all students have physical access. Due to the increased numbers, there will be two mazes at each site.

The pilot project was based on a loose-coupling between the separate instantiations of the project. As much as possible there was no constraint on the context at either institution, and as few additional requirements (on staff or for equipment) were imposed. For practical purposes in the given situation, the full-scale version will be based on a more tightly-coupled model.

The supervision model was deliberately designed to be a "lightweight" open system. The fewer costs and constraints that were required of participating institutions, the easier it would be to extend the use of the model to other ideas and settings. Consequently supervision was designed to mirror traditional practices as much as possible and, in the original model, students would have face-to-face access to a supervisor at their site.

Within the pilot project the supervisory set-up was not ideal. The US supervisor did not have the same level of technical knowledge and was not acquainted with the equipment. As the physical equipment was in Sweden, the US students felt disadvantaged and, in fact, obtained indirect supervision from the Swedish supervisor(s) via the Swedish students. Interestingly, the US students were reluctant to contact the Swedish supervisors directly (via e-mail or the like), even though they were well acquainted with one of them.

In the full scale version, the supervisory model will be constrained, to provide equality of contact to supervisors for all students. At each site there is a single person assigned to the project component of the course. They will act in a dual capacity with respect to the students. These capacities will be manifest in the ways in which they interact with the students, either face-to-face, or virtual.

• Virtual Each team will be assigned one of the two supervisors, and their supervisory input will be virtual. This will be implemented by supervisory meetings in "office hours". (Due to the physical locations of the two sites, there is a convenient "window" between the time-zones. However, this might be more of a problem with other partners, eg. Australia). Although this is the primary slot for help, we do expect that this will be supplemented by e-mail contact as necessary.

• Face to face An open weekly meeting will be held at both sites to discuss common problems. Of course, these will be technical, but we also anticipate that issues of staged deliverables, communication and collaboration may be raised.

The virtual supervision is designed to be the only team-specific contact that is available to the students. We are anxious to avoid problems manifest in the pilot project, where only students at one site were able to obtain face-to-face advice, which they then relayed to their counterparts. A positive side-effect is that it provides a mechanism for defining the expected supervisory hours of staff.

The pilot version was, throughout, an example of a loosely coupled project, but this was especially apparent when it came to assessment. Assessment in the pilot version was done completely according to the local standards, by the local academics, and was not part of the collaboration. There was no attempt to influence local practices, systems, standards or quality assurance mechanisms. At both sites (and as is common practice with project work) assessment was the responsibility of the supervisors. For both institutions this included staged deliverables and project documentation; additionally the Swedish students undertook an oral technical presentation.

In this iteration of the project, assessment will still be loosely-coupled in the sense that the US students will be graded on US criteria, and similarly with the Swedish students. However, the practical consequences of the revised supervision mechanism is that each supervisor will be responsible for the assessment of each member of all their teams. This means that each institution will delegate the responsibility for grading some of their students to the other supervisor. Hence, this is a more tightly-coupled scheme since we are reducing Location Independence because knowledge of local grading criteria is required at both locations.

There are several options available for doing assessment in joint projects. There are benefits and drawback with them all. In Runestone a shift from a loosely coupled scheme to a more tight version was an evolutionary response to an increased familiarity with the institutions involved in our project. This makes assessment in this instance easier and more efficient, but will make it harder to expand this project to other institutions.

Building research into teaching has particular value in the current academic climate; this paper considers why and reflects on the particular need for Computer Science education research in its own right. Co-operation between research and teaching is needed in order to understand learning in the Computer Science context. The accelerating convergence of technologies for computing, communications and teaching affords an opportunity to integrate research and teaching objectives in Computer Science education. The paper presents four example projects in which research was tied to changes in teaching, so that general lessons could be drawn from individual experiences.

In many Computer Science (CS) departments, there is a tension – often divisive – between teaching demands and research expectations, and a concomitant failure of communication between research and teaching. Yet understanding learning in context cannot be done without co-operation between the ‘two halves’ (research and teaching), in two forms: technical research (about the content of the discipline) and educational research (about the learning of the discipline). Technical research has an impact on teaching objectives, most effectively if we understand the learning process – through CS educational research – in order to bridge the gap between research outcomes and input to teaching. Making that bridge can improve motivation among ‘research-oriented’ teachers and increase ‘clarity among teaching-oriented teachers. Computer Science educators often claim their teaching to be ‘research-led’, but they typically mean led by CS technical research; by ignoring CS educational research, we impoverish our provision. Computer Science is sufficiently distinctive as a discipline to require that educational research come from ‘within’ the discipline. The accelerating convergence of technologies for computing, communications and teaching affords an opportunity to integrate research and teaching objectives. This paper presents the case for doing so, and then presents some example projects in which this is being done.

The effort to marshal the latest technology and to implement the latest teaching methods to serve educational aims must be balanced with the need to seek out and address questions about which concepts, strategies, and techniques are fundamental to a Computer Science education. The fast pace of technological change poses a double challenge for Computer Science education: developments affect both the subject and the mechanisms of teaching. Educational methods race to keep pace with the opportunities afforded by technology. We must understand ‘what Computing is’ in order to teach it – we must marshal appropriate tools and methods to teach it well – and what we teach will influence what Computer Science becomes. This requires that research looks deeper than merely evaluating implementations, deep enough to examine what changes in teaching practice reveal about underlying issues such as concept acquisition, development of skills and expertise, sources of misconception and superstition, learning processes, the roles of different types of interaction between teachers, students, and materials, and so on. We need to know not just the effect of introducing new technology or methodology, but also the price.

As computers infiltrate virtually every domain, the demand for education in computing concepts and skills increases. Computer Science education has knock-on effects to all other domains, not just to engineering domains. The past several years have seen changes in the academic climate in many countries, characterized roughly by:

• Higher demand but less funding, and hence pressure to find, attract, and take on ever more students, coupled with the difficulty of re-adjusting the distribution of funds between expanding and declining areas;
• The ambiguous status of teaching: whether to provide education or to provide training for work; whether to set standards in a discipline of thinking or to satisfy student ‘customers’;
• Changing modes of study: more re-education, more mature students, more non-majors, more hybrid degrees and study programmes.

Often, the university administration looks to technology as a panacea; the scramble to offer ‘distance education’ courses via the Web is an obvious example.

In Computer Science, the fast pace of change is not just technological, but also intellectual and methodological. The discipline of Computer Science, without a firm traditional underpinning or a firm educational tradition, is buffeted by changing definitions of the domain itself. The academic discipline is characterized by many tensions: between science and engineering; between theory and practice; between training and education. The tensions are exacerbated by the current climate; in the face of income-oriented institutional perspectives, the push to satisfy future employers, the competition for students, and so on, the tensions are a matter of continual debate. Hence, the discipline is characterized by an almost unmanageable diversity:

• academic perspectives: Degrees and courses in Computer Science cover a wide range of goals and values.
• representation systems: Changes in notations and programming paradigms are attended by the need to comprehend and have competence with more than one.
• technical context: Our artefacts must be understood in the current technological context.

This diversity and the pace of change mean that, not only must we provide students with a solid foundation, but we must also equip them for continual learning subsequently.

More fundamentally, the nature of what we study – that our tools are also our objects of study and are also a means of teaching – sets Computer Science apart:

• The objects of study, our artefacts, are abstract and difficult to observe. They are different from physical artefacts (e.g., what does a compiler look like? or an operating system?), although they interact with objects in the physical world.
• Concepts and artefacts encompass many levels of abstraction and complex inter-relations.
• Our artefacts are dynamic; we must reason not just about their properties, but about their behaviour (potentially complex behaviour) in time.

In this context of abstract, difficult to observe, dynamic, interacting objects of study, it is a particular challenge for educators to make theory concrete – without confusing technology with theory.

More fundamentally still, Computer Science is about thinking. The constraints to thinking within the discipline are not physical, but human: our artefacts are constrained primarily by our ability to invent. Hence Computer Science teaching is about what computer scientists have managed to think about so far, and in what manner: algorithms, paradigms, languages, engines, tools, solutions are all thought products. But they are thought products that interact crucially with the physical world, and the relationship between the reasoning discipline of Computer Science and its technology is central to its particular character.

Even so, why should Computer Science educators involve themselves in Computer Science education research? It makes sense for them as members of a discipline: both to inform their practice, and to draw upon their practice to inform discourse in their discipline. Even though not all CS educators need engage in CS education research, all teachers should be aware of the research that pertains to their practice – as part of professional "scholarship".

Given the current conditions, it is especially important to distinguish truth from assumption, to have practice that is well-founded. Evolving teaching practice is normal to good teaching, but evaluation reliant on anecdote is not good enough. Adding a research perspective allows educators to learn more from their practice, e.g., to consider how much of local practice generalizes, and to identify the important parameters governing effectiveness in given situations. It allows educators to combine individual experiences in a meaningful way in order to address bigger issues, e.g., assessing the balance – understanding the trade-off – between practice and theory in courses and programmes.

Moreover, undertaking CS education research makes sense for many educators as individuals, adding a dimension to their practice of teaching. It focusses on questions close to their vocation, and close to their daily work. It allows their investment in teaching to contribute to their research record, one of the principal factors by which they are judged.

Combining research and teaching objectives has a number of advantages: economy of effort, opportunity, credibility, and adding value.

• economy of effort: Building a research agenda into changes in teaching combines two sets of objectives in one activity. Education’s greatest research asset is its students and teachers; moreover, teaching practice is a readily accessible research resource.
• credibility: The evidence which arises from integrated research can provide solid ground in discussions about changes to courses and programmes. Practice-based research can contribute to arguments for support by providing credible evidence of improvement in outcomes. Well-designed studies are better than anecdote because they are documented and repeatable.
• opportunity: Although less controllable, the classroom has some advantages over an education laboratory. It affords the potential for discovering subtle interactions (e.g., contributing factors which are not visible without some change to practice or without in situ observation) and effects over time. Combining objectives brings the opportunity to build the bridges between practice and theory (and theory and practice), and to relate theory to CS education in particular.
• added value: Examining educational practice with a ‘research eye’ and a researcher’s toolkit can add value to changes in practice by providing the quality of documentation necessary for a reliable transfer of experience, providing the basis for establishing the generalizeability of experience, exposing students to research. Hence, educators can see greater outcomes of local changes and efforts.

The key question is how to achieve rigour in the face of human complexity and variability, and subject to the practical constraints of the educational setting. Rigour is plausible when research is viewed as a means of learning (i.e., adding information to the discourse in the community), rather than a means of proving. Scientific research is often not so much a process of getting answers as one of finding even better questions. This view leads to a healthy pragmatism, based on identifying the research question, considering what evidence is sufficient to address it, and accepting that constraints (and consequently trade-offs) are inevitable in the asking. It admits a variety of methods, theories, and accounts.

There may be no single reality to which claims made in research reports correspond;

phenomena are ‘constructed’ not just ‘discovered’. Rarely can one sort of evidence reveal everything that an educator wishes to know about the impact of introducing a change to practice, but less-than-exhaustive evidence may provide what an educator needs to know in order to gain insight about educational roles and value. A single technique reveals something of what’s needed, and a broader examination often relies on a succession of techniques that build a collection of evidence. Research can take place in a variety of formal and informal settings, including classroom and laboratory. It can also

be conducted according to a variety of learning, teaching, and methodological paradigms.

The space of research techniques is large [for a detailed exposition, see, e.g., Isaac and Michael (1989); Denzin and Lincoln (1994); Borg and Gall (1989)], including:

• case studies
• observation (natural tasks)
• think-aloud tasks
• in-depth interviews
• analysis of ‘naturally occurring’ artefacts such as documents and designs
• questionnaires & surveys
• diary studies
• focus groups
• walk-throughs
• constrained tasks
• quasi-experiments
• laboratory experiments

Overcoming the constraints of the educational setting is often a matter of combination: accumulating results from a series of studies; compiling results from studies at different sites; mixing qualitative and quantitative techniques [e.g., Bryman (1988), Brannan (1992)]. Combining methods allows a sort of ‘triangulation’ among multiple perspectives, which can achieve a more complete account. Consolidating results from different sources can improve representativeness and strengthen conclusions, as long as the data arises from cognate studies.

Typical criteria for research are relevance, importance or significance of topic, and contribution to existing knowledge. The value of a research contribution rests on its validity, the extent to which an account accurately represents the phenomena to which it refers. Our aim must be to establish that research results are valid ‘beyond reasonable doubt’: that they are plausible, credible, and supported by well-documented evidence. The value of evidence relies on clear and honest reporting of data that has been collected in a systematic manner using appropriate qualitative and/or quantitative methodologies. The crucial issues for research are whether it is representative, generalizeable, replicable, predictive, honest – and whether it recognizes its own assumptions and considers alternative interpretations or accounts. ‘Good’ evidence is appropriate evidence, i.e., data relevant to the question; ‘bad’ evidence fails to provide relevant information. We learn from ‘failure’ (i.e. refutation of our hypotheses) if the study is well-reported. Flawed evidence (i.e., evidence which is compromised by some limitation of the study) can still be useful, as long as we understand its limitations. A pragmatic approach to rigour relies on researchers knowing the value of different sorts of evidence and using available evidence within its value.

The evaluation component will both assess Runestone in terms of its own aims, and draw from the experience any lessons that enable the Runestone format to be transferred to other departments and institutions. The evaluation will for this reason aim to distinguish between domain-specific and general lessons. Key evaluation issues include: the efficacy of the method – whether students learned and to what extent they collaborated; the cost of using this form of education, both in money and time, both for students and staff, e.g., how much time is spent on becoming acquainted with new techniques for communication; whether (and if so how much) language problems impede progress; which aspects of the set-up contribute to or detract from the collaboration.

The two major aspects of the research component are peer-teaching and (changes in) cultural attitudes. The research will use largely qualitative methods to examine the processes students use in pursuing the project, the patterns of communication among the students and what their communication reveals about their attitudes and understanding.

One hypothesis of the project is that the different educational backgrounds between the two sets of students will promote interaction, because each group will need to draw on the other’s knowledge. Another hypothesis is that peer-teaching will contribute substantially to students’ understanding during the project. There are likely to be plenty of occasions for these students to explain things to each other. The project will examine occurrences of peer-teaching in order to try to determine what characterizes effective occasions (i.e., those in which understanding is improved), whether there are patterns of occurrence, and what factors might be used to promote effective peer-teaching. The anticipation is that peer-teaching will occur both spontaneously and in more formal settings when students in one group teach students in the other about topics not covered at that site.

There are, as described in section 3, several sources for data collection in this pilot study. A short summary:

• There are two weekly meetings that are audio recorded and followed up by a questionnaire. The questionnaires are one-page and focus on that meeting, asking about the organization of the meeting, the outcomes (decisions, learning, conflict resolution, clarification, etc), and the students' satisfaction with the proceedings, both overall and in terms of their own role in the meeting. The first of these meetings is with the students at both sides of the Atlantic, both ends taped. The second meeting is with the teachers during which students and teachers reflect on the course of the project during the week. This 'de-briefing' is loosely structured, following a standard script, but allowing the teachers to respond to developments or observations.
• The students, both the ones involved in the pilot study and the other students at the Swedish side fill in weekly project logs which ask for a daily log of their time on the project, their activities and interactions during that time, and the outcomes.
• The teachers are asked to keep a diary of their observations, making particular note of any evidence of peer-teaching, culture clashes or development of sensitivity, collaboration, effective or ineffective procedures, and technology issues.
• All student mail relating to the Runestone project and IRC logs from the weekly student meetings are collected and copied to a team member in the UK, who will keep an eye on it and ultimately analyze it for things like decision strands, student roles, evidence of peer teaching, and culture issues.
• Video tapes of the initial and final video conferences, as well as entry and exit questionnaires completed by all students.

The analysis of the pilot study will be qualitative and largely data-driven within the key topics. Results from the pilot study will be used to design more focussed research instruments for the subsequent years, facilitating more efficient analysis on a much larger student pool.

No reliable conclusions can be drawn from a small pilot study. However, this trial does suggest changes for the next phase in the Runestone project. Some example changes include having equal resources at the two sites, incorporating a clearer technical and project management briefing in order to achieve a faster and appropriately structured project start, making clearer suggestions about targets and milestones, and recommending a different team management structure.

Other means of collaboration will be investigated. For example, we are intrigued by the "CoWeb" (Collaborative Website) concept from Georgia Tech (see http://pbl.cc.gatech.edu:8080 /myswiki). In a Co Web, any page (including both text and graphics) can be edited by any user.

There is still much to understand about how international collaboration influences the learning process. We hope that the more detailed analysis of the full data will reveal more about factors that affected the nature of the social interactions within the team and will provide examples of peer-teaching opportunities taken and missed. The data collection schemes themselves will be evaluated, with the goal of introducing new methods that are faster to evaluate and more tuned in to this particular project. Scaling the project up to include the whole class in Uppsala next winter appears feasible.

Overall, the pilot study was a qualified success: the technology and interaction were demonstrated to be feasible. However, the project was not quite completed and the students experienced frustration associated with group interaction (particularly with the international interaction). Nevertheless, the students all report that they learned a great deal, and all but one reported that they would volunteer again for an international group project. Interestingly, the frustrations were largely attributable to individual differences (style, personality, commitment, and expectation), and a perceived imbalance of resources (key resources being located only in Sweden), rather than language, technical, or cultural factors. The need for faculty on both sides who know the technical content of the project was apparent, but would be interesting to find ways around this in future offerings.

The Runestone team and the American and Swedish student teams.

The Swedish Council for Renewal of Undergraduate Education, which sponsors the Runestone project.

NyIng.

Name: ____________________________________________________________________

How much have you used the following media?

For each medium, please tick the column which best describes your experience:

How comfortable do you think you will be using the following media to do collaborative work?

Please circle the rating that best applies, from 1 = uncomfortable to 5 = entirely comfortable)

Rank each of the media in terms of how helpful you think it will be in the following activities:

In general, how do you feel about working with other students?

Please tick the one that best applies:

In a typical week, what percentage of your study time (excluding lectures and laboratories) do you:

How often have you worked in a group?

For each activity, please tick the column that applies best to you:

Please name three advantages that you get from working with other students?

1. ________________________________________________________________________

2. ________________________________________________________________________

3. ________________________________________________________________________

Please name three disadvantages that you get from working with other students?

1. ________________________________________________________________________

2. ________________________________________________________________________

3. ________________________________________________________________________

How important do you think it is to be able to speak a foreign language well?

How important do you think it is to be able to understand a foreign culture well?

What do you think you can learn from foreign students that you can’t from students in your own country?

How likely do you think it is that you will learn the sort of things you’ve written above during this project?

For each of the following questions, please answer for group work in general, and international group work in particular (if you have no experience, please speculate):

When you have worked on teams before, which aspect has given you the most satisfaction?

When you work in a group (whether formally or informally), which of the following activities do you find yourself doing?

Please tick the column that best describes your behaviour:

Please describe an effective group:

For each attribute pair, please circle the one that best describes you:

Would you characterise yourself more as:

Why did you – or didn’t you – volunteer to join the international group?

What do you expect to gain from doing this group project?

The students were asked to state their experience with use of different media for collaboration. The results were (The Runestone group students within parenthesis):

• video-conferencing: 13 (4) never, 0 at least once and 0 many times.
• audio-conferencing: 8 (2) never, 5 (2) at least once and 0 many times.
• shared applications (e.g., electronic whiteboards): 6 (1) never, 5 (3) at least once and 1 (0) many times.
• e-mail: 0 never, 0 at least once and 13 (4) many times.
• computer conferences: 6 (1) never, 5 (3) at least once and 1 (0) many times.

We also asked the students to tell how comfortable they were with using the media to do collaborative work? The results were (Min 1 and Max 5, and the Runestone group students within parenthesis):

• video-conferencing: 2,9 (2,7)
• audio-conferencing: 3,3 (3,7)
• shared applications (e.g., electronic whiteboards): 4,1 (4,5)
• e-mail: 4,8 (5,0)
• computer conferences: 3,6 (4,5)

The final question about media was how helpful the students thought it would be in a few named activities, i.e. (Min 1 and Max 5, and the Runestone group students within parenthesis) for each of the five techniques above:

We wanted to know the attitude the students had towards working in groups. They were asked to choose a statement that best suited their attitude (The Runestone group students within parenthesis).

• 9 (3) said that they prefer it
• 3 (0) said that they like it sometimes
• 1 (1) said that they tolerate it – it’s OK
• 0 said that they don’t like it much
• 0 said that they strongly prefer to work alone

We also asked them to consider how they spent a typical week and state how they spent their study time (excluding lectures and laboratories), i.e. how much, in %, they (The Runestone group students within parenthesis)

• spend studying alone 62 (55)
• spend studying with one friend 15 (16)
• spend studying with a group of friends 23 (29)

Next we asked about their past experience with working in a group. They stated how often they had worked in a group to (The Runestone group students within parenthesis).

• do a team project, 5 (0) said at least once and 8 (4) many times
• solve problems (e.g., to work examples), all said they done that many times
• do a laboratory exercise, 2 (1) said at least once and 11 (4) many times

Finally we asked them to name three advantages and disadvantages they would get from working with other students. The suggested advantages were;

• More fun 7 (2)
• Get help when stuck 5 (2)
• Get others view on a problem 5 (2)
• Test ideas 4 (2)
• Work together 4
• Divide load 3 (1)
• Get access to others knowledge 3 (1)
• Easier to be disciplined 2
• Learn by explaining (1)
• Learn more 1
• Training for the future 1
• Less pressure 1
• Get to know persons 1

and when asked specifically about an international group they stated

• Contact with a new culture 6 (1)
• Use a foreign language 3 (1)
• Getting new experiences (2)
• More fun 2
• Learn to communicate 2
• A merit 1
• New ways to work 1

The suggested disadvantages were:

• Hard to schedule 7 (3)
• Slower 4 (1)
• Different commitment to the project 3 (1)
• Have to compromise 3 (1)
• Irrelevant discussions 2 (1)
• Hard to as "stupid" questions (1)
• Misunderstandings 1
• Hard to get credit 1
• Social pressure 1
• Don’t learn all parts 1
• May dislike members 1
• Some work is better done alone 1
• Hard to divide fairly 1

and when asked specifically about an international group they stated

• The long distance communication 3 (1)
• Cultural differences 3
• The time difference (2)
• Hard to divide the work (1)
• Being too different (1)
• No face-to-face meetings 1
• Slower 1
• The language 1

We asked them, which aspect has given them the most satisfaction in their past experience from working on teams, and gave a few examples.

• the social outcomes (making new friends, interacting, and so forth), 1 ticked this option
• the technical outcomes (the job was well done), 1 ticked this option
• both the social and technical outcomes in fairly equal amounts 11 (4) ticked this option
• simply completing the assignment or doing your job, no ticks
• have never really worked on a team, no ticks
• other, please specify, no ticks

The students also stated the activities they normally ended up doing when they worked in a group (whether formally or informally). They picked one that best described their own behavior in all of the following:

• initiating ideas, 9 (3) said about the same as others, 2 said more than others, and 2 (1) said often
• explaining, 1 said less than others, 6 (31) said about the same as others, 5 (3) said more than others, and 1 said often
• resolving differences, 2 said less than others, 7 (2) said about the same as others, 4 (2) said more than others
• asking for information, 2 (1) said less than others, 7 (1) said about the same as others, 2 (1) said more than others, and 2 (1) said often
• listening, (1) said less than others, 4 (3) said about the same as others, 7 said more than others, and 1 said often
• summarizing, 5 (2) said less than others, 4 said about the same as others, 4 (2) said more than others
• taking notes, 6 (2) said less than others, 6 (2) said about the same as others, 1 said more than others
• leading, 4 (1) said less than others, 5 (2) said about the same as others, 3 (1) said more than others, and 1 said often
• doing the work, 8 (2) said about the same as others, 5 (2) said more than others

The Swedish students needed to use a non-native language, and both sets of students would be confronted with a different culture. We asked them to state how important they thought it was to be able to speak a foreign language well. One put down 3 on a scale 1 to 5, all the rest marked 5 (i.e. very important). We also asked how important they thought it was to be able to understand a foreign culture well, where 2 marked 4 and the rest 5 (i.e. very important).

The students chose for each attribute pair below the one that best characterized themselves.

• theoretical 9 (4) or practical 5 (one ticked both options)
• an introvert 4 (2) or an extrovert 8 (2) (one didn’t tick an option)
• a do-er 5 or a thinker 9 (4) (one ticked both options)
• methodical 8 (3) or intuitive 5 (1)
• analytic (breaking things down into parts) 8 (2) or holistic (considering things as-a-whole) 5 (2)
• calm 10 (3) or excitable 3 (1)
• easy-going 4 (2) or stubborn 9 (2)
• tolerant of risks 9 (3) or avoiding risks 4 (1)
• doing things because you want to 8 (2) or doing things because they are expected 5 (2)
• talkative 6 (3) or quiet 9 (1) (two ticked both options)
• enjoying the beginning of a project more than the end 5 (2) or enjoying finishing a project more than beginning one 9 (2) (one ticked both options)

There is no time limit to this questionnaire, it will probably take you less than 10 minutes.

If you agree more than you disagree with a statement tick the item number.
If you disagree more than you agree mark the item with a cross.

Be sure to mark each item (unmarked items will be assumed to be crossed).

THANK YOU for taking time to complete this questionnaire.

If you wish to see which ‘learning style’ your answers reflect, please make a note here.

If you have comments or questions, feel free to contact the research team.

• Runestone was one of the examples in the paper A Smorgasbord of Pedagogical Dishes, that we presented at the ACM SIGCSE 2nd Australasian Conference on Computer Science Education, Melbourne in July 1997.

• Runestone, and especially the peer-teaching aspect, was discussed in a graduate course (Research Methods in Computer Science Education) at Uppsala University in October 1997.

• Runestone, and the aims of Runestone, was presented at a seminar in the seminar series "Research about the university" held by the Pedagogic department at Uppsala University in October 1997.

• Runestone, and especially the technique used, was presented at a work shop for educating leaders at the Technical and Natural Sciences sector at Uppsala University in February 1998.

• Runestone was discussed as a testbed for research about Computer Science Education at the Doctoral consortium at the ACM SIGCSE Computer Science Education Symposium in Atlanta in February 1998.

• Runestone was discussed at a meeting within the AAPS (Animated Aided Problem Solving) group at Helsinki University in March 1998.

• Runestone was the focus in the presentation held at Department of Computer Systems (DoCS) during the visit of the Minister of Education, Carl Tham, in April 1998. The presentation at DoCS was one third of the presentations during the Ministers visit.

• Runestone was used as an example of activities at Uppsala University at a presentation at ABB in April 1998.

• Runestone was discussed at the first CSERGI (Computer Science Education Research Groups International, http://www.docs.uu.se/csergi/) meeting in Uppsala in May 1998.

• Runestone was one of the examples of research done about our teaching in the paper Building a Rigorous Research Agenda into Changes to Teaching, that I presented at the third ACM Australasian conference of Computer Science Education in Brisbane, in July 1998.

• The results during the first year of running Runestone was presented at the second NyIng conference in Linköping in October 1998.

• Runestone, and possible extensions to Runestone, was discussed at the second CSERGI (Computer Science Education Research Groups International, http://www.docs.uu.se/csergi/) meeting in Stockholm in October 1998.

• Runestone was presented at LINKS'98 in Stockholm in October 1998.

• Runestone was the main focus of the paper RUNESTONE, an International Student Collaboration Project that I presented at the IEEE Frontiers in Education Conference in Tempe, Arizona in November 1998.

• Runestone was presented at the symposium about the role of the university in the future held at Uppsala University in December 1998.

• Runestone was presented at a seminar about increased use of IT in the education at Université de la Mediterranée, Marseilles in December 1998.

• Runestone, and especially the supervison aspect, was the theme of my invited presentation at an EPCoS (Effective Projectwork in Computer Science, http://www.cs.ukc.ac.uk/national/EPCOS/) workshop held in Canterbury, UK in January 1999.