How to make your assessment remote in Q4

How to make your assessment remote in Q4

Note: This site will be updated regularly with new material. Please check the site regularly for new insights and step-by-step guides. Current version: 15 june  2020 – 14:07. Click here to see what has changed.

This page contains advice to teachers with on-campus assessment in Q4 and Q5 on the possibilities to change their assessments to remote assessments. This document provides a (limited) overview of feasible and desirable possibilities.

Table of contents

  1. Procedure for changing your assessment to remote
  2. Available remote assessment types
  3. Available remote assessment tools
  4. Available fraud prevention options
  5. Assessment tips from Q3 lecturers to Q4 lecturers
  6. Guiding principles for assessment
  7. Page history

Links to specific manuals on assessment

1. Procedure for changing your assessment to remote

Planned assessment that required physical presence at the campus cannot take place in Q3 or Q4.
There are a lot of options for a remote assessment. Go to section  2. Available online assessment tools to learn more.

Important: Before you decide anything, please be aware that you need to consult your programme director and the Board of Examiners!

You cannot change the assessment of a course without consultation of your programme director and approval of your Board of Examiners. The latter has sent you information on their procedure. Here is a general representation of the procedure:Figure 1: procedure to change assessment of your course.

Procedure to follow:

  1. Consult your programme director
  2. Decide the form of the alternative assessment you want to use
  3. File a request to the applicable Board of Examiners according to the procedure of your Board of Examiners.
  4. Wait for approval from the Board of Examiners!
  5. Inform your students via email and a Brightspace announcement.
  6. Check the sites How to make your assessment remote in Q4 and Remote Teaching & Learning: Quick Guide – Brightspace Support regularly for new tools, practical teacher guides and new insights.
  7. Start redesigning your assessment

The main criteria which Board of Examiners will use to decide for an alternative assessment are listed in this drop down menu:

    1. Learning objectives need to be assessed sufficiently. If one learning objective cannot be assessed properly, dispensation may be given if that objective is tested elsewhere in the programme.
    2. The alternative assessment must be feasible for students in terms of preparation time and (technical) means, and have as few disadvantages for students as possible, for example in terms of inclusiveness and study delay.
    3. There should be sufficient safeguards against fraud.
    4. Students need to be informed as soon as possible, of the alternative assessment and procedure− but only after approval from the Board of Examiners.
    5. Examinations outside of the regular examination period can be considered. This could prevent exams piling up at the end of Q4, which could lead to overloading the students.
    6. Online proctoring may only be used if there is no alternative available (see here for possibilities of fraud prevention). Some Boards of Examiners do not approve of online proctoring in any case.

In case of (suspicion of) fraud, the Board of Examiners can sanction the student. In the event of serious fraud, the Board of Examiners is entitled to decide to propose to the Executive Board that the student’s enrolment on the degree programme be permanently terminated.

In case of suspicion of large-scale fraud, the Board of Examiners can declare all announced grades invalid, if it is impossible to determine who committed fraud and who did not. Any kind of participation in large-scale fraud is considered a case of serious fraud. In case of serious fraud, ‘the Board of Examiners is entitled to propose to the Executive Board that the student’s enrolment on the degree programme be permanently terminated’. This can be explicitly communicated to the students. Click to learn more about creating fraud resistant exams.

In case the reliability of the grade is not sufficient to express the results as a grade (1-10), the Board of Examiners can allow the examiner to change the grade to a pass/fail decision (fail-pass). Variations on pass/fail are not allowed.

Pass/fails do not count for the students’ average grade (GPA).

If a student passes a course with a 6.0, and wishes to retake the exam with a pass/fail exam, the highest grade counts. Currently, it is not totally clear what the ‘highest grade’ is when comparing a grade (>5.8) with a pass/fail. Regulations are under way that for this particular decision on which grade to keep, any pass grade is considered to be higher than a ‘pass’.

Students with special needs can be entitled to extra time (10 extra minutes per hour), as is stated in the TER (article 25). Adjustment can involve form, timing, aids permitted during testing and location.

There are two practical ways to organise extra time in Brightspace Quizzes and Brightspace Assignments: See here.

 

  1. Give ‘Special Access’ to groups of students. You can find the ‘Special access’ options under the ‘restrictions’ tab.
  2. Set the end-time of the test 10 minutes later per available 60 minutes.
  3. If you have multiple timed quizzes or assignments time-boxed together (i.e. you need the start-times of a single quiz to be the same), there should be time in between to facilitate students with a disability.
  4. Detailed instructions for quizzes can be found here and for assignments here.

If you prefer, you could also create two identical Brightspace Quizzes or Assignments that differ only in duration, split the students into a normal-time group and an extra-time group, and make each quiz available to either of the groups.
In case your students are entitled to other types of adjustments, like audio support, it might be useful to create separate exams, too.

2. Available remote assessment types

Here is a rough schedule that can help you to determine new, remote assessment types.

Frequently used changes of assessment type. For *written exams, there are multiple possibilities, see running text.

In the schedule with frequently chosen changes of assessment type, assignments, projects (including presentations, reports and code) and oral exams usually change into their remote version.

For written exams, it depends very much on the learning objectives, whether it can be changed into an assignment/project (individual or group), a remote oral exam (only for small numbers of students) or ‘simply’ a remote written exam. A combination of remote assessment types is of course also possible (strive for a spread of deadlines and exams for your course and the other courses that students take).

Remote written exams can be administered in two ways:

  1. as downloadable assignments: During the exam time-frame, students download the exam and upload their answers before the deadline, for example 45 minutes later. Exams can be split into 2-4 parts with each their own time window. The answers can for example be typed word-documents or pdfs, or scanned handwritten answers in pdf-format. Uploading movies is also possible (both for the student as well as for the teacher). For a how-to in Brightspace Assignments, see here.
  2. as a (partly) automatically graded digital exam: you and your team can either manually grade longer questions, which allows for partial grading (giving a student for example 2 out of 5 points for partially correct answers), or automatically graded questions with short answers. In the latter case, you would have to go over the incorrect answers to check whether unforeseen answers are nevertheless correct. For a how-to in Brightspace Quizzes, see here. For a webinar or a how-to Brightspace Support document on MapleTA, see here.

Larger projects and assignments can have multiple deliverables and assessment items, like reports, presentations (either live via YouSeeU or Skype for Business, or via a delivered recording in Brightspace Assignments), code, or observed skills.

Depending on your remote assessment type (see above), you can choose an assessment tool and fraud prevention measures.

3. Available remote assessment tools

Depending on your remote assessment type (see above), you can choose an assessment tool.

For summative assessment, you can choose from four unproctored tools and one proctored tool.

Unproctored (no camera surveillance) examination tools can be used for both formative and summative assessment.  Summative assessments should not contain multiple choice questions (including true/false questions) and preferably no knowledge questions. The following tools are available:

Download exam and upload results

  1. Brightspace assignments: you upload the exam as an assignment, which becomes available in the time slot of the exam. Students need to upload their results before the end of the exam. Students can upload handwritten work (preferably in pdf, so you can add comments), or text (Word/pdf/etc.). You can use annotations to visualize your feedback and score and communicate this to the student. You can use rubrics to implement an answer model.

Automatically graded exams

  1. Brightspace quizzes: A digital exam, with a lot of question types. Do not use multiple choice questions to prevent fraud.  parametrisation of numerical questions is possible, as well as deducting points for rounding errors and forgetting units. You can set up the exam itself, because it is easy to use. You can assign random questions from a pool of comparable questions per group of students. All students within this group receive the same questions (apart from the parametrization). Teaching assistants in your course have access to the quiz. Student assistants at the Brightspace support team have access as well.
  2. MapleTA: A digital exam, meant to automatically grade part of the questions.Do not use multiple choice questions to prevent fraud. Randomizing questions from question pools and parametrisation of numerical questions is possible, as well as deducting points for rounding errors and forgetting units. MapleTA exams are hosted on a separate assessment server. There is a Brightspace server available for formative testing. You will most likely need help from experienced TA’s to turn your questions into an exam, since the tool has a steep learning curve.
    Larger questions: Only grades the final answer with all/no points. In case of an incorrect answer, it can allow students a second chance for 50% of the points. However, students can become demotivated or insecure by the immediate feedback which might diminish performance and experience.
    You can anticipate some obvious mistakes by giving partial points for certain answers (e.g. ‘forgot to square the answer’), but only beforehand. Unforeseen partially correct answers require manually going through the answers and adding partial scores. Teaching assistants in your course and student assistants at the Brightspace support team have no access to the questions, since they are on a separate server with autonomous user rights (i.e. teaching assistants cannot access the server if you do not want them to).
  3. In some faculties, other systems are locally available, like  WebLab (EEMCS), ANS Delft and COZ (CEG).

Online proctored (with video surveillance) examination tools

Online proctoring may only be used if there are no unproctored options available. This may be the case if you really need to ask knowledge questions and if oral exams are not an option due to the amount of students. Please note that proctoring will cost around 10-15 euros per student and that these costs are for the faculty. Your Board of Examiners needs to give you explicit permission to use online proctoring. They will only give permission if there are no other options available. In some faculties, Boards of Examiners do not allow online proctoring.

The following tool is currently available for online proctored exams (if more tools become available, these will be added):

  1. MapleTA: exams with proctoring (see above).

For oral exams, you can use the following tools:

  1. YouSeeU: integrated in all Brightspace courses. Waiting room and break-out rooms are available to have TAs welcome students and help them test and setup their connection (handy in case of short exams or oral checks and large student numbers).
  2. Skype4Business: no recording available on Macs, no waiting room.
  1. Brightspace assignments: students upload their assignment or report individually or as a group (Brightspace allows for group assignments), before the deadline. The uploaded work is typically a report (pdf or .docx), code or a recording of a presentation (movie). You can find more information on how to create good assignments and how to create rubrics.

4. Fraud prevention

You can find an overview of all fraud prevention options on the sub-page on fraud prevention.

5. Assessment tips from Q3 lecturers to Q4 lecturers

Based upon experiences from teachers in Q3, we have developed the following do’s, don’ts and tips, per remote assessment type.

Do:

  1. Do a practice exam 1-2 weeks before the real exam.
    Goal: to test the procedure as well as students’ devices and apps. Furthermore, it will familiarize students with the procedure, which will diminish stress and improve performance.
  2. Honor pledge: For written remote exams, this can be done for example one day before the actual examination, by for example typing the text of a pledge in a Brightspace Quiz short answer and grade it automatically (students can have another go if they make a spelling mistake). For oral exams, you can do it at the start of the recording (if applicable). Adapt the pledge to your situation.
  3. Have students practice with representative questions/assignments before the examination.
  4. Don’t forget to enable extra time for students who are entitled to extra time.
  5. Helpdesk during exam: Be available (with backup TA’s if needed) to help students with questions on the examination and technical issues.
  6. Communicate to your students in detail how the exam procedure will look like. That way, they know what to expect and can focus on the content.
  7. Add fraud prevention methods like parametrization and question pooling.
  8. Take the Security and Privacy of Oral Examinations into account during oral authenticity checks.

Don’t:

  1. Do not ask students to upload or email their campus ID, nor any other ID or picture of themselves. It will not certify that they are the ones who take the exam and therefore provides no ground to process these privacy sensitive data.
  2. Do not add ‘surprise elements’ to the exam, to check the student’s identity. Students will be distracted and startled, or might miss the ‘call for surprise action’, which will influence their grade.

Tips:

  1. Have a peer review and/or an educational consultant check your exam questions, especially if you changed the set-up or type of answer model considerably compared to the on-campus examination.
  2. If you cut your exam in sections that students have to finish within a tight time slot, with short breaks between the time-slots, make sure that the time slots are long enough for students to get into a flow of concentration. Preferably not more than 4 time slots with a minimum duration of 30 -90 minutes (depending on the question length). Be aware that students get very stressed because of the extra deadlines and inflexibility to complete the exam in an alternative question order. This will influence their performance negatively, so only use if you deem it to be necessary.
  3. Make the examination available only during the examination time-slot for the students who subscribed for the exams. If different groups have different exams, make the exam available only to the correct group. Close the exam after the time-slot (due date) plus a short, extra time window (grace period, end date).  This flags all exams that were submitted late in Brightspace. See how-to for assignments: add time-slots to set-up the available time window and use ‘release conditions’ to make assignments available to specific groups only.

Brightspace assignments in general (digital and handwritten)

  1. More information and TU Delft links can be found in the Brightspace Support page on assessing assignments and gradingAnd in the guide how to set up an exam in Brightspace Assignments.
  2. Brightspace assignments can be used to give students larger assignments or projects and have them upload intermediate products for you to provide feedback, or for uploading final products (reports, code, drawings, videos, etc.).
  3. For peer feedback on intermediate products, use FeedbackFruits Peer review within Brightspace. Contact Brightspace Support to enable this in your Brightspace course.
  4. Reserve enough time per assignment to upload the assignment. This is especially relevant if students need to take pictures of handwritten work and combine these into a single pdf on their phone, before handing in the resulting file.
  5. Fraud prevention:
    1. You can split up your exam in 2-4 parts, that students have to take one by one in consecutive time slots.
    2. You can assign (slightly) different assignments for different groups, using release conditions.
    3. You can change the group composition per time slot to prevent students from identifying whom to contact. For example: Question has four versions. q1v1 is for the q1v1 group, which contains different students than the q2v1 group.
  6. Scoring
    1. Use a rubric to score the assignments and keep track of scores. If you use a Brightspace rubric could look like the one below. You can divide the rubric into steps and mandatory items (e.g. units and rounding) and/or criteria (e.g. explanation results and correctness of conclusion). When scoring the students’ work, click on the applicable cell for each row and the total score is automatically calculated.

      If you are using TurnItIn, you can also use their rubric and annotation tool to score the exams.
  7. Inspection of results: you can make the annotated pdfs, rubrics (if applicable) and/or points per subquestion (through Brightspace Grades) available, either during a specific time window, or for a longer period.

 

Handwritten Brightspace assignments:

  1. Instructions for the assignment
    1. Ask students to combine pages into a single pdf.
      1. This will save you time (you need to open one file only, only one file format)
      2. You can add scores in a pdf as annotations.
    2. During the practice exam run, have students test whether the resulting file size is of acceptable size (file size can slow down uploading assignments to 20 minutes, which is nerve wrecking for students and teachers).
    3. Ask students to bring and write on clean sheets.
    4. Ask students to number the sheets and check if each page contains the subquestion number (in case the order of the pages gets mixed up in the pdf).
  1. Scoring:
    1. If you prefer to do the scoring anonymously (to diminish bias) and online, ask the students to not write their name on the sheets.
    2. If you plan to print all assignments and score the exams on paper, then do ask students to provide their name and student number on each paper.
  2. Fraud prevention
    1. If available, you can compare handwriting of the student with earlier handwritten assignments to check whether the student authored the work.
    2. The plagiarism scan of TurnItIn is not compatible with handwritten assignments.

 

Digital Brightspace assignments:

  1. Fraud prevention:
    1. Turn on the plagiarism scan (TurnItIn) before publishing the assignment. This will detect plagiarism.
    2. Quickly go over the similarity report, to check if the student copied large parts of sections and only did minor paraphrasing.

More information on how to set up an exam in Brightspace Assignments can be found here.

The following advice is written for Brightspace Quizzes, but can be used in other digital exam tools as well.

Digital exam in Brightspace Quizzes

  1. Proof of identity
    1. Logging into Brightspace is enough proof of the student’s identity. It is forbidden to share one’s netid with others (this is considered identity fraud).
  2. Fraud prevention
    1. Use parameterization in arithmetic questions to give all students different numerical outcomes.
      • Goal: diminish fraud.
    2. Use question pools to draw random questions from a couple of question pools with interchangeable questions.
      • Goal: diminish fraud.
  3. Constructing questions
        1. Problem: Brightspace can only give students all points or no points for automatically graded questions, which diminishes the precision of the resulting grade. Students find it unfair if they lose a lot of points for small mistakes, especially if the digital exam replaces an exam in which students were given partial scores in case of small errors.
          • Solution 1: Split larger questions into smaller ones that are automatically graded.
          • Solution 2: Manually grade important questions so you can assign partial points, using the ‘Written Response’ question type in Brightspace Quizzes, or using Brightspace Assignments for a part or the whole of the exam.
          • Goal: to increase the precision of your grade.
        2. Exam questions:
          • Provide clear instructions on the answer format, including the use of spaces, capitals, etc. This is especially relevant if you use automated grading for short answer questions (for example using regular expressions, which you can test here).
            Have students practice with following these instructions and provide them with (automated) feedback. You do not want the grade to be influenced by this (unless it is part of the learning objectives).
  4. Progress during the exam:
    1. Check the logs of attempts to monitor who has started the exam. Answers will not be visible until a student submits the exam.
  5. Inspection of results:
    1. you can make the feedback to the Brightspace Quiz available, either during a specific time window, or for a longer period. This will show students their given answer and the correct answer, and feedback (if there is any).

Find all information in the How-to-Guide: Remote Oral Exams by Gillian Saunders (teacher at the TU Delft). Do not forget to take the regulations on Security and Privacy of Oral Examinations into account.

Oral checks and oral exams

  1. Identity check
    1. Inform students to keep their student ID ready.
    2. Check the students’ identity before you start the oral check or oral exam.
    3. Do not record campus cards or other proofs of identity.
  2. Recordings
    1. Do not forget to delete all recordings two months after grading, unless for students who filed complaints. Delete their recordings two months after the procedure has been finished. For details on how to record in a privacy compliant way (the use of Zoom is strongly discouraged, please read this document.
  3. Tool and TA support
    1. Use a tool with a main room and break-out rooms, like YouSeeU, or a tool with waiting rooms.
    2. Have TAs invite the (random) students, put them in the waiting room, help them with their audio and video, check their identity, and move them to your break-out room when you are available for the oral check.
    3. Goal: to diminish start-up time.

Oral exams only

  1. Use rubrics or other grading sheets to grade consistently. .
  2. If a student is slow in giving answers, you could prompt the student after a certain amount of time and deduct some points for this.
    Be transparent about this before the exam, in order for students to prepare and perform optimally for the exam. If feasible, standardize the prompt, time to prompt and point deduction.
    Have your students practice with answering quickly.
  3. Using two examiners can be helpful to alternate question asking (and registering the points for given answers), manage the time, solve technical issues, and discuss and decide on student performance and grade.

Oral checks only

  1. Choose the students for the oral check in a completely random way, or with a very clear, unbiased algorithm.
  2. Preparing for handwritten exam questions takes about 5 minutes per question due to readability issues. Doing an oral check takes about 10~15 minutes, unless you run into bad cases.
  3. Be clear about whether or not students can look at their answers and/or drafts or not during the oral check.
  4. If students are allowed draft paper during the exam, ask them to write on clean sheets and inform them that you may ask them to show these draft papers during the oral check.

6. Guiding principles for assessment

There are two guiding principles when you redesign your assessment: constructive alignment and fairness. The key elements to achieve alignment and fairness are displayed in the following picture, and explained in more detail in the text below.

Eight key elements for developing remote assessment

Figure 1: Constructive alignment of learning objectives, learning activities and assessment to stimulate (active) learning

Constructive alignment (Biggs, 1996) implies that the assessment of your course should match a) the learning objectives and b) the learning activities during the course. In other words, students should have been able to practice with what they will be assessed

Assessment and the resulting grade should be fair for students. This leads to the following principles:

 

Validity

  • The assessment should cover the learning objectives, and nothing more. Example: if an assessment is changed into an oral exam, the verbal capacities or English fluency of the student should not influence the grade, unless it is part of the learning objectives.

Reliability

  • Instructions in assignments and questions should be clear for students: they should e.g. know the amount of detail they should fill in, in order to get a full score
  • Assessors should receive clear grading instructions that should ensure objective assessment, minimize bias and align the grade with the learning objectives and assignment instructions.

Fraud prevention

  • The identity of the student who takes the assessment should be checked.
  • Since it is easier to cheat with fact questions, shift the questions (and practice materials) to higher levels of Bloom (Bloom, 1956, click here for the TU Delft adaptation for engineering education). That is, if the level of Bloom of your learning objective allows for it. If that is not possible, consider using oral exams to test factual knowledge.
  • For higher order learning objectives, open-book exams are preferred to closed-book exams.
  • If students trust that other students who try to cheat will be caught, they are less likely to consider cheating themselves.
  • The Board of Examiners can decide to declare the results of an exam invalid in case large fraud is detected (for example via WhatsApp). Communicating this to students might reduce the risk.
  • Students can read the code of conduct and should declare not to commit fraud at the start of the exam, or when receiving and/or handing in their assignment.

Transparency

  • It should be clear on what criteria students will be assessed in assignments, and what level of detail is expected in exams and assignments.
  • The grading of the assessment and weighing of assessment parts (or criteria) should be clear before, during and after the assessment.
  • One way of creating clear criteria for your students is a rubric. For more information on rubrics, go to: Rubrics to Grade Assignments

Feasibility

  • Students should be able to practice with the assessment tool and assessment examples, ample time before the assessment day.
  • Check for interaction with deadlines and assessment slots in other courses.
  • The proposed changes in assessment should be feasible for the teachers, examiners and assessors (during assessment construction and the grading).

Inclusiveness

  • Assessments should be as inclusive as possible, by stating the instructions and scoring as clearly as possible, and by allowing students to practice with the new type of assessment.
  • Students with special needs should have enough time to ask for and discuss special accommodations. See here for a practical way to implement this.
  • Special needs provisions are given by the Board of Examiners. Students can apply for these provisions via the student counsellors. The Boards of Examiners (via the study counsellors) should inform you which provisions need to be made available to which student. Lecturers themselves are not allowed to grant these provisions.
  • While assessing, grading or determining fraud, the lecturer should strive not to be unbiased. If possible, grading should be anonymous and oral authenticity checks should be based on an unbiased sample of students.

Privacy

  • Used tools have to be compliant with privacy rules and regulations. In case this is not possible, students have the possibility to refuse to use the tool, and other measures should be taken by the teacher. However, this implies extra work for the teachers, so this should be prevented.
  • In case recordings are made, this should be considered privacy sensitive data, and the recordings should be kept in a safe place. (More information will become available. Until then, store the recordings on your surfdrive.

7. Page history

The following guidelines were added and changes made:

The following guidelines were added and changes made:

  • Updated links, changed minor parts of the page, mostly textual.
  • Guidelines on the following topics were processed on this page:
    • Do’s, don’ts and tips based on experience of teachers with remote assessment in Q3
    • Postponement of assessment is no longer an option: deleted text that discussed the options to postpone assessments.

Guidelines on the following topics were processed on this page:

  • Update Taskforce Assessment (6 April 2020) on studying with a disability, security and privacy guidelines for oral exams, and complementary oral checks.
  • Advice on ‘poor man’s proctoring’ (30 March 2020)

The following manuals were added:

The previous guidelines on the Brightspace support website are quite strict, because Boards of Examiners were initially quite reluctant for remote assessment. Given the current situation, they are more open to remote assessment, if the quality of assessment is assured. Part of the quality of assessment is to minimize the risk of fraud. Therefore, we have adapted the homepage to reflect more lenient guidelines for remote assessment.

The first version of this page was published on 24 March 2020

Contact us

Please send us an email and we'll get back to you.

Sending

Log in with your credentials

Forgot your details?

By continuing to use this website, you agree with the use of cookies. More information

This website uses cookies of multiple categories: strictly necessary cookies, performance cookies and advertisement cookies. The strictly necessary cookies are used to verify whether you have given consent to the cookies before, whereas the performance cookies are used for insights in the usage of the website. Using the latter allows the admin of this domain to improve the experience on the website in the future. Some of the cookies are third-party, such as YouTube and Google Analytics.

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close