17. Assessment 3#
17.1. Replication and Reproduction Study#
17.2. Overview#
In this assessment you will draw on all the skills and knowledge you have developed over the course of this module to apply to a substantial coding project such as you may be required to develop or contribute to during the course of your PhD. This will be a group project and will take the form of a code reproduction exercise related to the game of Pig which was introduced earlier in the course.
You will be required you to replicate and reproduce the results from the article “Optimal Play of the Dice Game Pig” by Neller and Presser 2004 which proposes an approximate dynamic programming method for computing the optimal playing strategy for the game Pig. The result of this exercise should be a collection of work organised and prepared in a way such that it is suitable for submission as an article to the ReScience C journal. That is, the output will consist of a collection of code which can be used to reproduce numerical results from the original paper, and an accompanying report which outlines what was implemented, the new results, and discusses any issues encountered during the reproduction exercise.
Additionally, you will be required to critically reflect on your work in a group presentation. For this you will appraise the quality of your work and the original paper in terms of the 5 R’s. Finally, to ensure that each member of a group is awarded fairly for their contribution to the project, there will be a peer assessment component where you will be required individually to evaluate this contribution for each member of your group.
To help you and your group undertake this work there are three support tutorial sessions (8 hours in total).
The weighting of the each of the three components in the overall mark for the module is as follows:
Report and code repository: 30%
Presentation: 10%
Peer assessment: 10%
17.3. Report and code repository#
The report and code repository should be in a format suitable for submission to ReScience C. It is strongly encouraged that you familiarise yourselves with the submission guidelines for ReScience C journal, and you view some submissions to get an idea of what is required.
17.3.1. Format of Report#
The main report should include:
An introduction
A methodology
A results section
A discussion section
Like any other academic paper, you should provide references to other work where necessary, but a full literature review is not required for the purposes of this assessment.
If necessary, use an appendix to help organise your article.
17.3.2. Code#
You should submit all the code required to run the numerical tests presented in your report.
You are free to structure this in any way you wish.
For example, this could take the form of a Python package implementing core components of the methodology, with scripts or Jupyter notebooks used to run the numerical results appearing in your report.
The code will be assessed according to the 5 R’s, and as such, it should be appropropriately documented.
At a minimum, you should provide a README.md file which briefly overviews the contents and purpose of the code base, and in particular, provides instructions on how to rerun any code which reproduces the numerical results from your paper.
Although we do not expect every group member to have an equal contribution to the coding, we expect every member to have a significant contribution to this.
17.3.3. Submission of report and code#
All of the elements of your work (notebooks, data, LaTeX, source code etc) should be made available via a single repository on GitHub. The report should be included in the repository either as a PDF, or as LaTeX source files with instructions on how this can be compiled.
Formal submission of the work should be made by inviting Daniel Grose and Jamie Fairbrother to be collaborators to this repository by the submission date (if there are any issues here, you can alternatively email a copy of the repository including the hidden .git folder in the top-level).
Note, only files committed by the submission date will be considered for assessment unless other prior arrangements have been made.
17.3.4. Assessment Criteria#
This will be assessed against the Criteria for Publication as used by the ReScience C journal.
A rigorous description of the methodologies used in both your own work and the original article
Quality of source code with respect to the 5Rs
Evidence for replication of the original results or explanations why original results cannot be replicated.
Clarity and accuracy of exposition
Overall organisation and layout of the article
17.4. Group Presentation#
You will be required to give a short group presentation. This will consist of a 10-15 minute presentation followed by 10-15 minutes of questioning.
The presentation should cover the following:
A summary of the main findings of your ReScience C article.
An assessment of the quality of the original work with respect to the 5Rs.
An assessment of the quality of your replication/reproduction study with respect to the 5Rs.
The questioning will mainly cover the contents of the presentation, but may also relate to how the project was managed and contributions of individuals.
The group presentation component will be marked on a scale of 0-5 for each of the following criteria:
Content: Opening, clarity of argument or explanation, summary of work complete
Analysis: Understanding of 5 R’s, appraisal of report and paper with respect to these.
Presentation: Fluency, use of appropriate visual aids (diagrams etc.), body language etc.
Discussion skills: Listening, responding to questions, delegating answers to appropriate team members
17.5. Peer assessment#
Each member of the group will provide an assessment of their group members contribution to the overall assessment. For each member of the group you will be required to assign a mark between 0 and 10, including yourself, as well as some comments.
You should assign an overall mark based on the following criteria:
Professionalism: attendance to group-meetings, keeping to deadlines, respectful attitude to others etc.
Research: Gathering and researching additional information
Problem-solving: Ability to generate ideas and solve problems
Coding: contribution to writing and reviewing code, and associated documentation
Report writing: contribution to writing and reviewing report
The overall mark should be scaled as follows:
10-9 = outstanding contribution
8-7 = very good contribution
6-5 = good contribution
4-5 = fairly satisfactory contribution
2-1 = unsatisfactory contribution
0 = virtually no contribution
17.6. Time Line#
Release of Assessment: Monday 28/04/2025
Tutorial 1: Tuesday 29/04/2025
Tutorial 2: Thursday 08/05/2025 and Friday 09/05/2025
Tutorial 3: Tuesday 13/05/2025 and Wednesday 14/05/2025
Report and Code Submission Deadline: Tuesday 20/05/2025
Group Presentations: Friday 23/05/2025
Peer Assessment submission deadline: Thursday 22/05/2025 (a form will be distributed for this purpose closer to the date)
Your work needs to be submitted by 20/05/2025 using the method detailed in the Assessment Process section.
There are three timetabled tutorial sessions to provide support and advice during the assessment period. Attendance at these tutorials is mandatory.
Feedback and marks will be provided within two to four weeks after all components have been completed. However, your work may be subject to a review processing which case you will be informed about any delays this may incur.
Any queries regarding the assessment should, in the first instance, be by e-mail to Daniel Grose and Jamie Fairbrother.
17.7. Some Guidelines for Undertaking the Work#
Take some time to familiarise yourself with the article “Optimal Play of the Dice Game Pig”
Visit ReScience C journal and determine its rational regarding sustainable computational science
Check you understand the Criterea for Publication used byReScience C journal.
Briefly explore a small selection of articles from ReScience C, for example ReScience C 9, 1, #4. Make sure you look at how the supporting material, such as the code, is organised, documented, and made publicly available.
Check you understand how your work will be assessed (see the section Assessment Process in this document for details).
Once you have been allocated to a group,arrange a group meeting to discuss and agree how you are going to proceed with the replication / reproduction exercise. It might be useful to consider the following :
Agreement of objectives and outcomes
Division of responsibilities and tasks
How to assess progress
How to measure success or failure
How to decide when to stop !!
17.7.1. Please make sure that you and your group attend all of the support tutorials allocated to this assessment.#
17.8. Resources#
Re-run, Repeat, Reproduce, Reuse, Replicate: Transforming Code into Scientific Contributions