In England, some secondary-level qualifications comprise non-exam assessments which need to undergo moderation before grading. Currently, moderation is conducted at centre (school) level. This raises challenges for maintaining the standard across centres. Recent technological advances enable novel moderation methods that are no longer bound by centre. This study used simulation to investigate the feasibility of using comparative judgement (CJ) for moderating non-exam assessments. Our study explored the effects of CJ design parameters on the CJ estimates of script quality and how to assign moderator marks after the CJ procedure. The findings showed that certain design parameters had substantial effects on reliability and suggested minimum values for CJ protocols. The method used for assigning moderator marks maintained the rank order of scripts within centres and calibrated the centres to a common standard. Using CJ for moderation could transform current assessment practices, taking advantage of technological developments and ensuring reliability and fairness.