Funding Scheme

RGC Faculty Development Scheme

Project Title

An Automated Student Program Assessment Framework with Tailorable and Automated Test Oracle for Computer Science Education

Project Team (HSMC Staff) 

Prof POON Chung Keung, Department of Computing (PI)

Other Collaborating Parties

Douglas College, Canada

City University of Hong Kong

Project Period

1-1-2016 to 30-06-2019 (on-going)

Funding Amount (HKD)



Teaching and learning of computer programming in beginners' classes are known to be difficult. Students have to do a lot of exercises to practise their programming and debugging skills, and they need feedback on the correctness of their programs. However, assessing students' programming work manually is tedious, time-consuming and error-prone. With the large class and diverse background of students nowadays, it is increasingly challenging for instructors to provide students with adequate appropriate exercises and individual feedback, which are vital to the effectiveness of students' learning and sustenance of their learning motivation. In response, universities worldwide have developed automated program assessment systems (APASs), which free up instructors' time for other un-automated educational tasks. Moreover, APASs are found to be of tremendous benefit in many other aspects, such as facilitating the design of effective pedagogy, provision of instant and personalized feedback to both instructors and students, and enhancing students' learning motivation.


One core function of APASs is to assess the correctness of students' programs, typically by automatically executing them against a suite of pre-defined test cases and comparing the programs' actual outputs with the instructor's expected outputs. The latter task requires a test oracle, that is, a mechanism for determining the correctness of program outputs. In the field of software testing, the general problem of test oracle automation is well known to be challenging. Implementation of test oracles in existing APASs is often too simplistic, rigid and incapable of being tailored to support the intended educational outcomes of the exercises. For example, multiple correct (or admissible) programming solutions to an exercise may produce different outputs (called output variants). A program which the human instructor accepts to be correct (or admissible) could be inappropriately rejected by a rigid test oracle in an APAS. This technical limitation is common and has been a root cause of many educationally undesirable effects on teaching and learning that can substantially compromise the benefits of an APAS in practice. There is a clear and pressing need to address this limitation, which is said in a recent literature review to be the main disadvantage of using APASs.


In this project, we will develop an automatic program assessment framework which can recognize a variety of admissible output variants so that both instructors and students can focus on the essentials of the exercises instead of minor output deviations. The framework will satisfactorily handle a much broader range of programming exercises to suit different educational needs. We will build an online APAS platform to implement our framework that highly automates the program assessment process for use in real programming courses. The platform will also be instrumental for empirical evaluation and validation of our research. To fully utilize our expertise and collaboration, we will conduct cross-institutional experiments for higher reliability and validity.


Our study is significant in perfecting the state-of-the-art APAS technologies and the advancement of the teaching and learning environment for programming classes, as what the work of ours and others has previously contributed. The research is expected to benefit the learning of hundreds of students each year with visible impact in the form of their improved programming skills. This project will advance the research capability of the investigators so that they can transfer research experiences and new knowledge into teaching and learning in their institutions, which is also the main objective of this Faculty Development Scheme (FDS). Since the newly developed online APAS platform can be used by all academic staff of the institutions, the new knowledge and tangible outcomes arisen from this project will benefit the teaching and learning of all three institutions. Finally, we will actively publicize our research findings both locally and internationally so that their potential benefits and values can be deployed by the global computer education community.