Here we outline a few of the most notable models that demonstrate the contrast in approaches available. 0000009507 00000 n The process of evaluation is dynamic and ongoing. SROI aims to provide a valuation of the broader social, environmental, and economic impacts, providing a metric that can be used for demonstration of worth. 0000001087 00000 n New Directions for Evaluation, Impact is a Strong Weapon for Making an Evidence-Based Case Study for Enhanced Research Support but a State-of-the-Art Approach to Measurement is Needed, The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance, Evaluation in National Research Funding Agencies: Approaches, Experiences and Case Studies, Methodologies for Assessing and Evidencing Research Impact. The time lag between research and impact varies enormously. 0000007777 00000 n The fast-moving developments in the field of altmetrics (or alternative metrics) are providing a richer understanding of how research is being used, viewed, and moved. Definition of Evaluation "Evaluation is the collection, analysis and interpretation of information about any aspect of a programme of education, as part of a recognised process of judging its effectiveness, its efficiency and any other outcomes it may have." Mary Thorpe 2. It is time-intensive to both assimilate and review case studies and we therefore need to ensure that the resources required for this type of evaluation are justified by the knowledge gained. (2007), Nason et al. It is perhaps assumed here that a positive or beneficial effect will be considered as an impact but what about changes that are perceived to be negative? Definition of evaluation. Case studies are ideal for showcasing impact, but should they be used to critically evaluate impact? From an international perspective, this represents a step change in the comprehensive nature to which impact will be assessed within universities and research institutes, incorporating impact from across all research disciplines. Definitions of Evaluation ( by different authors) According to Hanna- "The process of gathering and interpreted evidence changes in the behavior of all students as they progress through school is called evaluation". The criteria for assessment were also supported by a model developed by Brunel for measurement of impact that used similar measures defined as depth and spread. different things to different people, and it is primarily a function of the application, as will be seen in the following. Evaluative research has many benefits, including identifying whether a product works as intended, and uncovering areas for improvement within your solution. Despite the concerns raised, the broader socio-economic impacts of research will be included and count for 20% of the overall research assessment, as part of the REF in 2014. Definition of Evaluation by Different Authors Tuckman: Evaluation is a process wherein the parts, processes, or outcomes of a programme are examined to see whether they are satisfactory, particularly with reference to the stated objectives of the programme our own expectations, or our own standards of excellence. 2007; Grant et al. There has been a drive from the UK government through Higher Education Funding Council for England (HEFCE) and the Research Councils (HM Treasury 2004) to account for the spending of public money by demonstrating the value of research to tax payers, voters, and the public in terms of socio-economic benefits (European Science Foundation 2009), in effect, justifying this expenditure (Davies Nutley, and Walter 2005; Hanney and Gonzlez-Block 2011). n.d.). The Payback Framework has been adopted internationally, largely within the health sector, by organizations such as the Canadian Institute of Health Research, the Dutch Public Health Authority, the Australian National Health and Medical Research Council, and the Welfare Bureau in Hong Kong (Bernstein et al. % Reviewing the research literature means finding, reading, and summarizing the published research relevant to your question. A comprehensive assessment of impact itself is not undertaken with SIAMPI, which make it a less-suitable method where showcasing the benefits of research is desirable or where this justification of funding based on impact is required. 0000007223 00000 n These sometimes dissim- ilar views are due to the varied training and background of the writers in terms of their profession, concerned with different aspects of the education process. What are the methodologies and frameworks that have been employed globally to evaluate research impact and how do these compare? Inform funding. Impact is derived not only from targeted research but from serendipitous findings, good fortune, and complex networks interacting and translating knowledge and research. The development of tools and systems for assisting with impact evaluation would be very valuable. Teresa Penfield, Matthew J. Baker, Rosa Scoble, Michael C. Wykes, Assessment, evaluations, and definitions of research impact: A review, Research Evaluation, Volume 23, Issue 1, January 2014, Pages 2132, https://doi.org/10.1093/reseval/rvt021. 0000001325 00000 n 0000001178 00000 n In the UK, UK Department for Business, Innovation, and Skills provided funding of 150 million for knowledge exchange in 201112 to help universities and colleges support the economic recovery and growth, and contribute to wider society (Department for Business, Innovation and Skills 2012). An evaluation essay or report is a type of argument that provides evidence to justify a writer's opinions about a subject. Such a framework should be not linear but recursive, including elements from contextual environments that influence and/or interact with various aspects of the system. It is possible to incorporate both metrics and narratives within systems, for example, within the Research Outcomes System and Researchfish, currently used by several of the UK research councils to allow impacts to be recorded; although recording narratives has the advantage of allowing some context to be documented, it may make the evidence less flexible for use by different stakeholder groups (which include government, funding bodies, research assessment agencies, research providers, and user communities) for whom the purpose of analysis may vary (Davies et al. And also that people who are recognized as authors, understand their responsibility and accountability for what is being published. From the outset, we note that the understanding of the term impact differs between users and audiences. Developing systems and taxonomies for capturing impact, 7. Why should this be the case? Co-author. 2007) who concluded that the researchers and case studies could provide enough qualitative and quantitative evidence for reviewers to assess the impact arising from their research (Duryea et al. 0000007967 00000 n Table 1 summarizes some of the advantages and disadvantages of the case study approach. Despite many attempts to replace it, no alternative definition has . These . The authors propose a new definition for measurement process based on the identification of the type of measurand and other metrological elements at each measurement process identified. To adequately capture interactions taking place between researchers, institutions, and stakeholders, the introduction of tools to enable this would be very valuable. This report, prepared by one of the evaluation team members (Richard Flaman), presents a non-exhaustive review definitions of primarily decentralization, and to a lesser extent decentralization as linked to local governance. Research findings will be taken up in other branches of research and developed further before socio-economic impact occurs, by which point, attribution becomes a huge challenge. The main risks associated with the use of standardized metrics are that, The full impact will not be realized, as we focus on easily quantifiable indicators. Assessment refers to the process of collecting information that reflects the performance of a student, school, classroom, or an academic system based on a set of standards, learning criteria, or curricula. Reviews and guidance on developing and evidencing impact in particular disciplines include the London School of Economics (LSE) Public Policy Groups impact handbook (LSE n.d.), a review of the social and economic impacts arising from the arts produced by Reeve (Reeves 2002), and a review by Kuruvilla et al. If this research is to be assessed alongside more applied research, it is important that we are able to at least determine the contribution of basic research. Thalidomide has since been found to have beneficial effects in the treatment of certain types of cancer. What is the Difference between Formative and Summative Evaluation through Example? 0000008591 00000 n 2006; Nason et al. (2007) surveyed researchers in the US top research institutions during 2005; the survey of more than 6000 researchers found that, on average, more than 40% of their time was spent doing administrative tasks. A comparative analysis of these definitions reveal that in defining performance appraisal they were saying the same thing, but in a slightly modified way. 0000348082 00000 n The aim of this study was to assess the accuracy of 3D rendering of the mandibular condylar region obtained from different semi-automatic segmentation methodology. For example, the development of a spin out can take place in a very short period, whereas it took around 30 years from the discovery of DNA before technology was developed to enable DNA fingerprinting. Citations (outside of academia) and documentation can be used as evidence to demonstrate the use research findings in developing new ideas and products for example. According to Hanna- " The process of gathering and interpreted evidence changes in the behavior of all students as they progress through school is called evaluation". Search for other works by this author on: A White Paper on Charity Impact Measurement, A Framework to Measure the Impact of Investments in Health Research, European Molecular Biology Organization (EMBO) Reports, Estimating the Economic Value to Societies of the Impact of Health Research: A Critical Review, Bulletin of the World Health Organization, Canadian Academy of Health Sciences Panel on Return on Investment in Health Research, Making an Impact. While valuing and supporting knowledge exchange is important, SIAMPI perhaps takes this a step further in enabling these exchange events to be captured and analysed. Collating the evidence and indicators of impact is a significant task that is being undertaken within universities and institutions globally. 0000004692 00000 n Evaluation of impact is becoming increasingly important, both within the UK and internationally, and research and development into impact evaluation continues, for example, researchers at Brunel have developed the concept of depth and spread further into the Brunel Impact Device for Evaluation, which also assesses the degree of separation between research and impact (Scoble et al. The introduction of impact assessments with the requirement to collate evidence retrospectively poses difficulties because evidence, measurements, and baselines have, in many cases, not been collected and may no longer be available. In the UK, there have been several Jisc-funded projects in recent years to develop systems capable of storing research information, for example, MICE (Measuring Impacts Under CERIF), UK Research Information Shared Service, and Integrated Research Input and Output System, all based on the CERIF standard. To evaluate impact, case studies were interrogated and verifiable indicators assessed to determine whether research had led to reciprocal engagement, adoption of research findings, or public value. Worth refers to extrinsic value to those outside the . To enable research organizations including HEIs to monitor and manage their performance and understand and disseminate the contribution that they are making to local, national, and international communities. 4 0 obj Husbands-Fealing suggests that to assist identification of causality for impact assessment, it is useful to develop a theoretical framework to map the actors, activities, linkages, outputs, and impacts within the system under evaluation, which shows how later phases result from earlier ones. 5. CERIF (Common European Research Information Format) was developed for this purpose, first released in 1991; a number of projects and systems across Europe such as the ERC Research Information System (Mugabushaka and Papazoglou 2012) are being developed as CERIF-compatible. (2005), Wooding et al. The University and College Union (University and College Union 2011) organized a petition calling on the UK funding councils to withdraw the inclusion of impact assessment from the REF proposals once plans for the new assessment of university research were released. Understand. In development of the RQF, The Allen Consulting Group (2005) highlighted that defining a time lag between research and impact was difficult. Time, attribution, impact. Media coverage is a useful means of disseminating our research and ideas and may be considered alongside other evidence as contributing to or an indicator of impact. 2009; Russell Group 2009). 0000007307 00000 n 3. This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research. The first attempt globally to comprehensively capture the socio-economic impact of research across all disciplines was undertaken for the Australian Research Quality Framework (RQF), using a case study approach. Where quantitative data were available, for example, audience numbers or book sales, these numbers rarely reflected the degree of impact, as no context or baseline was available. The basic purpose of both measurement assessment and evaluation is to determine the needs of all the learners. The Payback Framework is possibly the most widely used and adapted model for impact assessment (Wooding et al. They risk being monetized or converted into a lowest common denominator in an attempt to compare the cost of a new theatre against that of a hospital. HEFCE developed an initial methodology that was then tested through a pilot exercise. To understand the method and routes by which research leads to impacts to maximize on the findings that come out of research and develop better ways of delivering impact. 0000002109 00000 n The Social Return on Investment (SROI) guide (The SROI Network 2012) suggests that The language varies impact, returns, benefits, value but the questions around what sort of difference and how much of a difference we are making are the same. Impact is assessed alongside research outputs and environment to provide an evaluation of research taking place within an institution. Muffat says - "Evaluation is a continuous process and is concerned with than the formal academic achievement of pupils. A taxonomy of impact categories was then produced onto which impact could be mapped. For example, following the discovery of a new potential drug, preclinical work is required, followed by Phase 1, 2, and 3 trials, and then regulatory approval is granted before the drug is used to deliver potential health benefits. Given that the type of impact we might expect varies according to research discipline, impact-specific challenges present us with the problem that an evaluation mechanism may not fairly compare impact between research disciplines. Here we address types of evidence that need to be captured to enable an overview of impact to be developed. Replicated from (Hughes and Martin 2012). The university imparts information, but it imparts it imaginatively. The inherent technical disparities between the two different software packages and the adjustment . It is now possible to use data-mining tools to extract specific data from narratives or unstructured data (Mugabushaka and Papazoglou 2012). The . Figure 1, replicated from Hughes and Martin (2012), illustrates how the ease with which impact can be attributed decreases with time, whereas the impact, or effect of complementary assets, increases, highlighting the problem that it may take a considerable amount of time for the full impact of a piece of research to develop but because of this time and the increase in complexity of the networks involved in translating the research and interim impacts, it is more difficult to attribute and link back to a contributing piece of research. Assessment for Learning is the process of seeking and interpreting evidence for use by learners and their teachers to decide where the learners are in their learning, where they need to go and. In designing systems and tools for collating data related to impact, it is important to consider who will populate the database and ensure that the time and capability required for capture of information is considered. It has been suggested that a major problem in arriving at a definition of evaluation is confusion with related terms such as measurement, This might include the citation of a piece of research in policy documents or reference to a piece of research being cited within the media. 2007; Nason et al. Without measuring and evaluating their performance, teachers will not be able to determine how much the students have learned. Hb```f``e`c`Tgf@ aV(G Ldw0p)}c4Amff0`U.q$*6mS,T",?*+DutQZ&vO T4]2rBWrL.7bs/lcx&-SbiDEQ&. Capturing data, interactions, and indicators as they emerge increases the chance of capturing all relevant information and tools to enable researchers to capture much of this would be valuable. The following decisions may be made with the aid of evaluation. The Goldsmith report (Cooke and Nadim 2011) recommended making indicators value free, enabling the value or quality to be established in an impact descriptor that could be assessed by expert panels. only one author attempts to define evaluation. 2. Merit refers to the intrinsic value of a program, for example, how effective it is in meeting the needs those it is intended help. Collecting this type of evidence is time-consuming, and again, it can be difficult to gather the required evidence retrospectively when, for example, the appropriate user group might have dispersed. The process of evaluation involves figuring out how well the goals have been accomplished. These traditional bibliometric techniques can be regarded as giving only a partial picture of full impact (Bornmann and Marx 2013) with no link to causality. This presents particular difficulties in research disciplines conducting basic research, such as pure mathematics, where the impact of research is unlikely to be foreseen. To achieve compatible systems, a shared language is required. Findings from a Research Impact Pilot, Institutional Strategies for Capturing Socio-Economic Impact of Research, Journal of Higher Education Policy and Management, Introducing Productive Interactions in Social Impact Assessment, Measuring the Impact of Publicly Funded Research, Department of Education, Science and Training, Statement on the Research Excellence Framework Proposals, Handbook on the Theory and Practice of Program Evaluation, Policy and Practice Impacts of Research Funded by the Economic Social Research Council. Capturing knowledge exchange events would greatly assist the linking of research with impact. The Goldsmith report concluded that general categories of evidence would be more useful such that indicators could encompass dissemination and circulation, re-use and influence, collaboration and boundary work, and innovation and invention. Although metrics can provide evidence of quantitative changes or impacts from our research, they are unable to adequately provide evidence of the qualitative impacts that take place and hence are not suitable for all of the impact we will encounter.