Evaluation of sustainability using the AISHE Instrument: case study in a Brazilian University

The role of Higher Education Institutions (HEI), which are in the forefront of the challenges of sustainable development, is becoming increasingly predominant, given the importance of these institutions in the dissemination of knowledge and the implementation of sustainable conscientiousness in society. This article presents the application of the Auditing Instrument for Sustainability in Higher Education (AISHE) at the University of Passo Fundo, which is situated in Southern Brazil. The research evaluated the level of sustainability in 30 undergraduate courses at this university, focusing on Education for Sustainability. To do this, interviews were conducted with course coordinators and quickscans were utilized with professors and students. The way in which the University of Passo Fundo applied AISHE is innovative and interesting: first of all, it is important to emphasize that during the research, a wide variety of university departments were assessed; secondly, this approach allowed a comparison between the courses, producing interesting outcomes that can easily be used for a successive phase of sustainability integration. The most innovative element in the UPF approach is the way in which the outcome of the twenty AISHE criteria are related to the results of the quickscans, which are part of the ‘Check’ criteria of AISHE. Considering the priorities within sustainability in the UPF, the need exists of an initial drive in the managerial stage, such as definition of vision and policy for the institution.

According to Adomssent et al. (2007), in the inclusion of sustainability in higher education, the institutions should act as agents of change. Following this line of thought, it is important that the institutions divulge their implanted changes in benefit of sustainability and evaluate their effectiveness, serving as a model and inspiration for others and playing the role of an accelerator. To gauge this, the use of instruments of evaluation becomes necessary. Lozano (2006) mentions that one of the steps for the inclusion of SD in the curriculum is the use of tools to evaluate the effectiveness of the efforts that have been made.
In this context, mediation through evaluation indexes of involvement of the university with sustainability becomes important, beginning with the development of a general picture, which can facilitate evaluation of the current situation or progress. These indexes, however, should be instruments orientated towards the particularities of the higher education institution, for example, the Graphical Assessment of Sustainability in Universities -GASU (Lozano, 2006), the Sustainability Tool for Auditing Universities Curricula in Higher Education -STAUNCH (Lozano, 2010), the Framework-Level-Actors -FLA Analysis (Ferrer-Balas, Buckland, Mingo, 2007) and the Auditing Instrument for Sustainability in Higher Education -AISHE (Roorda, 2001;2008). Shriberg (2002) presented in his article an outlook of the strengths and weaknesses of the assessment tool and concluded that it is necessary in the future to think about a universal assessment tool which can facilitate the ranking of universities in terms of levels of sustainability. Shiberg (2002), also comments that the tool, AISHE, the focus of this study, is an excellent example of an approach which makes it possible to evaluate sustainability from evaluation indexes with orientation. The aim attached to the creation of AISHE was to construct a flexible platform where it is possible to stimulate and make sustainability operable at higher education level. In fact, this tool has the potential of achieving global appeal. However, the author also highlights that its weak point is that its criteria are quite abstract and that there may be difficulties to understand it.
Many universities find it appealing to use AISHE in a more informal way, i.e. as a self-assessment tool (Roorda, 2004;2008). Every university is free to do so, as all necessary equipment, such as the AISHE book (Roorda, 2001) and the computer application ' AISHE Reporter', can simply be downloaded. Consequently, universities in more than a dozen countries have applied AISHE in a variety of ways.
In 2008, the University of Passo Fundo (UPF) together with other universities from various other countries, committed itself to participating in the group of universities which applied the AISHE tool in its internal and external environments, with the aim of evaluating the methodology and contributing to future adjustments. The aim of this article was to evaluate environmental sustainability on the graduation course curricula of UPF, making it possible to identify priorities in relation to the endeavour for creating sustainable environments and to contribute to the knowledge of their evaluation.
This case study is innovative if compared to other AISHE applications (Roorda, 2004;2008). Firstly, it is important to emphasize that during the research, a wide variety of university departments were assessed, ranging from physics, chemistry and life sciences to, for example, Geography, History, Business Administration, Journalism and Music. This approach allows for a comparison to be made between all these subjects, delivering outcomes that can easily be used for a successive phase of sustainability integration.
Another distinguishing feature in the Passo Fundo approach is the way in which the outcomes of the twenty AISHE criteria are related to the results of the quick scans, which make part of the 'Check' criteria of AISHE. In these quick scans, the level of appreciation of the sustainability process and its results of several stakeholders could be assessed: course coordinators, professors, students. This idea of making a comparison between AISHE criteria and quick scans was never thought of before and it did not make part of the procedure when AISHE was originally developed in 2000-2001. Case description Aim of the study The University of Passo Fundo, in southern Brazil, is a community university with around 20 thousand students, 1100 teachers and 1000 employees. It is located in 7 campuses with 54 graduation courses, 7 technical courses, 45 specialization courses, 7 master degree courses and 1 doctorate.
The focus of the study was the 40 graduation courses of the main Campus I, 30 of which agreed to participate in the research, which represents 75% of the Campus I population. The sample size was random, depending on acceptance of participation in the study. It should be noted that, at this stage of the research, the 14 higher education technical courses were not included. The application of the questionnaires occurred in 2010.

Instrument for evaluation of sustainability
The Auditing Instrument for Sustainability in Higher Education (AISHE) is an instrument designed specifically for evaluating sustainability in educational institutions (Roorda, 2001;2008). When such assessments are used as repeated elements of an iterative process, a 'Deming cycle', this process has proved to be able to lead towards a true transition of the university towards 'System Integration of Sustainable Development' (SISD), a concept which was introduced and thoroughly discussed in Roorda (2010) PhD dissertation. The author also shows that this SISD state can very well act as the final goal for a university's sustainability strategy. This state can be considered as equal to the systemic integration of sustainable development described by Lozano-Garcia et al. (2006), at least in terms of the education and the underlying mission and philosophy of the institution.
As illustrated in Figure 1, AISHE is based on the PDCA cycle and within each of the phases of the cycle, fields of attention are evaluated, each one with four criteria. The criteria can be classified in accordance with five stages (level of sustainability). Each level has a specific description which reflects its level of sustainability for which the people that are responding classify their course.
The stages have an ordinal scale for the five phases: stage 1activity; stage 2process; stage 3system; stage 4part of a chain of process; stage 5society; In this way, it is only possible to conclude that a particular stage has been reached if the previous phases have been completely achieved because all the stages of a criterion are designed to be cumulative; for this reason intermediate values can be defined for each criterion.
The research was performed according to the scheme presented in Figure 2. The AISHE methodology was applied to campus I in Passo Fundo for the graduation course coordinators, with the presupposition that these people would be the most likely to respond in relation to their courses as they are directly involved in the day-to-day activities and are very familiar with them.
Criteria established in AISHE evaluation were answered individually by coordinators, for each (selected) stage the participant formed his/her own opinion about the situation on the course. For the appraisal of results, AISHE methodology allows the use of punctuation median.
After this, a quickscan, in the form of a questionnaire, was used for professors and students, with the aim of confirming the data obtained with the coordinators. In total, 60 professors (two from each course) and 90 students (three from each course) took part. The amount was defined in such a way as to make the research feasible and the sample was selected randomly, taking into consideration all courses in the institution.

Sustainability evaluation according to the coordinators
The sustainability level attributed by the coordinators to each one of the criteria when the situation of their course was evaluated is presented in the Additional file 1: Table S1.
In Figure 3 the different levels of sustainability reached for each course can be observed; this level was obtained by the median of the punctuation attributed for each criteria. Some attained considerably elevated levels (stage 4 e 5). Though it can be observed that these are exceptions, it can be seen that around 70% of the evaluated courses fit into stage 1, demonstrating that their actions are in line with ESD, from the point of view of isolated practices. Figure 4 shows the results from the median of each criterion in relation to the points attributed to the total sum of the evaluated courses. It can be observed that the majority of the criteria reached Stage 1.
The criterion educational methodology (3.2) showed the best results, reaching Stage 3, which signifies that the teaching and learning methodology gives the student the opportunity to encounter real situations, which gives rise both to reflection and development of his/her future professional development in a sustainable fashion.
The interdisciplinary criterion (4.2) attained Stage 2, which shows that the curriculum is structured in such a way that the subjects covered by the courses are interconnected. The other criteria of this field of attention (4.1 curriculum; 4.3 stage, graduation; 4.4 speciality), were met in Stage 1. This demonstrates that analysing the interdisciplinary criterion by itself does not make it possible to say that the subject of sustainability is embraced by the courses satisfactorily.
The criterion vision (1.1) presented a median of intermediate value between stages 1 and 2, signifying that the administration has taken a position on sustainability, even if only implicit, because it offers work opportunities with objectives which, aside from this awareness, can bring concrete consequences for the University, albeit through individual actions. It can also be observed that both this awareness and sustainability cease being merely implicit when they are formulated in documentation. The existence of vision is important but it does not necessarily produce changes; it merely represents an opinion. The policies are a structure which solidify this vision because it translates itself into plans and from these, goals are formulated The criteria policy (1.2) achived stage 1. The criteria Communication (1.3) attained 1.5, the communication of the university with society and it's stakeholders is fundamental although it is necessary that the subject of sustainability be properly approached.  The criteria of internal environmental management presented sustainability level 1 and should be given special attention because it plays a fundamental role. It demonstrates the respect which the institution has towards the environment, making the whole community learn and change its practices and develop a new life style concept that takes into account SD (Disterheft et al., 2012). There are various programmes and projects at the UPF for improving the quality of the environment, although these are in fact isolated practices because the university does not have an environmental management system (EMS).
The students' criterion (5.2) considers the existence of university data concerning students' perception in relation to the sustainability of the institution. As the university did not make available singificant data for valuation, the values stated in the evaluation of this criterion were low, resulting in a median value equal to zero.
Analysis of the items responded to by the students and professors Figure 5 presents the different responses of professors for each analysed item, where it can be seen that the professors opted more for 'partly agree'.
Some key points are possible to be concluded by the answers of the professors. The majority believes that a significant number of employees, students and even professors do not value and/or respect the principals of sustainable development, showing the weakness of the EMS of the institution. Also, that sustainability is still seen as a speciality and not as something interdisciplinary in teaching institutions. Lastly, they believed that what is being taught regarding sustainability is not perhaps ideal or sufficient to meet the requirements and form adults who have responsibility regarding sustainability. This is in compliance with the partial agreement demonstrated by the professors in relation to the presence of the subject of sustainability on the curriculum courses which they teach. Figure 6 presents the percentage of the replies for each of the items by the students. It can be observed that the students' replies are divided between 'neither agree nor disagree' and 'partly agree'.
The quickscan applied for the students showed a lack of knowledge about what the university, still in an isolated or partial way, is doing concerning its environmental sustainability. They state that environmental sustainability should be treated in a more effective manner in their curriculum, leading to the development of a sharper sense of ethics in relation to sustainability in their future professional lives. As the professors, they also seen sustainability merely as a speciality subject and not something interdisciplinary, such a position demonstrates uncertainty or even a lack of knowledge on the part of the students concerning the relationship between sustainability and the structure of the course curriculum. Considering that, environmental sustainability should be revised, principally within the curricula context, so that it can offer greater security for future graduates in relation to what represents sustainability attitudes around them.

Comparison between the replies of the coordinators, professors and students
The aspects evaluated by professors and students from the use of the quickscans are related to one or more criteria of the AISHE methodology which was answered by the coordinators. In this way it was possible to compare the opinions of students, professors and coordinators about the same subject. This is presented in Table 1.  Partly agree that sustainability is present in its statute.
Neither agree nor disagree: sign of lack of knowledge about the subject.
Observance of environmental guidelines by professors, employees and students.
1.4 Some members of the academic community pay attention.
Did not agree that the guidelines are observed.
Did not agree that the guidelines are observed.
Existence of specialists in sustainable development.

and 2.3
The university possesses few specialists and when they do exists is due to self initiative.
The university possesses few specialists and when they do exists it is due to self initiative.
Neither agree nor disagree: sign of lack of knowledge about the subject.
Level of information about aspects related to sustainability in the field of expertise. Addressing of ethical aspects towards SD.

3.3
Only some professors emphasize the need of having sustainable attitudes.
-Neither agree nor disagree: that believe the ethical aspects are not satisfactorily addressed.
Contribution of academic research and extension in the knowledge of SD.

2.4
Believe that some activities of research and extension involve SD.
Partly agree, once not all are inserted in these activities and state that a more expressive presence could exist.
-Relevance of aspects about SD in evaluations.

4.3
Believe only some projects include SD elements.
Partly agree. Partly agree.
Graduates ability to act in favour of SD.
3.3 and 5.2 Believe that not all professors emphasize the importance of SD in professional conduct.
Agree that they are preparing graduates for this.
Partly agree that they are being prepared on the subject.
When the opinions of the coordinators, professors and students are compared, it can be observed that the professors were the most positive group of the three because they partly agreed with most of the aspects related to the presence of sustainability in the institution. The students showed insecurity at the moment they were asked to affirm something about sustainability at UPF, showing a lack of knowledge about the subject or even about the activities that are being developed at the university. This became clear as most of the students' answers were "neither agree nor disagree". The more negative evaluation of the coordinators in some aspects could be a reflection of the instrument of evaluation. The coordinators evaluate the courses through the vision of AISHE, which is far more detailed than quickscan. Because of this, even if they are covering the same subjects, the instruments use different evaluation scales, which make it difficult to make comparisons.
Another aspect that can be identified by the comparison of the three responses is the unanimous opinion that internal environmental management is not being observed and respected by everybody and that even when the institution strives to implement action towards sustainability, the involvement of the employees, professors and students is ineffective both in their commitment and responsibility.
In addition, the students who graduate from the institution only have knowledge of certain aspects of sustainable development; the professors feel more secure about their level of knowledge but it depends on individual initiatives. The necessity for specialists in this area was also perceived and it is recognized that the demand for such professionals is on the increase. Currently, the graduate student does not have sufficient knowledge and only some curriculum material is concerned with sustainable development.

Definition of the priority actions to incorporate SD in the university
For sustainable development to be included in the courses of an institution, four distinct stages can be used which consist of motivation, planning, implementation and expected results. Figure 7 presents these 4 stages with their components. The motivators were applied to the case in study (UPF).
The motivators include requirements from the Ministry of Education (MEC), a Brazilian governmental body which requires that Law n o 9795 for Environmental Education at the Universities is applied; the necessity of an institution to prepare a professional adequately for the work market and to prepare him/her to deal with questions related to sustainability; the decade of UNESCO for SD (UNESCO, 2004); and the EMSU event, 2008 where the UPF committed itself to applying the AISHE 1.0 tool (results obtained with this application, which demonstrated the need for better integration of the subject in the teaching practices of the institution).
The planning for inclusion of SD on the course curriculums of the institution should be envisaged as long term. They should also be implemented with interdisciplinarity and transdisciplinarity as their base.
In relation to implementation, action can be developed, not just in the university management, education, research and extension but also in university life in general. The results evoke expectation in a change in the profile of the students as a result of the inclusion of sustainability in their competencies, responsibility and personal involvement. Likewise, as the image of the Institution is improved, the better its involvement with social, economic and environmental problems through its research and extension activities. Lozano (2006) states that "university policies and strategies must be designed to holistically integrate SD as the golden thread throughout the university system". Thus, the first stage would be the construction of an institutional policy for sustainability, through the Vice director of teaching, which would involve the commitment of the superior levels of the university and would extend to all the institution courses, regardless of their specialty.
For a university to be able to progress in this subject, it is suggested that a system of environmental management be implemented and structured in accordance with the specific norms and in accordance with its own context, having as a general directive aspect, a clear definition of its policies; in this way education about sustainability at the UPF should make part of a programme of systematic aims and a plan of action for concrete application of them. The environmental management system should include activities directed towards teaching, research, outreach and university life.
Sustainability should not be understood to be a specialist subject but should be incorporated into the practices of each course. This does not mean to insert into the curriculum a subject about environmental issues or sustainability but it means introducing an attitude and behavior pattern into each student as a future member of society and a future professional.

Conclusion
This paper showed a case study in which institutional approaches to sustainability are described in an Brazilian University. Even as, the article performed an evaluation of sustainability in the curriculums of the UPF and through this study, showed that the university faces many difficulties to include this theme of sustainable development in a satisfactory manner in its curriculums; a reflection perhaps of a lack of policy directed towards sustainability in its academic and operational practices. This case study, the UPF could contribute to knowledge for other universities, especially in Brazil, or even in other contexts where there is on going discussion about the incorporation of SD in their curriculums.
It has been observed during the research that the introduction of SD curriculums should be interdisciplinary and transdisciplinary. However, this is not sufficient; SD should also permeate into the research and extension activities and general university campus life. Only in this way, through daily practice in the academic activities of students, professors and employees, will SD be incorporated in the professional preparation of these individuals.
In relation to the application of AISHE for the course coordinators, it has been noticed that on numerous occasions, difficulties arise in the understanding of certain terminology used in the questionnaire, a fact that corroborates the results of other research, which may well reflect a general lack of familiarity in some areas.
Considering this, it can be confirmed that the 30 courses which were evaluated at the University of Passo Fundo using the AISHE tool have been shown to be at the beginning of a process of insertion of sustainability into their curriculums.
At the moment of publication of this paper, a new version of AISHE, called AISHE 2.0, is about to be published after a careful international development process. Whereas the first version of AISHE paid ample attention to the education within a university and the underlying mission and philosophy, AISHE 2.0 is also dedicated to the other roles of a university: the research, the operations and the societal role (the community outreach). These four main roles together form four modules, to which a fifth (or rather zero-th) module, 'Identity' is added, forming the new assessment tool.
The repeated application of AISHE 2.0 in an iterative process will allow a university to reach and verify the state of system integration of sustainable development (SISD), not only considering its education and its mission, but in all elements of the university activities and in its very identity. For this reason, it is recommended for the University of Passo Fundo to start using AISHE 2.0, as this will allow the institution to reach the full state of SISD.

Additional file
Additional file 1: Table S1. General results of the 30 graduation courses at UPF University according to AISHE.