HREC Self Evaluation Guide
2. Determine your methods
What method or series of methods will best suit your HREC’s approach to evaluation?
Overall, the methods include a range of informal and formal methods, or a combination of both.
2.1 Informal methods
Informal evaluation methods, usually some form of verbal feedback, are generally undertaken on a more regular basis than formal evaluations.
- Discussion about the effectiveness of meetings is included as a standing agenda item at the end of every meeting.
- Discussion that is scheduled to occur at regular times during the year (e.g. every 3 meetings), or an annual basis.
- Discussion facilitated by the Chair as required.
- Verbal feedback from a nominated committee member on a rotating basis (someone is tasked with providing observations at the end of each meeting).
There are advantages and disadvantages to informal methods.
Possible advantages of regular informal discussion:
- It provides more immediate and ongoing feedback, which provides opportunity to continually improve the committee procedures and processes.
- Its regularity means committee members become accustomed to the process and may feel free to give critical as well as positive feedback
- Can be easy to initiate and requires no additional resources other than people’s time
Possible disadvantages of informal discussion:
- the informality of the method can inhibit frank feedback on significant issues of concern
- some members may dominate and others not participate equally in the discussion
- members may not feel particularly energised to contribute to an end-of-meeting process
- discussion is confined to committee members only and excludes the input of HREC support staff, research office staff and other stakeholders.
2.2 Formal methods
Formal methods, both qualitative and quantitative, are successfully used by many types of committees and boards. The most common methods used are:
- survey or questionnaire
- facilitated group discussion
- interviews with individual committee members
There are advantages as well as disadvantages for choosing formal methods over informal options, so each approach is worth some consideration.
If you are choosing a more formal approach, it is common for it to involve a combination of tools. For example, data from surveys or interviews can be fed back to the committee via a facilitated discussion.
Here we provide examples of common methods and links to tools that can be used or adapted to your needs.
Some advantages of formal approaches:
- feedback is generally more structured and objective
- opportunity for all committee members to contribute feedback
- can provide for confidential feedback which may be more frank and honest
- provides more in-depth information or data
- provides opportunity for reflective and thoughtful responses and suggestions
chosen methods can offer opportunity for structured feedback from other stakeholders.
Some disadvantages of formal approaches:
- some level of resourcing is required, such as additional staff time, facilitator fees etc.
- some members, particularly external members, may consider it too demanding of their time
if there is disharmony or ‘factions’ within a committee, distrust may impact on the effectiveness of the tools.
Survey or Questionnaire
Depending on their length and style, surveys or questionnaires can produce a rich source of data for HREC evaluations. Like all methods they offer strengths and weaknesses.
- Surveys are a common feature of modern life, so people are generally comfortable and experienced in completing them.
- They can be a very efficient way of obtaining confidential feedback.
- People can complete them in their own time, rather than committee time.
- Questions that are specific to the needs of the committee can drafted or included.
When repeated with standardised responses they can provide useful comparative data that enables a committee to measure its performance over time.
- The data gathered is generally not detailed or in-depth, so may not provide sufficient information on differing viewpoints or suggestions for improvement.
- They rely on all members participating to ensure sufficient numbers for feedback purposes.
People can feel inhibited in their responses, or not offer useful criticisms, if they don’t trust the confidentiality of data input processes, or think that any comments will be identifiable.
The style and content of surveys vary enormously. They can be:
- very specific to the needs of the committee or very general tools
- short or detailed, requiring only a few minutes or more time for completion
- include open-ended questions to elicit further, more detailed, responses
Facilitated Group Discussion
This method provides HREC members with an opportunity to reflect upon and discuss their individual and collective efforts over a period (generally a year), to recognise the committee’s achievements, identify any issues, and discuss ideas for improvements in the way it operates.
Of course, there are both strengths and weaknesses to using this method, especially just on its own. Its strengths can be enhanced when used in conjunction with another method. For instance, it can be beneficial to have a facilitated discussion that is structured around the results of a survey, or interview data, or the results of an audit of policies and procedures.
- It provides time away from core work of the committee to consider its overall function.
- It can be scheduled as a relatively time-efficient process of review.
- The focussed discussion can effectively identify issues that are creating impediments to the quality of its work
- If well handled, it assists members to open up about, address and resolve any tensions in a supported or safe way
- Poorly developed aims and agendas can lead to frustration and disappointment among members.
- It relies on the expertise of a well-briefed facilitator, including their capacity to manage discussions and draw out themes.
- It does not provide the opportunity for confidential input from committee members, unless used in conjunction with other data sources.
Resources may be required e.g., to hire experienced external facilitators.
Facilitation is a process that requires good communication skills, impartiality, sensitivity and tact, and the ability to synthesise and summarise different strands of a discussion.
The ideal facilitator is someone who has no particular investment in the process other than to assist the group to formulate key issues through a managed discussion and reach agreed outcomes.
Regardless of whether the facilitator is sourced from within the host institution or externally, independence from the HREC is highly recommended. It can be helpful if the person has some knowledge of the HREC’s roles and the focus of their work, but not essential. If the objectives and parameters of the discussion are clearly articulated from the outset, an experienced facilitator can generally accomplish a satisfactory outcome for participants.
Interviews conducted with HREC members (and other stakeholders as required) create the possibility of exploring perspectives and contentious issues with a significant degree of depth and insight.
However, this method demands a considered approach, with approval from the whole committee, given the potential sensitivities involved.
There are strengths and weaknesses to consider. There are also basic planning questions to resolve if you choose this method.
- When done well, the method provides a rich insight into issues – obvious and underlying – that may be inhibiting the committee’s work.
- It promotes the emergence of useful and creative suggestions for continuous improvement.
- When sequenced to follow a committee survey, it can elaborate very well on issues that have emerged.
- It enables members that may feel less confident or excluded or intimidated by other members or by the committee process to contribute meaningful insights.
- HREC members are generally familiar with the method through their review work and therefore will be comfortable with the process.
- It can provide members with the opportunity to reflect upon and assess their individual contribution to the committee in a safe and confidential environment.
- The selection and performance of the interviewer can impact on the cooperation of interviewees and undermine trust in the outcomes.
- Buy-in from all participants is essential to worthy outcomes.
- Concerns about confidentiality can undermine its value.
- Members may have concerns that others will be privately critical of them, potentially sowing seeds of distrust in the committee.
- Resource requirements (e.g. budget and time allocation) can be a barrier to its successful execution.
If your HREC already conducts some form of regular review, then the Interview method may only be required once in each term of the committee (e.g. every three years). However, if your HREC does not use some established form of review, it can be very beneficial to implement a combined survey + interview method.
Questions to consider if choosing the interview method
- Who will be interviewed? Will the process be confined to committee members or expanded to include HREC administrators, research governance staff, others (eg. a sample set of researchers who have submitted applications)?
- Who will conduct the interviews? For instance, some committees utilise a process of one-to-one discussions between Chair and committee members; others seek a more independent process.
- What are the themes to be explored in interviews? Has the HREC contributed its own ideas or priority areas? How structured or open-ended will the process be? Will committee members be asked to provide comments on their own performance? Other members’ performance?
- How will confidentiality be managed and maintained? Who will collate and analyse the interview notes or transcripts? Is that solely the responsibility of the interviewer or will others have involved? How will the collated data be reported back? Who is responsible for deleting the raw data after the report is finalised and the results are fed back to the HREC?