• No results found

Programme assessment and improvement requirements

5. ESTABLISHMENT AND ADMINISTRATION OF FOREIGN MATERIAL

5.5. Establishing and describing administrative controls

5.5.11. Programme assessment and improvement requirements

Management (and frontline) needs to be aware and informed of the strengths and weaknesses of their own organisation, other organisations and interfaces in establishing, administering, maintaining and applying FMM and FMMP. Lack of an effective and structured process to identify, analyse and correct (and keep records) issues, deficiencies and weaknesses in FMMP and FMM practices, as well as the expression of views to the decision and programme authorities, would result in accumulation of errors and deficiencies in conduct of and processes for FMM. Consequently, this accumulated errors and deficiencies, some of which would go unnoticed until an impact appears, could lead to significant FMI events that jeopardise safety and reliability and/or efficient and effective performance of the nuclear power plant.

Therefore, the plant/project management needs to proactively check, identify and correct weaknesses in FMMP and associated programmes, processes, practices and procedures, which requires their (and relevant plant staff’s) active support for routinely observing, reporting, monitoring, trending and assessing to identify any areas for improvement.

More importantly, improvement process also necessitates the establishment, application and improvement of tangible measures and metrics to assess the status of FMMP, its implementation and FMM practices regarding the detection of declining (or improving) performance or measuring effectiveness, which are discussed in the following Sections.

5.5.11.1. Measuring effectiveness

To maintain an effective FMMP, the organisation needs to develop measures to identify whether the activities relating to FM and FMM are deteriorating or are being maintained or improving, and more importantly, to determine actions to control, manage and improve the programme and its implementation, as stated in a widely referred quote by Dr H. James Harrington:

“Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it”.

138

Before any further discussions on the methods and tools for measuring, it is important to emphasise a few points for the measurement process and efforts:

— The purpose of measuring programme effectiveness is to understand, and consequently, to identify actions to manage and improve, the programme and its implementation;

— Measuring programme effectiveness is a part of the learning and informing culture towards continuous improvement and sincere organisational commitment to the safe and efficient operation.

Moreover, numerical measures (e.g. indicators and metrics) for measuring programme effectiveness:

— Are to understand and to interpret programme’s status and trends;

— Are not criteria for rewarding or punishing individuals or organisations;

— Are not for counting events/errors/failures.

As stated by the INSAG, in the publication INSAG-13:

“Numerical measures must always be subject to careful interpretation and be used as part of an overall judgement about performance. They should not be regarded as an end in themselves” [66].

Therefore, the measures (e.g. indicators and metrics) are to be selected and tailored primarily towards identification of the underlying causes and precursors of any defect and deficiency (as well as the strength) in FMMP and associated process and procedures. This identification, in turn, will lead to determination and implementation of corrective (or, in cases of strengths, cultivating) actions.

Additionally, the performance indicators serve to ensure that decision and programme authorities become, and remain, aware of actual practices and values in the field, including those of external organisations (working either at the owner/operating organisation’s site and facilities site or at their offsite facilities). Measures also provide avenues to set/revise performance goals and expectations and gaps in overall performance, and to communicate them to plant/site personnel.

Therefore, effective organisations develop and use metrics/indicators to monitor, measure and improve performance  including the FMMP performance  both reactively and proactively, by observing, collecting, analysing and assessing such indicators and their trends. Development of metrics for activities and processes could be undertaken in a manner similar to that described in Ref. [66]. For example, a set of indicators for FMMP may reactively measure effectiveness of:

— Recent and current performance of FM and FMM related tasks;

— Current awareness of past performance;

— Effectiveness of determination and implementation of preventive and corrective actions;

— The attitudes and behaviour of staff, managers and authorities.

It is also important to establish and use forward looking (sometimes referred to as ‘proactive’

or ‘leading’) metrics/indicators to measure the awareness, identification and recognition of potential efforts to improve FMMP, the associated processes and FMM practices in the conduct of future activities. While such proactive measurements provide opportunities to anticipate, predict and pay attention, for example, to developing or accumulating issues, they indicate

early signs of declining performance and processes in order to take proactive measure to correct or change path and trend towards inevitable failures.

The indicators could also be quantitative or qualitative, latter of which, particularly forward looking, could be very difficult to develop and tangibly assess. For example, measurements of general, or task specific, personnel behaviour and attitudes in FMM are typically qualitative in nature. On the other hand, the quantitative indicators (metrics) are relatively easier to develop and assess and provide a more practical input to the monitoring and assessment of performance of activities with the observance of FMM. Quantitative indicators can also be tailored to measure the health of FMMP, and associated processes and procedures, by monitoring, for example:

— Number or consequences of errors in communication of FMM information and errors in the execution of activity (such as both in its performance and in its inspection) which may provide indications of:

 Lack of competencies and skills;

 Insufficient or deficient work instructions, procedures;

 Insufficient or inadequate training;

 Values/importance of FMM to staff and management;

— Repeated FMM incidents and deficiencies  when underlying errors and deficiencies could be quantified  which typically provide a measure/indication of:

 Failures or defects in the CAP as to the determination and implementation of adequate, timely or effective corrective actions after the earlier incidents/deficiencies;

 Organisational problems, such as not being a learning organisation, particularly in such cases where the errors are the same as the ones that occurred in the past (internally or externally);

 Issues with the correctness and adequacy of previous assessments or with analysis and communication of OPEX (which may also point to issues, for example, with OPEX programmes and processes).

Concerning the last bullet on repeated incidents and deficiencies, it should be noted that both consequential (FMI event) and non-consequential (FM near miss or close call) incidents usually have similar, if not the same, underlying causes. Therefore, being aware and correcting the causes of non-consequential also contributes to the measurement of the effectiveness of the programmes and processes and help to take corrective or improving actions to prevent potential future events. Accordingly, the monitoring and assessment of all incidents need to be included in the FMMP improvement.

5.5.11.2. Key performance indicators

If one considers the extent of administering, maintaining and applying FMM and FMMP in addition to interfacing/interacting programmes, processes and procedures, there may be a large amount of performance indicators either in the FMMP or, since the management system integrates them, in other relevant programmes. However, a smaller set of high value (as to measure and be informative towards the improvement/correction) performance indicators/metrics that would represent substantial portion of characteristics of past, current and future status and performance with low impact (as to requiring resources), to monitor and

140

assess FMM and FMMP, as the ‘key’ indicators  which are called key performance indicators (KPIs). The KPIs are, therefore, selected and used to measure the significant portion13 of overall FMMP health status and trend.

The selection and use of KPIs are primarily based on the corporate strategy, safety and performance goals and FMM commitment and policy as to why something needs to measure and what needs to be measured for that purpose. Therefore, FMMP KPIs maybe different for each owner/operating organisation; however, an organisation does not need to start from scratch to establish their FMMP KPIs since:

— There have been joined efforts by nuclear plant owner/operating organisations that established a set of common FMMP KPIs. For example, INPO [9] and WANO [11] have published industry developed and adopted metrics which categorise FMI events based on the severity or consequence of the events. A points system is then used in conjunction with level of significance and is presented on a monthly basis as well as a rolling average which is used as an indicator for measuring the overall number and significance of FMI events. Appendix III provides an example of this KPI and the metrics associated with it.

A variation of this indicator, in addition to others, can also be found in EPRI guidelines [10];

— There may be already existing KPIs for the other programmes and processes within the plan/project organisation;

— There are already established KPIs for FMM, FMMP in many other industries, such as aviation, medicine, pharmaceutical.

Accordingly, an organisation could use such existing KPIs (or adopt a derivation of/from them) for their FMM and FMMP performance measure, if the existing KPIs or a link/derivative of those are adequate and sufficient. For example, there might be an existing KPI used by the plant/project organisation for measuring the Maintenance Programme performance, say:

‘actual versus expected maintenance duration’, which can be derived into a FMMP KPI as

‘outage extension due to FMI’.

Some organisations may still desire, however, to measure other aspects that is not associated with any existing or derived indicators and, thus, need to create a totally new KPI indicator that is more suitable and applicable to their specific or unique conditions, strategies and areas of interest/attention to help with continual improvement.

It should be noted, from the OPEX, that sometimes performance indicators that is not significant for programme/process improvement are erroneously considered as ‘key’, which can be detrimental as follows:

“As a result of considering performance indicators that have no or little value to the improvement of FMM or FMMP erroneously considered as ‘key’, tracking, managing, assessing, reviewing and reporting efforts, and as such dedicated resources (human and financial), unnecessarily increase. Large amounts of unfocused/untargeted/useless (i.e. non-key) indicators that deemed as ‘key’ without a basis or with no application to improvement, eventually become a burden for the entire organisation. This burden continues to increase to a point that the management could totally stop using

13 As a commonly known rule of thumb, the 80-20 approach (commonly referred as the Pareto principle), i.e. 80 per cent of consequences come from 20 per cent of causes, could be a good practice in starting the selection of KPIs.

all programmatic KPIs, including the good ones, as performance indicators seem not adding too much value but having too much impact on the human and financial resources”.

Regardless of numbers or types of KPIs, the FMMP governing procedure needs to clearly specify and describe programme effectiveness measurement KPIs as to:

— Meaning of (i.e. what it is measuring) and reason (i.e. why it is measuring) of KPI;

— Grading scale and associated criteria/thresholds/rules;

— Weighting and the basis of weighting;

— Analysis and assessment methods and tools;

— Reporting requirements (as to role and responsibilities, scope and content, communication type, media and periodicity, etc.);

— Management review response and feedback process;

— Process of reporting and resolving adverse results and management feedback.

Again, it is prudent to reiterate that the reasons for selecting and the application of KPIs are:

— Foremost and certainly, not about establishing criterion for rewarding or punishing individuals or organisations;

— Not for counting events/errors/failures14;

— Not about having or creating artificial or ‘to go on the books’ metrics that have no use for programme assessment and improvement  as unnecessary KPIs will cause a blurred vision of the programme status and trend resulting in incorrect or ineffective management decisions and actions that are not effective or that create more issues.

5.5.11.3. Observation and reporting programmes and processes

The primary input to programme assessment and improvement is the checked, identified and recognised issues, challenges, adverse conditions and weaknesses (and strengths) in activities, people or environment involving FMM and associated programme, processes, practices and procedures. The collection of this input requires management’s and relevant plant staff’s active support for routinely monitoring, observing and reporting to identify and correct/improve.

Typically, monitoring and observation of FMM activities at the field need to be done at several levels:

— By workers during their (and their peers’) activities;

— By FMM experts (i.e. the FMM Conscience of the plant/project) during, administration, coordination and field verification of the programme and associated activities;

— By supervisors, managers and executives during their field visits and inspections;

— By certain plant staff that are performing routine non-FMMP related tasks in the general areas, for example, operations and security personnel;

— By all plant/project employees during their general routes, areas and activities.

As discussed in Section 5.3.2, (although the clear ownership of the FMMP itself is assigned to certain organisation and people) people at all levels within the organisation have ownership of

14 Noting that, in some industry groups, this is one of the KPIs for FMM and FMMP ; however, it has a particular

purpose, which is to have a standard criterion to compare/grade plants and to assess overall industry performance; and is not considered directly to indicate/measure the improvement of people or programme by a specific organisation, within the context of their corporate strategy and culture.

142

their activities and ownership in oversight, assessment and improvement of the programme.

Accordingly, everyone (including the management) is responsible to ensure that periodic observations are conducted to monitor compliance with (and practice of) FMM requirements, expectations and behaviours.

For example, plant managers and supervisors conducting observation by way of frequent tours of plant areas can confirm, by seeing it first hand, that standards are maintained and that any deficiencies are identified, controlled and eliminated and activity workers; and other frontline personnel can observe and report conditions that may cause FM and FMM problems. Security personnel can be provided with FMM training on types of items and situations that they can observe during their patrols, so that the FMMP can be enhanced by taking advantage of these additional field observations.

To ensure observations are detecting and correcting at risk behaviours, observers need to be adequately knowledgeable of the FM, FMM and FMMP, to identify, recognise and interpret the observed/monitored issue. Therefore, it is necessary to consider, for example, to:

— Provide appropriate and sufficient training (as described in Section 5.5.8) at each level in accordance with their required, expected and desired involvement in monitoring, observing and reporting;

— To conduct paired observations (for example, walk down teams consisting of FMM/FMMP expert(s) and frontline and management staff) to ensure those performing observations are adequately skilled to spot deficiencies and provide the proper timely input for correction needed.

Therefore, plant’s/project’s observation programme(s) need to specifically identify the needs to observe FMM practices during appropriate work activities. As such, the FMMP governing documents need to define or describe observation activities as to where and when to look for what and how to report it. These governing descriptions include the settings, e.g. during job briefings, activity implementation and general area walkdowns, to discuss and ascertain whether, for example:

— Pre-job briefings address specific FMM activities and expectations;

— FMCA controls are determined and implemented;

— Appropriate barriers and signs are installed and clear to understand;

— Personal items are secured or controlled;

— Tools going in and out of FMCAs are inspected, are clean and serviceable;

— Lanyards are used as required or tools are failsafe;

— Lay down or disassembly areas are established as necessary;

— FMM documentation in the work packages is complete;

— FM reconciliation logs are established and kept properly;

— The use of FMC devices and techniques conform to the plant/project standards;

— ‘Clean as you go’ and other behaviours for cleanliness and housekeeping is practiced.

These definitions and descriptions from the FMMP governing documents might be incorporated into the implementing procedures or formulated to provide quick and easy reminders of what to look for during job briefings, activity implementation and general areas.

Overall, it is essential that the tracking of observations and findings are sincere and systematic to allow, for easy review and assessment to assist with program health evaluation, and more importantly, to encourage desired ownership, observing and reporting behaviours.

5.5.11.4. Assessments and audits

In an effective and continuously improving FMMP, all the elements of the FMMP are subject to periodic or causal assessments and audits, in accordance with the established programme guidelines and processes. In depth assessments and audits may be performed regularly or in response to trend or event findings and may involve document reviews, observations or interviews with individuals who regularly perform, coordinate, oversee manage and direct work, tactics and strategies with FMM implications.

It is important to note that, the assessment and audits ought to focus on the programme, processes and procedures and deficiencies/defects in the management, coordination and implementation system, rather than the examination and critique of the individual performances.

Assessments and audits can be performed by parties who are internal or external to the programme owner within the owner/operating organisation (i.e., self-audit) or the site organisation, at large (i.e., peer audit/evaluation). Self-evaluations, i.e. the self-assessments and audits conducted in a target organisation or at the overall owner/operating organisation levels, achieve improvement through:

— Building common commitment to corrective action and key attributes within the owner/operating organisation;

— Getting management and staff of all involved entities in the identification of problems in their own and peer organisations;

— Providing experience and assistance in identifying, assessing and correcting problems;

— Focusing on programmatic performance problems and deficiencies at all levels;

— Providing training in observation, assessment and auditing skills, competencies and expertise.

On the other hand, the peer evaluations, audits and assessments are very effective in identification and documentation of problem (or are to be problem) areas. Even in the self-assessments, it is a good practice to involve some external experts to provide an objective and unbiased peer view as well as the assistance with identifying and introducing good practices used elsewhere. (Here, it should be noted that when they are considered to be adopted, ‘Good’

practices of other plant/project owner/operating organisations need to be carefully evaluated for their effective applicability with the same process or to obtain same results. One’s good practice may be ‘not good’ when applied to own organisations owing to the differences in organisational culture, skills, competencies, experience, work processes, corporate strategy).

Whether it is self or peer assessments or audits, the line management of the FMMP line management (and the line management of the department(s) that was targeted in the assessment) is responsible and accountable to resolve the identified issues, i.e. finding.

Similarly, only high-level management is capable of making the changes to improve decision making processes. Moreover, for these assessments and audits to ultimately result in improved FMMP and FMM performance of the entire plant/project organisation, there needs to be a strong executive and line management commitment for finding the ‘truth’, for receiving and accepting the results and timely implementation of appropriate corrective actions within the organisation. Furthermore, in the case of findings pointing to external organisations, the executive management ensures that they are communicated, and the corrective actions are implemented in those organisations.

144

The assessments need to be designed to verify how the FMMP, requirements, expectations, principles and objectives are being implemented and cover the following aspects, as a minimum:

— Standards, guidelines, procedures and expectations used for the FMMP;

— The definition and implementation of responsibilities for the FMMP;

— The incorporation of FMM and FMCs into work plans and work instructions;

— Maintenance practices and techniques;

— Inclusion of FMM specifics in job briefs;

— Availability and accessibility of FMC/FMM materials and tools;

— Management and supervision knowledge, behaviours and practices;

— FMM specific training for the direct staff and contractors;

— Contractor and vendor FMMP knowledge and compliance;

— Warehouse storage and transport standards;

— Housekeeping and cleanliness standards, behaviours and practices;

— Identification and reporting criteria and practices for FMI incidents and trending;

— Communication and use of OPEX, internal feedback, including the analysis of FMI events;

— Effectiveness of previous corrective actions;

— Results of self-assessments, independent reviews, peer reviews and benchmarks;

— Performance indicators and their effectiveness for FMMP status and improvements.

In addition, results of regulatory reviews and inspections, as well as results of peer reviews and benchmark need also be taken into account in the assessments of the overall effectiveness of the FMMP. Corrective actions and opportunities for improvement need to be identified and implemented.

The FMMP administrative procedure needs to identify and define the requirements and expectations for conducting an assessment or audit. These definitions include the type, scope and periodicity, as well as the possible causes, for conducting and receiving assessments (self-assessment, peer assessment) or independent audits at all levels of the organisation. The administrative procedure also needs to identify and define (or refer to other programmes/processes) the methods for evaluating the assessment/audit results and implements corrective actions.

5.5.11.5. Recognition of declining programme effectiveness and conduct

The results of regular and consistent monitoring, observing, assessing and trending provide important input to the viewing overall instructional and compliance picture, while the results of assessments and audits provide in relation to FMMP administration, coordination and implementation in an organisation. However, they all need to be interpreted and understood in order to recognise and act upon the attributes of declining performance for the improvement FMMP. Based on OPEX and past observations, typical attributes of declining performance include:

— Lack of management commitment and support for the FMMP;

— Inadequate or no communication of expectations and standards for FMM to those concerned (plant/project staff and contractors);

— Bypassed or worked around FMCs because either seen as obstacles or delaying meeting schedule demands;

— FMM and FMMP are seen as a burden and useless;