Vic Murray,

University of Victoria

Nonprofit organizations are under the gun from funders, government contractors and clients to show that they are “making a difference”. They eagerly seek valid and reliable measurements, which will yield clear indicators of how well they are doing. Many also yearn for proven “best practices” for managing their organizations which, when implemented, will ensure that they run as efficiently and effectively as they can. This way, if they are unable to show directly the impact of their efforts, at least they can assure concerned stakeholders that they are doing things the way they “ought” to be done.

Is there a set of performance indicators which can be easily applied by many different kinds of charitable organizations and which will yield the clear picture of impact that stakeholders want? Are there universally agreed-upon “best practices” applicable to all nonprofits which have been proven to increase performance? This short article looks at the answers to these questions by reviewing a few of the systems, which have been put forward as usable for assessing performance in the nonprofit sector. In a nutshell this critical appraisal concludes that the answers to both of the above questions is: “not yet”. But there are some promising leads in the form of processes which a nonprofit organization and its stakeholders can adopt which could help improve the analysis of performance and, most importantly, lead to better decisions on needed changes that are right for it (if not for others).

In order to “evaluate the evaluation systems”, we need first to have a clear idea about what an ideal evaluation system would look like. In a nutshell it would need to have:

1. A clear statement of its objectives. They should be S.M.A.R.T.: Specific, Measurable, Achievable, Relevant and Timebound (specific in terms of the time in which they are to be achieved).

2. A clear statement of the desired outcomes —both positive effects and the absence of negative “side effects”. This means creating:

· Indicators or measures that fully reflect the desired outcomes and possible side-effects;

· A process for interpreting the results of these measurements, i.e. what are “good”, “average” or “poor” results? This means there must be one or more pre-established, absolute standards against which performance can be measured in a given time period, or relative standards that allow comparisons with others or over time periods.

3. Methods for producing the data on the indicators which are timely and feasible to use in terms of cost and effort. They must also be valid (measure what they intend to measure) and reliable (produce consistently accurate results every time they are used).

4. Predetermined agreements as to how to interpret the results obtained by the measurements used. Professional evaluators call these agreements “logic models” There are two basic logic models required:

(a) “Measurement” logic models make clear the assumed links between input indicators and process indicators, process indicators and outcome indicators and outcome indicators and goals. They should also identify the other factors that affect the indicators at each stage but are not directly controllable by those responsible. Finally, they should attempt to predict possible side effects and show how these will be measured.

(b) “Level of focus” logic models link evaluations between levels. That is, they make explicit the links between evaluations of individuals, programs or organizational units, organizations and systems or sub-systems.

5. Once the system has been designed, it must be implemented with integrity by all those involved.

The only problem with this ideal is that it is virtually impossible to reach if we are talking about the performance of nonprofit organizations. There are rarely clear causal links between inputs and outputs (e.g. putting more money into something does not guarantee results) or between outputs and outcomes (e.g. more drug enforcement activity does not seem to be reducing illicit drug dependency). Because of this it is difficult to develop the needed logic models. As well, it is often difficult to gather information on long-term outcomes. And when information is obtained, it is often unclear as to how to interpret it—what is it really telling us about how well the organization’s mission is being achieved and why the results came out as they did. There is also the problem that those who provide the information may not have the time, resources or motivation to do it well. In fact, insofar as they believe, even unconsciously, that they themselves might be in any way punished for the results being reported, many will be tempted to distort or “spin” the information that report so as to look good and avoid blame. In a nutshell, evaluation will always be a subjective process involving a negotiated interpretation of reality between all the parties involved in it.

This said, what can we say about the pros and cons of the existing evaluation tools and approaches? We will look briefly at three of the best of those that are offered for evaluating the organization as a whole. Space limits prohibit a full description of each of these so those interested in more detail are invited to check them out and make their own judgments. Websites for each of them are noted.

First some general comments. Two of the evaluation tools focus on measuring processes, activities or outputs, rather than outcomes. They make the implicit assumption that if the processes are performed well the outcomes will follow. The problem is that there is no publicly available systematic research that actually checks this assumption. It is not known whether getting “high scores” on the process indicators in these evaluation systems actually results in an organization performing more effectively or efficiently.

Another problem is that very little attention is paid in any of them to how the evaluation system is to be implemented. In our research (Cutt and Murray, 2000), we concluded that, unless all those to be affected by an evaluation system have a strong voice in its design and accept the final product, there is a high probability that the system will fail.

Finally, none of the systems discussed below have themselves been evaluated. There is no information on how widely they have been adopted, how long they have remained in place once adopted or how satisfied the users of the systems are. Given these general caveats, let us now look at the systems themselves. They are all intended primarily for use within the organization by its board or management team as a way of assessing organizational performance.

1. Program Outcomes: The United Way Approach (http://http://www.unitedway.org/outcomes)

The United Way's is the one evaluation system that focuses explicitly on the identification and measurement of program outcomes for United Way funded agencies. While the system starts by evaluating results at the program level, these are supposed to be aggregated at the organizational level by member agencies to report on their effectiveness.

The outcome information is intended to be used by the United Way to help member agencies improve program performance, identify and achieve United Way priorities (funding allocations) and to broaden the base of financial and volunteer support (fundraising).

Recent research reported on the United Way website suggests that there is general satisfaction with the six stage process recommended by the United Way and the proper way to develop the outcome measurement system. This process is the most sensitive to the importance of implementation of all those reviewed. It is strongly recommended as a model to consider but would be best used in conjunction with either or both of the two process-oriented systems discussed below.

2. The Balanced Scorecard (http://www.balancedscorecard.org/)

This is a multi-attribute system for conceptualizing and measuring performance designed originally for business organizations and currently being adapted for non-profit organizations. In its original form it assumes that the primary goal of a business is long-run profit maximization. It argues that this will be achieved through a “balanced scorecard of performance attributes” grouped around 4 “perspectives”:

· The Financial Perspective measuring various financial performance indicators of primary interest to shareholders;

· The Customer Perspective comprising measures of customer satisfaction;

· The Internal Business Perspective which measures internal efficiency and quality; and

· The Innovation and Learning Perspective which attempts to measure the organization’s ability to adapt to changes required by a changing environment.

In the case of nonprofit organizations, their mission statement becomes the endpoint to be reached through these perspectives. “Customers” must be replaced by “clients” or “users” of the organization’s services and the “financial perspective” becomes that of the funders or potential funders. While there is still a great deal of work to do to flesh out exactly how to apply this system to nonprofit organizations, it does show considerable promise.

3. Canadian Comprehensive Auditing Foundation Framework for Performance Reporting (http://www.ccaf-fcvi.com/) is the most significant Canadian effort at tackling the “value-for-money” issue in both the public sector and the non-profit sector. It puts forward 12 “attributes of effectiveness” suggesting that organizations can be audited in terms of how well they manifest these attributes. In this sense it can be focused at either the organization or program/function levels and is intended to help evaluators get a clear picture of how effectively and efficiently the mission is being achieved. It is similar in many ways to “The Balanced Scorecard”, however the details of implementation are more thorough.

The system involves developing indicators for the following 12 attributes of effectiveness:

1. Management Direction: The extent to which programmatic objectives are clearly stated and understood;

2. Relevance: The extent to which the organization or program continues to make sense with respect to the problems or conditions to which it was intended to respond;

3. Appropriateness: The extent to which the design of the organization or program and the level of effort are logical in relation to their objectives;

4. Achievement of intended results: the extend to which the goals and objectives have been achieved;

5. Acceptance: The extent to which the stakeholders for whom the organization or program is designed judge it to be satisfactory;

6. Secondary Impacts: The extent to which significant consequences, either intended or unintended and either positive or negative have occurred;

7. Costs and Productivity: The relationship between costs, inputs and outputs;

8. Responsiveness: The capacity of the program or organization to adapt to changes in such factors as markets, competition, available funding and technology;

9. Financial Results: Accounting for revenues and expenditures and for assets and liabilities;

10. Working Environment: The extent to which the organization or program provides an appropriate work environment for staff, and staff have the information, capacities and disposition to serve the objectives;

11. Protection of Assets: The extent to which the various assets entrusted to the organization or program (physical, technological, financial and human) are safeguarded; and

12. Monitoring and Reporting: The extent to which key matters pertaining to performance and organizational or program strength are identified, reported and monitored.

It can be seen from the above that the system encompasses both process and outcome elements though the former are dominant. While, again, there is no information on how effective this evaluation system is, there is potential in trying to merge it with The Balanced Scorecard Approach.

Space limitations prohibit discussion of several evaluation systems developed to help funders and the public decide how effective various charities are. See, for example: the National Standards Information Bureau (www.give.org), the American Institute for Philanthropy Charity Rating Guide (www.charitywatch.org), and the Charities Review Council of Minnesota (www.crcmn.org). All of these systems try to offer generic process or output “standards” that will reveal how well a nonprofit organization is managed. Regrettably, there is little or no published research which supports these assumptions and all of them fail to discuss outcomes.

In conclusion, it appears that there is still a long way to go before there will be available a tried and tested evaluation system that can be applied by most nonprofit organizations to reveal a valid picture of how well the organization is performing. Some would argue there is no point in trying yet decisions are made every day based on assumptions and idiosyncratic perceptions of performance. The process of trying to move the dialogue on this process to a more rational, data-based approach is worth pursuing.