Monitoring & Evaluation,
Follow the trail
- Planning tips.
- Monitoring and Quality The various standards and initiatives.
- Resources - publications, software, web-based.
- Management Publications for related titles.
- Marketing speak.
- Organisational Management - structure, leadership, stakeholders, change management etc.
- Information Management page has some thoughts on what Knowledge Management actually is.
A briefings page, within the Management grouping.
This is not just something you get consultants or professionals to do! There is an increasing trend towards evaluation, quality management, strategic planning etc. Perfomance measurement (and reporting) is increasingly a funding requirement, and should be integrated into a performance improvement (or maintenance) strategy as far as possible.
Some people think all this gets in the way of getting on and doing the work, which can be true, but don't you want to be sure you are actually doing useful stuff effectively, and getting the most out of your limited resources? Below we give some basics on the jargon and ideas which consultants use - while they can often bring in a wider view and feed in other organisations experiences, their main benefit is often forcing you to step back from the daily grind and take stock. Note that some jargon can be used differently depending on people's background and experience - stopping to agree meanings at the start of an exercise can be worthwhile, if tedious!
You can do this yourselves if you are serious. Just make sure that the time, money, formality, expertise etc. that you use are appropriate for the size and complexity of the organisation (or unit) and the issues you wish to tackle. This page should be a starting point in deciding how to do this.
Where should you be concentrating your energies? Assess what is important in your operations, and assess their performance (in your terms). Fit into the matrix below (adapted by OUBS course B752 from Slack), and take appropriate action. For example if you are good at activities which are of only low importance (to your goals), you may be able to release this 'excessive' attention, and use it to take urgent action on bad performers which will have greater impact overall.
SWOT One of the classic approaches to where you are and where you might go. Do a matrix examining the organisation's Strengths and Weaknesses, Opportunities and Threats. Concentrate on exploiting the S and O, reducing or avoiding the W and T. STEP analysis is used alongside this - look at the various factors impacting on the organisation - Sociological, Technological, Economic, Political. Sometimes Environment and Values are added to this collection. We've also seen PESTLE with the L for Legal.
Mission, Goals, Objectives, Targets Terminology here can be particularly problematic - Vision, values, aims, activities could equally well be used. Basically you need some structure that goes from the long-term Broad Purpose (why) through medium-term 'what can we achieve' to the immediate (within next year) and specific hows which can be costed out in detail. A 'pyramid of purpose' is illustrated below (bottom level of targets left off as needs better drawing software!).
More on Mission on the Organisation Management page.
SMART Are your proposals (objectives, targets) Specific, Measurable, Achievable, Relevant and Timed?(Have also seen items 3 and 4 stated as Actionable, Realistic.) If they can't be phrased in this way, maybe they are operating principles - e.g. 'working in partnership' is an underlying approach rather than an activity in its own right.
Three dimensions Activities, Resources and Goals/Objectives all inter-relate. While you can look at one of these areas at a time, you must then see what effect any changes or developments in one will have on the others and re-visit your original ideas if this throws up problems or further issues.
Another three dimensions
Adapted from Open University B752 'competitive performance management' summary diagram. These three functional areas have to work together in achieving organisational objectives. The precise split and labels may differ (eg marketing may be called fundraising or public relations and, rather than determining the mix of products and services, will be responsible for 'selling' the desired mix to potential funders), but the necessity for having a balanced approach remains.
Measuring a project's success, to satisfy funders, trustees but also to address shortcomings, make changes as part of learning process. Not just inputs (resources used, such as money, volunteer time) but outputs (e.g. level of activities), outcomes (what actually happened/changed as a result) and even impact (moving towards achieving the mission). What happened along the way on personnel, finances, unexpected spin-offs, changing environment etc. and how this can be used/improved (learning from failure), influence on priorities.
Approaches and standards
PQASSO - a practical quality assurance system for small organisations - is now a nationally recognised standard used by many local authorities as the recommended quality system for funded organisations, and is being used as a working model by various national voluntary organisations, e.g. NCH Action for Children have adopted it for their projects. CES (see next section) and others do training in how to use it.
Quality First, published by Birmingham Voluntary Service Council, is a quality system for organisations with no paid or only part-time staff. It is more of an ongoing process than a standard of attainment, looking at nine quality areas, and can form the basis for moving on to more advanced systems. It was developed by Tony Farley, original author of PQASSO, who now acts as a consultant through PQA (Practical Quality Assurance, email: firstname.lastname@example.org - note that as web site has disappeared we are not sure this info is current (Nov 04). The workbook is £25 incl p&p (free to Birmingham groups) from BVSC, phone 0121 678 8808.
NCVO has a section on the various quality assurance options.
Proving and Improving: a quality and impact toolkit for social enterprise has been produced by New Economics Foundation. Three (downloadable) toolkit books plus an overview chart outline basics of measuring organisation impact, the pros and cons of over twenty quality measures plus practical tools and exercises.
The European Foundation for Quality Management has an 'Excellence Model' designed to be applicable to all organisations. Worth a look if you are interested in this subject. The British Quality Foundation has various publications relating to this model. See Resources: Publications below too.
ISO 9000 (was BS5750), administered by the British Standards Institute, is about having a 'systems' quality standard - it doesn't guarantee anything about what you produce, other than consistency of the process. It is quite a complex and bureaucratic to register to this standard, and unlikely to be the best route forward for small to medium voluntary organisations (in our opinion).
See People Management for Investors in People standard. It can now cover organisations with no paid staff, but isnt cheap to pursue.
- Charter Mark was a government award scheme for recognising
and encouraging excellence in (public) service delivery, extended
to voluntary organisations receiving at least 10% public funding (at April 2000). The following info is out of date, but may still be of help. There are 10 criteria used:
- Set standards
- Be open and provide full information
- Consult and involve
- Encourage access and the promotion of choice
- Treat all fairly
- Put things right when they go wrong
- Use resources effectively
- Innovate and improve
- Work with other providers
- Provide user satisfaction
The Chartermark website now forwards to Customer Service Excellence.
Sustainability: Integrated Guidelines for Management (SIGMA) may be of interest - involves BSI and others. A 'work in progress' rather than a finished standard.
Charities Evaluation Services 'Training in approaches to monitoring, self-evaluation and quality assurance.' Second Edition of PQASSO is £65 plus p&p, CDROM also available. Course prices are very reasonable for small organisations. Also seven evaluation discussion papers. 4 Coldbath Square, London, EC1R 5HL, phone 020 7713 5722, email: email@example.com
Evaluation Trust closed 2013.
UK Evaluation Society UKES exists to promote and improve the theory, practice, understanding and utilisation of evaluation and its contribution to public knowledge.
Also see Further Resources below.
BEDE - Building Evidence, Demonstrating Effectiveness - for local community organisations was from bassac, now Locality. No longer available?
Homeless Outcomes"a one-stop resource for homelessness agencies who are interested in taking an outcomes approach to their work".
Service Audit Partnership auditing toolkit for the homelessness sector. Uses a peer audit approach.
- Monitoring and Evaluation News A news service focusing on developments in monitoring and evaluation methods relevant to development programmes with social development objectives.
A buzz word which can be taken in a number of ways. Basically the idea is to find a comparator to 'benchmark' your operations against - usually the idea is to find 'best practice', identify the gaps between you and them, and work to improve. It isn't easy in the voluntary sector to find the partners or agree on peformance indicators, and some would say that the value is looking further afield - if you want to benchmark a phone counselling service, say, why not compare with a call centre handling customer complaints?
As Benchmarking Plus (Australia) says, 'a survey may tell you where you rank, but it won't help you improve your position' (but it might prompt you to ask some questions). We reproduce a page of theirs giving ten common benchmarking mistakes. Unfortunately their web site no longer contains any useful resources (autumn 2011).
We have extracts from a survey investigating the need for and practicalities of a benchmarking club for UK charities (Open University, 1997).
NCVO has some more on benchmarking, including resources.
- The annual Charity Finance magazine Charity Shops Survey (published July?) provides some ratings in that area to measure performance against.
- Charity Finance Group runs benchmarking exercises and workshops.
- 18 major British voluntary organisations have formed an HR Benchmarking Club. People Count: Benchmarks for the Human Resources Function in Voluntary Organisations, a publication from Compass Partnership (June 00), gives information on over 165 aspects of personnel management from this club.
There are some good training courses from DSC, plus CES mentioned above.
NCVO's Third Sector Foresight project has closed, spring 2013? (certainly its website has).
CES has a resources section on its website.
See Buy Management Books Direct for titles - almost certainly the best introductory book is Complete Guide to Business and Strategic Planning by Alan Lawrie.
International Development Research Centre has made its publication Organizational Assessment: A Framework for Improving Performance available online. Not sector specific but does go beyond the commercial.
On the web, IT
Also see Social Return on Investment Network, with a SROI GUide published May 2009 with government support.
Strategic Planning Society is probably only really useful for the big voluntary bodies (they seem to bracket the voluntary sector with the public one), but you may be able to pick up some ideas from their web site. The Voluntary Sector Special Interest Group appears to have disappeared. 17 Portland Place, London, W1N 3AF, phone 020 7636 7737, email: firstname.lastname@example.org
Here are initial findings on useful software:
- MS Project is the big one to plot out your implementation of a project, allocate tasks, monitor progress, slippage and the effects etc.
- To help you brainstorm etc, look at Inspiration. A trial version can be downloaded from the web and is available for Macs and PCs (using Windows) - we havent checked out the current version 6. '...A powerful but easy-to-use visual thinking and learning tool that helps you brainstorm ideas, organize thinking, develop concepts and plan. Use its Diagram view to create concept maps, webs, diagrams, knowledge maps etc. or its Outline view to prioritize and rearrange ideas, leading to clear, concise writing.' It looks good and our limited testing of the demo was positive. UK distributors are PMI, phone 024 7641 9089, email: email@example.com, cost £90 plus VAT (July 99).
Non-Profit Evaluation - international resources
The following,mainly American, web sites seem promising, if you have the time - we have only done basic checking on them. All are non-profit specific.
- Center for Excellence in Nonprofits Programs in leadership development, systemic change, continuous improvement and best practices. A Silicon Valley learning community of non-profits.
- Center for What Works Aims to help public and nonprofit organisations to identify and replicate best practice. Links to promising practices, and is developing an On-line Benchmarking Centre.
- Kellogg Foundation. Their Evaluation Handbook (in pdf format) is designed for their 'grantees' but undoubtedly of interest to others. A participatory, multidisciplinary process, not just about outcomes but also building capacity. In the authors' view, "Project evaluation should not be conducted simply to prove that a project worked, but also to improve the way it works."
- An online evaluation tool which "takes about 10 minutes to complete if you are very familiar with your organization's operation", according to a posting on Digital Divide forum, is Innovation Network's Workstation 2.0 - this has moved, spring 05, but the site is still worth a check.