Insights & Resources

No data, no problem

It’s never too late to collect the data you need.

In the ideal evaluation world, an evaluation occurs in the context of a beautifully planned and executed monitoring and evaluation framework.

You have time and resources and oodles of beautiful, relevant, high quality data to inform your evaluation……

But who lives in this ideal world?  Oftentimes, you are notified late of the requirement, there’s a tight budget and timeframe, and for the final kicker…. you have no data.

DON’T PANIC!

You can still move forward with an evaluation even when you have limited or no data right now – it just means that you will need to develop a data collection approach as part of your evaluation.

To get you started we have outlined:

  • six considerations for planning data collection
  • different types of data to consider
  • the pros and cons of data collection methods.

Six considerations for planning data collection

It’s important to identify the most appropriate methods of data collection and the types of data required at the planning stage, as:

  • selected methods will impact on your budget, resources and timeframe
  • resources and timeframes are a key factor in determining what data collection methods are feasible
  • the data you collect, along with the methods and sources, can constrain what evaluation questions can be addressed if it is not fit for purpose.

Careful planning will help ensure you have the data you need: on time, on budget and to the right level of quality.

When planning your approach to data collection, consider:

  1. Balance – aim for a balance of information collected from different sources and of different types. For example, complementing quantitative data with qualitative data. A general rule of thumb is to aim for data collected from two or more sources/types to allow for triangulation and evenness. Collecting data from multiple sources and of multiple types enhances your ability to validate your findings.
  2. Credibility – will the methods and types of data be considered credible by those who will be using the results? There is no point planning a qualitative evaluation approach if the findings and conclusions drawn from this data will be dismissed by the evaluation’s decisions makers.
  3. Quality – will the collection methods used yield sufficiently reliable data to draw conclusions to the degree of certainty required? Not all evaluations will require the same level of data quality. The level of quality required depends a lot on what the data is being used for. Large scale and/or costly policy and program decisions may demand higher levels of data quality compared to small scale or easily amended decisions. Data quality itself has a number of dimensions including: accuracy, completeness/duplication, consistency, currency and integrity.
  4. Feasibility – this is about the practicality, logistics and cost of collecting and using the data. Consider if there will be enough time as some data collection methods are more time consuming than others. Will there be enough people with the relevant skills to undertake the proposed methods of collection and to analyse the data once you have it? What is the impact of the data collection on others – is the burden on stakeholders fair and appropriate or will is cause complaints and distress? And lastly is there sufficient budget to undertake the planned data collection?
  5. Legal requirements and ethics – In any evaluation activity is important to ensure ethical practices are followed and legal requirements are complied with. This includes privacy, confidentiality, data security and storage, compiling with police or security clearance requirements, seeking participant consent, providing transparent information to participants on the use of the information they provide and respecting individual values and cultural differences. In some cases, you may need to seek clearance from a relevant ethics committee for your evaluation. The Australasian Evaluation Society has published guidelines on ethical conduct in evaluations.
  6. Analysis approach(es) – How will you assess your program or policy? The approaches used to analyse performance will drive your data requirements and collection and may be constrained by feasibility, data quality, and legal and ethical requirements. Consider, for example, an approach which compares current results with the results achieved prior to your intervention – this will require relevant baseline data. Ideally baseline data will have been collected prior to implementation of your program, however this doesn’t always happen. In these instances, there may be ways to obtain secondary data such as national studies, feasibility studies and program reporting that will indicate conditions at a prior point in time.  Alternatively, surveys and interviews can be conducted that collect pre-program information.

Types of data

When planning data collection for your evaluation, it’s vital to identify and select a balance of data that is appropriate to your evaluation needs.  There are various types of data you can consider:

Quantitative vs Qualitative

  • Quantitative information is useful describing incidence and trends. For example, large scale surveys and patterns of use.
  • Qualitative information is useful for understanding why things are or are not working. Qualitative information is useful for understanding and explaining what lies behind the numbers.

Subjective vs Objective

  • Subjective information relates to people’s opinions and perceptions.
  • Objective information relates to observable facts.

Intrusive vs Unobtrusive

  • Intrusive methods of data collection are those where the people involved are aware that the information is being collected. For example, interviews, surveys and focus groups.
  • Unobtrusive methods are those where people involved are not aware. For example, analysis of information from files, records or policy documents and some forms of observation.

Individual vs Groups

  • Methods for collecting information from individuals are used when the evaluation requires information unaffected by peer pressure and privacy for the participants.
  • Methods for gathering information from groups involve interaction between group members, eg focus groups, groups interviews. These methods can be useful to generate ideas, explore feasibility of various options and to compare and contrast different experiences and perceptions.

General vs Particular

  • Sampled surveys are an example of generalisable data, which is needed when it is important to comment on general practices across a program, population or stakeholder group.
  • Case studies are an example of particular data. Particular data focuses on one example and is useful for gathering a better understanding of key factors for success and any barriers, but these factors may not be an accurate reflection of other experiences.

Methods for collecting data and their pros and cons

There are different advantages and disadvantages associated with different data collection methods.  Consider the following pros and cons when planning your evaluation’s approach to data collection.

1. Face to face

Pros:

  • Qualitative and in depth
  • Allows you to explain, clarify and cross check
  • Good for small sample sizes
  • Allows you to develop rapport

Cons:

  • Produces large volume of qualitative data which takes time to analyse
  • It is expensive and labour intensive
  • Can be vulnerable to interviewer bias

2. Telephone interviews

Pros:

  • Allows contact with large numbers of people and can produce both quantitative and qualitative data
  • Can be less threatening than face to face
  • Allows you to explain, clarify and cross check

Cons:

  • Can be expensive to administer
  • Can make it more difficult to establish rapport
  • Not as good for complex questions

3.  Submissions

Pros:

  • Can be a powerful mechanism for bring issues to the surface for further investigation
  • Inviting submissions according to a terms of reference or discussion paper makes it easier to report evaluation findings against specific themes or topics

Cons:

  • Reliance on submissions can solicit responses from those with extreme views. This can lead to undue attention being given to polarised minorities
  • Can be effort intensive for respondents and impact response numbers
  • Opinions may not be substantiated by evidence
  • Can be difficult to analyse and sort information

4. Focus groups

Pros:

  • Allows collection of qualitative data about attitudes, perceptions and opinions
  • Can get reasons for views – particularly useful for developing or improving services
  • Can generate new ideas and build understanding of a variety of needs
  • Can explain and clarify questions

Cons

  • Small sample size and can be difficult to compose groups
  • Requires a highly skilled facilitator, to make sure opinions don’t reflect peer pressure, or views of dominant participants, and to keep discussions on track
  • Can be expensive to administer and conduct

5.  Surveys

Pros

  • Good for collecting quantitative data
  • Low cost way of reaching large numbers of people
  • A well constructed survey is simple and not very time consuming for participants to complete and for evaluators to analyse
  • Individuals respond in private, so views expressed are less subject to peer pressure. Further responses can be confidential

Cons:

  • If responses are not from a representative sample, it can lead to a bias of results
  • Considerable front end work in design and piloting of survey instrument
  • Considerable follow up required if response rates are low
  • Does not allow for development of rapport
  • May not allow for exploring reasons behind actions or opinions
  • Not suited to all stakeholders – eg, language barrier, lack of access to internet

Start planning your next evaluation

Now that you are well prepared, use our data collection matrix to start planning the data collection for your next evaluation.