HOME 5 Applied Tools 5 Data Collection and Analysis

DATA COLLECTION AND ANALYSIS

The Data Collection and Analysis Tool is one of a suite of applied Community Well-being tools developed by SGIGs to form part of the online Data Governance and Management toolkit. The Data Collection and Analysis Tool outlines key considerations for effective primary data collection. It is closely related to the Indicator Guide & Recommended Indicators Tool, which describes a suite of indicators developed by SGIGs to measure well-being, many of which may need to be informed by primary data

Below is a summary of the Data Collection and Analysis Tool, including a webinar video for further guidance. The full Tool can be downloaded as a PDF and contains detailed information to support SGIGs through a primary data collection exercise, from early data collection considerations including survey design through to reporting out on results. 

Considerations for collecting data

What are primary data?

Primary data are information that do not already exist and that need to be collected for specific research purposes. In the context of socioeconomic research, primary data are useful when precision about the experiences of members of a specific population is important and when the data need to be current. In contrast, secondary data are pre-existing information collected for purposes other than the specific research in question.

When to collect primary vs. secondary data?

When choosing between primary and secondary data, consider budget, timeline, research needs, and whether existing data can answer the research question. Primary data collection can be difficult, but when it’s required, the benefits can be tremendous! After SGIGs have gone through a proper primary data collection exercise, they will have moved along the data use continuum towards expertise, and primary data collection exercises can serve as a catalyst for organizational change more broadly.

Scoping a primary data collection exercise

Getting started

The most important step in acquiring primary data is deeply understanding the information needs that are driving data collection. The two most important guiding questions to answer ahead of undertaking a primary data collection exercise are:

  1. What are our priority areas for data collection? 
  2. What do we have the capacity (people, time, and funding) to undertake?

Should you run a census or a survey?

Once a decision has been made to collect primary data and a clear understanding exists of the information needs, resources, and capacity, a subsequent key decision is whether to collect data from the entire population or from a sample of the population. Collecting data from the entire population is called a census. The main alternative to a census is a survey. A survey is used to collect information from a subset, or a fraction, of the population. This subset of the population is called a sample. With both a census and a survey, the information collected is used to calculate statistics for the population as a whole. Consider Table 1 when deciding whether a census or a survey is necessary.

 

Table 1. Census vs. survey

 

Census

Survey

ScopeData are collected from the entire population. If the population is small, or there are strategic considerations, a census may be preferable.Data are collected from a subset of the population. If the population is large, accurate results can usually be derived from a relatively small sample.
TimelinessTime-intensive. A census can take several years to complete and so it is typically conducted only once or twice in a decade.Can be completed in a short timeframe, ranging from weeks to months. Can be conducted frequently.
CostMore costly. Primary data collection exercise costs typically scale with the size of the sample because enumeration is usually a main cost-driver.Less costly overall, and more resources can be devoted to each individual response.

Additional  information on whether to conduct a census or a survey is available in the full Data Collection and Analysis Tool.

SULINGITUK GOVERNMENT: Decision to undertake a census

Survey Design

The goal in running a survey is to ensure that the sample is representative of the population of interest. In other words, the data collected should accurately reflect the characteristics of the entire population. The process of defining a sample can be broken down into four main steps: (1) identify the survey frame, (2) decide the sample design, (3) determine the sample size, and (4) produce the sample randomly.

Steps one through four of defining a sample are broken down in detail in the full Data Collection and Analysis Tool. The Tool also reviews common sources of bias and discusses the importance of SGIG’s citizenship/membership lists for survey design.

Questionnaire Design

A questionnaire (or form) is one of the survey instruments through which primary data are systematically collected. It is a group of sequences of questions, prompts, or statements designed to obtain information from a respondent. A questionnaire can include open ended questions, close ended questions, or a combination of both. Questionnaires are both critically important to data quality and create an impression on the survey respondents. The goal is to make sure respondents understand what they are being asked and can provide the answers easily in a form that is suitable for subsequent data processing and analysis. This is achieved through thoughtful questionnaire design, where question topics are determined and precisely worded and ordered. SGIGs that are also referencing the Conceptual Well-being Framework Tool and the Indicator Guide & Recommended Indicators Tool —including the starting point questions they provide— may use these as guidance in the planning, development, and customization of a questionnaire. 

In the full Data Collection and Analysis Tool, SGIGs can read more on:

  • Questionnaire best practices
  • General rules of thumb
  • Layout
  • Testing 
  • Customization
  • Privacy 
  • Software 

SULINGITUK GOVERNMENT: Drafting a questionnaire

Fictional Case Study

The Sulingituk Government recently decided to introduce a new tool to manage information and to help with data collection. This will involve the transition away from paper filing towards an online, cloud-based system.

read more

Communication Strategy

Key considerations

Effective communication when conducting primary data collection ensures potential survey participants understand the importance of the data collection exercise, how the data will be used and safeguarded, and important timelines. This is the opportunity to connect with potential survey participants, express gratitude for their time and participation, and let them know if you are offering any incentives. The communication strategy may also address concerns, build trust, and maximize participation by providing transparency to participants. When the respondents feel that their input is valued and their concerns are being addressed, they may be more likely to participate fully in the survey. 

 

To achieve this, consider these guiding questions while creating the communication strategy:

  • What is the primary goal of the data collection efforts and why is it important?
  • Who is the target audience for the collected data and what are their information needs and preferences?
  • What is the level of trust amongst participants in data collection and in the SGIG holding citizen data? How can trust be built or reinforced?
  • Who are the opinion leaders in the community and within families that can encourage people to participate?
  • Who are trusted messengers or people with large followings that respondents will be connected to?
  • What potential barriers or misconceptions may participants have about the data collection and how can the concerns be addressed proactively? What are the messages people need to hear about how the information will be treated? 
  • How can the messaging be tailored to resonate with the values and interests of participants? 
  • What are the key events and times of year to target the messaging and audience?
  • What are the likely questions people will ask? What feedback mechanisms can be implemented to continually gather input and insights from participants to refine the communication strategy?
  • What channels of communication do participants use to obtain information? What content formats are most engaging? (e.g., reports, infographics, videos)

Think of the communication timeline as being composed of multiple waves, each with its own time frame, target responses, and audience. As the project moves through each wave, there is the opportunity to refine the approach based on successes and challenges.

 

To get a clearer sense of communication methods, whether to provide incentives to participants, and how to address participant FAQs, see the full Data Collection and Analysis Tool.

SULINGITUK GOVERNMENT: Engagement Strategies

SULINGITUK GOVERNMENT: Engagement Strategies

Fictional Case Study

The Sulingituk Government recently decided to introduce a new tool to manage information and to help with data collection. This will involve the transition away from paper filing towards an online, cloud-based system.

read more

Enumeration

Enumeration methods

Enumeration refers to the process of collecting information from survey participants, usually through questionnaires or interviews. Having varied enumeration methods allows people to participate in the way most convenient and comfortable for them. Consider providing various options, such as:

  • Self-directed online survey
  • Phone enumeration appointments where staff enumerate respondents
  • Door-to-door
  • Events
  • Email, phone, social media, text
  • Other approaches like mail-in questionnaires

When selecting an enumeration method, SGIGs should consider the trade-offs between respondent preference and cost, data quality, and timely responses. Depending on the project objectives and constraints, SGIGs may find certain methods are more suitable than others. In addition to selecting an enumeration method, SGIGs will have to decide on the recruitment of enumerators, onboarding and training, and how to develop trust with their participants.  

Each of these steps are broken down in the full Data Collection and Analysis Tool.

SULINGITUK GOVERNMENT: Trust building

SULINGITUK GOVERNMENT: Trust Building

Fictional Case Study

The Sulingituk Government recently decided to introduce a new tool to manage information and to help with data collection. This will involve the transition away from paper filing towards an online, cloud-based system.

read more

Processing Data

Once the data have been collected and the enumeration phase is closed, analysts should “clean” the data. Data cleaning means identifying and correcting errors and inconsistencies. Then, analysts can move on to data processing, which transforms survey responses obtained during collection into a form that is suitable for tabulation and data analysis.

In the full Data Collection and Analysis Tool, there are helpful tips and in-depth explanations for data cleaning and weighting to guide SGIGs through the process. 

SULINGITUK GOVERNMENT: Data cleaning

SULINGITUK GOVERNMENT: Data cleaning

Fictional Case Study

The Sulingituk Government recently decided to introduce a new tool to manage information and to help with data collection. This will involve the transition away from paper filing towards an online, cloud-based system.

read more

Analysis

Data analysis involves summarizing the data and interpreting their meaning in a way that provides clear answers to the questions that initiated the survey. Often, it consists of interpreting tables and various summary measures, such as frequencies, means and ranges, or more complex statistics when relevant.

The analysis of the data should follow an analysis plan. The analysis plan is produced prior to enumeration and shows how the data generated by the survey will meet the information needs. It includes how the data generated by the survey will be processed, which relationships will be examined, which statistical methods will be used to examine those relationships, what criteria will be used to interpret the results, and how the results will be reported. A successful analysis plan ensures that each aspect of the survey works together to meet the objectives of the survey: that the variables used in the survey meet the needs the analysis, that the planned survey sample will meet the needs of the statistical methods, that the outputs of the statistical methods have objective criteria they can be evaluated against, and that the results will meet the information needs of the survey.

For additional considerations in creating an analysis plan see the full Data Collection and Analysis Tool.

SULINGITUK GOVERNMENT: Weighting

SULINGITUK GOVERNMENT: Weighting

Fictional Case Study

The Sulingituk Government recently decided to introduce a new tool to manage information and to help with data collection. This will involve the transition away from paper filing towards an online, cloud-based system.

read more

Reporting

Reporting is about taking all of the work that’s been done and making it useful for different audiences. Importantly, reporting is one of the major cost-drivers of the project overall, so understanding your reporting requirements and then translating those into a set of products that meets those requirements and budget is key. The initial set of reports that will be generated from the dataset should largely be determined by the work done earlier in the project timeline. The questionnaire was developed with a specific set of statistics in mind, the impetus for the project overall would have been clearly articulated, and the government’s specific priorities that lead to the desire to conduct the data collection exercise in the first place should all be answered in an initial set of reporting products. That said, new reporting requirements will invariably emerge through the course of the project. Those new reporting requirements should be scoped through a planning process. 

The planning process does not need to be too involved, but at a minimum, the following questions should be answered:

  • What are the key objectives and intended uses for reporting?
  • Who are the target audiences and what formats are likely to resonate most?
  • What are the main insights and takeaways to communicate?
  • What level of detail is appropriate for different audiences?
  • How will respondent privacy be protected in public reporting?
  • What is the budget and timeline that can be allocated to this product?

Reporting products end up being the most substantive legacy of the entire project. When people talk about the project in the future, they’ll be thinking about the reporting products that were generated. 

Learn more about various reporting products such as summary reporting, ad hoc data requests, knowledge transfer and capacity building, and interactive web tools in the full Data Collection and Analysis Tool.

SULINGITUK GOVERNMENT: Reporting

SULINGITUK GOVERNMENT: Reporting

Fictional Case Study

The Sulingituk Government recently decided to introduce a new tool to manage information and to help with data collection. This will involve the transition away from paper filing towards an online, cloud-based system.

read more

External Support

Should you recruit external support?

When considering the hiring of external support for primary data collection, several factors come into play, including the project’s scope and objectives, the time and capacity of any in-house expertise, budget, project timelines, and the existence and availability of trusted service providers, among others. Primary data collection is made up of several activities, and it’s possible to go to market on some, but not all, of these activities, using one or several service providers:

  • Scoping/Statement of objectives
  • Survey design and analysis plan
  • Questionnaire design
  • Communications strategy
  • Survey enumeration
  • Processing, analysis, and reporting
  • Specialized products, including websites or dashboards, and reporting, including audio, video, events, or other communication formats

Do you need to create a Request For Proposals to recruit external support? Looking for guidance on how to evaluate the proposals? Download the full Data Collection and Analysis Tool for more information.

Downloadable documents

Data Collection and Analysis Tool