CPiE Situation and Response Monitoring

This e-learning course is a basic orientation to the "Child Protection in Emergencies Monitoring Toolkit". The toolkit provides guidance on how to develop and implement systems for monitoring child protection issues that affect children in humanitarian settings, as well as for monitoring the child protection response. This e-learning course is designed to be complemented with regular group discussions, led by a facilitator.

Module 1: Overview

1.1 Introduction

This module is an overview of the Child Protection Situation and Response Monitoring Toolkit

You have three options in terms of accessing this overview:

  • Option 1: You can click "next" and go through this module;
  • Option 2: You can watch the video through the [WMA version] or the [MP4 version - Part 1] and [MP4 version - Part 2]; or
  • Option 3: You can look at this ppt and listen to these MP3 files: [Part 1] and [Part 2].

​You also have the option to view the video and ppt at the end of this module.

1.2 Background

  • The need for a CP monitoring toolkit was identified through feedback from the field (mostly CP coordinators).
  • Its development was made possible through generous contribution from OFDA and the German government.
  • An advisory committee was formed (Columbia University, CPWG, OCHA, Save, and UNICEF).
  • Five country visits were conducted to initiate the process: Burkina Faso, CAR, DRC, Mali, and South Sudan.*
  • 4 rounds of revisions have happened and the current version is ready for use by any interested party.
  • 1 pilot testing is ongoing (in South Sudan) and up to 2 more are in the pipeline.

    * At the time of development, no coordination mechanism outside of Africa was able to host an initial visit (hence the choice of these 5 countries). To address that issue, we are ensuring that we pilot the tool in a non-African context as well.

1.3 Definitions

  • Situation Monitoring: Ongoing and coordinated data collection and analysis of child protection risks, concerns, violations and capacities in a given context to inform programming.
  • Response Monitoring: Ongoing and coordinated measurement of coverage and quality of emergency response in a humanitarian context.

Check your understanding

  • Situation Monitoring
    Ongoing and coordinated data collection and analysis of child protection risks, concerns, violations and capacities in a given context to inform programming.
  • Response Monitoring
    Ongoing and coordinated measurement of coverage and quality of emergency response in a humanitarian context.

1.4 Current Structure of the Toolkit

  1. Introduction
  2. Fundamentals
  3. Situation monitoring for child protection in emergencies
    3.1 Secondary data collection
    3.2. Primary data collection
  4. Response monitoring for child protection in emergencies
    4.1. Coverage monitoring
    4.2. Monitoring of Quality
  5. Key steps for establishing a monitoring system

  6. Indicators

  7. Sampling and selection of participants

  8. Data collection and staffing requirements

  9. Data analysis and sharing

1.5 Four Main Components

Matching Game

  • Basics
    Section 1: Introduction Section 2: Fundamentals
  • Methodology
    Section 3: Situation Monitoring Section 4: Response Monitoring Section 5: Sampling and Selection
  • Planning and Roll out
    Section 5: Key Steps for Establishing a Monitoring System Section 6: Indicators
  • Data Management
    Section 8: Data Collection and Staffing Section 9: Data Analysis and Sharing

1.6 Section 1 - Introduction

The purpose of the toolkit is to provide guidance on how to develop and implement systems for monitoring child protection issues that affect children in humanitarian settings, as well as for monitoring the child protection response.

This section also describes:

  • What the toolkit is and what it is not.

    This toolkit provides guidance on how to set up situation and response monitoring mechanisms in emergency affected contexts. It provides tools and methodologies that need to be adjusted to the country and each humanitarian context.

     

    This toolkit is not intended to replace other guidelines and tools on: a) the measurement of the impact of child protection programs; b) identification of cases or mapping of services; or c) setting up a monitoring and reporting mechanism on grave violations against children in armed conflict.
     

  • Who it is for.
    This toolkit is for anyone working to identify protection issues for children in emergencies and/or planning to monitor interventions that are responding to child protection issues in emergencies.
     
  • Where it should be used.
    This toolkit is best suited for contexts where the child protection response is coordinated either through the cluster/sector approach or other coordination mechanisms. 

1.7 Section 2 - Fundamentals

  • This section provides definitions for:

    1. Child Protection in Emergencies
    Child protection in emergencies, as agreed by the global level Child Protection     Working Group (CPWG), is “the prevention of and response to abuse, neglect, exploitation and violence against children.”

    2. Situation Monitoring
    Situation monitoring is the ongoing and systematic data collection and analysis of child protection risks, concerns, violations and capacities in a given humanitarian context.

    3. Response Monitoring

    Response monitoring is the ongoing and coordinated measurement of the humanitarian response in a humanitarian context; i.e. activities planned and carried out by humanitarian actors.
     

  • Describes linkages between situation and response monitoring and other measurement efforts (such as assessments and evaluations)
    > Situation and response monitoring are two sides of the same coin. They produce complementary information. 
    > Assessments, such as the Child Protection Rapid Assessment (CPRA), provide a snapshot of and information about a situation at a moment in time. The child protection situation evolves over time as a result of the response and/or other factors, and monitoring allows the humanitarian community to observe trends and changes in the situation.
    > Program evaluations are in most cases a one off exercise used to assess the effectiveness and impact of a program. Evaluation is not a substitute for response monitoring, or vice versa.
    > While situation monitoring is not meant to work as a case-finding mechanism, it can support case management in several ways. For example, situation monitoring can help managers identify hotspots. 
     
  • Explains ethical considerations during data collection for situation and response monitoring
    In designing and implementing any data collection system, ethical considerations should be taken into account. “Do No Harm,” “Best Interest of the Child,” and “Confidentiality of Information” are the core principles to be considered for a child protection monitoring system.

1.8 Section 3 - Situation Monitoring

  • Introduces the two main approaches to data used in situation monitoring:

    1. Secondary Data Review
    Secondary data is any information extracted from existing sources of information, such as reports, assessment data, case management data, etc. 

    2. Primary Data Collection
    Primary data refers to any data that is collected directly from its original source for the objective in question. The objective of primary data collection is to establish a reliable source of information from the affected populations and areas, including where the affected population might have moved to. 
     
  • Describes how secondary data should be collected, compiled and analyzed.
    For situation monitoring, it is advisable that you reach out to a wide range of actors for relevant information on the situation.  
     
  • Describes the two methodologies proposed for primary data collection:
    The Primary Data Collection component can be done using two methods: community-based and/or agency-based situation monitoring. 

    1. Community-Based Situation Monitoring
    This approach requires identification and training of community focal points to become active data collectors in a sample of communities. 

    2. Agency-Based Situation Monitoring
    Agency-based situation monitoring requires data collection from a systematically sampled group of communities by operational agencies. 
     
  • Provides sample tools
    > Sample list of what we need to know and indicators for situation monitoring
    > Secondary data review tool
    > Sample framework for community-based situation monitoring
    > Data collection form for community based situation monitoring
    > Sample framework for agency- based situation monitoring
     

1.9 Section 4 - Response Monitoring

  • Introduces the two components of response monitoring:

    1. Coverage Monitoring
    This component focuses on reach—in the geographical sense—and coverage—both in terms of thematic areas and provision of services to all children in need.
     
    2. Monitoring of Program Quality
    Therefore a complimentary data collection and data management system is required to help capture the quality of the response. Quality monitoring is also meant to reflect the voice of the beneficiaries.
     
  • Describes how 5W tool can be adapted and used for coverage monitoring.
    When using a 5W tool, it is important to first ensure that the information collected in this tool is sufficient to inform the indicators defined in country for response monitoring and reporting needs of the HPC. The tool has to be adapted and tested before the data collection process begins.
     
  • Describes the three methodologies proposed for monitoring of quality:

    1. Independent Monitoring
    Independent monitoring is when data collection is undertaken by a group of trained independent monitors, meaning they are not connected to the project being monitored. 

    2. Agency Self Monitoring
    This option will require commitment from implementing agencies to collect data on a regular basis, using commonly agreed indicators of quality and data collection tools. 

    3. Peer-to-peer Monitoring
    This approach is in effect a mix of the two options presented above (3.2.a & 3.2.b). This option requires implementing agencies to monitor each other’s’ work.
     
  • Provides sample tools
    Tools for coverage monitoring
    > Tools for quality monitoring
    > Agreed indicators and targets to measure “adherence to standards” on UASC program
    > Data collection form to measure “adherence to standards” on UASC program
    > Agreed indicators and targets to measure “accountability” on UASC program
    > Data collection form to measure “accountability” on UASC program

1.10 Section 5 - Twelve Steps to Establish a Monitoring System

1.11 Section 6 - Indicators

  • This section provides definitions and guidance on what qualitative and quantitative indicators are and how they should be developed for use within CP monitoring.
  • Annex 1 provides more detailed definitions of different types of indicators and related topics, including:

  • Input indicator: Measures the financial, human, and material resources used for the intervention. For example “number of tents set up for CFSs in the affected area”. Input indicators are only relevant to response monitoring.
  • Process indicator: Measures activities that have taken place to move the program forward. For example: “number of social workers trained on case management.” Process indicators are only relevant to response monitoring. Note: Some experts consider process indicators as being a sub-set of ‘output’ indicators.
  • Output indicator: Measures products, goods and services, which result from an intervention. For example: “number of children reunified with their families by project staff.” Output indicators are only relevant to response monitoring.
  • Indicator to measure quality:  Measures quality of the products, goods and services delivered by the intervention.  For example: “Percentage of registered unaccompanied children that have been reunified within 6 weeks from their identification.” Quality indicators are only relevant to response monitoring. Note: Some experts consider quality indicators as subset of ‘input,’ ‘output’ and ‘outcome’ indicators. 
  • Outcome indicator: Measures short-term and medium-term effects of an intervention. For example a girl who has been reunified with her family.  “Percentage of reunified children that stayed with their family for more than six months.” Outcome indicators can be used for both response and situation monitoring.
  • SMART indicator: SMART stands for Specific, Measurable, Achievable, Relevant, and Time bound. Avoid trying to gather too much information under a single indicator. Example for a NON-SMART indicator: “percentage of separated children who were identified and registered and reunified”. This indicator has too many components (not-specific), hard to measure, and not time-bound. It also has the potential of double or triple counting a single child.
  • Proxy Indicator: A proxy indicator is an indicator that does not directly measure what we want to know, but gives an approximation of the issue by measuring something related to it (such as a symptom or a consequence). For example, to know if children are psychosocially distressed, we may not be able to measure their actual level of distress. But we often measure whether they show behavioral signs of distress, such as bedwetting or unusual crying.
  • Numerator: The expression written above the line in a common fraction to indicate the number that represents the count of the issue of concern. For example if the indicator is: “percentage of children who participated in x”, the “of children who participated in x” is the numerator.
  • Denominator: The expression written below the line in a common fraction that represents the total population of concern. In the example above, “total number of targeted children” is the denominator.
  • Target: A target is the value assigned to an indicator that is set as a goal by program managers or coordination bodies. It is used by program staff as a determinant for the success of the intervention. For example, target for the quality indicator mentioned above can be 90%.
  • Unit of measurement: Is the level at which the measurement of the phenomenon in question takes place. In humanitarian contexts, this can be community, household, individual, education facility, health center, etc. For example (a) for the ‘number of identified unaccompanied children’ the unit is ‘child’; (b) for the ‘percentage of communities with at least one functioning CFS’ the unit is ‘community.’ Unit of measurement should be well defined for each indicator.

Match the indicator

Determine whether the following indicator belong to: 1) situation monitoring; 2) response monitoring; or 3) both). Please refer back to the definitions of situation and response monitoring when matching this indicator to the type(s) of monitoring.

Indicator 1: Child to facilitator ratio in CFS

 

Match the indicator

Determine whether the following indicator belong to: 1) situation monitoring; 2) response monitoring; or 3) both). Please refer back to the definitions of situation and response monitoring when matching this indicator to the type(s) of monitoring.

Indicator 2: % of children placed in foster families who received at least 1 home visit during the past month

 

Match the indicator

Determine whether the following indicator belong to: 1) situation monitoring; 2) response monitoring; or 3) both). Please refer back to the definitions of situation and response monitoring when matching this indicator to the type(s) of monitoring.

Indicator 3: % of surveyed households that report at least 1 missing child since the earthquake

 

Match the indicator

Determine whether the following indicator belong to: 1) situation monitoring; 2) response monitoring; or 3) both). Please refer back to the definitions of situation and response monitoring when matching this indicator to the type(s) of monitoring.

Indicator 4: % of communities that report an increase in the number of incidents of sexual violence against children since the beginning of intervention Z.

 

Match the indicator

Determine whether the following indicator belong to: 1) situation monitoring; 2) response monitoring; or 3) both). Please refer back to the definitions of situation and response monitoring when matching this indicator to the type(s) of monitoring.

Indicator 5: % of communities that report an increase in number of recruitment of children into armed groups since Christmas.

 

1.12 Section 7 - Sampling and Selection of Participants

  • This section describes why we sample, what a unit of measurement is and what the proposed sampling approach is for situation and response monitoring.
  • Sampling for situation monitoring depends on the selected data collection method. But in general, the approach can be referred to as purposive sampling of sentinel sites.
  • Sampling for response monitoring also depends on the selected data collection method. If agency self monitoring is selected, a purposive sample will be drawn. 

1. 13 Section 8 - Data collection and staffing requirements

  • This section lays out some of the considerations that will determine frequency of data collection, staffing requirements and data collection process.
  • It also describes 3 separate roles: project manager/coordinator, information manager, and enumerators.

1.14 Section 9 - Data analysis and sharing

  • This section describes three data management tools that have been developed/adapted for the CP Monitoring process:

    1. Adapted 5W tool;

    2. Program quality data management tool;
    For response monitoring the existing data management tools (5W and program quality data management tool) provide an easy to use platform for tabulation and analysis of the data to facilitate interpretation and report writing. 

    3. Situation monitoring data management tool.
    Like the tools for response monitoring, the situation monitoring data management tool provides the possibility of easy data entry, cleaning and analysis. 
     
  • All these tools need to be adapted after contextualization of the CP monitoring process and tools.
     
  • Examples from South Sudan include: Situation Monitoring Data Management Tool and the Response Monitoring Data Management Tool

Options

This module was an overview of the Child Protection Situation and Response Monitoring Toolkit

If you would like to review the video, there are two options: [WMA version] or [MP4 version - Part 1] and [MP4 version - Part 2]. You can also look at this ppt and listen to these MP3 files: [Part 1] and [Part 2].​

Module 2: Situation Monitoring

2.1 Reminder

  • Situation Monitoring is the ongoing and coordinated collection and analysis of data on child protection risks, concerns, violations and capacities.
  • The objective of collecting situational data on child protection is to inform CPiE programming.

2.2 Situation Monitoring vs. Assessment

  • Assessments, such as the Child Protection Rapid Assessment (CPRA), provide a snapshot of and information about a situation at a given moment in time.
  • In many humanitarian contexts, especially in protracted emergencies, child protection situation evolves over time as a result of the response and/or other factors. Therefore, the snapshot from six months ago, may not reflect the reality today.
  • Situation monitoring allows the humanitarian community to observe trends and changes in the situation, to continuously adjust their programs to the needs of the affected population.
  • Assessments may be used to set a baseline at the start of a monitoring process. 

Can the CPRA be used as a situation monitoring system?

  • Yes, completely
  • Yes, under certain conditions
  • No

2.3 Linkages between case management information system and a situation monitoring system

  • Case management data only reflects information about children that have been reached by service providers. This is often a fraction of those in need.
  • While case management data should be used as one valuable source of information for a monitoring system, relying solely on case management data will likely lead to exclusion of a significant portion of the affected population. This can introduce a significant bias in our understanding of the situation of children.
  • Situation monitoring, however, can support case management in several ways.
  • Use of situation monitoring as a case-finding system is also not recommended. As an exception to the rule, in well-resourced, small-scale emergencies, these two functions can be merged.

In what ways can Situation Monitoring support case management?

Why is the use of Situation Monitoring as a case-finding system not recommended in larger scale emergencies?

2.4 What does Situation Monitoring do? What does it not do?

  • Situation monitoring generates data on emerging or changing child protection risks and threats so that the necessary response can be organized. For example, if children start disappearing in an area, situation monitoring is meant to capture the increasing/changing trend. However to identify the causes and details of each case, follow up investigation is required.
  • Situation monitoring is not designed to provide comprehensive or in depth understanding of any one child protection issue. 
  • It is meant to produce a general sense of emerging and changing child protection needs and risks.

Which of the following is generally NOT expected from a CP situation monitoring system? (select all that apply)

  • a. Identification of emerging child protection risks and threats
  • b. Comprehensive picture of underlying causes of sexual violence in communities
  • c. General sense of the trend in terms of the scale at which children are affected by different child protection issues
  • d. Identification of children affected by abuse and violence
  • e. Observing improvement in the situation of children
  • f. None of the above

2.5 Who should be involved?

  • Establishing a situation monitoring system is recommended as an inter-agency activity. If a CP coordination mechanism exists, all members should be invited to take part in this initiative.
  • It is advisable that you reach out to a wide range of actors before establishing a situation monitoring system. Stakeholders may operate existing information management or surveillance systems that collect relevant child protection data and/or are willing to add one or more relevant child protection indicators to their existing situational data collection mechanisms.

2.6 Situation Monitoring: The Toolkit

2.7 Secondary Data Review

  • Secondary data is any information extracted from existing sources of information, such as reports, assessment data, case management data, etc.
  • The Secondary Data Review (SDR) template is an excel-based tool developed by the global level Child Protection AoR that can be used to compile and organize secondary information.
  • An inclusion criteria should be developed for information sources that will be used for secondary data review. Particular attention should be paid to the reliability of information sources.
  • Download an example of a SDR from South Sudan:

2.8 Primary Data Collection

  • Two options for primary data collection are recommended in this toolkit:

    1. Community-based Situation Monitoring
    2. Agency-based Situation Monitoring
     
  • To ensure simplicity and feasibility, only data on general trends and patterns should be collected.
  • Once a change in patterns or alarming trends are observed, a technical team from agencies who are active in the corresponding area should be deployed to gather more in depth information of the situation and prepare for response accordingly.

2.8.1 Community-based Situation Monitoring

  • This approach requires identification and training of community focal points to become active data collectors in communities.

    1. Trained agency staff will be deployed to a sample of communities to    select/elect focal points in consultation with community members.
    2. Community focal points should then be trained on data collection and reporting; urgent action procedures; and ethical considerations.
     
  • Community focal point will be asked to submit reports through agreed upon channels on a regular basis.
  • Depending on the reliability of cellular phone or internet coverage, a phone-based, internet-based or paper-based reporting structure can be established.
  • If a phone-based system is selected, a series of codes or short questions can be developed to represent different risks to children.  Choice of codes versus questions has to be made based on the mobile platform that will be used. For example, for platforms such as RapidPro, questions are more appropriate, while for platforms such as Frontline SMS, series of codes may work better.
  • An example of a series of short questions:

2.8.2 Agency-based Situation Monitoring

  • Agency-based situation monitoring approach requires commitment from agencies to dedicate staff to act as data collectors.
  • This approach requires data collection from a systematically sampled group of communities.
  • Trained agency staff will be deployed to the sampled communities on agreed upon intervals to collect data.
  • Data collection methods have to be agreed upon based on the type of WWNKs and Indicators. Methods may include: key informant interview, direct observation, focus group discussions, participatory methods, etc. 

2.8.3 Primary Data Collection Comparision

2.9 Sampling for Situation Monitoring: Unit of Measurement

  • For situation monitoring, the recommended unit of measurement is the community. “Community” has to be defined in context based on the realities on the ground.
  • Once the unit of measurement is defined, a comprehensive sample frame should be developed.

  • For purposes of sampling, community should not be too large so that key informants can be identified who have knowledge of the whole community.
  • As a rule of thumb, a community should not be larger than 5000 individuals.  Also very small communities cannot serve as part of the sampling.
  • If you are in an IDP or refugee context, and camps have been set up, community should be set up based on existing division of the camps. 

2.10.1 Sampling for Situation Monitoring

  • If primary data collection is integrated in existing regular activities (such as monthly meeting of community members), no sampling is necessary. This means that data will be collected from all communities that participate in those community meetings on specified intervals.
  • If a new mechanism is being established for primary data collection, under either of the data collection options, a comprehensive sample frame needs to be developed based on the distribution of units of measurement across the affected area.
  • To ensure diversity and variation in the data, the sample frame should be built such that it disaggregates the units of measurement based on the most important distinct characteristics of the affected population in terms of level of needs, risks, vulnerabilities, existing capacity and availability of services. These are called “sampling scenarios” and will be used to stratify the sample frame.

     

    For example, if there are some affected areas that are hosting IDPs and other affected areas where there are no IDPs, the sample frame has to disaggregate the communities based on the presentence of IDPs.

How do we sample so the needs of the most affected are represented?

2.10.2 Sampling for Situation Monitoring

  • The general sampling approach for both data collection options is purposive sampling of sentinel sites.
  • Sentinel sites are communities that are likely to produce cases related to one or several child protection issues that are being measured in each reporting period. Therefore they should have a large enough population to allow for meaningful detection of child protection cases.
  • Example: if the projected incidence of separation in an emergency is 0.5% of children per month, in a community with only 150 children we will not see a single case of separation during many of the reporting periods (e.g. months). Whereas, if a community with 500 children is selected as the sentinel site, it is likely that we will see about 2 or 3 cases of separation in that site during each reporting period.
  • If the entire area is affected in the same way (i.e. only one scenario is identified), a minimum of 30 units of measurement should be selected for the sample. If more than one scenario exists in the sample frame, a minimum of 15 units of measurement should be selected for each scenario.

  • If your sample frame represents sites that cannot be considered a sentinel site due to their small size, they should be excluded from the selection of sites. However, if this means that all sites in a given scenario will be excluded, sites can be combined to create larger units.

  • Example of sample frame with scenarios:

True or false?

  • It is recommended that implementing agencies engage in situation and response monitoring individually.

True or false?

  • Secondary Data Review excel file can analyze secondary information and provide tables and graphs.

True or false?

  • Situation monitoring does not provide in depth knowledge of specific child protection issues.

True or false?

  •  Publically-available Monitoring and Reporting Mechanism (MRM) documents can be used as a source of secondary information.

True or false?

  • Sentinel sites are sites that are selected because they are easily accessible by NGOs so that they can visit them regularly.

True or false?

  • If we have three scenarios in our sample frame, we should select at least 45 sites (15 in each scenario) for our situation monitoring.

True or false?

  • Imagine that your two scenarios in a conflict and drought affected area are IDP versus non-IDP (two scenarios). In the area of coverage for situation monitoring, you have about 200 sites that host non-IDPs and about 40 sites that host IDPs (including camps). You should select at least 25 sites for the non-IDP areas and at least 5 from IDP areas (total of 30) to ensure a proportional sample size.

Module 3: Response Monitoring

3.1 Reminder

  • Response Monitoring is the ongoing and coordinated measurement of coverage and quality of emergency response in a humanitarian context.
  • The objective of collecting data on emergency response is to inform the adaptation/improvement of CPiE programs

3.2 Response monitoring vs. evaluations?

  • Monitoring and Evaluation (M&E) is a broad term used to describe measurement activities linked to programs.
  • Program evaluation seeks to either assess the process or outcome of a project. Process evaluations can greatly benefit from response monitoring data. Outcome evaluations can also indirectly benefit from response monitoring data.
  • Response monitoring collects data on program implementation to inform decisions on programmatic adjustments during the lifetime of the project.
  • Evaluation is not a substitute for response monitoring, or vice versa because their objectives are different.

Can Response Monitoring data be used to inform program evaluation?

  • Yes
  • No

Explain: How can Response Monitoring data be used to inform program evaluation?

3.3 What does Response Monitoring do?

  • Response monitoring tracks the achievements of the child protection response so that shortcomings  can be identified and rectified in a timely fashion.
  • A response monitoring process includes two components: 

    1. Coverage Monitoring: to measure reach and coverage of interventions and

    2. Quality Monitoring: to assess the quality of responses.

Which of the following is NOT generally expected from a CP response monitoring? (select all that apply)

  • a. Tracking improvements in the satisfaction of beneficiaries with the way CFS activities are undertaken.
  • b. Show geographic coverage of affected areas as per different CP thematic area (e.g. FTR, SV, PSS, etc).
  • c. Determine the impact of CP interventions.
  • d. Determining eligibility of children affected by abuse and violence to receive CP services.
  • e. Show trends (increase or decrease) in number of beneficiaries reached across time.

3.4 Situation and Response Monitoring: how do they relate to each other?

  • Situation and response monitoring are two sides of the same coin. They produce complementary information. Without knowing the needs, our response may not target the most pressing issues.
  • Almost all humanitarian programs are designed and implemented against a backdrop of a theory of change. It is assumed that our response or programs are going to bring about a positive change to the lives of affected population. Situation monitoring can help us see that change (or lack thereof). Response monitoring can help us track how well we are doing. 

In Country x, response monitoring data shows you that both in terms of quality and coverage, you are doing well. But the situation of children is not improving (and may even be deteriorating in some areas). Reflect on why this might be happening?

3.5 Who should be involved?

  • Establishing a response monitoring system almost only makes sense as an inter-agency activity. If a CP coordination mechanism exists, all members should be invited to take part in this initiative.
  • It is advisable that a wide range of actors are involved in the establishment of a response monitoring system.  It is important to use already existing indicators to avoid unnecessary additional burden on participating agencies.

3.6 Response Monitoring: The Toolkit

3.7 Coverage Monitoring

This component of response monitoring focuses on two things:

  • Reach—in the geographical sense; and
  • Coverage—both in terms of thematic areas and provision of services to all children in need.

3.8 5W - Who Does What Where When and for Whom

Many child protection coordination groups use the “Who does What, Where, When and for Whom” (5W) tool to monitor the CPiE response. The Child Protection AoR has developed an adaptable version of the tool, along with guidance, for use in any context.

However, some minor modifications are necessary to make sure the generic 5W tool collects enough data to inform coverage indicators for response monitoring.

3.9  Adapting the 5W matrix for Response Monitoring

  • Reporting only the number of children reached does not fully represent the activities carried out by child protection actors. This is because many child protection services (such as FTR) require weeks or months of follow up and involve multiple practitioners to serve one single child. To effectively measure the response, it is important to break down child protection services to easily-measurable units.
  • The generic 5W matrix measures coverage through capturing the number of children reached. But the unit of service provided to a single child does not get captured in the current format. Therefore, 5W matrix  has to be adapted to create space for reporting of units of services. 
  • Example of a 5W from Lebanon:

If you were asked to break down an FTR program into units of service, what components would you break it down to?

3.10 Why is coverage monitoring not enough?

  • Coverage monitoring helps us measure reach and coverage of our activities. For example, through the 5W tool, we try to determine how many children have received psychosocial services during a given period.
  • But a 5W matrix will not be able to tell us whether protocols were followed, what the quality of services was, and whether children and their parents were satisfied with the services.
  • Therefore, the CP monitoring process employs a second component to reflect Quality.

3.11 Monitoring Program Quality

  • There are three optional methods suggested for montoring program quality:

    a) independent monitoring,
    b) agency self-monitoring, and
    c) peer-to-peer monitoring.
     
  • Each of these methods has advantages and limitations, which have to be considered in each context. Sometimes the best option for your context may be a mix of these suggested methods.

3.11.1 Independent Monitoring

  • Independent monitoring refers to data collection by a group of trained independent monitors. In other words, the monitors are not in anyway linked to the project being monitored. The monitors will travel to different sites on a regular basis and collect data against defined indicators.
  • Note: this option will require dedicated and trained staff and logistical support for traveling and carrying out the monitoring. Therefore, dedicated funds will be necessary.

3.11.2 Agency-self Monitoring

  • Agency-self monitoring refers to data collection by agencies themselves.
  • This option will require commitment from implementing agencies to collect data on a regular basis, using commonly agreed upon indicators of quality and data collection tools.
  • Focal points from implementing agencies should be trained on how to use the agreed tools. If this option is selected, measures should be taken to minimize potential biases. 

If you choose agency-self monitoring methodology for monitoring program quality, what measures can you take to reduce potential bias?

3.11.3 Peer-to-peer monitoring

  • This approach is in effect a mix of the two options presented above. This option requires implementing agencies to monitor each others’ work.
  • It can be done in a reciprocal manner (i.e. agency X monitors agency Y and vice versa), or in a rotational manner (i.e. agency X monitoring agency Y, then agency Y monitors agency Z and agency Z monitors agency X).
  • It is based on the premise that agencies not only have the technical expertise to conduct such monitoring, but will also be able to exchange learning and best practices with each other. 

What do you think may be the main disadvantage of the peer-to-peer monitoring approach for monitoring program quality?

3.11.4 Program Quality Monitoring Approaches Comparison

3.12 Defining Program Quality

One of the most important steps in setting up a response monitoring system is to define program quality in context. Below is an sample criteria for program quality:

  • Targeting: Are we reaching the right children with the right services?
  • Adherence to standards: Does our service meet the standards, protocols and standard operating procedures?
  • Accountability: Are children, their family and their  communities satisfied with our services?

3.13 Sample indicators for adherence to standards

3.14 Sampling

  • The main objective of sampling is to make data collection manageable, while maintaining some level of representativeness. Sampling approaches recommended here do NOT lead to statistically representative data. But they do ensure that the data reflect the diversity of threats and capacities that may exist in the sample frame.
  • If primary data collection is integrated in existing activities or data collection procedures, no particular sampling is needed. If data collection is being set up independent of existing data collection systems, follow the instruction in the sampling section (sec 7). 

3.14.1 Sampling for Response Monitoring : Unit of Measurement

  • Unit of measurement depends on the indicator. It could be a ‘project site’ or an ‘agency’ or an ‘individual.’
  • For example, if your indicator is: ‘% of CFSs that have involved children in development of their monthly activity plan,’ the unit of measurement is a ‘project site’ (in this case CFS).
  • If the indicator is: ‘% of agencies who have provided child safe-guarding training for their CP staff,’ your unit of measurement is ‘agency.’
  • If the indicator is ‘% of reunified children who have expressed satisfaction with the FTR process,’ your unit of measurement is an ‘individual child.’
  • Note: The Unit of analysis can be different from unit of measurement. But it can only be larger than the unit of measurement (not smaller).

  • Level of disaggregation: As one of the first steps during the sampling process, the coordination group should decide on the level of disaggregation for the data.
  • These should be based on OCHA’s definition of “admin levels.” For example if you want to use the data to inform the Humanitarian Needs Overview (HNO), ensure that your data can be disaggregated at the level mandated by the HNO process (often Admin level two). 

Decide which unit of measurement best fits the following indicators

Unit of Measurement: Individual, Project Site or Agency?

  • % of reunified children who received at least two follow-up visits within the first quarter of reunification

     
  • % of reunified children who remain within their families 6 months after the reunification
     
     
  • % of NGOs operating FTR projects that meet the standard for Caseworker to child ratio (1 caseworker to 20 children)
     
     
  • % of FTR projects that meet the standard for caseworker-to-child ratio (1 caseworker to 20 children)
     

Reformulate this indicator such that the unit of analysis is a “project site”: % of children participating in CFS activities who have expressed satisfaction with the activities of the day.

3.15 Sampling for Response Monitoring : Sampling Approach

  • Coverage monitoring does not usually need a specific sampling approach. This is because 5Ws or online activity tracking platforms are often designed to collect data from all project sites.
  • If no coverage monitoring system is in place in the context, attempt to set up a ‘Who does What Where, When and for Whom’ (5W).

  • There are two stages to the sampling for response monitoring. The first stage of sampling is when a project site gets selected for a visit during a specific reporting period.
  • The second stage is only relevant for indicators that have the ‘individual’ as their unit of measurement (e.g. beneficiary satisfaction). During the second stage individual respondents will be selected in a random way to reduce bias.

  • First stage: if agency self-monitoring or peer-to-peer monitoring is opted for, the first stage sampling will basically be an agreed upon plan to ensure coverage of all project sites over time. The actual project sites to be visited in each reporting period can either be selected randomly or can be agreed upon in advance.
  • If independent monitoring is opted for, the first stage sample should be drawn randomly from a list of project sites.

  • First stage selection can be done using the 5W information that outlines all child protection agencies, their activities and their geographical coverage. If 5W matrix is not in use in the context, a simple list can be developed that has all participating agencies, disaggregated by location and activities.
  • It is recommended that at least three (3) project sites be selected for each administrative unit (as per the selected admin level).
  • Note: Ensure that all project sites are visited at least once during the period of 3 to 6 months (depending on the size of the response and capacity of monitors). This will allow for meaningful aggregation and comparison of data across time.
  • Example sample frame:

Which of the following indicators will require second stage sampling?

  • a) % of registered Unaccompanied children receiving at least one follow-up visit every month
  • b) % of CFS or CBPSS sites where daily age-appropriate activities are organized as per the activity plan
  • c) % of CFSs that meet the animator to child ratio (1 to 20)
  • d) % of reunified children who express satisfaction with their case manager

Module 4: Establishing a Monitoring Mechanism

4.1 - Four Main Components

4.2 - Twelve Steps to Establish a Monitoring System

4.2 - Step 1: Convene a monitoring coordination group/ taskforce

The main objective of this step is to ensure buy-in and representation by key stakeholder. Tasks that should be undertaken under this step may include:

  • Engaging representatives from other relevant groups (such as the protection cluster, the education, health clusters and/ or IOM).
  • Presenting what CP Monitoring can and cannot do;
  • Clarifying contributions that will be needed from agencies.

Why is it important to implement a CP monitoring system in an inter-agency manner?

4.3 - Step 2: Consult with CP and other humanitarian agencies to determine feasibility

The main objective of this step is to generate consensus on the general direction of the monitoring project. Tasks that can be undertaken under this step may include:

  • Discussing the need for a monitoring system;
  • Agreeing on the overall scope of the monitoring system, including whether situation or response monitoring (or both) will be implemented;
  • Develop a clear objective statement for CP monitoring;
  • Deciding on a “home” for the monitoring system.

4.4 - Step 3: Identify existing mechanisms that can be used as data collection forums for child protection monitoring

The main objective of this step is to determine the most feasible ways of implementing situation and response monitoring. Tasks to be considered under this step may include:

  • Meeting with staff at national and field levels to discuss existing data collection processes;
  • Examining existing programs to determine how best data can be collected from the field;
  • Exploring the usability of existing monitoring mechanisms.

If there is a Monitoring and Reporting Mechanism (MRM) system in place in the context where you are establishing the monitoring system, will you recommend the use of the MRM data collection mechanism for situation monitoring?

Note: In 2005, the Security Council requested in Resolution 1612 the UN Secretary-General to establish a monitoring and reporting mechanism (MRM), managed by country-based task forces co-led by UNICEF and the highest UN representative in the country, to provide timely and reliable information on six grave children’s rights violations. For more on MRM see: http://www.unicef.org/protection/57929_57997.html

  • Yes
  • No

Why or why not?

4.5a - Step 4: Decide on & adapt methodology(ies) to be used for situation and response monitoring

Based on the findings of steps 2 and 3, the coordination structure must decide on the most appropriate/feasible methodologies for situation and response monitoring. Some tasks may include:

  • Assessing the capacity of communities and implementing agencies in collecting and transmitting data;
  • Feasibility of the use of technology in the context;
  • Data management capacity and possibilities.

4.5b - Step 4, Cont. Practical considerations to decide whether or not to use mobile technology for data collection

  • Does technology improve efficiency and effectiveness?
  • Are mobile phone and internet networks functioning and/or will they be likely cut during a natural disaster or conflict?
  • How long does it take to set up with the system, including software development, etc?
  • Does the use of technology-based monitoring tools put staff security at risk?
  • Is data confidentiality guaranteed when using a mobile device or an online system?
  • Can the technology in question function offline as well?
  • Does it duplicate any existing system?

4.6 - Step 5: Develop and/or contextualize indicators

The main objective of this step is to decide on data needs and develop / contextualize indicators for situation and response monitoring. Please consider the following tasks below: 

  • Agree on the data needs to achieve the stated objectives - otherwise known as ‘What We Need to Know’ (WWNK).
  • On the basis of WWNKs, select / develop / adapt SMART indicators. Consider contextualizing in-country or global indicators instead of coming up with new ones. Sources may include: OCHA indicator registry; CP minimum standards ; HPC, etc. 

Resources:
 

4.7 - Step 6: Develop & adapt data collection tools and procedures and context specific protocols

This step culminates the work from previous steps in data collection tools and context specific protocols. Tools can either be adapted from sample tools or be developed from scratch if necessary. Protocols should also be developed to guide data collection and analysis as well as report writing.

In many contexts, tools will have to be developed in or translated to local language(s). When translated, back translation is also necessary to ensure accuracy of terms and concepts.

Check your understanding (Part A)

You are supporting the establishment of a CP monitoring system in Mali where the official language is French. The Information Manager argues that since all your data collectors speak French and the local languages of concern (Arabic and Tamasheq), you do not need to translate the tools to the local languages. He says: “they can translate to Arabic or Tamasheq on the spot and record all the answers in French.” He argues that having all the answers in French will facilitate data management. Would you agree with him? Yes or no?

 

Why?

Check your understanding (Part B)

You convince your colleague that the data collection tools should be translated into the local languages. But when speaking to a translator, you realize that Tamasheq does not have a strong written tradition. The translator says: “even if I translate the tools, many Tamasheq speakers won’t understand what I have written as people are not used to seeing Tamasheq in writing.” How would you resolve this issue?

4.8 - Step 7: Prepare human resources: train the trainers

The objective is to ensure that you have a cadre of data collectors and those who can coach and support them. In most humanitarian contexts, it may be more efficient to create a cascading training process, starting with a training of trainers and then replicating the training in different areas affected by the emergency. 

4.9 - Step 8: Carryout field-testing

The main objective of step 8 is to ensure that tools and protocols work well in the context. Field testing is meant to not only test the tools, but also the whole process of data collection and management. Field testing should ideally happen after the training of trainers and before the full roll-out of the training for all data collectors. This will allow you to use the trained trainers in pilot testing the tools.

4.10 - Step 9: Finalize tools and monitoring protocol(s)

Once field testing is completed, tools and protocols have to be revised and finalized. If major changes are affected, training material may also need to be adapted to the changes. 

During this step, and before finalizing, sign offs should be requested from all participating agencies as well as the internal review board (IRB). 

4.11 - Step 10: Collect and manage data: rolling out the monitoring system

This step is at the heart of the monitoring system. Therefore it needs to be managed and supervised well to ensure high quality data. Some tasks under this step are:

  • Collecting data from identified primary and secondary sources;
  • Ensuring data quality and accuracy through supervisory visits and random spot-checks;
  • Managing data (including data entry and cleaning) using data management tools.

4.12 - Step 11: Analyse, interpret and share data through periodic reports to inform programming and advocacy

The main objective of this step is to use the data that has been collected to:

  • Interpret the data to make sense of it;
  • Produce and share periodic reports;
  • Inform decision-making;
  • Advocate when, where and with whom necessary.

Note: Analysis, interpretation and report writing is often a bottleneck in effective use of data

Some tasks under this step are as follows:

  • Analysing data. This can be done quantitatively or qualitatively (or a mix), depending on the nature of the data;
  • Triangulate data through interpretation workshop;
  • Use data to inform programming, make strategic decisions, advocate and raise funds.

4.14 - Step 12: Review the functioning of the monitoring system and adjust monitoring protocol(s)

The main objective of this step is to ensure that the monitoring system is functioning well and that adjustments are done systematically, when necessary. Some tasks are:

  • Critically analyse the implementation of the monitoring system and suggest improvements.
  • Assess the timeliness of reporting and validity of reported data.
  • Evaluate the ability of the monitoring system to detect protection risks and threats.
  • Make necessary adjustments to improve the system.

4.15 - Indicators

An indicator is a quantitative or qualitative variable that provides a simple and reliable means to measure achievement, to reflect the changes in the situation connected to an event or intervention or to help assess the performance of organizations.

Indicators used in CP Monitoring either describe an aspect of the situation (referred to as situational indicators) or the CP response (referred to as response indicators). 

4.16 - Definitions of Indicators

  • SMART indicator: SMART stands for Specific, Measurable, Achievable, Relevant, and Time bound. Example for a NON-SMART indicator: “percentage of separated children who were identified and registered and reunified”. This indicator has too many components (not-specific), hard to measure, and not time-bound.
  • Baseline: Baseline determines the starting point of measurement. It tells us where we are when we begin the measurement. Therefore, baseline is not a type of indicator per se.
  •  Target: A target is the value assigned to an indicator that is set as a goal by program managers or coordination bodies. It is used by program staff as a determinant for the success of the intervention. For example, target for the quality indicator mentioned above can be 90%.

  • Input indicator: Measures the financial, human, and material resources used for the intervention. Input indicators are only relevant to response monitoring.
  • Process indicator: Measures activities that have taken place to move the program forward. Process indicators are only relevant to response monitoring.
  • Output indicator: Measures products, goods and services, which result from an intervention. Output indicators are mostly relevant to response monitoring, but can sometimes apply to situation monitoring.
  • Outcome indicator: Measures short-term and medium-term effects of an intervention. Outcome indicators can be used for both response and situation monitoring.

  • Indicator to measure quality:  Measures quality of the products, goods and services delivered by the intervention. Quality indicators are only relevant to response monitoring. Quality indicators can be considered a subset of ‘input,’ ‘output’ and ‘outcome’ indicators.
  • Incidence: Incidence captures new cases. The incidence rate is the number of new cases per population at risk in a given time period.  Incidence information can be collected through situation monitoring.
  • Prevalence: Prevalence captures all existing cases of interest at the time of the measurement (including old and new cases). Prevalence information can be collected through assessments or through the baseline survey of a situation monitoring system.

Check your understanding

Directions: Match the term with the definition
  • SMART indicator
    Indicator that is Specific, Measurable, Achievable, Relevant, and Time bound
  • Baseline
    Determines the starting point of measurement. It tells us where we are when we begin the measurement. Therefore, baseline is not a type of indicator per se.
  • Target
    The value assigned to an indicator that is set as a goal by program managers or coordination bodies. It is used by program staff as a determinant for the success of the intervention.
  • Input indicator
    Measures the financial, human, and material resources used for the intervention. This is only relevant to response monitoring.
  • Process indicator
    Measures activities that have taken place to move the program forward. This is only relevant to response monitoring.
  • Output indicator
    Measures products, goods and services, which result from an intervention. This is mostly relevant to response monitoring, but can sometimes apply to situation monitoring.
  • Outcome indicator
    Measures short-term and medium-term effects of an intervention. It can be used for both response and situation monitoring.
  • Indicator to measure quality
    Measures quality of the products, goods and services delivered by the intervention. It is only relevant to response monitoring. It can be considered a subset of ‘input,’ ‘output’ and ‘outcome’ indicators.
  • Incidence
    This captures new cases. The rate is the number of new cases per population at risk in a given time period. This information can be collected through situation monitoring.
  • Prevalence
    This captures all existing cases of interest at the time of the measurement (including old and new cases). This information can be collected through assessments or through the baseline survey of a situation monitoring system.

4.17 - Data collection and staffing requirements

There are three main roles for a monitoring system:

  • Project Manager/coordinator holds the overall responsibility of the monitoring project.
  • Data/Information manager: Data manager is primarily responsible for compiling, cleaning, and analyzing the data.
  • Enumerators/data collectors are responsible to collect primary data for each data collection period. 

4.18 - Data analysis and sharing

There are four data management tools that can be adapted and used for data management in a CP Monitoring system:

Response Monitoring:

Situation Monitoring:

These tools have to be adapted to the data collection tools in each context.