The NUSAF YOP evaluation thus far ...

This evaluation has thus far consisted of the design process and first phase of the baseline, conducted in February to March 2008. I will briefly outline both of these in this post, starting with a short description of the NUSAF project.

1. Introduction to NUSAF

NUSAF is funded by the World Bank and Government of Uganda in order to target the north of Uganda, which has thus far significantly lagged behind the development of the rest of the country, in part due to a 20 year civil war based in Acholiland.

To this end, NUSAF has created a grant program to target community based development. Each district in the north is responsible for the funding of projects within a given budget, including large scale construction (such as roads and bridges) and group project grants to increase livelihoods through vocational training and tool purchases. Group projects normally consist of 15-30 members identified by communities as vulnerable and are given on average $10,000.

One component of these group projects is the Youth Opportunities Project (YOP), designed for youth aged 15 to 35 (in the north, youth is considered a broad range of ages). This is the program we are evaluating.

YOP groups that apply for funding from NUSAF go through a rigorous vetting process. Their first step is to bring their proposals to the district headquarters, who reviews the paperwork and passes those deemed most needy to the head NUSAF office. There the groups go through another auditing process, and a final selection of groups is decided on to fund. When the groups recieve funding in a group bank account, the money is their responsiblity to use according to their budgets for training and purchasing tools. A later follow-up by the NUSAF district offices ensures the money is being spent appropriately.

2. Evaluation deisgn and methodology

As there has been more interest in the program than available funds, we are faced with over-subscription, and thus have the opportunity to run a randomized impact evaluation. A total of 530 groups were identified for funding. Of those, 265 were randomly selected for funding, with the remaining 265 acting as a control.

As each district is constrained by their own funding, randomization was done at the district level. Groups were given a random number and sorted by that number. Groups were then selected for funding until the full funding limit was reached. For most districts this meant funding between 40% to 60% of groups.

The adavantage of a randomized treatment and control group is that it helps to control for unobservable characteristics between the two groups. When running statistical tests, it is therefore more likely that the two groups are comparable, and thus ensuring we can identify the effect of recieving NUSAF funding. There is of course no way to ensure that all unobservable differencees have been eliminated, but we can test this for observables from the baseline.

3. Phase 1 baseline questionnaire

The first phase of the baseline was conducted before the groups were told if they are to be in the treatment or control groups in order to avoid biases in responses. Some simple questions were asked of the entire group in order to identify the group characteristics. 5 random individuals from the group were then chosen to be given a detailed questionnaire.

The individual questionnaire was modeled on the SWAY interviews and was conducted in private with detailed questions on household composition and demographics, income, work history, conflict history, psycho-social well-being, community group participation, loan history, risk preferences and health.

When the baseline was completed, balance tests were conducted to ensure the answers from people in the treatment group were not statistically different than those in the control. Most of the tests suggest that there are no significant differences.

Of course, it is important to note that, unless we have reason to believe there are non-linear effects of differences between the two groups, we can control for any differences and still get a consistent estimation from OLS. Using a baseline in conjunction with a randomized design helps to ensure there are no major biases in the results, but of course it cannot guarentee it.

The final empirical goal will then be to take the results of the baseline, in conjunction with the follow-up of both the treatment and control groups, and statistically determine the average treatment effect of being funded by NUSAF.

4. Cross-cutting design

In addition to the normal structure of this impact evaluation, we have added an additional cross-cutting design component in order to identify some of the potentially important determinants of success in NUSAF.

The funded youth groups have thus been further randomly split into three additional experimental groups of approximately 90 youth groups each. The first group is treated by NUSAF according to standard opperating procedures. The second and third are given funds to hire an additional facilitator for the groups as recieving consistant support from an outside agent was suggested as a potentially important indicator of group success.

One set of groups contracts with a person of their chosing from the community and, pending an evaluation of services, pays this person 150,000 USH ($100) at the end of 6 months. The other set also chooses a person from the community, but it is up to the districts to evaluate and pay this person.

As this is another randomized experiment, this allows us to identify the impact of these two additonal components in much the same way we are doing with the full evaluation. We hope then to be able to determine not just if and how much NSUAF impacts peoples lives, but what some of the determining factors may be.

5. Phase 2 of the baseline

The next phase of the baseline is set to begin at the end of October through the beginning of December 2008. I am now in Uganda finalizing the preperation. This blog will be updated with documentation of the next phase, with more details to come.