The second component of the second phase of the baseline is a study of micro-enterprise profits and is a little more complicated to explain because it is not standard in the literature. In fact, I am only aware of work by De Mel et al. dealing with this issue. I do not know of any work in Africa.
Any study looking at the outcomes of a business grant program must deal with the issue of measuring profits as it is one of the direct implications of a program on a person’s economic status. That few people have looked into this in detail is surprising to me.
This project came out of my time living in northern Uganda last year and assisting on designing an evaluation of an NGO that does very small scale grant programs. I noticed that most people made some small but serious mistakes in calculating their profits (not including transport and communication being the most popular). Then, in June of this year, I met with some people from a micro-finance group in Kampala and was shocked to learn that they give their clients no training. Not surprisingly, the majority of clients have to refinance their loans, and so get trapped in a cycle of debt.
Before getting to the research design, there are a number of research questions we want to address with this component. A short list includes: do people calculate their profits correctly, and if not, what are the mistakes they are making? Do these mistakes have long-run implications for the business? Are there better ways of eliciting correct profit data? Do the mistakes correlate with other variables that are easy to measure? How are the NUSAF businesses different than businesses in the communities? Do too many businesses in one area adversely affect profits, as one might expect? How much time do people spend on their businesses, and is there a household constraint to this time? How does this time constraint affect women? Do women make similar profits in similar industries as men, or less? What are the returns to business in the north, and how do they compare to loan interest rates? Do people have an innate business skill, like Yunnus believes, or do they need help? If they need help, what is the most cost effective way to address these needs? Are new businesses more prone to these mistakes than older businesses?
This is an incredibly long list of questions, but I think we can answer them using a research design similar to that of De Mel et al. mentioned above.
First, we will draw from three populations. I have data on the old and newly funded NUSAF groups and can identify different types of businesses from each of these. We will then conduct a community census of businesses in a number of areas where the NUSAF groups are. This will be done with a short (5 minute) questionnaire of about 1000 businesses to ensure a large enough sample (which also helps with generalization beyond NUSAF).
From both of these groups we will randomly select 400 businesses to conduct a detailed questionnaire that I have adapted from De mel et al. This is still in draft form; I will update on the progress of the pre-testing.
From these we will randomly select 200 businesses to give bookkeeping tools to. This will prove to be a difficulty as many people are illiterate. I will be looking into a method to get the household/community involved to help with increasing compliance.
It is possible that bookkeeping tools are an intervention in themselves, and so 50 businesses from this group will be randomly selected, along with 50 random businesses without bookkeeping tools, to be given a detailed follow-up. An enumerator will visit these selected businesses every other day for 30 days to collect daily business interaction records. This will then constitute our most accurate measure of profits, against which we will make all comparisons.
There are of course some potential difficulties. The NUSAF projects are group projects, so separating out assets and profits for each person will be difficult. Also, business involvement may be erratic, so the detailed follow-up may run into a lot of “no shows”. Identifying “shadow labor” (implied labor availability of household member, like the kids) of households could also be difficult. There is also a need to balance cost with dispersion to increase the generalizability of the findings. Finally, we probably can’t make strong claims on women’s involvement as we cannot identify individual preferences and social traditions that may complicate the results. This is an especially big problem as women likely engage in more informal work, and so this could confuse the data.
Besides pre-testing the instrument, I also want to think about possible very low cost interventions to assist businesses. Last year I came up with the idea of a low cost radio program to teach basic profit calculations. I am now leaning towards getting businesses to use the bookkeeping tools more. Both of these could then be compared to larger interventions, such as multi-day business skills training. Within this is the possibility of using the cross-cutting design to increase assistance to the groups, as well as compliance with the bookkeeping tools.
An outstanding question for me is how to connect the mistakes people make with profits with the outcomes of the business. I still do not have a strong enough design to make this clear as omitted variables and bad proxies of outcome could be important.
Tuesday, September 30, 2008
Second phase of the baseline: Group Dynamics
As NUSAF YOP funding is going to groups rather than individuals, the success of each individual is closely tied in a number of important ways to the group. The second phase of the baseline includes a few ways we have come up with to capture group effects.
First, the skills of the other group members may be important for an individual’s success. Data on education of everyone in the group and details on 4 other people in the group was captured in the first baseline. In the second phase, we will expand some of our original questions to more people with a short questionnaire to capture more information on skills training and the qualities of the leaders of the groups.
Second, we will run experiments derived from game theory in order to capture the dynamics of the interactions between group members. We will measure such qualities as altruism towards other members, trust and the ability to overcome collective action problems by using the dictator, ultimatum, investment and public goods games. I have some preliminary group and individual scripts of the investment and public goods games that I will use on a selection of volunteers in West Nile and Iteso regions over the next two weeks. I have run them in Kitgum with good success, but want to give them another run.
The timing of this phase of the baseline is important. The groups have just been given their funding and so are now either in training or have begun work, and so are interacting closely. By measuring their quality of interaction now, we hope to be able to capture data that may give some clues as to the success of individuals, as well as predict how long the groups will be together in the future. We are also curious to see the interaction of group leaders with the group, perhaps giving some important clues as to preference of equality, from both individuals in the group and the group leaders. Put an-other way, does corrupt leadership and willingness to accept this corruption has a negative impact on group performance?
I will update the progress of the pre-testing of this component in the next few weeks.
First, the skills of the other group members may be important for an individual’s success. Data on education of everyone in the group and details on 4 other people in the group was captured in the first baseline. In the second phase, we will expand some of our original questions to more people with a short questionnaire to capture more information on skills training and the qualities of the leaders of the groups.
Second, we will run experiments derived from game theory in order to capture the dynamics of the interactions between group members. We will measure such qualities as altruism towards other members, trust and the ability to overcome collective action problems by using the dictator, ultimatum, investment and public goods games. I have some preliminary group and individual scripts of the investment and public goods games that I will use on a selection of volunteers in West Nile and Iteso regions over the next two weeks. I have run them in Kitgum with good success, but want to give them another run.
The timing of this phase of the baseline is important. The groups have just been given their funding and so are now either in training or have begun work, and so are interacting closely. By measuring their quality of interaction now, we hope to be able to capture data that may give some clues as to the success of individuals, as well as predict how long the groups will be together in the future. We are also curious to see the interaction of group leaders with the group, perhaps giving some important clues as to preference of equality, from both individuals in the group and the group leaders. Put an-other way, does corrupt leadership and willingness to accept this corruption has a negative impact on group performance?
I will update the progress of the pre-testing of this component in the next few weeks.
Monday, September 29, 2008
Some of the challenges we have faced
Overall, the main NUSAF office has appreciated the impact evaluation and recognizes its importance for understanding the impact of the program. Difficulties though remain.
The biggest problem for the evaluation has been that some people in the districts do not appreciate the randomized design, which is understandable. While there has not been one standard solution to convincing the districts to cooperate, we have re-iterated with them how, faced with over-subscription, a randomized selection process is the most fair.
I do though agree with their common critique that, while we have more people eligible than we can afford, there are many groups that are more deserving of funding than others. This of course begs the question, what does more deserving mean?
For some groups, it means they are significantly more vulnerable. For instance, some groups are composed of single child mothers, or minority ethnic groups. It was decided that these groups should be kept out of the evaluation and given automatic funding.
For some groups though, it is not clear that the most vulnerable are the most deserving. I think it is an open question, given the way NUSAF is currently designed, that the most vulnerable will benefit as much as they could from the program (see my previous comments on the cross-cutting design).
There is also a programing problem that affects group selection. District officers are not responsible to the NUSAF main office, but instead to the district Chief Administrative Officer (CAO), who is a political appointment by the district leaders. There is thus both a political element to NUSAF, as well as an oversight blackhole for the main NUSAF office.
This setup has undoubtably lead to the numerous corruption problems in NUSAF, which can be found almost every day in the papers. Some district offices, rather than using their funds to help the people, are embezziling funds through "ghost projects" that do not actually contain real members, but instead the money is sent to phony accounts. When it gets real bad, and evidence has piled up against an office, NUSAF files formal charges. For instance, many people in the Acholiland offices have been sent to prison, including the Gulu district officer, who is sitting in a cell right now.
All of this then means that a randomized evaluation is not only more fair for participants, but it is likely the ONLY way to accurately measure the impact of NUSAF. The decision of who to fund, while officially an organized process, is actually quite messy at the district level, as I am finding out during this trip. The decision often targets the needy, but can also be motivated by politics and cronyism, and even a confused sense of the most needy.
While I believe we have successfully addressed all of the challenges we have had for this evaluation, I think these issues point to important program design issues that should be resolved in future NUSAF programs.
The biggest problem for the evaluation has been that some people in the districts do not appreciate the randomized design, which is understandable. While there has not been one standard solution to convincing the districts to cooperate, we have re-iterated with them how, faced with over-subscription, a randomized selection process is the most fair.
I do though agree with their common critique that, while we have more people eligible than we can afford, there are many groups that are more deserving of funding than others. This of course begs the question, what does more deserving mean?
For some groups, it means they are significantly more vulnerable. For instance, some groups are composed of single child mothers, or minority ethnic groups. It was decided that these groups should be kept out of the evaluation and given automatic funding.
For some groups though, it is not clear that the most vulnerable are the most deserving. I think it is an open question, given the way NUSAF is currently designed, that the most vulnerable will benefit as much as they could from the program (see my previous comments on the cross-cutting design).
There is also a programing problem that affects group selection. District officers are not responsible to the NUSAF main office, but instead to the district Chief Administrative Officer (CAO), who is a political appointment by the district leaders. There is thus both a political element to NUSAF, as well as an oversight blackhole for the main NUSAF office.
This setup has undoubtably lead to the numerous corruption problems in NUSAF, which can be found almost every day in the papers. Some district offices, rather than using their funds to help the people, are embezziling funds through "ghost projects" that do not actually contain real members, but instead the money is sent to phony accounts. When it gets real bad, and evidence has piled up against an office, NUSAF files formal charges. For instance, many people in the Acholiland offices have been sent to prison, including the Gulu district officer, who is sitting in a cell right now.
All of this then means that a randomized evaluation is not only more fair for participants, but it is likely the ONLY way to accurately measure the impact of NUSAF. The decision of who to fund, while officially an organized process, is actually quite messy at the district level, as I am finding out during this trip. The decision often targets the needy, but can also be motivated by politics and cronyism, and even a confused sense of the most needy.
While I believe we have successfully addressed all of the challenges we have had for this evaluation, I think these issues point to important program design issues that should be resolved in future NUSAF programs.
The NUSAF YOP evaluation thus far ...
This evaluation has thus far consisted of the design process and first phase of the baseline, conducted in February to March 2008. I will briefly outline both of these in this post, starting with a short description of the NUSAF project.
1. Introduction to NUSAF
NUSAF is funded by the World Bank and Government of Uganda in order to target the north of Uganda, which has thus far significantly lagged behind the development of the rest of the country, in part due to a 20 year civil war based in Acholiland.
To this end, NUSAF has created a grant program to target community based development. Each district in the north is responsible for the funding of projects within a given budget, including large scale construction (such as roads and bridges) and group project grants to increase livelihoods through vocational training and tool purchases. Group projects normally consist of 15-30 members identified by communities as vulnerable and are given on average $10,000.
One component of these group projects is the Youth Opportunities Project (YOP), designed for youth aged 15 to 35 (in the north, youth is considered a broad range of ages). This is the program we are evaluating.
YOP groups that apply for funding from NUSAF go through a rigorous vetting process. Their first step is to bring their proposals to the district headquarters, who reviews the paperwork and passes those deemed most needy to the head NUSAF office. There the groups go through another auditing process, and a final selection of groups is decided on to fund. When the groups recieve funding in a group bank account, the money is their responsiblity to use according to their budgets for training and purchasing tools. A later follow-up by the NUSAF district offices ensures the money is being spent appropriately.
2. Evaluation deisgn and methodology
As there has been more interest in the program than available funds, we are faced with over-subscription, and thus have the opportunity to run a randomized impact evaluation. A total of 530 groups were identified for funding. Of those, 265 were randomly selected for funding, with the remaining 265 acting as a control.
As each district is constrained by their own funding, randomization was done at the district level. Groups were given a random number and sorted by that number. Groups were then selected for funding until the full funding limit was reached. For most districts this meant funding between 40% to 60% of groups.
The adavantage of a randomized treatment and control group is that it helps to control for unobservable characteristics between the two groups. When running statistical tests, it is therefore more likely that the two groups are comparable, and thus ensuring we can identify the effect of recieving NUSAF funding. There is of course no way to ensure that all unobservable differencees have been eliminated, but we can test this for observables from the baseline.
3. Phase 1 baseline questionnaire
The first phase of the baseline was conducted before the groups were told if they are to be in the treatment or control groups in order to avoid biases in responses. Some simple questions were asked of the entire group in order to identify the group characteristics. 5 random individuals from the group were then chosen to be given a detailed questionnaire.
The individual questionnaire was modeled on the SWAY interviews and was conducted in private with detailed questions on household composition and demographics, income, work history, conflict history, psycho-social well-being, community group participation, loan history, risk preferences and health.
When the baseline was completed, balance tests were conducted to ensure the answers from people in the treatment group were not statistically different than those in the control. Most of the tests suggest that there are no significant differences.
Of course, it is important to note that, unless we have reason to believe there are non-linear effects of differences between the two groups, we can control for any differences and still get a consistent estimation from OLS. Using a baseline in conjunction with a randomized design helps to ensure there are no major biases in the results, but of course it cannot guarentee it.
The final empirical goal will then be to take the results of the baseline, in conjunction with the follow-up of both the treatment and control groups, and statistically determine the average treatment effect of being funded by NUSAF.
4. Cross-cutting design
In addition to the normal structure of this impact evaluation, we have added an additional cross-cutting design component in order to identify some of the potentially important determinants of success in NUSAF.
The funded youth groups have thus been further randomly split into three additional experimental groups of approximately 90 youth groups each. The first group is treated by NUSAF according to standard opperating procedures. The second and third are given funds to hire an additional facilitator for the groups as recieving consistant support from an outside agent was suggested as a potentially important indicator of group success.
One set of groups contracts with a person of their chosing from the community and, pending an evaluation of services, pays this person 150,000 USH ($100) at the end of 6 months. The other set also chooses a person from the community, but it is up to the districts to evaluate and pay this person.
As this is another randomized experiment, this allows us to identify the impact of these two additonal components in much the same way we are doing with the full evaluation. We hope then to be able to determine not just if and how much NSUAF impacts peoples lives, but what some of the determining factors may be.
5. Phase 2 of the baseline
The next phase of the baseline is set to begin at the end of October through the beginning of December 2008. I am now in Uganda finalizing the preperation. This blog will be updated with documentation of the next phase, with more details to come.
1. Introduction to NUSAF
NUSAF is funded by the World Bank and Government of Uganda in order to target the north of Uganda, which has thus far significantly lagged behind the development of the rest of the country, in part due to a 20 year civil war based in Acholiland.
To this end, NUSAF has created a grant program to target community based development. Each district in the north is responsible for the funding of projects within a given budget, including large scale construction (such as roads and bridges) and group project grants to increase livelihoods through vocational training and tool purchases. Group projects normally consist of 15-30 members identified by communities as vulnerable and are given on average $10,000.
One component of these group projects is the Youth Opportunities Project (YOP), designed for youth aged 15 to 35 (in the north, youth is considered a broad range of ages). This is the program we are evaluating.
YOP groups that apply for funding from NUSAF go through a rigorous vetting process. Their first step is to bring their proposals to the district headquarters, who reviews the paperwork and passes those deemed most needy to the head NUSAF office. There the groups go through another auditing process, and a final selection of groups is decided on to fund. When the groups recieve funding in a group bank account, the money is their responsiblity to use according to their budgets for training and purchasing tools. A later follow-up by the NUSAF district offices ensures the money is being spent appropriately.
2. Evaluation deisgn and methodology
As there has been more interest in the program than available funds, we are faced with over-subscription, and thus have the opportunity to run a randomized impact evaluation. A total of 530 groups were identified for funding. Of those, 265 were randomly selected for funding, with the remaining 265 acting as a control.
As each district is constrained by their own funding, randomization was done at the district level. Groups were given a random number and sorted by that number. Groups were then selected for funding until the full funding limit was reached. For most districts this meant funding between 40% to 60% of groups.
The adavantage of a randomized treatment and control group is that it helps to control for unobservable characteristics between the two groups. When running statistical tests, it is therefore more likely that the two groups are comparable, and thus ensuring we can identify the effect of recieving NUSAF funding. There is of course no way to ensure that all unobservable differencees have been eliminated, but we can test this for observables from the baseline.
3. Phase 1 baseline questionnaire
The first phase of the baseline was conducted before the groups were told if they are to be in the treatment or control groups in order to avoid biases in responses. Some simple questions were asked of the entire group in order to identify the group characteristics. 5 random individuals from the group were then chosen to be given a detailed questionnaire.
The individual questionnaire was modeled on the SWAY interviews and was conducted in private with detailed questions on household composition and demographics, income, work history, conflict history, psycho-social well-being, community group participation, loan history, risk preferences and health.
When the baseline was completed, balance tests were conducted to ensure the answers from people in the treatment group were not statistically different than those in the control. Most of the tests suggest that there are no significant differences.
Of course, it is important to note that, unless we have reason to believe there are non-linear effects of differences between the two groups, we can control for any differences and still get a consistent estimation from OLS. Using a baseline in conjunction with a randomized design helps to ensure there are no major biases in the results, but of course it cannot guarentee it.
The final empirical goal will then be to take the results of the baseline, in conjunction with the follow-up of both the treatment and control groups, and statistically determine the average treatment effect of being funded by NUSAF.
4. Cross-cutting design
In addition to the normal structure of this impact evaluation, we have added an additional cross-cutting design component in order to identify some of the potentially important determinants of success in NUSAF.
The funded youth groups have thus been further randomly split into three additional experimental groups of approximately 90 youth groups each. The first group is treated by NUSAF according to standard opperating procedures. The second and third are given funds to hire an additional facilitator for the groups as recieving consistant support from an outside agent was suggested as a potentially important indicator of group success.
One set of groups contracts with a person of their chosing from the community and, pending an evaluation of services, pays this person 150,000 USH ($100) at the end of 6 months. The other set also chooses a person from the community, but it is up to the districts to evaluate and pay this person.
As this is another randomized experiment, this allows us to identify the impact of these two additonal components in much the same way we are doing with the full evaluation. We hope then to be able to determine not just if and how much NSUAF impacts peoples lives, but what some of the determining factors may be.
5. Phase 2 of the baseline
The next phase of the baseline is set to begin at the end of October through the beginning of December 2008. I am now in Uganda finalizing the preperation. This blog will be updated with documentation of the next phase, with more details to come.
Impact evaluation in Uganda
Welcome to yet one more blog on development.
The goal of this blog is to document the Northern Uganda Social Action Fund Youth Opportunities Program impact evaluation that I am conducting with Chris Blattman of Yale University and Sebastian Martinez of the World Bank. All of the posts here though reflect my own opinion, and may not necessarily be those of the World Bank.
Please check back for upcoming posts. I am putting this together while in the north of Uganda, and internet is not always reliable, so please have patience.
The goal of this blog is to document the Northern Uganda Social Action Fund Youth Opportunities Program impact evaluation that I am conducting with Chris Blattman of Yale University and Sebastian Martinez of the World Bank. All of the posts here though reflect my own opinion, and may not necessarily be those of the World Bank.
Please check back for upcoming posts. I am putting this together while in the north of Uganda, and internet is not always reliable, so please have patience.
Subscribe to:
Posts (Atom)
