Surveys are an invaluable research tool, they can help us understand non-observable phenomena such as perception, preferences, and beliefs which are the major determinants of social, political, and economic outcomes in all societies. Understanding citizens' preferences about policies is crucial; it allows for legislations that are sustainable and it paves the way for policies to go unchallenged in courts.

The policies around taxes especially, have been a cause of major contention in Pakistan; the luxury property tax policy of 2013 was challenged and ended up in the high court and something in a similar vein is currently unfolding in Islamabad. When matters of taxation are taken up in courts, they lead to significant losses in terms of revenues and moreover, such policies reduce the trust in the state of the citizens.

To better inform those in charge of drawing up property tax policies of what the citizens of Lahore want, we took to field starting in December 2023 in liaison with the Survey Wing at IDEAS, our aim with the survey is to understand how much citizens of Lahore are aware of the current property tax policies, what their preferences are and how various treatment arms move those preferences.

However, the initial response rates of our survey pilots were low, at a little over 10%. Over the past few months, we conducted several pilots and kept incorporating feedback from the field. The changes and tweaks we made allowed us to massively improve on the initial response rates. The following was our approach and what we learned along the way:

1.     Getting the Preamble Right. The preamble is one of the most important components of the survey as this is where the enumerators introduce themselves and the survey to the respondent. It is important to mention here a few key elements about the survey:

a. The institutions that the research is being conducted by; mentioning the affiliated institutions upfront builds credibility and trust.

b. The purpose of the study; it is important to give the respondents some preface about the study and the premise of the research. Taking a direct approach here helps, if the survey is about taxes, then that should be mentioned upfront, this prepares the respondents for the questions to come.

c. Estimated total time it will take to complete the survey; this is another thing which helps prepare the respondents beforehand and can help reduce partial completions.

d. Ask for respondent consent & assure anonymity; these are also the Institutional Review Board (IRB) requirements for all surveys including a human subject and should be adhered to strictly.

e. Mention any reward/compensation that the respondent can get in exchange for their time.

Mehek Khaliq & Ali Shahab from the IDEAS Survey Wing – Pictured here introducing the survey to the respondent near Mozang, Lahore.

2.     Keep the Survey Short & Direct.  The longer the survey is, the more respondent fatigue it will cause and the more it will make them want to drop out. It is best to keep only the questions that are essential for the survey. It is also a good practice to word the questions so that they are more direct and less redundant, redundancy can also lead to respondent fatigue.

3.     Training the Enumerators. After designing the survey, the next step is to sit with the enumerators and go over the survey with them. The enumerators depending upon their experience will provide some initial feedback of how things are likely to be received on ground. The enumerators also need to understand the survey in and out, if there are any complicated components within the survey, they need to familiarize themselves with them, so that when they are in the field things go smoothly.

4.     Pilot, Make Changes & Pilot Again. Pilots provide the opportunity to test how the survey is being received by the audience and they offer key insights into the on-ground realities. By analyzing the pilot data and incorporating the insights of the survey’s reception from the field; the right adjustments can be made that can allow for a much higher uptake. While the pilots are underway, it can be extremely fruitful to accompany the enumerators on the field or to conduct some sample surveys yourself to better communicate the findings to the rest of the research team.

5.     Identify the Target Audience. A strong identification of the target group can greatly help increase the response rates. In our case the target group were the members of the household in-charge of paying taxes and we found that their availability was higher over the weekends as compared to weekdays when they were not away for work. Analyzing data to look for any trends across different days of the week is a good practice.

6.     Measure Attrition. To ensure that there is no systemic refusal for the survey, which can lead to non-response bias, it is important to have some identifiers before the main components are introduced, these can include, age, gender, education, and occupation; these can later be analyzed to ensure that there is no one specific group that is deciding to opt out of the survey based on the topic of the survey or the questions being asked. While analyzing data it is also important to note the parts of the survey where there are sudden jumps in attrition so that changes can be made around those components during pilots.

7.     Have Different Strategies for Different Groups. If your sample contains any significant stratifications based on income, education, age or location then the need to have different strategies to cater to each group will make themselves felt during the pilots. For instance, if you are offering a compensation of 500 to someone from a low-income group in exchange for their time and that is wielding a high response rate for that group; the same strategy may not work for someone from a high-income stratum.  

8.     Take Appointments. If at the time of the survey the respondent refuses because they are preoccupied, it is good practice to ask them for another suitable time based on their preferred availability. Similarly for longer survey, the approach of splitting the survey into 2 separate halves can be adopted, and appointments can then be taken to complete the second half of the survey at a later convenience of the respondent.

Mehek Khaliq from the IDEAS Survey Wing Team – Asking the respondent for a suitable time to conduct the survey – Picture taken near Garhi Shahu Locality in Lahore

9.     Offer Compensation as Budget Allows. Offering respondents some form of compensation for their time can boost the response rates greatly. Incorporating lotteries in the survey is a great way to offer compensation; quick cash transfer services like Jazzcash/Easypaisa can also be utilized for on the spot compensation. It is important to note also that compensation does not have to be always monetary, as mentioned earlier having a cash prize might not have any effect on someone wealthy; there a different approach of outlining that their individual responses can help change the policies in the country can be more efficient.

10.      Timing Matters. A reason for the low initial response rates for our survey was that our initial pilot roll outs intersected with the election campaigns of different political parties leading up to the general election in March 2024, this led people to think that we had come on behalf of some political party to ask for votes. The distrust at the time near elections was so high that on a couple of instances people gathered around our survey team telling others in their neighborhood that we had come to ask for votes and even threatened to call the police if we did not leave. It is recommended to plan the roll-out ahead of time in such a way that it does not coincide with any major social, religious or political event.

Muhammad Faizan Imran is a Field Research Coordinator at Mahbub ul Haq Research Centre (LUMS).