When I joined StreetLight Data, I learned that household transportation surveys – that’s physical pieces of paper mailed to residences – are one of the most popular transportation data collection methods. According to my new hire training, these studies can be problematic: they’re difficult to design, the sample sizes are small, and their accuracy depends on the memories of survey respondents. In fact, our product, StreetLight InSight®, was created in part to fill these studies' gaps with locational data from mobile devices.
Serendipitously, the city of San Francisco sent me a survey (see below) on my transportation behavior soon after I joined the team, which gave me the chance to put my colleagues’ claims about household surveys’ flaws to the test.
The picture above shows the materials that I received from the San Francisco Dept. of the Environment.
The $2 incentive to submit the survey by June 17th was the first thing I noticed. Given that I’m writing this post in mid-July, it’s clear that wasn’t effective. In fact, if I worked elsewhere, that survey would still be on my To-Do list. What if everyone like me reacted similarly, or never submitted the survey at all? How would the lack of responses influence the results?
Where Did I Go, and How Did I Get There?
Once I stopped procrastinating, the real work started: remembering every trip I’ve made for “Work, School, Errands, or Other” in the last three days, and indicating my mode of transport, i.e.: whether I drove, walked, took public transit, or biked.
I tackled Work first, but struggled to categorize my daily commute because it comprises two walks and a public transit ride - I'm a multimodal kind of woman. I split the difference and categorized my morning commute trip as a walk, and the evening trip as public transit. I wondered if my response would confuse the statisticians who examined the results, and, more importantly if my compromise would impact the survey’s accuracy.
The next type of trip that applied to me is “Errands.” I wasn't sure if stopping for coffee on the way to work counted as a separate trip from my commute, so I didn't include it. But just how many times did I run out for lunch because I forgot my leftovers in the fridge at home? Two out of the last three days - I think.
Next came “Other.” I stared hard at that survey for a solid 5 minutes struggling to remember the other trips I’d made in the past three days. Oh right – that early morning trip back from Las Vegas! It seemed like a big one to me, but "airplane" was not a mode that the survey allowed me to select. I decided to pretend that trip to Las Vegas just didn’t happen. After all, what happens there, stays there, right?
Checking My Work
In the end, my survey was completed with just 10 trips. Setting aside the coffee and Las Vegas trips I that I consciously left unrecorded, I still wasn’t confident that my survey accurately represented my full mobility behavior. So, I checked my work by turning to my trusty smart phone and reviewing its location records to jog my memory. It turns out that I forgot:
- (2) trips to the gym (one by car, one by foot)
- (1) trip to a networking event (by foot)
- (1) trip to Walgreens (by public transit)
I’m almost embarrassed to admit how poor my recall was, but the fact is that it's not important for me to remember exactly where I go and how I get there to this level of detail. That data may be valuable to the city of San Francisco, but remembering these trips will not make me any happier or more successful. Beyond that $2, my only incentive was the vague promise that the survey results would "inform decisions for future transportation programs and service in your neighborhood."
Of course, surveys are valuable in that they can provide much more context about than the list of addresses in my mobile phone can. However, they cannot tell the full story of transportation behavior unless the people that respond can both remember and want to tell their stories. I sincerely hope that completing that survey helps my city improve transportation, but I cannot say with certainty that the data I provided will be useful.
Do you want to learn more about how Big Data provides a full picture of community’s travel patterns? Request your free, personal StreetLight InSight demo from one of our Big Data experts here.