This blog is for people who really enjoy getting into the weeds about data methodologies!  

With StreetLight’s end-of-year product updates (if you’re an existing customer you can see release notes here) we included even more new data months, and we updated some of our methodologies for processing data. Our methodologies are documented in detail on our white papers page but we thought it would be helpful to summarize the updates here and discuss how the methodology changes might affect results of certain analyses, using some of our own previous blogs and ebooks as examples.  

While these changes make our results better, we know methodology updates can be tricky for customers, so we want to be as transparent and helpful as possible, so customers can better navigate and understand any differences they may see. If you have questions about any of your own analyses, please contact our support team by visiting help.streetlightdata.com and selecting “Contact Support”   or clicking the “Contact Support” link under the Help menu in your StreetLight InSight® account.  

TL;DR: Overall the changes are moderate and the updates mean our results more accurately reflect the real world. The most significant impacts are: 

• Improved spot speeds and sample sizes for road segments analyzed with Network Performance in lieu of Segment Analysis. 

• Improved All Vehicles Volume estimates for data periods in 2019 nationally, as well as volume estimates for late 2023 for select states.  

• Improved differentiation between weekday and weekend vehicle volume metrics.

For Analyzing Speeds and Volumes on a Road Segment – Moving to our New Network Performance Analysis Type

During 2024, many users will have seen a new analysis type called “Network Performance” in StreetLight InSight®. This offers many of the same outputs as “Segment Analysis” but with improved methodology and data inputs, ultimately yielding better results. We are now recommending that clients analyzing vehicular movement on road segments transition to Network Performance, in particular for use cases involving measuring changes across time. 

What’s Different: Network Performance relies on Aggregated GPS data (AGPS) as its underlying data source. Segment Analysis relies on a combination of Connected Vehicle Data (CVD) and Location-Based Services (LBS) data sources, both of which have smaller sample sizes than AGPS.  

AGPS data has a few benefits, including a very high sample size (18-40% of vehicles on the road) and availability in both the US and Canada. Most notably, AGPS data has been continuously available since 2019, allowing for a better comparison across time since “data source” is no longer a variable that could account for differences measured.  In addition, Segment Analysis metrics are not available beyond May 2023.  

NOTE – As of January 2025, Network Performance can only be run on OpenStreetMap (OSM) segments. While it cannot currently be run on a customer’s LRS if they’ve uploaded that to the system, we are working on adding this capacity as soon as possible. Additionally, you can always work with our services team for a custom analysis that matches the metrics to your LRS.

Exploring Impact: Taylor Swift’s Eras Tour analysis

To explore the impact of shifting to Network Performance, we reran the results of our all-time-most-popular blog, which originally analyzed the Taylor Swift Eras Tour traffic jams using Segment Analysis (you can read the updated Eras Tour analysis here). 

The analysis of congestion on typical days across each of the cities shows almost no difference between the two methodologies (Segment Analysis and Network Performance). The only notable difference where Network Performance shows less delay on a typical day is in New York City. We think this reflects Network Performance’s better differentiation of cars from subways and buses, and thus is an improvement. 

On Eras Tour days, for most cities, Network Performance picked up a little more of an impact from the concerts as shown in Figure 1. Again, we consider this a positive reflection of Network Performance, as the data source is showing improved differentiation between typical activity and disruptive activity. This is one of the key benefits of big data — to analyze and react when events do not follow typical patterns.

Notably, these changes aren’t big enough to impact the overall story: Looking at excess VHD, the concerts in Vegas followed by Dallas, then Phoenix and then Tampa had the biggest impact on traffic compared to a typical day. The concert in New York City (with the most transit alternatives to driving) had the smallest.  Figure 1 shows the changes. 

Looking at excess VHD % change (i.e., the percentage difference between typical VHD and VHD on the day of the concert), the Boston concert shows the biggest percent change for both methodologies, followed by Dallas-Fort Worth and Phoenix. New York City still shows the smallest change. The table below shows how these rankings vary between Segment Analysis and Network Performance, with venue positions shifting by 1 rank at most.

Network Performance Volume Model Updates

In our end-of-year release, we also updated our U.S. Network Performance Volume model for all road segments in the U.S. for all months starting in 2019.
 
What’s Different: The volume estimates are derived from a machine learning model trained on over 14,000 unique permanent vehicle counts across all states in the contiguous U.S. The updated model uses more training locations than the first version of the model as more states published 2023 data after our initial release. We also used more historical data from 2019 and 2020 to refine the algorithms for those years. In general, these improvements yield:

1. Reduced bias and improved error in all years, especially on low volume roads
2. Improvements to the volume model for 2019
3. Improved weekday vs. weekend comparisons for all years

Figures 3 and 4 compare MAPE (Mean Absolute Percent Error) for various bins of roads for each data year. Deeper dive white papers are available here.

The new release also includes Network Performance volume estimates for Canada.

Exploring Impact: VMT Report

Last fall, we published a report ranking VMT changes from Spring 2019 – Spring 2023  for metro areas in the U.S. VMT relies on our volume model, and when the report was published, we were still using our V1 model. In hindsight, for a metric as critically important as VMT, we should not have developed a report with V1 when we knew V2 was coming soon! It created unnecessary confusion for our customers. This was an error we regret and will not repeat. We may publish a more comprehensive update of that report with v2 metrics in the future.

When we reran the results with our improved volume model, we saw some changes:

• A number of metro areas showed increased VMT totals for 2019, while most had similar results for 2024. This means that the percentage change in some metros between 2019 and 2024 was overstated in our initial report (Overall, Spring 2019 was in fact closer to Spring 2024, than initially reported by approximately 4-7 percentage points depending on region).
• The increase in 2019 was most often attributable to improvements in low/medium volume road accuracy.

This granularity ensures agencies of all sizes, as well as firms and businesses, can get actionable insights to prioritize projects, evaluate impact, and anticipate future needs. It’s also particularly important for transportation modeling, which requires granular, empirical data to help predict how conditions will change over time, or in response to specific infrastructural and policy changes.

Let’s use a few metros in Connecticut to illustrate the change.

VMT Musings: How do we know what is “right” or “better”?

For our volume metrics, we can publish very precise estimates of overall accuracy based on thousands of “ground truth” permanent counters, as shown in Figure 5.

But VMT over a large area is trickier — there’s no such thing as ground truth. Instead, there are various methodologies, and thoughtful comparisons can be made based on known strengths and weaknesses of each one.
 
FHWA publishes two different reports on statewide VMT (and individual states have their own methodologies): the Traffic Volume Trends (TVT) and within the Highways Statistics Series (HSS). These are published at the state level, not MSA.

Method Summary:

• The TVT is updated faster than HSS and is based on Continuous Count Stations (CCS), extrapolating changes seen on them to the rest of the roads.   
• For the Highway Statistics Series (HSS), FHWA “estimates national trends by using State reported Highway Performance and Monitoring System (HPMS) data, fuel consumption data (MF-21), vehicle registration data (MV-1), other data such as the R. L. Polk vehicle data, and a host of modeling techniques”  and since HSS hasn’t come out yet for 2023, we can’t compare the most recent data.  
• Like the TVT, StreetLight uses CCS counters from the state in question as well as from similar roads (similar by volume, rural/urban context, weather patterns, and more) in nearby states to create a machine learning model to scale up a ~25% sample to a full count. We estimate each individual road segment’s volume independently using this method, multiply that volume by road segment length, then sum all road segment VMT values up in a given area to estimate that area’s total VMT.

Sticking with Connecticut to illustrate differences:

Looking at these two FHWA methods, we feel our V2 model performs well based on common sense and COVID experience.
 
Methodologies like TVT often extrapolate growth in VMT in a region from measured growth on CCS counters (which are most often on busy roads). If the balance of growth between highways and more local roads has changed since the pandemic — and we believe it has — then our industry needs to update the methodology used to estimate region-wide VMT. We believe the big data–driven approach offers just such an opportunity and we will explore this in future publications and posts.
 
And, as always, the more up-to-date, well-maintained permanent counters that are available (especially on lower volume roads) the better everyone’s estimates will be!