why won't most transportation agencies share demand models & data readily?

Hi Kara

As you know, we have been reviewing traffic consultants’ technical work for over 15 years. There is INCREDIBLE push-back when we ask to see their models. Please be very clear, this has nothing to do with proprietary information or too much data etc. etc. (we’ve heard all of the excuses). There is only one reason and it is simple. Their work is generally poor and it is easy to find weaknesses and deficiencies that can be used against them.

We know because, even when we don’t get their models, putting a simple set of questions to them generally starts to reveal the weaknesses and deficiencies (many of them really fundamental and basic - often laughable).

Another reason for wanting (needing) to hide their technical work is that, having promised the world to their clients (“look at our great, complex model that can do everything - and can even tell the future”) they find that their model outputs don’t make sense. This regularly happens on managed lane projects - but elsewhere too. Then [having boxed themselves into a corner] they have to bend the model out of shape to get it to start producing sensible numbers. On MLs, that means eg. using simply crazy values of time etc. On regular toll roads it is fiddling with ‘helpful’ model levers like penalties, bonuses, scaling factors etc.

If one of OUR models isn’t making sense, we simply throw it in the trash and start again. Our experience suggests that this behaviour is not the sector norm.

At the end of the day their results look plausible so few people bother digging deeper. We should have kept a list of all the nonsense we have found (and find on an almost weekly basis).

That’s the commercial due diligence work we do.

Turning to expert witness work, in arbitrations consultants are required to share their models. In litigation, we simply subpoena them. That’s when the fun really begins.

That said, everything is easiest with spreadsheet models. With big 4-step models, you often get into practical problems. Do you have the capability to run the software? Do you own your own version of the software. Is the version you have the same as that used by the consultants? Etc. For this reason, we generally don’t even ask. Our questions give us all we need to know.

Like your point about publicly-funded studies. I entirely agree that this is outrageous. These models should be made available and the onus should be on the (publicly-paid) consultant to make sure that the models are accessible - not just available - to others (especially those with limited technical knowledge).

One question that always puzzles those of us who work internationally, is why T&R studies in the US cost so much? Agencies pay millions yet - having seen the end product - it’s not that different from studies that elsewhere cost $100-$200k. And it’s often boilerplate stuff with considerable cut-and-paste from other reports (we know because we have a library of over 300 T&R studies for comparison purposes).

Hope this finds you well.

Kind regards,

Rob

Robert Bain
RBconsult Ltd
www.robbain.com

NOTICE OF CONFIDENTIALITY. This communication is intended only for the use of the addressee and may contain confidential and privileged information. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately or, alternatively, immediately destroy this communication.

From: KKOCKELM=mail.utexas.edu@mg.tmip.org On Behalf Of kkockelm@mail.utexas.edu
Sent: Wednesday, August 23, 2023 12:05 PM
To: TMIP
Subject: [TMIP] why won't most transportation agencies share demand models & data readily?

Great point, Krishnan! I would think property values should be softening a LOT in the southern US (& elsewhere, around the globe) during this death march (from our climate emergency-generated heat wave). And I was thinking that perhaps those in the southern US walk a lot less/drive a lot more & harm more pedestrians per VMT since they get into a bad habit of driving (& being sedentary – sort of like one long siesta) during the hottest months (much like so many people got used to working at home during covid, and had a difficult time getting back into an office-work groove).

Hijacking your thread, since I can’t post directly to this listserv, I wonder if you & others can tell us why you feel many/most public transportation agencies don’t readily share their travel demand models (& perhaps many or most data sets they’ve generated)? I realize that many DOTs (state & city & US) pay consultants to create the code (& files), but usually those files are not protected (not property of the consultants, in my experience). My experience is that progressive states are much better about sharing, but I have still hit walls in some unexpected places (perhaps because they were modeling very deeply – at the level of small grid cells [for land use, for example, across SF Bay Area], so the data sets give away too much intel?).

I believe that all data collected & models created or purchased with public monies should be public (unless protected by some consultant-required proprietary protection or posing security risks, like with water treatment plants or power lines, I guess). Are they worried we’ll find flaws and embarrass the agencies? Are they worried they will get too many questions & have no one to respond to those? Are they worried we’ll run too many new scenarios and get asked about those by policymakers & the public at large? Or that our results will differ from their own and they definitely don’t have time to try & figure out why?

I wonder what we can do to make agencies much more open to sharing such things, so that we can improve & expand upon them. Do we have to promise to not publicly (or privately?) mention any flaws we find, for example? And add protective language to all results we generate that absolves them of association?

For all of you at public agencies & who have served public agencies (or have other experiences that illuminate this issue’s basis): thank you for any insights you can share!

Kara

Note: I think we all have to promise not to share the files forward. (Any such requests have to go through a contract with the agency directly.)

From: krisviswanathan=gmail.com@mg.tmip.org On Behalf Of krishnan
Sent: Tuesday, August 22, 2023 3:12 PM
To: TMIP
Subject: Re: [TMIP] Impact of Extreme Heat and other Weather on Travel (trips, modes, etc.)

Your question raises another question. What happens to our forecast models?
Do regions that are going to face increased heat and humidity, excessive
flooding or droughts, wildfires reduced water supply continue to grow at
the same rate (looking at the South and the Southwest in the US)? Do
demographers account for potential loss of population in their projections
and do politicians/jurisdictions accept them? How to incorporate climate
conditions in forecast scenarios? Should we start looking at insurance
data/models (i am pretty sure those are proprietary) to include in our
forecast models?

On Wed, Aug 16, 2023 at 7:00 AM jabunch wrote:

> As we are now going through the hottest summer on record and many urban
> areas are experiencing multiple days of greater than 100 or even 110 degree
> heat, I've started to wonder how these conditions are impacting travel.
> Are people making fewer overall trips? Are they changing when they
> travel? Are they changing their mode of travel, particularly their walking
> and biking habits? How will or should these extreme conditions impact our
> future planning in urban areas?
>
> Is anyone studying this in places like Dallas, Houston, or other southern
> and western cities? It would be interesting to see.
>
> Thanks
>
> Jim Bunch
> --
> Full post:
> https://tmip.org/content/impact-extreme-heat-and-other-weather-travel-tr...
> Manage my subscriptions: https://tmip.org/mailinglist
> Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/14028
>

--
Krishnan Viswanathan
5628 Burnside Circle
Tallahassee FL 32312
--
Full post: https://tmip.org/content/impact-extreme-heat-and-other-weather-travel-tr...
Manage my subscriptions: https://tmip.org/mailinglist
Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/14028
--
Full post: https://tmip.org/content/why-wont-most-transportation-agencies-share-dem...
Manage my subscriptions: https://tmip.org/mailinglist
Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/14035

1 - The main reason Ohio does not share everything is that the variable files are based on confidential QCEW data.  Zonal data do not meet confidentiality requirements, and therefore cannot be shared.  All of our consultants (as well as us) sign confidentiality agreements for access to the data.

2 – Some of the more advanced models are not solely in the public domain.  i.e. some of the consulting firms spent private money to develop code that is then used in the model.  You would need to license the software from the consultant (similar to that of buying commercial software).

Other than that, I suppose you’re welcome to make a public records request to get the model.  But, you’ll probably not get the datasets to run it.  (Although the network should be available.)

Ohio has created a 2010 public variable file for Lima.  This is based on LEHD and not QCEW.  You can download the SHRP2 C10 model for Lima using Dynus-T at: Dynamic Traffic Assignment | DynusT  Choose the “Lima Download” option at the top.  I believe that you can contact Caliper if you’d like the TransModeler version of it.  I believe that you will also need a license of Cube to run either of these. (And obviously either a license of Dynus-T or TransModeler.)  If you want the 4-step model, I suppose you can contact me and I’ll get to it when I have time.  It also requires a license of Cube.

Honestly, it’s in our best interest to have our models used as classroom examples as it gets students familiar with actual models.  That’s the main reason we spent so much time and money creating public versions of our models, but I’m not sure if they are at all being used.

Hi, Kara.  No, this is at TAZ level.  Some of our TAZs are single employers.  (E.g. Proctor & Gamble or Honda plant)  Additionally, our variables are at the 2-digit NAICS code.  To not be confidential, there must be at least 3 employers in a TAZ in every 2-digit NAICS code category with more than 0 employment with no one employer having more than 80% of the total employment in that 2-digit NAICS code category.  Many of our TAZs do not meet those disclosure requirements.  Add in that our advanced models are now using MAZs and there is no possible way that the MAZ file will be disclosable. 

My Task Order to create a public dataset for Lima was ~$50k, and that was something like 8 years ago.  So, this is not a cheap endeavor, and it’s hard to justify the use of public funds to pay for toy models. 

QCEW is point data with # of employees and wage data for each individual employer, based on their unemployment insurance record.  We aggregate those up to TAZs/MAZs.  And yes, each person and firm working for us must sign a confidentiality agreement with ODJFS in order to access our variable files (same with the MPO staff).  Because we have a legal contract with them (and the state has legal recourse), ODJFS allows them to be signatories to our agreement.  That obviously does not hold true for researchers or universities.  (But the legal recourse for us releasing the data is severe.  I have no intention of losing my pension and going to jail and paying fines.  We joke that it is akin to Rapunzel – where you sign away your first born.)

- Rebekah

We’ll happily share our models with anyone who asks.

---
Jonathan Ehrlich
Pronouns: he/him/his
Senior Manager: Transportation Research
Metropolitan Council Transportation Services
P. 651-602-1408

From: KKOCKELM=mail.utexas.edu@mg.tmip.org On Behalf Of kkockelm@mail.utexas.edu
Sent: Wednesday, August 23, 2023 6:05 AM
To: TMIP
Subject: [TMIP] why won't most transportation agencies share demand models & data readily?

Great point, Krishnan! I would think property values should be softening a LOT in the southern US (& elsewhere, around the globe) during this death march (from our climate emergency-generated heat wave). And I was thinking that perhaps those in the southern US walk a lot less/drive a lot more & harm more pedestrians per VMT since they get into a bad habit of driving (& being sedentary – sort of like one long siesta) during the hottest months (much like so many people got used to working at home during covid, and had a difficult time getting back into an office-work groove).

Hijacking your thread, since I can’t post directly to this listserv, I wonder if you & others can tell us why you feel many/most public transportation agencies don’t readily share their travel demand models (& perhaps many or most data sets they’ve generated)? I realize that many DOTs (state & city & US) pay consultants to create the code (& files), but usually those files are not protected (not property of the consultants, in my experience). My experience is that progressive states are much better about sharing, but I have still hit walls in some unexpected places (perhaps because they were modeling very deeply – at the level of small grid cells [for land use, for example, across SF Bay Area], so the data sets give away too much intel?).

I believe that all data collected & models created or purchased with public monies should be public (unless protected by some consultant-required proprietary protection or posing security risks, like with water treatment plants or power lines, I guess). Are they worried we’ll find flaws and embarrass the agencies? Are they worried they will get too many questions & have no one to respond to those? Are they worried we’ll run too many new scenarios and get asked about those by policymakers & the public at large? Or that our results will differ from their own and they definitely don’t have time to try & figure out why?

I wonder what we can do to make agencies much more open to sharing such things, so that we can improve & expand upon them. Do we have to promise to not publicly (or privately?) mention any flaws we find, for example? And add protective language to all results we generate that absolves them of association?

For all of you at public agencies & who have served public agencies (or have other experiences that illuminate this issue’s basis): thank you for any insights you can share!

Kara

Note: I think we all have to promise not to share the files forward. (Any such requests have to go through a contract with the agency directly.)

From: krisviswanathan=gmail.com@mg.tmip.org On Behalf Of krishnan
Sent: Tuesday, August 22, 2023 3:12 PM
To: TMIP
Subject: Re: [TMIP] Impact of Extreme Heat and other Weather on Travel (trips, modes, etc.)

Your question raises another question. What happens to our forecast models?
Do regions that are going to face increased heat and humidity, excessive
flooding or droughts, wildfires reduced water supply continue to grow at
the same rate (looking at the South and the Southwest in the US)? Do
demographers account for potential loss of population in their projections
and do politicians/jurisdictions accept them? How to incorporate climate
conditions in forecast scenarios? Should we start looking at insurance
data/models (i am pretty sure those are proprietary) to include in our
forecast models?

On Wed, Aug 16, 2023 at 7:00 AM jabunch wrote:

> As we are now going through the hottest summer on record and many urban
> areas are experiencing multiple days of greater than 100 or even 110 degree
> heat, I've started to wonder how these conditions are impacting travel.
> Are people making fewer overall trips? Are they changing when they
> travel? Are they changing their mode of travel, particularly their walking
> and biking habits? How will or should these extreme conditions impact our
> future planning in urban areas?
>
> Is anyone studying this in places like Dallas, Houston, or other southern
> and western cities? It would be interesting to see.
>
> Thanks
>
> Jim Bunch
> --
> Full post:
> https://tmip.org/content/impact-extreme-heat-and-other-weather-travel-tr...
> Manage my subscriptions: https://tmip.org/mailinglist
> Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/14028
>

--
Krishnan Viswanathan
5628 Burnside Circle
Tallahassee FL 32312
--
Full post: https://tmip.org/content/impact-extreme-heat-and-other-weather-travel-tr...
Manage my subscriptions: https://tmip.org/mailinglist
Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/14028
--
Full post: https://tmip.org/content/why-wont-most-transportation-agencies-share-dem...
Manage my subscriptions: https://tmip.org/mailinglist
Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/14035