Computing Platform to Support Travel Forecasting?

7 posts / 0 new
Last post
jabunch
Computing Platform to Support Travel Forecasting?

Where I work is thinking about updating the computer platform that 

we use for all of our travel forecasting projects.  Currently, we have a large (24 core, 128 GB RAM, 2 1TB SSD, TB internal hard drive) PC and 2 smaller (14 core, 32 GG Ram, 1TB SSD) PCs running Windows 10 64x Pro.  We remote in to these PCs to run jobs, and access files through shared folders on out network.  To date this set up has met all of our computing needs.

My questions regard how do others set up their environment, especially when running projects in different regions around the country (As we are starting to do). 

  • Do you have 1 or more very large servers at headquarters  (or regional hubs) for all of the travel forecasting?
  • Do you have your own server running VMWARE or something similar that hosts virtual PCs that can be spun up/down  for different regional models and/or projects? Does this create data management challenges?
  • Do you use the commercial cloud services (Amazon AWS, Microsoft Azure) for supporting your travel forecasting?
  • What issues do the above raise, particularly regarding licensing and running different travel forecasting platforms (Citilabs Cube, TRANSCAD, EMME, etc.)?
  • How do the above impact run times over having a stand alone dedicated computer (on our system running the Maryland MSTM takes ~24 hours, the Florida SERPM-7 takes ~17 hours).

So I understand, that for specific short term projects it may be  cost effective to set up a virtual PC using the Amazon AWS or Microsoft Azure.  But, does anyone use the Cloud services to maintain their overall modeling processes?  How does this effect the monthly marginal costs for the platform, or per project costs particuarly with the more advanced ABM and MesoScopic (DTA) forecasting models with higher processing and storage demands?

Thanks in advance for all your responses.

Jim Bunch

 

Paul Sittig

I think you also need to know at least two more factors when scoping a computer or computers for modeling:

1. What is the size of the model or models you’re running, and

2. How often are you running the models?

In New Mexico DOT, we’re gearing up to upgrade our model, but that’s a change from ~900 to ~3,000 TAZs, where I know some other states already have much larger and more detailed models. And because we’re working our way up to more intensive model utilization, we’re only using moderately robust laptops (Win10, 64bit, 16gb ram, SSD) as our modeling computers at the moment. They’re able to run the current model in about an hour, with PTV Visum.

I’m looking forward to hear what others say. Would you mind gathering the responses and circulating a summary document?

Thanks!
Paul Sittig

New Mexico Department of Transportation
Technical and Freight Planning Supervisor
(505) 490-2410

From: jabunch.work=gmail.com@mg.tmip.org On Behalf Of jabunch
Sent: Wednesday, September 25, 2019 9:48 AM
To: TMIP
Subject: [EXT] [TMIP] Computing Platform to Support Travel Forecasting?

Where I work is thinking about updating the computer platform that

we use for all of our travel forecasting projects. Currently, we have a large (24 core, 128 GB RAM, 2 1TB SSD, 4 TB internal hard drive) PC and 2 smaller (14 core, 32 GG Ram, 1TB SSD) PCs running Windows 10 64x Pro. We remote in to these PCs to run jobs, and access files through shared folders on out network. To date this set up has met all of our computing needs.

My questions regard how do others set up their environment, especially when running projects in different regions around the country (As we are starting to do).

* Do you have 1 or more very large servers at headquarters (or regional hubs) for all of the travel forecasting?
* Do you have your own server running VMWARE or something similar that hosts virtual PCs that can be spun up/down for different regional models and/or projects? Does this create data management challenges?
* Do you use the commercial cloud services (Amazon AWS, Microsoft Azure) for supporting your travel forecasting?
* What issues do the above raise, particularly regarding licensing and running different travel forecasting platforms (Citilabs Cube, TRANSCAD, EMME, etc.)?
* How do the above impact run times over having a stand alone dedicated computer (on our system running the Maryland MSTM takes ~24 hours, the Florida SERPM-7 takes ~17 hours).

So I understand, that for specific short term projects it may be cost effective to set up a virtual PC using the Amazon AWS or Microsoft Azure. But, does anyone use the Cloud services to maintain their overall modeling processes? How does this effect the monthly marginal costs for the platform, or per project costs particuarly with the more advanced ABM and MesoScopic (DTA) forecasting models with higher processing and storage demands?

Thanks in advance for all your responses.

Jim Bunch

--
Full post: https://tmip.org/content/computing-platform-support-travel-forecasting
Manage my subscriptions: https://tmip.org/mailinglist
Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/13294

jabunch

I will gladly gather responses offered.

As a consultant we run models of all sizes and in different regions, but
all of our main clients have "large models" that take 20 hours or more to
run on our modeling PC (The 24 core 128 GB ram Windows 10 PC). The ABM
models or horizon years with signficant congestion can take longer. This
increases significantly when we run on a standard 4 to 8 core desktop.
We are running several model runs a week for various projects.

On Wed, Sep 25, 2019 at 11:58 AM Sittig, Paul, NMDOT <
Paul.Sittig@state.nm.us> wrote:

> I think you also need to know at least two more factors when scoping a
> computer or computers for modeling:
>
>
>
> 1. What is the size of the model or models you’re running, and
>
> 2. How often are you running the models?
>
>
>
> In New Mexico DOT, we’re gearing up to upgrade our model, but that’s a
> change from ~900 to ~3,000 TAZs, where I know some other states already
> have much larger and more detailed models. And because we’re working our
> way up to more intensive model utilization, we’re only using moderately
> robust laptops (Win10, 64bit, 16gb ram, SSD) as our modeling computers at
> the moment. They’re able to run the current model in about an hour, with
> PTV Visum.
>
>
>
> I’m looking forward to hear what others say. Would you mind gathering the
> responses and circulating a summary document?
>
>
>
> Thanks!
>
> Paul Sittig
>
>
>
> New Mexico Department of Transportation
>
> Technical and Freight Planning Supervisor
>
> (505) 490-2410
>
>
>
> *From:* jabunch.work=gmail.com@mg.tmip.org gmail.com@mg.tmip.org> *On Behalf Of *jabunch
> *Sent:* Wednesday, September 25, 2019 9:48 AM
> *To:* TMIP
> *Subject:* [EXT] [TMIP] Computing Platform to Support Travel Forecasting?
>
>
>
> Where I work is thinking about updating the computer platform that
>
> we use for all of our travel forecasting projects. Currently, we have a
> large (24 core, 128 GB RAM, 2 1TB SSD, 4 TB internal hard drive) PC and 2
> smaller (14 core, 32 GG Ram, 1TB SSD) PCs running Windows 10 64x Pro. We
> remote in to these PCs to run jobs, and access files through shared folders
> on out network. To date this set up has met all of our computing needs.
>
> My questions regard how do others set up their environment, especially
> when running projects in different regions around the country (As we are
> starting to do).
>
> - Do you have 1 or more very large servers at headquarters (or
> regional hubs) for all of the travel forecasting?
> - Do you have your own server running VMWARE or something similar that
> hosts virtual PCs that can be spun up/down for different regional models
> and/or projects? Does this create data management challenges?
> - Do you use the commercial cloud services (Amazon AWS, Microsoft
> Azure) for supporting your travel forecasting?
> - What issues do the above raise, particularly regarding licensing and
> running different travel forecasting platforms (Citilabs Cube, TRANSCAD,
> EMME, etc.)?
> - How do the above impact run times over having a stand alone
> dedicated computer (on our system running the Maryland MSTM takes ~24
> hours, the Florida SERPM-7 takes ~17 hours).
>
> So I understand, that for specific short term projects it may be cost
> effective to set up a virtual PC using the Amazon AWS or Microsoft Azure.
> But, does anyone use the Cloud services to maintain their overall modeling
> processes? How does this effect the monthly marginal costs for the
> platform, or per project costs particuarly with the more advanced ABM and
> MesoScopic (DTA) forecasting models with higher processing and storage
> demands?
>
> Thanks in advance for all your responses.
>
> Jim Bunch
>
>
>
> --
> Full post:
> https://tmip.org/content/computing-platform-support-travel-forecasting
> Manage my subscriptions: https://tmip.org/mailinglist
> Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/13294
>

--
James (Jim) Allday Bunch, JABunch Transportation Consulting
411 Penwood Road, Silver Spring Maryland, 20901
240-271-3534 jabunch.work@gmail.com

Alex Bettinardi

This is a great question that I personally have put a lot of thought in, and I would love to share what I learned with the community.

We (Oregon DOT – TPAU) recently up graded a set of machines that is similar to the 24 core, 128GB Windows machine you describe. Without doing our own research, we went along with our IS departments recommendations for the next set of boxes. Their recommendation was more processors is better, and we got a 7000 series dell with dual processors 40 cores and a bunch of other impressive specs. These new machines that were a good 5 years newer than our previous boxes slowed our models down. What we later learned is that basically zero of the processes in our models (which are threaded), were optimized to the point to take advantage of 40 cores. Our threading sweet spot was between 8 and 16 cores. The extra distribution across two internal processors was greatly reducing our speed. On top of that these processors with all their cores had a slower processor speed than the previous boxes – and our latest lap tops (which now ran our models faster with 4 cores).

We returned those new dell 7000 series boxes and did a lot of extra work with machines available in the agency to fine tune the specs that we needed.

We don’t run models as big as the rest of the country, but our statewide model (which has a halo), does run a simple activity based model for ~7M people along with a land use model, population synthesizer… a bunch of processes that are likely similar to most transportation modelers.

Our lesson – get the fastest processor that your budget will allow. Stick to a single processor, not dual processors. Somewhere between 8-16 threads/cores is the sweet spot. We got 128GBs of RAM in the last purchase, which we should be able to utilize, but we found out later that we have some coding issues that are not allowing us to truly optimize on that. So I think RAM needs are going to be area / application specific. (A note that we ended on a high-end dell 3000 series).

I think solid state drives are in general better, although we have done a lot of testing with drives and have NOT found a significant speed improvement in overall run time with solid state drives. For that reason, we have a 512GB SSD for the C drive which is responsible for just software. We then add extra drives (as fast as we can, but not SSD) for TBs of storage. We generally keep a running “D” drive, and then a third drive for archived (none active) runs / information.

Our model run data is too big to put on our backed up servers in full. So we put model runs that are heavily used on server back-up, and other runs that are one-off questions or side investigations on duplicated machine drive back-up, with small summary information backed up to our servers.

We have not gotten into cloud computing yet (in any real way), and licensing is one of our limiting factors. Most of the machines in our office (~20) are now fast and large enough to run any of our models, but we are limited in the number of active runs we can have by the number of traffic assignment software licenses that we have access to.

Again, an amazing question that I hope other respond to as well – as I would love to learn more on this.

Alex Bettinardi, P.E.
503.986.4104
http://www.oregon.gov/ODOT/Planning/Pages/default.aspx

From: jabunch.work=gmail.com@mg.tmip.org On Behalf Of jabunch
Sent: Wednesday, September 25, 2019 9:23 AM
To: TMIP
Subject: Re: [TMIP] Computing Platform to Support Travel Forecasting?

I will gladly gather responses offered.

As a consultant we run models of all sizes and in different regions, but
all of our main clients have "large models" that take 20 hours or more to
run on our modeling PC (The 24 core 128 GB ram Windows 10 PC). The ABM
models or horizon years with signficant congestion can take longer. This
increases significantly when we run on a standard 4 to 8 core desktop.
We are running several model runs a week for various projects.

On Wed, Sep 25, 2019 at 11:58 AM Sittig, Paul, NMDOT <
Paul.Sittig@state.nm.us> wrote:

> I think you also need to know at least two more factors when scoping a
> computer or computers for modeling:
>
>
>
> 1. What is the size of the model or models you’re running, and
>
> 2. How often are you running the models?
>
>
>
> In New Mexico DOT, we’re gearing up to upgrade our model, but that’s a
> change from ~900 to ~3,000 TAZs, where I know some other states already
> have much larger and more detailed models. And because we’re working our
> way up to more intensive model utilization, we’re only using moderately
> robust laptops (Win10, 64bit, 16gb ram, SSD) as our modeling computers at
> the moment. They’re able to run the current model in about an hour, with
> PTV Visum.
>
>
>
> I’m looking forward to hear what others say. Would you mind gathering the
> responses and circulating a summary document?
>
>
>
> Thanks!
>
> Paul Sittig
>
>
>
> New Mexico Department of Transportation
>
> Technical and Freight Planning Supervisor
>
> (505) 490-2410
>
>
>
> *From:* jabunch.work=gmail.com@mg.tmip.org gmail.com@mg.tmip.org> *On Behalf Of *jabunch
> *Sent:* Wednesday, September 25, 2019 9:48 AM
> *To:* TMIP
> *Subject:* [EXT] [TMIP] Computing Platform to Support Travel Forecasting?
>
>
>
> Where I work is thinking about updating the computer platform that
>
> we use for all of our travel forecasting projects. Currently, we have a
> large (24 core, 128 GB RAM, 2 1TB SSD, 4 TB internal hard drive) PC and 2
> smaller (14 core, 32 GG Ram, 1TB SSD) PCs running Windows 10 64x Pro. We
> remote in to these PCs to run jobs, and access files through shared folders
> on out network. To date this set up has met all of our computing needs.
>
> My questions regard how do others set up their environment, especially
> when running projects in different regions around the country (As we are
> starting to do).
>
> - Do you have 1 or more very large servers at headquarters (or
> regional hubs) for all of the travel forecasting?
> - Do you have your own server running VMWARE or something similar that
> hosts virtual PCs that can be spun up/down for different regional models
> and/or projects? Does this create data management challenges?
> - Do you use the commercial cloud services (Amazon AWS, Microsoft
> Azure) for supporting your travel forecasting?
> - What issues do the above raise, particularly regarding licensing and
> running different travel forecasting platforms (Citilabs Cube, TRANSCAD,
> EMME, etc.)?
> - How do the above impact run times over having a stand alone
> dedicated computer (on our system running the Maryland MSTM takes ~24
> hours, the Florida SERPM-7 takes ~17 hours).
>
> So I understand, that for specific short term projects it may be cost
> effective to set up a virtual PC using the Amazon AWS or Microsoft Azure.
> But, does anyone use the Cloud services to maintain their overall modeling
> processes? How does this effect the monthly marginal costs for the
> platform, or per project costs particuarly with the more advanced ABM and
> MesoScopic (DTA) forecasting models with higher processing and storage
> demands?
>
> Thanks in advance for all your responses.
>
> Jim Bunch
>
>
>
> --
> Full post:
> https://tmip.org/content/computing-platform-support-travel-forecasting
> Manage my subscriptions: https://tmip.org/mailinglist
> Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/13294
>

--
James (Jim) Allday Bunch, JABunch Transportation Consulting
411 Penwood Road, Silver Spring Maryland, 20901
240-271-3534 jabunch.work@gmail.com
--
Full post: https://tmip.org/content/computing-platform-support-travel-forecasting
Manage my subscriptions: https://tmip.org/mailinglist
Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/13294

vipoqijyt

Related to this question of computing infrastructure for travel models, I
would like to ask who else is doing work to port travel models to GPU
computing? We are doing some experimental research as part of a DOE funded
Smart Mobility project to adapt travel models to run on GPUs (commodity
gaming graphics cards are a few hundred dollars, their bitcoin mining and
machine learning GPU cousins are several thousand dollars). Our
experiments so far are beginning to show some pretty interesting results
in terms of computational performance on GPUs compared to multi-core CPU
approaches. I don't want to say much more until we have had sufficient time
to test these results and continue improving on them. Our goal is to get
integrated land use, activity-based travel demand, and traffic
microsimulation running together efficiently on high detail networks and
parcel-level land use representation, allowing us to trace individuals
through the whole model set, and get the travel models running in under one
hour. Pretty ambitious targets, but we're converging on them more quickly
than I expected. Working to have a paper to share on this project later
this fall.

If others are working on related research, please drop me a note.

Paul Waddell
Professor, City and Regional Planning
University of California, Berkeley
President, UrbanSim Inc.

On Wed, Sep 25, 2019 at 10:36 AM Alex Bettinardi <
alexander.o.bettinardi@odot.state.or.us> wrote:

> This is a great question that I personally have put a lot of thought in,
> and I would love to share what I learned with the community.
>
> We (Oregon DOT – TPAU) recently up graded a set of machines that is
> similar to the 24 core, 128GB Windows machine you describe. Without doing
> our own research, we went along with our IS departments recommendations for
> the next set of boxes. Their recommendation was more processors is better,
> and we got a 7000 series dell with dual processors 40 cores and a bunch of
> other impressive specs. These new machines that were a good 5 years newer
> than our previous boxes slowed our models down. What we later learned is
> that basically zero of the processes in our models (which are threaded),
> were optimized to the point to take advantage of 40 cores. Our threading
> sweet spot was between 8 and 16 cores. The extra distribution across two
> internal processors was greatly reducing our speed. On top of that these
> processors with all their cores had a slower processor speed than the
> previous boxes – and our latest lap tops (which now ran our models faster
> with 4 cores).
>
> We returned those new dell 7000 series boxes and did a lot of extra work
> with machines available in the agency to fine tune the specs that we needed.
>
> We don’t run models as big as the rest of the country, but our statewide
> model (which has a halo), does run a simple activity based model for ~7M
> people along with a land use model, population synthesizer… a bunch of
> processes that are likely similar to most transportation modelers.
>
> Our lesson – get the fastest processor that your budget will allow. Stick
> to a single processor, not dual processors. Somewhere between 8-16
> threads/cores is the sweet spot. We got 128GBs of RAM in the last purchase,
> which we should be able to utilize, but we found out later that we have
> some coding issues that are not allowing us to truly optimize on that. So I
> think RAM needs are going to be area / application specific. (A note that
> we ended on a high-end dell 3000 series).
>
> I think solid state drives are in general better, although we have done a
> lot of testing with drives and have NOT found a significant speed
> improvement in overall run time with solid state drives. For that reason,
> we have a 512GB SSD for the C drive which is responsible for just software.
> We then add extra drives (as fast as we can, but not SSD) for TBs of
> storage. We generally keep a running “D” drive, and then a third drive for
> archived (none active) runs / information.
>
> Our model run data is too big to put on our backed up servers in full. So
> we put model runs that are heavily used on server back-up, and other runs
> that are one-off questions or side investigations on duplicated machine
> drive back-up, with small summary information backed up to our servers.
>
> We have not gotten into cloud computing yet (in any real way), and
> licensing is one of our limiting factors. Most of the machines in our
> office (~20) are now fast and large enough to run any of our models, but we
> are limited in the number of active runs we can have by the number of
> traffic assignment software licenses that we have access to.
>
> Again, an amazing question that I hope other respond to as well – as I
> would love to learn more on this.
>
> Alex Bettinardi, P.E.
> 503.986.4104
> http://www.oregon.gov/ODOT/Planning/Pages/default.aspx
>
> From: jabunch.work=gmail.com@mg.tmip.org On Behalf Of jabunch
> Sent: Wednesday, September 25, 2019 9:23 AM
> To: TMIP
> Subject: Re: [TMIP] Computing Platform to Support Travel Forecasting?
>
> I will gladly gather responses offered.
>
> As a consultant we run models of all sizes and in different regions, but
> all of our main clients have "large models" that take 20 hours or more to
> run on our modeling PC (The 24 core 128 GB ram Windows 10 PC). The ABM
> models or horizon years with signficant congestion can take longer. This
> increases significantly when we run on a standard 4 to 8 core desktop.
> We are running several model runs a week for various projects.
>
> On Wed, Sep 25, 2019 at 11:58 AM Sittig, Paul, NMDOT <
> Paul.Sittig@state.nm.us> wrote:
>
> > I think you also need to know at least two more factors when scoping a
> > computer or computers for modeling:
> >
> >
> >
> > 1. What is the size of the model or models you’re running, and
> >
> > 2. How often are you running the models?
> >
> >
> >
> > In New Mexico DOT, we’re gearing up to upgrade our model, but that’s a
> > change from ~900 to ~3,000 TAZs, where I know some other states already
> > have much larger and more detailed models. And because we’re working our
> > way up to more intensive model utilization, we’re only using moderately
> > robust laptops (Win10, 64bit, 16gb ram, SSD) as our modeling computers at
> > the moment. They’re able to run the current model in about an hour, with
> > PTV Visum.
> >
> >
> >
> > I’m looking forward to hear what others say. Would you mind gathering the
> > responses and circulating a summary document?
> >
> >
> >
> > Thanks!
> >
> > Paul Sittig
> >
> >
> >
> > New Mexico Department of Transportation
> >
> > Technical and Freight Planning Supervisor
> >
> > (505) 490-2410
> >
> >
> >
> > *From:* jabunch.work=gmail.com@mg.tmip.org gmail.com@mg.tmip.org> *On
> Behalf Of *jabunch
> > *Sent:* Wednesday, September 25, 2019 9:48 AM
> > *To:* TMIP
> > *Subject:* [EXT] [TMIP] Computing Platform to Support Travel Forecasting?
> >
> >
> >
> > Where I work is thinking about updating the computer platform that
> >
> > we use for all of our travel forecasting projects. Currently, we have a
> > large (24 core, 128 GB RAM, 2 1TB SSD, 4 TB internal hard drive) PC and 2
> > smaller (14 core, 32 GG Ram, 1TB SSD) PCs running Windows 10 64x Pro. We
> > remote in to these PCs to run jobs, and access files through shared
> folders
> > on out network. To date this set up has met all of our computing needs.
> >
> > My questions regard how do others set up their environment, especially
> > when running projects in different regions around the country (As we are
> > starting to do).
> >
> > - Do you have 1 or more very large servers at headquarters (or
> > regional hubs) for all of the travel forecasting?
> > - Do you have your own server running VMWARE or something similar that
> > hosts virtual PCs that can be spun up/down for different regional models
> > and/or projects? Does this create data management challenges?
> > - Do you use the commercial cloud services (Amazon AWS, Microsoft
> > Azure) for supporting your travel forecasting?
> > - What issues do the above raise, particularly regarding licensing and
> > running different travel forecasting platforms (Citilabs Cube, TRANSCAD,
> > EMME, etc.)?
> > - How do the above impact run times over having a stand alone
> > dedicated computer (on our system running the Maryland MSTM takes ~24
> > hours, the Florida SERPM-7 takes ~17 hours).
> >
> > So I understand, that for specific short term projects it may be cost
> > effective to set up a virtual PC using the Amazon AWS or Microsoft Azure.
> > But, does anyone use the Cloud services to maintain their overall
> modeling
> > processes? How does this effect the monthly marginal costs for the
> > platform, or per project costs particuarly with the more advanced ABM and
> > MesoScopic (DTA) forecasting models with higher processing and storage
> > demands?
> >
> > Thanks in advance for all your responses.
> >
> > Jim Bunch
> >
> >
> >
> > --
> > Full post:
> > https://tmip.org/content/computing-platform-support-travel-forecasting
> > Manage my subscriptions: https://tmip.org/mailinglist
> > Stop emails for this post:
> https://tmip.org/mailinglist/unsubscribe/13294
> >
>
> --
> James (Jim) Allday Bunch, JABunch Transportation Consulting
> 411 Penwood Road, Silver Spring Maryland, 20901
> 240-271-3534 jabunch.work@gmail.com
> --
> Full post:
> https://tmip.org/content/computing-platform-support-travel-forecasting
> Manage my subscriptions: https://tmip.org/mailinglist
> Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/13294
> --
> Full post:
> https://tmip.org/content/computing-platform-support-travel-forecasting
> Manage my subscriptions: https://tmip.org/mailinglist
> Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/13294
>

Pedro Camargo

Jim,
I know I am late to the party, but this is something I looked
quite closely a year ago, so I think I can add a couple of interesting
points.

*1. Assignment Vs. demand model* - Most models have two very distinct
components when it comes to the provenance of the software doing the code.
While assignment is often highly optimized (more on that later) and written
by professional software developers, demand models (even the newest stuff)
is written by modelers (and some software developers) that are not
necessarily worried about computing efficiency or scalability. Optimizing
your machine for traffic assignment will likely have sharply
diminishing returns due to the less-than-efficient scalability of demand
models when it comes to CPUs.

*2. Not all traffic assignments are the same* - Origin-based assignment is
often (e.g. TransCad & Emme) single-threaded, and even FW/BFW alternatives
have single-threaded components that highly impact running times. The most
relevant case we saw was the "BFW" assignment in VISUM, which has a very
time-consuming network contraction step that is run at each iteration,
which makes scaling past 12 or 16 threaded not really interesting. In ALL
cases, however, we saw diminishing returns when raising the number of cores
for assignment, and CPU profiling showed really poor use of resources when
we were to 14/28 cores/threads and beyond.

*3. NVMe drives*: We saw significant improvements when using NVMe drives
(highly quality, 4-lanes), especially in cases with very large networks and
many traffic classes

*4. New architectures*: We were not particularly successful with the first
generation of Ryzen chips, but I'd like to hear about people using CPUs
with 2 NUMA nodes (Threadripper)

*5. Resource contention:* We found that running two models at the SAME TIME
fighting for computer resources yielded a combined run time 25% shorter
than sequential runs, but I am not sure if that is possible with the likes
of VMWARE

In the end, I have the impression that our Industry was caught particularly
off-guard when it comes to a high number of CPU cores (in fairness, AMD
caught everybody by surprise), and the impression I got from more than one
software house is that they are not too invested in migrating to a
completely new world of potentially 100+ cores.

Cheers,
Pedro

On Wed, Sep 25, 2019 at 4:47 PM jabunch wrote:

> Where I work is thinking about updating the computer platform that
>
> we use for all of our travel forecasting projects. Currently, we have a
> large (24 core, 128 GB RAM, 2 1TB SSD, 4 TB internal hard drive) PC and 2
> smaller (14 core, 32 GG Ram, 1TB SSD) PCs running Windows 10 64x Pro. We
> remote in to these PCs to run jobs, and access files through shared folders
> on out network. To date this set up has met all of our computing needs.
>
> My questions regard how do others set up their environment, especially
> when running projects in different regions around the country (As we are
> starting to do).
>
> - Do you have 1 or more very large servers at headquarters (or
> regional hubs) for all of the travel forecasting?
> - Do you have your own server running VMWARE or something similar that
> hosts virtual PCs that can be spun up/down for different regional models
> and/or projects? Does this create data management challenges?
> - Do you use the commercial cloud services (Amazon AWS, Microsoft
> Azure) for supporting your travel forecasting?
> - What issues do the above raise, particularly regarding licensing and
> running different travel forecasting platforms (Citilabs Cube, TRANSCAD,
> EMME, etc.)?
> - How do the above impact run times over having a stand alone
> dedicated computer (on our system running the Maryland MSTM takes ~24
> hours, the Florida SERPM-7 takes ~17 hours).
>
> So I understand, that for specific short term projects it may be cost
> effective to set up a virtual PC using the Amazon AWS or Microsoft Azure.
> But, does anyone use the Cloud services to maintain their overall modeling
> processes? How does this effect the monthly marginal costs for the
> platform, or per project costs particuarly with the more advanced ABM and
> MesoScopic (DTA) forecasting models with higher processing and storage
> demands?
>
> Thanks in advance for all your responses.
>
> Jim Bunch
>
>
> --
> Full post:
> https://tmip.org/content/computing-platform-support-travel-forecasting
> Manage my subscriptions: https://tmip.org/mailinglist
> Stop emails for this post: https://tmip.org/mailinglist/unsubscribe/13294
>

barrycolleary

Compute

We currently are working with a core fleet of 10 compute rack mounted servers (Win 2012 R2)  for running our models (see J.Nott notes). These consist of two grades of system (due to age not requirements), and are accessed via RDP.

·         2 Socket / 36 core / 72  logical processor 64 GB Ram / 1TB local storage.

·         2 Socket / 44  core / 88 logical processor 128 GB Ram / 1TB local storage.

In addition to these we have two UAT servers, (Win 2012 R2).  

·         2 Socket / 32 core / 64 logical processor 64 GB Ram / .5 TB local storage.

Burst capacity

·         Additional burst capacity for compute is accessible via AWS EC2 systems running a custom AMI (image).

Storage

Network storage is provided by a .500TB  SAN. Data is copied to the local machine, for processing and then saved back to the SAN.

SAN data is now backed up with AWS S3.

Network.

Direct connect and backup VPN connect corporate network to AWS.  

Additional Remote workspaces access has been ran  as a POC and will be live in 2020.

 

As per the specific questions

 

Do you have 1 or more very large servers at headquarters  (or regional hubs) for all of the travel forecasting?- As above we essentially have two centres for when work load requires this. With our existing data centre being our current backbone.

 

Do you have your own server running VMWARE or something similar that hosts virtual PCs that can be spun up/down  for different regional models and/or projects? Does this create data management challenges? Current issues are essentially with manual data transfer for data. As the process is refined, more scripting is being used, with the long term strategy to get to processes such as batch processing.

 

Do you use the commercial cloud services (Amazon AWS, Microsoft Azure) for supporting your travel forecasting. Yes as burst capacity currently.

 

What issues do the above raise, particularly regarding licensing and running different travel forecasting platforms (Citilabs Cube, TRANSCAD, EMME, etc.)? Current licensing model covers the number of connections required. Once networking was in place all systems ran as expected.

 

How do the above impact run times over having a standalone dedicated computer (on our system running the Maryland MSTM takes ~24 hours, the Florida SERPM-7 takes ~17 hours). Current run time for largest model is 3.5-4 Days. When Amazon introduced the C5 range of servers we saw an reduction of 25% approx. We are planning to test clustering the cube application, and possible use of GPU as opposed to CPU .

Kind regards 

 

Barry Colleary