Why Autonomous Valet Parking, not robo-taxis, will lead the adoption in self-driving technology
Looking back on 2018, the press have reported it to be the year when the hype around self-driving “came crashing down” with the first driverless fatality in March 2018. The first driverless taxi service was rolled out but it didn’t quite have the impact that the industry was expecting.
Research on self-driving cars has been continuing for more than 30 years, starting with the pioneering work by Ernst Dickmanns on the PROMETHEUS project. A lot of work has taken place since then and is still ongoing, but the question remains: why has the problem of self-driving still not been solved in 30 years?
The biggest challenge faced by the developers of general purpose self-driving technology is the requirement to handle complex environments with unpredictable interactions. Waymo’s director of engineering recently summarised the challenge by saying that the final 10% of technology development is requiring 10x the effort required for the first 90%. Where the environment is simpler or constrained, then self-driving technology reduces to that of autonomous mobile robots like the kiva.
A more realistic approach to deploying self-driving is therefore needed. The two major places where self-driving car technology are likely to be deployed are on motorways – constrained environments with very strict rules, limitations on cyclists and pedestrians – and low speed restricted environments like retirement villages and car parks.
Autonomous Valet Parking (AVP)
Parking is one of the most important challenges for a traveller, with a parking pain point experienced on 12% of UK journeys (19% in London); the average driver spends 6.45 mins looking for a parking space during each journey. With nearly 1 in 5 journeys already experiencing problem finding a space, AVP represents a way of solving not only the current parking pain point, but also improving the overall parking experience for the other 81% of drivers. BCG’s 2015 report showed that 67% of drivers are interested in “capabilities such as automated searching for parking spots and autonomous valet parking”. Bosch’s 2017 study found that two thirds of consumers want an automated parking feature.
A study by Allensbach in 2016 asking the question “When would you want a driver assistance feature to take over for you?” overwhelmingly showed parking as the most desirable feature.
When German consumers were asked by Bitkom in 2016 when they would be willing to hand over control to the vehicle, the answer similarly was for parking.
The idea of Autonomous Valet Parking is to mimic the Valet function available in selected car parks. After driving to a suitable drop-off location at or near the entrance of the car park, and similar to handing the keys to a valet, the driver presses the “PARK” button on a specially designed app. The car then drives off under autonomous control and finds a suitable place to park. When the driver wants the vehicle back, they will press “SUMMON” and the vehicle will navigate to the pick-up zone.
The Society of Automotive Engineers (SAE) classifies this as a Level 4 feature, in that it provides total automation under specific circumstances.
Based on publicly available information, almost all premium OEMs (Daimler, Audi, JLR, VW, BMW) are working on AVP pilot projects . The reason this feature is so desirable is that it:
Improves the parking experience by allowing drivers to be dropped off at a convenient location (e.g. at the entrance of the car park closest to the desired location such as shops or food court), avoiding the inconvenience and stress of having to find a parking space.
Utilises parking spaces more efficiently by tighter/double parking of autonomous cars, and optimally distributing these vehicles within the available parking real estate.
Avoids unnecessary congestion and pollution through real-time dissemination of parking space availability to the connected autonomous cars.
In addition to the economic benefits, there are clear social and environmental benefits. Driving around looking for parking causes stress and frustration, costs, wasted time, missed appointments, accidents and congestion, noise pollution and CO2 emissions. IBM’s parking survey found that in addition to the typical traffic congestion caused by daily commutes and gridlock from construction and accidents, it is estimated that over 30 percent of traffic in a city is caused by drivers searching for a parking space. By reducing the need to circle looking for a space, AVP has the potential to significantly reduce unnecessary congestion and pollution.
With space at a premium in busy city centres, vehicles equipped with AVP technology could make use of the less desirable spaces that are further from the entrance, freeing up parking spaces closer to a desired destination for those without the technology.
In addition to the economic, social and environmental benefits to AVP, there are also some technical reasons why it is a good candidate feature for large scale public rollout.
1) low speeds mean much lower risk of damage to people, cars and infrastructure.
2) a constrained environment means that the complexity of interactions with other actors has the potential to be significantly reduced.
3) the cost of the required sensor suite and hardware platforms is lower because of the reduced risk and lower speeds.
This consortium’s key objective is to identify obstacles to the full deployment of AVP through the development of a technology demonstrator. It aims to achieve this goal by
Developing automotive-grade indoor parking maps, required for autonomous vehicles to localise and navigate within a multi-storey car park.
Developing the associated localisation algorithms – targeting a minimal sensor set of cameras, ultrasonic sensors and inertial measurement units – that make best use of these maps.
Demonstrating this self-parking technology in a variety of car parks.
Developing the safety case and preparing for in-car-park trials.
Engaging with stakeholders to evaluate perceptions around AVP technology.
Autonomous Valet Parking is a low cost, low risk and high reward feature that consumers want. It makes sense then to expect that this feature will be the first fully autonomous feature (at level 4 or 5) available to the general public. Through Parkopedia’s autonomous valet parking project, we are actively working to make that desire a reality.
You know how it can be a real challenge to understand the complexity in emerging multi-modal and automated transport systems?
By harnessing Systems Engineering best practices, our Systems Engineers at the Connected Places Catapult (CPC) use their expertise in systems requirements gathering, safety management, and verification and validation, ensuring that all aspects and interactions in the AVP system are clearly defined, integrated, evaluated and tested, thus improving our ability to deliver a safe, efficient and innovative system.
Systems Engineering is an established practice capable of delivering technically complex systems, and Systems Engineers swear by the V lifecycle model, which shows the logical relationship between the different System Engineering activities.
In the V model, time and system maturity proceed from left to right:
In each stage of the system life cycle, the Systems Engineering process iterates to ensure that a concept or design is feasible, and that the stakeholders remain supportive of the solution as it evolves.
The process is applied sequentially, one level at a time, adding additional detail and definition with each level of development.
1. Concept Development
The concept development is the first step of the life cycle of the AVP project. This stage explores concepts, identifies stakeholders’ needs and proposes viable solutions.
Requirements definitions are the key to success in the design and development of any complex system. Our Systems Engineers have carefully elicited requirements from all the project stakeholders to ensure the final product meets everyone’s needs, from technical feasibility to budget considerations and testability.
The total set of requirements encompasses the functional, performance, and non-functional requirements and the architectural constraints.
3. System Architecture
The architecture of a system is its fundamental structure. The purpose of drawing an architecture is to define a comprehensive solution based on principles, concepts, and properties logically related and consistent with each other. The architecture defines the system boundary and functions, from which more detailed system requirements can be derived.
step of the cycle is focused on developing:
At the start of the AVP project, both of these architectures were drawn. The system level architecture refers more to the overall system and its operating environment. The functionalities that enable the AVP technology are mapped on the functional architecture, based on the Autoware ROS and showing modules such as perception, sensor fusion, path planning, and others.
4. System design and development
Once the initial opportunities and constraints have been identified in the steps above, it is time to create and describe in detail the AVP system to satisfy the identified needs.
While the Parkopedia Engineers work on developing the AVP system in accordance with the ConOps, Requirements and System Architecture, the Systems Engineers at the CPC focus on the functional system analysis and producing the Safety Case for the AVP testing.
This lifecycle step expands on ISO 26262, which addresses the functional safety in road vehicles. The analysis of the system level safety is carried through system-level functional analysis, such as the Failure Mode and Effect Analysis (FMEA) and the Hazard Risk Assessment (HARA) , which results in providing the system safety goals (translated as the safety requirements for the system). These form the system safety argument for the Safety Plan for the AVP project. This step is crucial to ensure the risk is minimised if and when the system fails. Every malfunction leads to a warning and degradation strategy and the safety driver should be able to gain control of the vehicle.
Note: the system analysis also involves reviewing all of the previous deliverables as the design is being developed.
5. System Integration
The AVP system is made of different system elements provided by different parties: – The vehicle is provided by StreetDrone – The software, application and maps are provided by Parkopedia – The localisation system is developed by the University of Surrey
This step ensures that all of the disparate system elements integrate to create an effective whole and allows the different teams to work in parallel confident that all of the pieces fit and work together. The interface management activity relate to the steps taken to integrate the different architecture elements:
Reviewing and refining the previously mentioned deliverables (ConOps, System Architecture, Requirements, FMEA, HARA)
Assessing the criteria and methods for Requirements verification and validation (Analysis/Inspection/Test/Demonstration)
The Interface Control Document (ICD), which describes the integration strategy of the AVP system. This is a live document and is continuously updated until the end of the system development and testing.
6. Test & Evaluation
Prior to driving in the desired environment, the vehicle has to demonstrate that it is capable of safely navigating and responding to diverse situations and scenarios, such as emerging pedestrians, vehicles, obstacles, etc. By combining simulation and testing in a real environment (open field track and car parks), the AVP system can be validated.
This step focuses on ensuring that the developed system is operationally acceptable and that the responsibility for the effective, efficient, and safe operations of the system is transferred to the owner.
Can the system be tested, demonstrated, inspected, or analysed to show that it satisfies requirements?
Are the requirements stated precisely to facilitate specification of system test success criteria and requirements
Testing will always be performed with a highly trained safety driver, who will monitor the vehicle behavior at all times and ready to take over in the event of a failure. An engineer will monitor the path on a HMI, so that the safety driver is not distracted. The system must be able to detect and respond to a wide range of static obstacles in the testing environment.
It examines the evidence required to show that acceptable safety has been achieved for any given testing activity, and how that evidence is to be collected, such that when all the evidence is taken together, there is a convincing argument that the project as a whole is safe.
It summarises the evidence collected prior to commencing Autonomous Valet Parking testing in accordance with the Safety Plan. There will be a Safety Case for each test phase: testing in a controlled environment (open field track), testing in car parks and the demonstration.
This document sets out the safety procedures that all participants are required to follow during testing of the autonomous control system and ensures the requirements and procedures are read and understood by all involved in the trial, and adhered to at all times. The Risk Assessment for the trial is also included in this document. Similarly to the Safety Case, each testing phase will be prepared with a RAMS.
After testing, the requirements are reviewed and given a pass/fail mark depending on the acceptance criteria set. In both cases, a justification needs to be provided.
Once again, the previous deliverables are reviewed and refined as required.
7.Operation and maintenance
The AVP system has now been tested in different test environments and settings. The system is ready to be deployed, and as part of this activity, several demonstrations are organised as evidence of the operational effectiveness of the deployed AVP system.
The Systems Engineers carry the below activities, verifying and validating the concept one last time:
Safety Case close out
Verification and Validation close out
Review and refine as required previous deliverables
The self-driving car industry is a fast-moving and uncertain environment, which makes safety the number one priority. This is also why the AVP project consortium is encouraging the industry to work together, collaborate, share use-cases and lessons learnt. Self-driving cars have the potential to ensure safety of passengers, reduce road congestion, drive smarter and in a more sustainable way. By implementing a robust Systems Engineering process, the self-driving car industry can mitigate risks and drive society safely into the future.
IV2019 brings together researchers and practitioners from universities, industry and government agencies worldwide to share and discuss the latest advances in theory and technology related to intelligent vehicles.
The Autoware foundation is hosting a workshop on Sunday 9th June 2019, with the aim of discussing the current state of development of Autoware AI and Autoware Auto, and considering various technical directions that the Foundation is looking to pursue. Parkopedia’s contribution is around maps and specifically, the integration of indoor maps for Autonomous Valet Parking.
Parkopedia’s Angelo Mastroberardino will be presenting our work on maps, answering questions like “Why do we need these maps?”, “How do we represent geometry and road markings within maps?”, and naturally leading towards the question of how we use these maps for path planning within indoor car parks.
Later, Dr Brian Holt will be joining Tier 4, Apex.AI, Open Robotics, TRI-AD, and Intel on a panel to discuss Autoware and its impact on autonomous driving.
Parkopedia joined the Autoware Foundation as a premium founding member, because we believe in open source as a force multiplier to build amazing software. We are contributing maps, including for the AutonomouStuff car park in San Jose, USA, which you can download for use with your own self-driving car in simulation. Find out more
Connected Places Catapult (CPC) have conducted research with
UK car drivers and stakeholders to better understand public and industry readiness
for autonomous valet parking (AVP).
The key questions guiding the research were:
What are the key parking pain points that can be
resolved by AVP?
What are other likely benefits of AVP to users
and parking stakeholders?
What are the key barriers to AVP deployment and
uptake, from a social and behavioural point of view?
What will be the likely impact of AVP, on the
environment, the economy, and the parking industry?
The report produced details the findings, conclusions and
recommendations from a suite of research activities:
A literature review to explore existing
knowledge about the chosen research topics and questions;
Stakeholder interviews with parking
professionals and OEMs to explore their views of AVP;
Focus group interviews to explore the needs and
attitudes of drivers in-depth; and
A UK wide survey of 1025 car drivers to examine
differences between user groups, and to gauge how common certain attitudes or
The research found that car drivers would be more receptive
to the car taking control in a car park than on the roads and a technology
solution that can reduce the stress of parking and make parking easier would be
appealing. One in five drivers would like to use AVP now with a further 40%
open to the idea but wanting to know more about it. Likely early adopters would
be younger male drivers and those who have previously used driver assistance
technology or have previously used a valet parking service.
We successfully completed our first tests at Turweston Aerodrome last week.
The plan was to check and ensure the robustness of the drive-by-wire system, to train our safety drivers and to do basic path following.
We also took the opportunity to collect some data from the ultrasonic sensors that are on the StreetDrone.ONE which we will use for system safety.
For testing the drive-by-wire system, we carried out a number of test runs using teleoperation from the driver. We drove on the track forward, backward and changing steering at various speeds. The system performed satisfactorily. We also performed a full brake test to work out a safe driving speed and stopping distance in a case of emergency stop. Further details are presented in this blog post.
On the final day, we tested a basic path following to make sure everything worked together. We integrated the drive-by-wire (dbw) system with a path follower, PID motion controller and a basic gps and imu localisation in this open space environment.
We managed to achieve the objective of testing the dbw with the feedback control for the path following. However, precision was not there as we expected. The basic imu and gps sensor localisation would not give very accurate positioning and tends to drift away or jump around to within 5m accuracy. To resolve this issue, we are working on a better localisation using RTK GPS (like a simpleRTK2B) using RTK corrections over NTRIP.
Safety is the highest priority at the Autonomous Valet Parking project. As we seek to demonstrate that Parkopedia’s maps are suitable for localisation and navigation within covered car parks, our safety case must ensure the safety of the people, vehicles and infrastructure inside our test vehicle and without. There have been some highly publicised incidents involving other organisations’ autonomous vehicles over the past year or two, so what is the AVP project doing around safety?
Our Safety Case involves a combination of System Safety and Operational Safety, to achieve the required assurance levels around our activities. System Safety covers any and all aspects of the system (hardware, software or both) that contribute to the safety case and is the focus of this post. Operational Safety is all the operational decisions taken to ensure that we are conducting the tests safely and you can read more about that here.
Our test vehicle, a StreetDrone.ONE is a converted Twizy, and comes with ultrasound range sensors. There are eight Neobotix ultrasound sensors in total, three to the front, three to the rear, and one in each door looking sideways. The front sensors are slightly fanned out, and in the rear the outer two are corner mounted. The signals from each sensor are gathered together by a Neobotix board and this publishes the range data as an automotive industry canbus signal. These can be monitored and action taken when a significant measurement is made.
The sensors create a “virtual safety cage”. This virtual safety cage can be imagined as an invisible cuboid around the StreetDrone.ONE, slightly wider and longer than the car itself. If anything, be it a car, pedestrian or wall, intrudes inside this cage, the car should stop immediately, thus acting as a belt and braces to the perception and navigation parts of the AI driving the car.
The video below shows a demonstration of the drive-by-wire system of the StreetDrone at 14 mph. The brake is applied by the actuators at the very beginning of the 3 metres wide white strip. Based on a 14mph start speed, we calculated the braking distance to be 4.5 metres. This is obviously an approximation, and the actual braking distance and time depends on many factors including brake disk wear and tear.
Remembering our high-school equations of motion:
Where “v” is the initial velocity, “u” the final velocity, “a” is the acceleration and “s” the distance.
Now we can rearrange this to obtain the acceleration, remembering in our case the final velocity is zero:
These numbers match well with the similar experiments carried out by StreetDrone using IMU data, which have shown peak deceleration of 0.67g and an average deceleration of 0.46g.
The maximum range of the Neobotix ultrasound sensor is 1.5 metres, so we could do the above calculation in reverse and calculate the maximum safe speed. Allowing for a safety buffer of 0.5m:
The distance “s” is 1.0 metres, the acceleration “a” is 4.3m/s^2, and again the final velocity “u” is 0 m/s.
The present AVP plan calls for a maximum speed within car parks of 5 mph which is well within the safety margin calculated above. The next step now is to process the data we’ve captured from the ultrasonic sensors and to develop the software that will automatically apply the maximum brake whenever the virtual safety cage is breached.
Exciting times: this week marks the start of testing for our StreetDrone autonomous vehicle, as we build towards an Autonomous Valet Parking demonstrator.
This first testing phase will be in a controlled environment to minimise risk. For that reason, we’ve chosen Turweston Flight Centre, which has previously been used by our friends at StreetDrone who have done some of their testing there.
In accordance with our project Safety Plan and the Safety Case for this phase of the project, we’ve also been busy collecting safety evidence prior to starting the live tests. These documents will be made publicly available as part of our goal to be as transparent as possible, and for those who wish to use them as a starting point for their own safety case.
For this phase, our safety evidence documents are:
Parkopedia’s mission is to improve the world by delivering innovative parking solutions. Our expertise lies within the parking and automotive industries, where we have developed a solid reputation as the leading global provider of high quality off-street and on-street parking services.
Parkopedia helped found the AVP consortium because we believe that Autonomous Valet Parking will become an important way in which we can serve our customers, by reducing the hassle of the parking experience. Parkopedia are providing highly detailed mapping data for off-street car parks, one of the critical components to a car being able to successfully park autonomously.
To make Autonomous Valet Parking a reality, the consortium first selected the StreetDrone.ONE as its car development platform. We are now developing the software stack to run on our StreetDrone with NVIDIA Drive PX2. The University of Surrey, another founding member of the AVP consortium, provides the camera-based localisation algorithms needed for the car to navigate autonomously inside a parking garage, which will support vision, in addition to LiDAR-based localisation.
Parkopedia has joined the Autoware Foundation as a premium founding member, along with StreetDrone, Linaro/96Boards, LG, ARM, Huawei and others. We believe in open source as a force multiplier to build amazing software, and the AVP consortium is committed to using Autoware as the self-driving stack which will run on our StreetDrone and PX2 to demonstrate Autonomous Valet Parking.
Autoware was started in 2015 by Professor Shinpei Kato at the Nagoya University, who presented it at ROSCon 2017. Autoware.ai is based on ROS 1, which has certain fundamental design decisions that make it impractical for production autonomous cars. ROS 2, backed by Open Robotics, Intel, Amazon, Toyota and others, is quickly maturing, and from the very beginning was designed to fulfill the needs of not only researchers in academia, but also the emerging robotics industry.
Autoware.Auto launched in 2018 as an evolution of Autoware.AI, based on ROS 2, applying engineering best practices from the beginning, such as documentation, code coverage and testing, to build a production-ready open-source stack for autonomous driving with the guarantees in robustness and safety that the industry demands. We want to modularise Autoware.ai and align with Autoware.Auto and move to ROS 2.
We want high quality software, we care about safety and we want to do things right. Parkopedia’s main contributions so far have been to improve the quality of the code by fleshing out the CI infrastructure, adding support for cross-compiling for ARM and the NVIDIA Drive PX2, modernising the message interfaces and developing a new driver to support 8 cameras, among other improvements.
Our plan for 2019 is to keep contributing to Autoware.AI and Autoware.Auto to support the StreetDrone ONE and to make whatever changes necessary to support our Autonomous Valet Parking demonstration.
We’re very grateful to Shinpei Kato and the Tier4 team for open-sourcing Autoware and for welcoming our contributions.
One of the key objectives of this Autonomous Valet Parking project is to demonstrate our autonomous vehicle parking itself in a covered car park. The Transport Systems Catapult is responsible for the safety work package which ensures that all activities undertaken during the project are done in a systematic and safe manner. One of the important deliverables to ensure safety is the Risk Assessment and Method Statement (RAMS).
The RAMS document generally includes:
1. An overview of the project and key objectives to provide the reader with a background of the project
2. The activity being assessed, including:
Roles and responsibilities
Limits of the operation and trial details (route planned, scenarios, vehicle specifications, time of day, limits, weather, specifications)
Legal considerations such as vehicle insurance and laws
Training achieved (eg. driver training on the StreetDrone.ONE vehicle, taking over manually)
3. A risk assessment listing hazards, consequences, mitigation methods and detailing who might be harmed. Following the ISO 26262 standard, a hazard analysis and risk assessment is required in order to determine the criticality of a system.
The risk analysis is focused on:
Risks related to the ongoing operation of the vehicle
Risks related to the operation of external factors that affect current operation
Risks arising from the new equipment that may affect the safety of the vehicle or other
The method statement part describes in a logical sequence how a task will be carried out in a safe manner. It includes all the risks identified and the measures needed to control those risks.
The purpose of the method statement is to ensure that:
The trial is carried in a structured, controlled and safe manner
The hazards and associated risks are understood and mitigated
While the ultimate goal of the project is to demonstrate Autonomous Valet Parking, we will build up to this demonstration through smaller manageable steps and a separate RAMS will be produced at each stage:
At Parkopedia’s Autonomous Valet Parking team, we will be creating indoor car park maps for autonomous vehicles. Throughout this 2.5 year project, in collaboration with Transport System Catapult and the University of Surrey, we expect to be doing a significant amount of deep learning. We estimate that over the course of this project there will be intensive training on large datasets, for multiple days, weeks or even more. As any deep learning project there are three distinct phases in the research and development pipeline, which can be loosely described as (1) prototyping; (2) hyperparameter search and (3) intensive training. While there will be a lot of prototyping involved, which we believe any high-end desktop computer equipped with one GTX Nvidia 1080 Ti will fit our needs, the next question is how do we efficiently solve (2) and (3). Typically, in (2) one picks different parameters of the model and trains it against the dataset (or part of it) for a few iterations. In our case, a reasonable estimate could be perhaps 10 different runs, each taking a day of processing time. This insight (best hyperparameters) is then used to apply heavy computation in (3).
We are at the early stages of this project, but due to accounting reasons, we need to have a clear vision now on where we want to apply the budget for computing power for deep learning. Given that we know beforehand that we’ll need to have it for a relatively long period and we will do a lot of experimentation on large amounts of data, we want to investigate what is the most cost effective solution. On the other hand, because of the high processing times involved, it is important that the solution has enough flexibility to the research team, in that heavy computation times don’t hinder the ability to focus on other tasks that might require a GPU too. This blog post is about finding the sweet spot between these two.
The discussion is generally around on-premise hardware vs cloud based solutions. A cloud based solution is appealing because resources are readily available, maintenance-free, and one can launch as many computing machines as desired at any point in time. This flexibility also has a much wider hardware choice: a multitude of flavours are available so one is not restricted to working with a certain hardware architecture. Of course, these very interesting features come with a price. Whilst an office-based solution doesn’t offer these characteristics, it is interesting to see how they compare, especially in the long run. Therefore, it comes down to asking the following question:
What infrastructure gives us the best compromise between time-to-solution, cost-to-solution and availability of the resources ?
The goal of this document is to do a performance vs cost study on different possible architectures, both cloud-based and office-based. We ran benchmarks on various AWS GPU instances, and extrapolated some of the results to other similar cloud based architectures on Google Cloud. We compared the results against a system with 4 Nvidia GTX 1080 Tis, by extrapolating the results obtained for a single GTX 1080 Ti, which we currently have. An analysis was made both in terms of performance and cost.
2. Experimental Setup
We ran the tensorflow benchmark code repository which according to their description, “contains implementations of several popular convolutional models, and is designed to be as fast as possible. tf_cnn_benchmarks supports both running on a single machine or running in distributed mode across multiple hosts.”, which is perfect for running it against various infrastructures, both in single and multiple GPU scenarios. In terms of software all the infractures had the exact same configuration, namely:
2.1. Amazon Web Services (AWS)
Amazon’s GPU options are very simple: you choose an instance which comes with one (or more) GPUs of any given type. We compared three GPU families on AWS p2s (K80s), g3s (M60s) and p3s (V100s).
On demand price $/hour
2.2. EC2 Spot Instances
Spot instances are a solution offered by Amazon’s EC2, which defines it as an: “(…) unused EC2 instance that is available for less than the On-Demand price. Because Spot Instances enable you to request unused EC2 instances at steep discounts, you can lower your Amazon EC2 costs significantly (…)”. One thing to have in mind is that its pricing scheme is based on offer/demand model, resulting in a fluctuant hourly rate. In a nutshell, the user selects the hardware (say p3.2xlarge), defines how much he/she is willing to pay per hour, and the machine is accessible, provided the price set is below AWS’ price threshold. In such an event, the machine is automatically shutdown by Amazon. On the financial side, this is obviously very appealing as one gets the same hardware for a fraction of the cost. However, on the technical side, this solution requires a more complex architecture, which can handle interruptions (on a DL scenario this means most likely saving the network’s weights and gradients often to a location outside the local instance). The user must also accept downtimes – even if there is an orchestration mechanism that automatically launches on-demand instances to recover the loss of an instance and continue training/inference, there’s an initialization stage (setting up the instance, downloading the last state of the network, downloading the dataset, etc) – that cannot be ignored. Moreover, there’s a considerable effort in coming up with such orchestration mechanism, which cannot be disregarded as well, as it brings to advantages to the deep learning researcher; in fact, it’s quite the opposite.
Because of these reasons, this solution doesn’t directly compare with the two main solutions in debate. Nevertheless, we believe that in order to have a more balanced post, it should be added here. Below is a screenshot of the last 3 months obtained from AWS’ EC2 Management Console for a p2.8xlarge instance.
In order to work out an hourly rate for EC2 spot instances, we’ve taken the cheapest curve for all the availability zones, and we came up with a ballpark estimate for its average price throughout the 3 months period. The following table summarizes this information for all the instances, as well as the discount with respect to the on demand prices.
2.2. Office GPU box
For a proper comparison against cloud-based solutions, we need to come up with an hourly rate for the office GPU. Working out the equivalent price/hour involves making assumptions on the utilization (the more we use it, the cheaper it becomes, per hour). In addition to the hardware acquisition, we want a maintenance contract. This is because we want to minimize downtimes when a certain GPU goes out of order temporarily (unfortunately, it is not that unusual, and is in itself a disadvantage of this solution). Because this project will run for 30 months, we can amortize this price per month and depending on the average utilization in terms of the number of days the box is used (24 hours per day, uninterruptedly), we can reach a final figure for the hourly rate. To be even fairer, one might consider adding in the electricity costs, which we thought it would make sense.
Broadly speaking, in our scenario, a box of 4 x GTX 1080 Ti (about $6.5k USD with a maintenance contract), assuming 20 USD cents per kwh in the UK, being used 10 days per month equates into about $1.30/hour. We appreciate this 10 days is a strong assumption, especially in this early stages of the project. Therefore we test different utilization figures later. A breakdown of this figures is shown below:
Cost estimation of office GPUs
w/ 5 Year Warranty, 3 Years Collect & Return
Total Acquisition Cost
Months (length of project)
Price / month
Price per kwh
Utilization (days per month running 24/7)
Cost / month
Global cost / month
(assuming previous utilization)
2.3. Google Compute Engine (GCE)
Google Cloud’s model is slightly different than AWS’ model, in that GPUs are attached to an instance, so one can combine a certain GPU with different host characteristics. As per Google Cloud’s pricing page: “Each GPU adds to the cost of your instance in addition to the cost of the machine type”. We did not run any benchmarks on google cloud, but because their prices are different we thought it would make sense to add GCE’s scenario. Because of this, we only took into account hardware setups that would be comparable to the AWS’ offer, in terms of the GPUs.
Their GPU offering for GPUs consists on the following boards: NVIDIA Tesla V100, NVIDIA Tesla K80 and NVIDIA Tesla P100. The latter was discarded as there’s no counterpart on aws. Also, regarding the K80’s one can only select 1,2,4 and 8 GPUs and as for the NVIDIA V100’s, only 1 or 8 GPUs can be selected
In order to obtain a similar instance to those in AWS, we selected the “base instances” so that the number of vCPUs is the same, as follows:
Price/h ($) [first term is the instance price/h]
Equivalent aws instance
0.38+2.48 = 2.86
3.04+8*2.48 = 22.88
0.04+0.45 = 0.49
1.52+8*0.45 = 5.12
In terms of performance benchmarks, Tensorflow has done this comparison already. Amazon’s p2.8xlarge was compared against a similar instance, which we denote as GCE_8xK80. Although there appears to be a minor difference, the results are approximately similar, as expected. In our context, we believe we can safely assume the same for the three remaining cases, so we simply used the same performance metrics.
We also took Google TPUs into account. This blog post compared AWS’ p3.8xlarge against a cloud TPU. They show that on ResNet50 with a batch size of 1024 (the recommended size by Google), the TPU device is slightly faster, but slower at any other smaller batch size. We omitted this fact, and used our measurements obtained on the p3.8xlarge.
3. Performance Results
We ran the benchmark code in three different standard models (Inception3, VGG16 and ResNet50) using three different batch sizes per GPU: 16, 32 and 64, meaning that in a multi-GPU training session the effective global number of images per batch is #GPUs x batch size. Half-precision computation was used. A typical command line call would look like this:
As per TensorFlow’s performance guide, “obtaining optimal performance on multi-GPUs is a challenge” […] “How each tower gets the updated variables and how the gradients are applied has an impact on the performance, scaling, and convergence of the mode”. As mentioned earlier, the goal of this analysis is to obtain a ballpark estimate on cloud vs on-premise in terms of performance and cost, so gpu-related optimization issues, are out of scope of this post (this issue has been addressed in various other blog posts, e.g. see: Towards Efficient Multi-GPU Training in Keras with TensorFlow). That said, when we ran the benchmarking code with its default configuration (as show in the python command above), we soon realised we were having poor scalability in terms of throughput, i.e. the throughput didn’t scale well with the number of GPUs, so that wouldn’t be a fair comparison in a more optimized scenario. After reading Tensorflow’s tips, and a bit of experimentation we reached a setup which we were happy with. Note also that using different gpu optimizations might have an impact on training convergence, but we will ignore that for the remainder of this post. Despite this, we only fined-tuned g3.16xlarge and p2.16xlarge and p3.16xlarge instances, which was where the scalability issues were seriously being noted. Note that we don’t claim this to be by any means the optimum configuration, nor did we put serious efforts in it, but it allows us to make a more coherent comparison. The table below shows the scaling efficiency obtained after optimizing each of the three multi-GPU instances, with respect to the respective instance that has a single GPU. For each configuration (AWS instance / model ) the associated letter (A, B, C, x ) refers to the configuration in use, as follows:
A : variable_update:replicated | nccl:no | local_parameter_device:cpu
B : variable_update:replicated | nccl:yes | local_parameter_device:gpu
C : variable_update:replicated | nccl:no | local_parameter_device:gpu
x - default configuration
g3.4xlarge -> g3.8xlarge
g3.4xlarge -> g3.16xlarge
p2.xlarge -> p2.8xlarge
p2.xlarge -> p2.16xlarge
p3.2xlarge -> p3.8xlarge
p3.2xlarge -> p3.16xlarge
In the perfect world these numbers would all be close to 1.0, showing a perfectly linear efficiency in terms of performance as the number of GPUs scale from 1 to many. This table shows that scaling is not perfect but some setups can get quite close.
As mentioned in the introduction, we have one GTX 1080 Ti where we can run these tests on, therefore we had to estimate the performance figures on a box with four of these GPUs. We simply used the average obtained in the earlier table for each of the datasets and batch sizes (see row in red). We believed this to be a relatively conservative estimate. These are the figures obtained.
3.1. Throughput comparison
The following charts show the samples / second for each of the models for all the mentioned scenarios.
4. Cost results
Based on the throughput and the hourly costs we can estimate the number of images we can process with $1, which is shown below. Note that the EC2 spot prices were also included on these charts.
5. Additional Comparisons
We don’t know yet the amount of data we will be processing, so we thought the ImageNet dataset would be a good choice to give us some real-world intuition on what to expect in terms of time to train and global cost in dollars, as it is pretty much the “industry standard” in deep learning and also because of its respectful size. For each of the models, we their respective papers give us the batch size and the number of iterations, which allows us to work out the total number of samples seen when training.
To estimate these figures, we simply used the cost and time for a batch of 64 for all the models. Note however that in reality the figures would be lower (both for the time and consequently, the cost). This is because we are using a smaller batch size in our estimations, so a better utilisation of the GPU memory (bigger batch size) would increase the performance, so this is just for illustration purposes. Of course, this choice would impact convergence, but we’re disregarding that too.
The two charts below shows us the cost and time to train. Considering only “persistent” solutions (all except EC2 Spot instances), the office-based solution offers a much cheaper solution – almost 2x cheaper than the Google’s GCE_8xK80 (the cheapest persistent cloud-based solution). It is still about 36% cheaper than the cheapest solution on the “spot instances” model. Another interesting fact is that it is the forth best performing solution in terms of time to train.
5.2. Gain vs utilization
On previous estimates, we were using an utilisation figure of 10 days – running the box on average 240h per month. As this is just a guess, it is interesting to know how much do we gain (or lose) as we change the number of days per month when compared to a cloud-based solution. To simplify we’ve compared different utilisations against the three best scenarios on VGG16 (64). Again, by doing this comparison on VGG16, is itself a conservative choice, as out of all the models, VGG16 is the one where the images/cost gain of 4x1080tis versus the second best scenario (GCE_8xK80) is smaller – it is better by only approximately 1.7x versus 1.9x on ResNet50 and Inception3, on batch sizes of 64. The comparison is shown on the chart below.
Out of all the the persistent solutions, Google’s GCE_8xK80 is clearly the one with the best value for money. Yet, after 5 days of usage per month, the office GPU outbids Google’s all other cloud based instances. After about 8.5 days per month the box is twice as cheap as TPUv2 – 1.8x and 1.9x respectively for p3.x2large and p3.8xlarge instances. This increases slightly to 10 days for GCE_8xK80. The cheapest spot instance price get outbid only after 8 days of continuous usage.
6. Closing remarks
We believe that throughout our analysis, we were highly conservative in the estimates for an office GPU machine. Essentially, when assumptions had to be made, we tried to compare an office based solution against worst performing scenarios. An accurate estimate depends a lot on the actual scenario but here we tried to be slightly pessimistic with regards to an office based solution.
What we believe matters most is the time and cost to reach a certain solution and our analysis on ILSVRC shows that the 4xGTX1080tis lies perfectly on the sweet spot between these two factors. Another interesting angle is the utilisation chart which clearly shows that, if a GPU box will be used for slightly more than 5 days per month, then cost wise, the on-premise solution is the clear winner. So, on long projects such as the one we’re about to embark, an office based solution is a no-brainer. Therefore the choice is mostly obvious if you need to use GPUs extensively.
Nethertheless, we would like to highlight the following points:
In terms of costs, when compared to AWS, Google’s cloud GPU offering is worth investigating.
We are mindful EC2 spot instances are very appealing, but we believe this scenario didn’t fit our purposes because to gain the value from spot instances a maximum price needs to be set and the job will be terminated if that threshold is breached. Even if it its drawbacks can be surpassed using additional logic to handle the termination and re-start of instances gracefully, we believe such configuration is non-optimal and wouldn’t be appropriate for us.
Another point for consideration regarding EC2 spot prices is that although this analysis was done over a relatively long period of 3 months, effectively reducing the fluctuations, there’s no guarantee cost will stay more or less fixed in the future. In the best scenario it can even be lower, but predictability is required in when budgeting for a 2.5 year project.