VW’s attempts at emulating Tesla are the talk of the industry. CEO Herbert Diess’ publicly avowed bromance for Elon Musk and Tesla have been on display over the past few years, and Diess has committed some 90+ billion euros in a massive plan to remake the company with not just EVs, but ones that incorporate an all-inclusive and readily over-the-air updatable centralized software. In other words, the German Tesla. This feature is a core technology and one of the keys to the success of Tesla, and is considered to be 5-6 years ahead of the industry. But VW is determined to try to catch them.
But things are not going well at all. The heart of the program is their new ID family, which will be the basis of over 27 offspring across the various owned brands. Massive software problems are seriously threatening to delay its launch this summer. Just over a week ago I showed you an Audi etron dash alert for a required oil change, and I said No wonder Volkswagen is desperately hiring as many software engineers as possible. Now VW has some 10,000 of them desperately trying to fix the problems.
From an autonews article today:
The problems stem from the basic architecture of the ID3’s software, which was developed “too hastily,” the magazine said, citing VW’s own experts. Many of the system’s elements do not understand each other, leading to dropouts and other difficulties, the report said.
Hundreds of test drivers of the car are reporting up to 300 errors a day, the magazine said.
The ID was intended to be the first true Tesla Model 3 competitor, meaning designed from the ground up to incorporate the kind of features that have made the Tesla the head of the pack, and a hot seller in Europe. A crossover variant, the ID4, is intended for the US market, and VW is expanding its Tennessee plant in order to build it here.
Diess has said that the ID is as strategically important to VW’s future as were the Beetle and the Golf. VW started limited series production of the ID3 at its plant in Zwickau, Germany, in November. It said it is using the months until deliveries of the first cars to fix issues that may come up during further field testing. Meaning, the ones being produced now are not really feature complete, and will require substantial upgrading.
Interestingly, Tesla never had any real issues with its software, but then it is a Silicon Valley technology company at heart. Many have said that it’s easier for existing automaker to adopt new technology than for a tech company to master traditional auto making skills. That argument is increasingly being put to the test, and failing.
Clearly they have never heard of Brook’s Law – Adding manpower to a late software project will make it later.
https://en.wikipedia.org/wiki/Brooks's_law
Tom,
So true.
And, in my first software development manager’s office I saw a sign that would follow me throughout my career in IT/SW development:
. Good
. Fast
. Cheap
Pick two.
Interesting article. I especially liked this point:
Brooks used this example to point out that certain tasks were not divisible: While it takes one woman nine months to make one baby, “nine women can’t make a baby in one month.”
The part about scheduling is so true. Whereas humans are very bad at estimating and judging risk they are also very poor at estimating how long something will take. I learned that long ago and now my general rule is to take my estimate and then double it. So when it seems like 6 months I double to 12 months and that ends up being close to right.
Over the decades I came to modify that rule-of-thumb on estimates to the following: with a clear and concrete set of specifications any estimate greater than two weeks should be doubled. If the specs are nebulous or “evolving,” double the number and increase the units by 1. This allows you to explain why an “I have an idea” project originally estimated at 4 weeks turns into an 8 month mess.
To much stuck in tradition, with to many old (sorry) people who have a say in the company.
You have to make a real radical break to make a successful EV, as many more are finding out……
It may take some time for VW to get their EVs on the road. I don’t find this surprising since it is a completely new technology.
When VW is pumping out EVs are the rate of several million per year, we are going to see a proliferation of these cars, especially in Europe.
Finally, to anyone complaining the EU’s exhaust emission regulations, there are approximately 320,000,000 cars on the road in the EU, in an area less than that of one Canadian province. All one needs to do is go up a hill in Paris and look down; the entire city is enveloped in diesel smoke. These changed were made at the request of EU voters.
I think you may be a little off there. Three simple Google searches show that:
The landmass of the entire EU is about 1.7 million square miles.
The landmass of ALL OF CANADA is about 3.8 million square miles.
Quebec is Canada’s largest province by area, it covers about 15% of that total for about half a million square miles.
So the EU appears to have around three times the area of the largest province, not half the area, a factor of six.
Good to hear that they aren’t treating customers like beta testers.
I don’t know that I trust 10,000 desparate engineers. I’d never heard of Brook’s Law but I was thinking that just wrangling them all would take a lot of time.
The engineering leader of the Prius development team said the hardest part was the software.
Regarding the last paragraph in the article above – having been involved in the Model S, X, and 3 development and rollout, I can definitely vouch for the accuracy of the final sentence. The Tesla Engineers that I worked with were always adamant that Tesla is a Technology company, not an automobile company. They spent a lot of money up front, but getting the production equipment up and running was not all that different from any of the established automakers that I’ve been involved with. I think that the VW experience outlined above showcases the challenges that a conventional automaker encounters when trying to adapt a whole new operating paradigm.
Tesla’s Model 3 is one of those “overnight successes” that took a generation of hard work to get to.
A 21st century electric drivetrain depends heavily on software in every stage, from lithium battery management through driving the motor windings. Not to mention all the non-drivetrain capabilities a modern car is expected to have.
Tesla’s direct ancestor was the AC Propulsion tzero roadster (below), launched in 1997, and converted to lithium-ion in 2003. Even this first car has software complex enough to decrease the rear-wheel regenerative braking in hard cornering.
Tesla was founded in 2003 and came out with the its own Roadster, originally based on the tzero, in 2008. Model S in 2012. Model 3 shipped in 2017, a full 20 years after the first tzero.
All these cars had an increasing degree of software capability and complexity, each building on road-tested cars that came before. You can’t just throw money at an army of people and catch up to that in a few years, as VW appears to be finding out.
I’m a bit surprised though given that VW has been selling the all-electric e-Golf for several years and I haven’t heard of lots of problems with them. I would have expected that experience to carry over to the ID3.
You have a very good point. VW does make their own motor and transmission for eGolf. This according to a detailed article published at introduction back in 2013. Doesn’t say who developed the controller and its software. Batteries are now from Samsung. Seems likely VW has been directly involved in eGolf’s powertrain design and manufacturing all along. (Unlike GM who gets a complete drivetrain from LG).
I see lots of eGolfs here in metro Portland, in fact a neighbor just bought one.
Mike, this is not at all about just the software to properly make an EV function. Everyone has figured that out some time ago. There’s lots of perfectly well-functioning EVs from numerous companies.
Tesla’s software is a whole different thing. It’s an operating system for the whole car; everything, from safety functions, entertainment, security, Autopilot, etc., etc. it’s roughly comparable to Windows, or Apple Ios. it’s drastically more comprehensive than just controlling the basic EV functions. And it all runs off a micro-super computer, that runs Tesla’s own chip. Yes, they designed the chip themselves. Fully capable of interfacing with the neural net for autonomous learning/function, etc.
Other manufacturers have literally dozens of microprocessors all over the car, most of them developed by different suppliers to control the various areas of their responsibilities. A completely different environment. And that includes all other existing EVs.
VW has seen the light about Tesla’s fully integrated OS, and wants one too. That’s why they hired 10,000 software engineers, literally. But it’s not going as easily as they thought.
The software that ran the AC Propulsion Roadster is like comparing a 15 year old flip phone with a current iPhone. Or even more apart. I’m not sure why you’re even bringing it into the conversation. It has zero relevance to Tesla’s OS. Even the Tesla Roadster doesn’t run the current Tesla OS.
It’s a whole new ball game, but an absolute critical aspect of what a Tesla is. The best comparison is to dumb cell phones (all other EVs) and iPhones, infinitely upgrade able and highly capable.
The point of bringing up the AC Propulsion tzero is that the Tesla software was developed on the 20 years’ experience of developing and running all the software that came before it, including the tzero’s. Any successful complex system evolves over many generations of development, based on all the experience gained along the way.
I agree it’s “not at all about just the software to properly make an EV function.” Now it’s about the software to make an EV function well, at maximum efficiency. Model 3 has a big jump in efficiency, which translates into range. It has a different motor design called “switched reluctance”, which suffered from torque ripple before Tesla tamed it with motor hardware and motor controller software.
The eGolf may just be good enough technology, and VW (eGolf: 36 kWh, 125 miles = 288 W/mi) may be straining to reach high enough technology to match Tesla’s efficiency (54 kWh, 250 miles = 216 W/mi).
Mike, it’s a lot more complicated than that. But I’ll leave it at that.
Oops, Wh/mi, not W/mi.
Paul, I’d love to hear more about the complications, please.
Anyway I certainly agree, it’s harder for an established company to catch up with fundamental innovation, than for the innovator to catch up with traditional skills. That’s how disruption works.
Volkswagen must have hired the same software team that worked on the Boeing 737Max
It looks a lot more like VW executives are making missteps comparable to Boeing’s, out of comparable arrogance and hubris.
An intelligent person looks at a tough problem and thinks “I can make this work.”
An intelligent person with experience looks at the problem and thinks “Making this work is gonna be a lot harder than everyone thinks.”
My first reaction to this was mismanagement. While this project is large and ambitious, 10,000 software engineers alone are too many and too much. Putting up more resources into the project without proper planning and good leadership are just wasteful spending. And natural law of diminishing margin return will also catch up. Some readers may have had experience in the large scale projects before, they will probably say the same. I don’t believe Mircosoft put 10,000 software engineers into development of Windows 10, so as did Apple on orginal iPhone, Boeing for its 787, Trecent for its WeChat.
With so many software engineers working on the project, managing them is a very difficult task. I expect them at least generating 100k lines of code a year, how large of the overall software package will be? And highly complex thing tend to be less stable.
Been in software for 20 years… this isn’t going to end well for VW.
My usual axiom to managers is “programming isn’t digging ditches. You can’t just add labor and get it done faster.”
Musk had the vision and intellect to make Tesla happen organically. He used the brightest in the valley and probably pressure cooked/cooks them in rapid succession. Like any great chef he will gladly tell you what the ingredients/ patents are, less the special seasoning. Making it beautifully come together takes much trial and error we may never see (but I hope was well documented for future learning.) Longtime automakers reverse engineering parts in second nature but coding and such has got to be driving them bananas.
Note: I’m not a Telsa owner or Musk fanboy. I personally think Tesla would be much further along if a “traditional” President had been installed years ago and Musk’s Lutz-esk personality was put on a short leash.
“readily over-the-air updatable centralized software”
I’ve never understood why this is seen as a “feature”. And the recent story about Tesla stealing installed options off a car they’d sold, when they retroactively decided that they shouldn’t have included them with the sale goes a long way toward explaining why.
A friend has owned a Tesla Model S for a number of years, and he gets a big kick out of new features showing up in his driveway. That is pretty cool, and must surely be a great help in developing and maintaining their software.
But I’d be uneasy about it. The factory pulling options off a used car is not good. I wonder about security too. Surely cracking into a Tesla’s computer systems has been tried by now. There’s something to be said for physical access required.
+10.
Tesla made a mistake there, and re-installed the option they removed.
That’s backwards. All the cars in question have the equipment installed, and it was originally enabled, then Tesla disabled it (over the air) and now they’re allowing “owners” to have it re-enabled for $300.
If Tesla did that to me, I’d already be down at the Nissan dealer buying a Leaf and posting this reply while they finish the paperwork, then arranging with a contractor to convert my charger. F- that! Don’t hold my options hostage! Does the word Danegeld mean anything to anyone else here?
I have your same eff-that reaction, but it’s the wave of the future. Audi are very pleased with themselves for having figured out they can put the same hardware on all cars, lock it all down, and unlock whatever features and capabilities the buyer will pay for upfront and/or on an ongoing-subscription basis. Then they can zero it all out when the car gets sold—they are vague on whether this just means “traded in at the Audi dealer” or also means “sold privately”—and charge for the options again.
What’s a Danegeld?
Wow, how many engineers does it take to change a light-bulb moment?
Only tangentially related: are we seeing the start of a singularity that comes at the end, when all the car names and designations have been used up? Audi, BMW, Mazda, Tesla, and VW are all offering a “3”, along with some automakers local to China and India. Or is this just bench hypermarking?