Early days self-driving car we build a machine to drive through the desert,
Stanley, for the Dawawine Challenge and the machine drove really nicely.
Except every 40 miles or so,
it would see a magic big obstacle in front of
it and freak out and attempt to jump down a cliff.
It was very rare.
It was every 40 miles. It was like every two or three hours.
It was very hard to catch and we were flabbergasted like why all of a sudden,
everything's going perfect, all of a sudden,
dust of all would see this random massive obstacle.
It took me months of my own time to figure that out and fast forward, here is the answer.
We had a laser scanning the terrain and to make
sure that when laser data overlapped over time,
that you could bounce up and down,
that we wouldn't fall into a trap to compare laser data,
that wasn't in the same cone of frame.
We just said look, the less laser reading and some time elapsed.
The more time elapsed, the less confident we are.
It was actually a good rule so we could actually have
a situation if data was acquired very successfully in time, very confident.
We're telling between, we compared all data points, less confidence.
Then, we imported this architecture to multiple computers and exporting,
we had to sync the clocks of those computers.
In the syncing of clocks of those computers,
time would occasionally go backwards.
What that did to our confidences was catastrophic.
It was, all of a sudden, instead of relaxing confidence over time,
tighten confidences to the level that could never see no obstacle.
It was really a combination of clock being backwards, as far as syncing,
and the car bouncing up and down at
the same time that would cause that catastrophic situation.
Now I, as a software engineer,
I wouldn't have asked the question of these bizarre obstacles,
does the clock go backwards?
It took us forever to even look at the right number to understand
that this backwards-going clock could actually cause the car to crash.
That is one of these examples of a really,
really hard to find bug.