How did Geoffrey Hinton come up with the idea of Dropout?
Hinton says he was inspired by, among other things, a fraud-prevention mechanism used by banks.
In his own words:
I went to my bank. The tellers kept changing and I asked one of them why. He said he didn’t know but they got moved around a lot. I figured it must be because it would require cooperation between employees to successfully defraud the bank. This made me realize that randomly removing a different subset of neurons on each example would prevent conspiracies and thus reduce overfitting.
The core idea is that introducing noise in the output values of a layer can break up happenstance patterns that aren't significant (what Hinton refers to as conspiracies), which the network will start memorizing if no noise is present.