Make sure that no callback is running before we teardown the module.

Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Dmitry Torokhov <dmitry.torokhov@gmail.com>