Fix race conditions with unplug/replug behavior, in particular
take care not to hold up USB probe/disconnect for long-running
framebuffer operations and rely on usb to handle teardown.

Fix for kernel panic reported with new F17 multiseat support.

Reported-by: Kay Sievers <kay.sievers@vrfy.org>
Signed-off-by: Bernie Thompson <bernie@plugable.com>