During IB device removal, cancel the event worker before the device
structure is freed.

Fixes: a4cf0443 ("smc: introduce SMC as an IB-client")
Reported-by:  <syzbot+b297c6825752e7a07272@syzkaller.appspotmail.com>
Signed-off-by: Karsten Graul <kgraul@linux.ibm.com>
Reviewed-by: Ursula Braun <ubraun@linux.ibm.com>
Reviewed-by: Leon Romanovsky <leonro@mellanox.com>
Signed-off-by: David S. Miller <davem@davemloft.net>