Make debugging real problems easier by not trying to disable an EP that
was not yet enabled.

Reviewed-by: Peter Chen <peter.chen@nxp.com>
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
Signed-off-by: Felipe Balbi <balbi@kernel.org>