Clean up the unnecessarily complicated blacklist implementation
somewhat.

Signed-off-by: Johan Hovold <johan@kernel.org>