**DO NOT MERGE** swap to librdkafka for kafka consumer/producer

[natedogs911]

This is the start of the switch for kerberos support,

items that still need work:

• i am not selecting which partitions to consume or produce to, looking at using multithreading with logging to simplify this approach and allow 1 consumer as opposed to multiple python workers.py
• topic creation, if auto create is enabled what is the logic behind creating topics in advance with multiple partitions? how can this be done better?
• i've added a lot of exception handling during testing and debug, will look to simplify this
• should some handling be added to the for msg consumer loop under pipeline/$TYPE/worker.py?

[kpeiruza]

Kafka topic auto-creation is disencouraged in production environments as it degrades performance and increases file handlers. Despite LinkedIn says 5k topics should be okay, many people suggests keeping them below 100-200.

If topic auto-creation is enabled, buggy producer code could degrade performance or even DoS Kafka, and you could end up delivering messages not consumed by any process.

So, keep auto-creation disabled :-)

[natedogs911]

Thanks @kpeiruza that is good info.