Adding Apache Kafka capability to Apache Pulsar

KOP - Kafka On Pulsar is an interesting project that adds Kafka capabilities to Pulsar. I did play around with KOP release prior to 0.3.0 and unfortunately it supported only Kafka protocol version 0 to 2. This seemed to have changed with KOP version 0.3.0. I have not had a chance to try it out.

Nevertheless, to get KOP working is straight forward. On your broker nodes add the following to broker.conf. For listeners parameter substitute correct value for each node:


Place KOP’s .nar file into /opt/pulsar/protohandlers. The directory does not matter, as long as Pulsar can access it. Do not forget to change ownership and/or permissions, if necessary. Follow up by restarting broker process. That’s it.

Now grab Kafka binaries and try testing the functionality by running producer…

[root@broker1 ~]# bin/ --broker-list --topic kafkatest --producer-property security.protocol=PLAINTEXT

…and consumer:

[root@broker1 ~]# bin/ --bootstrap-server --topic kafkatest --from-beginning --consumer-property security.protocol=PLAINTEXT