Nifi - How to put data into Hive database?

I am building a Nifi flow to get json elements from a kafka and write them into a Have table. However, there is very little to none documentation about the processors and how to use them. What I plan to do is the following:

kafka consume --> ReplaceText --> PutHiveQL

enter image description here

Consuming kafka topic is doing great. I receive a json string. I would like to extract the json data (with replaceText) and put them into the hive table (PutHiveQL). However, I have absolutely no idea how to do this. Documentation is not helping and there is no precise example of processor usage (or I could not find one).

Is my theoretical solution valid ?
How to extract json data, build a HQL query and send it to my local hive database ?

asked Dec 17, 2019 at 14:57 Itération 122442 Itération 122442 2,920 2 2 gold badges 39 39 silver badges 88 88 bronze badges

2 Answers 2

basicly you want to transform your record from kafka into HQL request then send the request to putHiveQl processor.

I am not sur that the transformation kafka record -> putHQL can be done with replacing text ( seam little bit hard/ tricky) . In general i use custom groovy script processor to do this.

Edit

enter image description here

EvaluateJsonPath

This extract the properties timestamp and uuid of my Json flowfile and put them as attribute of the flowfile.

enter image description here

ReplaceText

This set flowfile string to empty string and replaces it by the replacement value property, in which I build the query.