Splitting Large Files to Smaller Files in NiFi

Apache NiFi, Apache NiFi Developers / 17/10/2020

Splitting Large Files With Nifi is quite easy when you know the right approach to do that. In this article I will show you step by step on how you can split a large file to smaller files.

See the complete Video or scroll down to continue reading.

Thumb Rule is, You should always use split record processor to split your total records divided by 10. So for example, If you have a file with 1,000,000 records and you want final output with only 10 records per FlowFile. Then you should first split it to 100,000 and then split it to 10,000 and then split it to 1000 and then split it to 100 and then finally to 10 records.

Splitting Large Files With Nifi data flow example


Input FlowFile is having 10,000 records per FlowFile. This is processed by NiFi Split Record Processors and Splitted into from 10,000 records to 1,000 records, then to 100 records and then finally to 10 records per FlowFile

Splitting Large Files With Nifi Input file

Final Output FlowFile is having only 10 Records Per FlowFile. You see how easy it is to split big files to smaller files with this approach. Also it gives you total control on the number of records you want to see in each file.

Splitting Large Files With Nifi Output File

Download The Template Here

I hope you liked Splitting Large Files With Nifi article. You can read more about Apache NiFi Here.

 
%d bloggers like this:

Privacy Policy - Terms and Conditions