Splitting Large Files With Nifi is quite easy when you know the right approach to do that. In this article I will show you step by step on how you can split a large file to smaller files.
See the complete Video or scroll down to continue reading.
Thumb Rule is, You should always use split record processor to split your total records divided by 10. So for example, If you have a file with 1,000,000 records and you want final output with only 10 records per FlowFile. Then you should first split it to 100,000 and then split it to 10,000 and then split it to 1000 and then split it to 100 and then finally to 10 records.

Input FlowFile is having 10,000 records per FlowFile. This is processed by NiFi Split Record Processors and Splitted into from 10,000 records to 1,000 records, then to 100 records and then finally to 10 records per FlowFile

Final Output FlowFile is having only 10 Records Per FlowFile. You see how easy it is to split big files to smaller files with this approach. Also it gives you total control on the number of records you want to see in each file.

Download The Template Here
I hope you liked Splitting Large Files With Nifi article. You can read more about Apache NiFi Here.
- Installing Apache NiFi Cluster on LinuxInstalling Apache NiFi Cluster on Linux is pretty straightforward to be honest. If you follow the steps mentioned in this article, you can pretty …
- Installing NiFi Cluster on Windows ServerIn this article I am going to show you steps for Installing NiFi Cluster on Windows Server. Please follow all the steps as mentioned …
- Apache NiFi Cluster in DetailWhat’s a cluster? To learn about Apache NiFi Cluster in Detail first you should understand about cluster. A cluster is a set of loosely …
- Splitting Large Files to Smaller Files in NiFiSplitting Large Files With Nifi is quite easy when you know the right approach to do that. In this article I will show you …
- Apache NiFi Best PracticesApache NiFi Best Standards and Other Tips In this article I am going to show you Apache NiFi Best Practices. These best practices will …
You must be logged in to post a comment.