NiFi’s ExecuteStreamCommand Processor executes an external command (can be a shell script, python script or java script etc) on the contents of a flow file, and creates a new flow file with the results of the command.
Many a times, we may end up in a scenario when existing NiFi processors are not enough to handle the custom logic which we have. In situations like these we’re left with one of below two options:
- Create a custom NiFi Processor. This can be tough if you don’t know Java, as NiFi processors are written in Java.
- Write a script in our choice or programming language and then call this script using NiFi
So in this article we will learn about the second approach, calling a custom script (python script) using Apache NiFi ExecuteStreamCommand Processor.
You can watch the complete video here, If you like to read, then continue scrolling down.
1: Use ExecuteStreamCommand Processor
2: Right-Click on ExecuteStreamCommand Processor and go to “Configure”
3: In “Command Path” property, Provide the full path of the script. In the example below, I assume that there is “check_duplicate.py” python script already exists at “/home/itpanther” directory.
4: If your script is accepting any arguments, you can provide that in “Command Arguments” property. If your program accepts multiple arguments, then you can pass multiple arguments by separating them with semicolon.
Step 5: Please note that ExecuteStreamCommand processor passes the flow file content to system buffer, so your custom program should be able to read it from “stdin”. Each programming syntax has its own way to read from “stdin”.
Step 6: Once you process the data within your script, you can return the output back by sending the output to “stdout”.
I hope you liked this “Apache NiFi ExecuteStreamCommand Processor” article. Do not forget to say thanks in the comment box if you liked this article. You can read more about Apache NiFi here.
Get any of the courses at a very special price. The offer is available only for a limited time.
- Apache NiFi
- Apache NiFi – Admin Guide
- Liferay – Create Your Website Without The Need to Code
- Grafana – Learn to do Analytics with Grafana
- Redis – Learn World’s Fastest Database
- BMC Control-M for Beginners
1 thought on “Apache NiFi ExecuteStreamCommand Processor”
May you share the pipeline (xml format)?
Comments are closed.