--- description: Apache Flume is a system used for moving massive quantities of streaming data into HDFS. We will Stream Twitter data using flume-ng command flume agent. title: Apache Flume Tutorial: What is, Architecture & Hadoop Example image: https://www.guru99.com/images/flume-program.png --- [Skip to content](#main) ## What is Apache Flume in Hadoop? **Apache Flume** is a reliable and distributed system for collecting, aggregating and moving massive quantities of log data. It has a simple yet flexible architecture based on streaming data flows. Apache Flume is used to collect log data present in log files from web servers and aggregating it into HDFS for analysis. Flume in Hadoop supports multiple sources like – * ‘tail’ (which pipes data from a local file and write into HDFS via Flume, similar to Unix command ‘tail’) * System logs * [Apache log4j](https://www.guru99.com/tutorial-on-log4j-and-logexpert-with-selenium.html) (enable Java applications to write events to files in HDFS via Flume). ## Flume Architecture A **Flume agent** is a **[JVM](https://www.guru99.com/java-virtual-machine-jvm.html)** process which has 3 components –**Flume Source,** **Flume Channel** and **Flume Sink**– through which events propagate after initiated at an external source. [![Flume Architecture](https://www.guru99.com/images/Big_Data/061114_1038_Introductio2.png)](https://www.guru99.com/images/Big%5FData/061114%5F1038%5FIntroductio2.png) Flume Architecture 1. In the above diagram, the events generated by external source (WebServer) are consumed by Flume Data Source. The external source sends events to Flume source in a format that is recognized by the target source. 2. Flume Source receives an event and stores it into one or more channels. The channel acts as a store which keeps the event until it is consumed by the flume sink. This channel may use a local file system in order to store these events. 3. Flume sink removes the event from a channel and stores it into an external repository like e.g., HDFS. There could be multiple flume agents, in which case flume sink forwards the event to the flume source of next flume agent in the flow. ## Some Important features of FLUME * Flume has a flexible design based upon streaming data flows. It is fault tolerant and robust with multiple failovers and recovery mechanisms. Flume Big data has different levels of reliability to offer which includes **‘best-effort delivery’** and an **‘end-to-end delivery’**. **Best-effort delivery** does not tolerate any Flume node failure whereas **‘end-to-end delivery’** mode guarantees delivery even in the event of multiple node failures. * Flume carries data between sources and sinks. This gathering of data can either be scheduled or event-driven. Flume has its own query processing engine which makes it easy to transform each new batch of data before it is moved to the intended sink. * Possible **Flume sinks** include **[HDFS](https://www.guru99.com/learn-hdfs-a-beginners-guide.html)** and **[HBase](https://www.guru99.com/hbase-architecture-data-flow-usecases.html)**. Flume Hadoop can also be used to transport event data including but not limited to network traffic data, data generated by social media websites and email messages. ## Flume, library and source code setup Before we start with the actual process, ensure you have Hadoop installed. Change user to ‘hduser’ (id used while Hadoop configuration, you can switch to the userid used during your Hadoop config) [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM1.png) **Step 1)** Create a new directory with the name ‘FlumeTutorial’ sudo mkdir FlumeTutorial 1. Give a read, write and execute permissions sudo chmod -R 777 FlumeTutorial 2. Copy files **MyTwitterSource.java** and **MyTwitterSourceForFlume.java** in this directory. [Download Input Files From Here](https://drive.google.com/uc?export=download&id=0B%5FvqvT0ovzHcendnWXlfbThMejQ) Check the file permissions of all these files and if ‘read’ permissions are missing then grant the same- [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM2.png) **Step 2)** Download ‘Apache Flume’ from a site- Apache Flume 1.4.0 has been used in this Flume tutorial. [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM3.png) Next Click [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM4.png) **Step 3)** Copy the downloaded tarball in the directory of your choice and extract contents using the following command sudo tar -xvf apache-flume-1.4.0-bin.tar.gz [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM5.png) This command will create a new directory named **apache-flume-1.4.0-bin** and extract files into it. This directory will be referred to as **** in rest of the article. **Step 4)** Flume library setup Copy **twitter4j-core-4.0.1.jar, flume-ng-configuration-1.4.0.jar, flume-ng-core-1.4.0.jar, flume-ng-sdk-1.4.0.jar** to **/lib/** It is possible that either or all of the copied JAR will have to execute permission. This may cause an issue with the compilation of code. So, revoke execute permission on such JAR. In my case, **twitter4j-core-4.0.1.jar was** having to execute permission. I revoked it as below- sudo chmod -x twitter4j-core-4.0.1.jar [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM6.png) After this command gives ‘read’ permission on **twitter4j-core-4.0.1.jar** to all. sudo chmod +rrr /usr/local/apache-flume-1.4.0-bin/lib/twitter4j-core-4.0.1.jar Please note that I have downloaded- **– twitter4j-core-4.0.1.jar** from **–** All flame JARs i.e., **flume-ng-\*-1.4.0.jar** from ### RELATED ARTICLES * [Hadoop MapReduce Join & Counter with Example ](https://www.guru99.com/introduction-to-counters-joins-in-map-reduce.html "Hadoop MapReduce Join & Counter with Example") * [Sqoop Tutorial: What is Apache Sqoop? Architecture & Example ](https://www.guru99.com/introduction-to-flume-and-sqoop.html "Sqoop Tutorial: What is Apache Sqoop? Architecture & Example") * [9 BEST Big Data Analytics Tools & Software (2026) ](https://www.guru99.com/big-data-analytics-tools.html "9 BEST Big Data Analytics Tools & Software (2026)") * [Top 30 Talend Interview Questions and Answers (2026) ](https://www.guru99.com/talend-interview-questions.html "Top 30 Talend Interview Questions and Answers (2026)") ## Load data from Twitter using Flume **Step 1)** Go to the directory containing source code files in it. **Step 2)** Set **CLASSPATH** to contain **/lib/\*** and **\~/FlumeTutorial/flume/mytwittersource/\*** export CLASSPATH="/usr/local/apache-flume-1.4.0-bin/lib/*:~/FlumeTutorial/flume/mytwittersource/*" [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM8.png) **Step 3)** Compile source code using the command- javac -d . MyTwitterSourceForFlume.java MyTwitterSource.java [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM9.png) **Step 4)**Create a jar First, create **Manifest.txt** file using a text editor of your choice and add below line in it- Main-Class: flume.mytwittersource.MyTwitterSourceForFlume .. here **flume.mytwittersource.MyTwitterSourceForFlume** is the name of the main class. Please note that you have to hit enter key at end of this line. [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM10.png) Now, create JAR ‘**MyTwitterSourceForFlume.jar’** as- jar cfm MyTwitterSourceForFlume.jar Manifest.txt flume/mytwittersource/*.class [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM11.png) **Step 5)** Copy this jar to **/lib/** sudo cp MyTwitterSourceForFlume.jar /lib/ [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM12.png) **Step 6)** Go to the configuration directory of Flume, **/conf** If flume.conf does not exist, then copy flume-conf.properties.template and rename it to flume.conf sudo cp flume-conf.properties.template flume.conf [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM13.png) If **flume-env.sh** does not exist, then copy **flume-env.sh.template** and rename it to **flume-env.sh** sudo cp flume-env.sh.template flume-env.sh [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM14.png) ## Creating a Twitter Application **Step 1)** Create a Twitter application by signing in to [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM15.png) [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM16.png) **Step 2)** Go to **‘My applications’** (This option gets dropped down when **‘Egg’** button at the top right corner is clicked) [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM17.png) **Step 3)** Create a new application by clicking **‘Create New App’** **Step 4)** Fill up application details by specifying the name of application, description, and website. You may refer to the notes given underneath each input box. [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM18.png) **Step 5)** Scroll down the page and accept terms by marking **‘Yes, I agree’** and click on button**‘Create your Twitter application’** [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM19.png) **Step 6)** On the window of a newly created application, go to the tab, **‘API Keys’** scroll down the page and click button **‘Create my access token’** [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM20.png) [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM21.png) **Step 7)** Refresh the page. **Step 8)** Click on **‘Test OAuth’**. This will display **‘OAuth’** settings of the application. [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM22.png) **Step 9)** Modify **‘flume.conf’** using these **OAuth settings**. Steps to modify **‘flume.conf’** are given below. [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM23.png) We need to copy Consumer key, Consumer secret, Access token and Access token secret to updating ‘flume.conf’. **Note: These values belong to the user and hence are confidential, so should not be shared.** ## Modify ‘flume.conf’ File **Step 1)** Open **‘flume.conf’** in write mode and set values for below parameters- sudo gedit flume.conf Copy below contents- MyTwitAgent.sources = Twitter MyTwitAgent.channels = MemChannel MyTwitAgent.sinks = HDFS MyTwitAgent.sources.Twitter.type = flume.mytwittersource.MyTwitterSourceForFlume MyTwitAgent.sources.Twitter.channels = MemChannel MyTwitAgent.sources.Twitter.consumerKey = MyTwitAgent.sources.Twitter.consumerSecret = MyTwitAgent.sources.Twitter.accessToken = MyTwitAgent.sources.Twitter.accessTokenSecret = MyTwitAgent.sources.Twitter.keywords = guru99 MyTwitAgent.sinks.HDFS.channel = MemChannel MyTwitAgent.sinks.HDFS.type = hdfs MyTwitAgent.sinks.HDFS.hdfs.path = hdfs://localhost:54310/user/hduser/flume/tweets/ MyTwitAgent.sinks.HDFS.hdfs.fileType = DataStream MyTwitAgent.sinks.HDFS.hdfs.writeFormat = Text MyTwitAgent.sinks.HDFS.hdfs.batchSize = 1000 MyTwitAgent.sinks.HDFS.hdfs.rollSize = 0 MyTwitAgent.sinks.HDFS.hdfs.rollCount = 10000 MyTwitAgent.channels.MemChannel.type = memory MyTwitAgent.channels.MemChannel.capacity = 10000 MyTwitAgent.channels.MemChannel.transactionCapacity = 1000 [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM24.png) **Step 2)** Also, set **TwitterAgent.sinks.HDFS.hdfs.path** as below, **TwitterAgent.sinks.HDFS.hdfs.path = hdfs://://flume/tweets/** [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM25.png) To know ****, **** and **** , see value of parameter **‘fs.defaultFS’** set in **$HADOOP\_HOME/etc/hadoop/core-site.xml** [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM26.png) **Step 3)** In order to flush the data to HDFS, as an when it comes, delete below entry if it exists, **TwitterAgent.sinks.HDFS.hdfs.rollInterval = 600** ## Example: Streaming Twitter Data using Flume **Step 1)** Open **‘flume-env.sh’** in write mode and set values for below parameters, JAVA_HOME= FLUME_CLASSPATH="/lib/MyTwitterSourceForFlume.jar" [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM27.png) **Step 2)** Start Hadoop $HADOOP_HOME/sbin/start-dfs.sh $HADOOP_HOME/sbin/start-yarn.sh **Step 3)** Two of the JAR files from the Flume tarball are not compatible with Hadoop 2.2.0\. So, we will need to follow below steps in this Apache Flume example to make Flume compatible with Hadoop 2.2.0. **a.** Move **protobuf-java-2.4.1.jar** out of **‘/lib’.** Go to **‘/lib’** **cd /lib** sudo mv protobuf-java-2.4.1.jar ~/ [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM28.png) **b.** Find for JAR file ‘guava’ as below find . -name "guava*" [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM29.png) Move **guava-10.0.1.jar** out of **‘/lib’.** sudo mv guava-10.0.1.jar ~/ [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM30.png) **c.** Download **guava-17.0.jar** from [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM31.png) Now, copy this downloaded jar file to **‘/lib’** **Step 4)** Go to **‘/bin’** and start Flume as- ./flume-ng agent -n MyTwitAgent -c conf -f /conf/flume.conf [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM32.png) Command prompt window where flume is fetching Tweets- [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM33.png) From command window message we can see that the output is written to **/user/hduser/flume/tweets/** directory. Now, open this directory using a web browser. **Step 5)** To see the result of data load, using a browser open **http://localhost:50070/** and browse the file system, then go to the directory where data has been loaded, that is- **/flume/tweets/** [](https://www.guru99.com/images/Big%5FData/061114%5F1213%5FHANDSONFLUM34.png) #### Summarize this post with: ChatGPT Perplexity Grok Google AI **Stay Updated on AI** **Get Weekly AI Skills, Trends, Actionable Advice.** ##### Sign up for the newsletter Subscribe for Free You have successfully subscribed. Please check your inbox. ![AI-Newsletter]() Chosen by over **350,000+** professionals [Scroll to top ](#wrapper)Scroll to top × Toggle Menu Close Search for: Search ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://www.guru99.com/#organization","name":"Guru99","sameAs":["https://www.facebook.com/Guru99Official","https://twitter.com/guru99com"],"logo":{"@type":"ImageObject","@id":"https://www.guru99.com/#logo","url":"https://www.guru99.com/images/guru99-logo-v1-150x59.png","contentUrl":"https://www.guru99.com/images/guru99-logo-v1-150x59.png","caption":"Guru99","inLanguage":"en-US"}},{"@type":"WebSite","@id":"https://www.guru99.com/#website","url":"https://www.guru99.com","name":"Guru99","publisher":{"@id":"https://www.guru99.com/#organization"},"inLanguage":"en-US"},{"@type":"ImageObject","@id":"https://www.guru99.com/images/flume-program.png","url":"https://www.guru99.com/images/flume-program.png","width":"355","height":"200","inLanguage":"en-US"},{"@type":"BreadcrumbList","@id":"https://www.guru99.com/create-your-first-flume-program.html#breadcrumb","itemListElement":[{"@type":"ListItem","position":"1","item":{"@id":"https://www.guru99.com","name":"Home"}},{"@type":"ListItem","position":"2","item":{"@id":"https://www.guru99.com/bigdata","name":"Big Data"}},{"@type":"ListItem","position":"3","item":{"@id":"https://www.guru99.com/create-your-first-flume-program.html","name":"Apache Flume Tutorial: What is, Architecture & Hadoop Example"}}]},{"@type":"WebPage","@id":"https://www.guru99.com/create-your-first-flume-program.html#webpage","url":"https://www.guru99.com/create-your-first-flume-program.html","name":"Apache Flume Tutorial: What is, Architecture & Hadoop Example","dateModified":"2024-06-13T17:23:07+05:30","isPartOf":{"@id":"https://www.guru99.com/#website"},"primaryImageOfPage":{"@id":"https://www.guru99.com/images/flume-program.png"},"inLanguage":"en-US","breadcrumb":{"@id":"https://www.guru99.com/create-your-first-flume-program.html#breadcrumb"}},{"@type":"Person","@id":"https://www.guru99.com/author/prafulla","name":"Prafulla Ranadive","description":"I am Prafulla Ranadive, a seasoned professional in Big Data, specializing in crafting comprehensive guides to help you master your data analysis and management skills.","url":"https://www.guru99.com/author/prafulla","image":{"@type":"ImageObject","@id":"https://www.guru99.com/images/prafulla-ranadive-author-120x120.png","url":"https://www.guru99.com/images/prafulla-ranadive-author-120x120.png","caption":"Prafulla Ranadive","inLanguage":"en-US"},"worksFor":{"@id":"https://www.guru99.com/#organization"}},{"@type":"NewsArticle","headline":"Apache Flume Tutorial: What is, Architecture & Hadoop Example","keywords":"bigdata","dateModified":"2024-06-13T17:23:07+05:30","articleSection":"Big Data","author":{"@id":"https://www.guru99.com/author/prafulla","name":"Prafulla Ranadive"},"publisher":{"@id":"https://www.guru99.com/#organization"},"description":"Apache Flume is a system used for moving massive quantities of streaming data into HDFS. We will Stream Twitter data using flume-ng command flume agent.","copyrightYear":"2024","copyrightHolder":{"@id":"https://www.guru99.com/#organization"},"name":"Apache Flume Tutorial: What is, Architecture & Hadoop Example","@id":"https://www.guru99.com/create-your-first-flume-program.html#richSnippet","isPartOf":{"@id":"https://www.guru99.com/create-your-first-flume-program.html#webpage"},"image":{"@id":"https://www.guru99.com/images/flume-program.png"},"inLanguage":"en-US","mainEntityOfPage":{"@id":"https://www.guru99.com/create-your-first-flume-program.html#webpage"}}]} ```