What is Hive? Architecture & Modes

By : David Carter David Carter
Hours Updated

⚡ Smart Summary

Hive is the SQL layer of the Hadoop ecosystem, turning HiveQL statements into distributed jobs that read structured data already sitting on HDFS, so analysts query petabyte tables without writing MapReduce code themselves.

  • 🐝 What it is: An ETL and data-warehousing tool that adds tables, rows and columns on top of files stored in HDFS.
  • 🗂️ Metastore: Schema information lives in a relational metastore, backed by Derby for one user and by MySQL for shared access.
  • 🆚 Against an RDBMS: Hive validates schema on read, scales horizontally, and favours large scans over the row-level updates a database handles.
  • 🏗️ Three layers: Clients, services and storage form the architecture, with the driver coordinating compiler, metastore and execution engine.
  • 🔀 Two modes: Local mode suits a single data node and small files, while MapReduce mode spreads execution across a multi-node cluster.
  • 🔌 HiveServer2: The Thrift-based HS2 interface adds multi-client concurrency and authentication for remote sessions.
Read More

Hive architecture and modes

What is Hive?

Hive is an ETL and data warehousing tool developed on top of the Hadoop Distributed File System (HDFS). Hive makes the job easy for performing operations like

  • Data encapsulation
  • Ad-hoc queries
  • Analysis of huge datasets

Important characteristics of Hive

The points below explain what separates Hive from a plain MapReduce program.

  • In Hive, tables and databases are created first and then data is loaded into these tables.
  • Hive is a data warehouse designed for managing and querying only structured data that is stored in tables.
  • While dealing with structured data, MapReduce does not have optimization and usability features like UDFs, but the Hive framework does. Query optimization refers to an effective way of query execution in terms of performance.
  • Hive’s SQL-inspired language separates the user from the complexity of MapReduce programming. It reuses familiar concepts from the relational database world, such as tables, rows, columns and schema, for ease of learning.
  • Hadoop’s programming works on flat files. So, Hive can use directory structures to “partition” data to improve performance on certain queries.
  • An important component of Hive is the metastore, used for storing schema information. This metastore typically resides in a relational database. We can interact with Hive using methods like
    • Web GUI
    • Java Database Connectivity (JDBC) interface
  • Most interactions tend to take place over a command line interface (CLI). Hive provides a CLI to write Hive queries using Hive Query Language (HQL).
  • Generally, HQL syntax is similar to the SQL syntax that most data analysts are familiar with. The sample query below displays all the records present in the mentioned table name.
    • Sample query : Select * from <TableName>
  • Hive supports four file formats, those are TEXTFILE, SEQUENCEFILE, ORC and RCFILE (Record Columnar File).
  • For single-user metadata storage, Hive uses the Derby database, and for multiple-user or shared metadata Hive uses MySQL.

For setting up MySQL as the database and to store metadata information, check the tutorial on configuring the Hive metastore with MySQL, which follows on from the Hive installation guide.

Some of the key points about Hive:

  • The major difference between HQL and SQL is that a Hive query executes on Hadoop’s infrastructure rather than on a traditional database.
  • Hive query execution is a series of automatically generated MapReduce jobs.
  • Hive supports partition and bucket concepts for easy retrieval of data when the client executes the query.
  • Hive supports custom UDFs (user-defined functions) for data cleansing, filtering and similar work. According to the requirements of the programmers, one can define Hive UDFs.

Hive Vs Relational Databases

Hive performs functions that relational databases cannot. When data runs to petabytes, returning results in seconds matters, and Hive does this efficiently. The key differences are the following.

Relational databases are of “schema on read and schema on write”: first a table is created, then data is inserted into that table. On relational database tables, functions like insertions, updates and modifications can be performed.

Hive is “schema on read only”. So functions like update and modification did not work in early releases, because a Hive query in a typical cluster runs across multiple DataNodes, which made it impossible to update and modify data across multiple nodes (Hive versions below 0.13).

Hive also supports the “read many, write once” pattern, so in recent versions a table can be updated after insertion.

NOTE: Newer versions of Hive come with updated features. Hive 0.14 introduced UPDATE and DELETE as new features, and later releases added full ACID transaction support.

The table below condenses the comparison.

Aspect Relational database Apache Hive
Schema validation On write On read
Typical data volume Gigabytes to terabytes Terabytes to petabytes
Workload Row-level OLTP reads and writes Full-scan batch analytics
Query language SQL HQL, an SQL-inspired dialect
Execution Database engine Distributed cluster jobs

RELATED ARTICLES

Hive Architecture

The diagram below explains the Apache Hive architecture in detail.

Apache Hive architecture diagram showing clients, services, storage and computing

Hive consists of mainly 3 core parts:

  1. Hive clients
  2. Hive services
  3. Hive storage and computing

Hive Clients

Hive provides different drivers for communication with different types of applications. For Thrift-based applications, it provides a Thrift client for communication.

For Java related applications, it provides JDBC drivers. For any other type of application it provides ODBC drivers. These clients and drivers in turn communicate with the Hive server in the Hive services.

Hive Services

Client interactions with Hive are performed through Hive services. If the client wants to perform any query-related operation in Hive, it has to communicate through Hive services.

The CLI acts as the Hive service for DDL operations. All drivers communicate with the Hive server and the main driver in Hive services, as shown in the architecture diagram above.

The driver in Hive services is the main driver: it communicates with JDBC, ODBC and other client-specific applications, then passes their requests to the metastore and the file system for further processing.

Hive Storage and Computing

Hive services such as the metastore, the file system and the job client in turn communicate with Hive storage and perform the following actions:

  • Metadata information about tables created in Hive is stored in the Hive metastore database.
  • Query results and the data loaded into the tables are stored in the Hadoop cluster on HDFS.

Job execution flow

The diagram below traces one query from the user interface to the DataNodes and back.

Hive job execution flow diagram tracing a query from the user interface to the DataNodes

From the above diagram we can understand the job execution flow in Hive with Hadoop. The data flow in Hive behaves in the following pattern.

  1. Executing a query from the UI (user interface)
  2. The driver interacts with the compiler to get the plan. (Here plan refers to the query execution process and its related metadata information gathering.)
  3. The compiler creates the plan for the job to be executed. The compiler communicates with the metastore to get the metadata request.
  4. The metastore sends metadata information back to the compiler
  5. The compiler communicates with the driver with the proposed plan to execute the query
  6. The driver sends execution plans to the execution engine
  7. The execution engine (EE) acts as a bridge between Hive and Hadoop to process the query, including DFS operations:
    • The EE first contacts the NameNode and then the DataNodes to get the values stored in tables.
    • The EE fetches the desired records from the DataNodes. The actual table data resides on the data nodes only; from the NameNode it fetches only metadata for the query.
    • It collects actual data from the data nodes related to the mentioned query
    • The EE communicates bi-directionally with the metastore to perform DDL (data definition language) operations such as CREATE, DROP and ALTER on tables and databases. The metastore stores database, table and column names only.
    • The EE in turn communicates with Hadoop daemons such as the NameNode, DataNodes and job tracker to execute the query on top of the Hadoop file system
  1. Fetching results from the driver
  2. Sending results to the execution engine. Once the results are fetched from the data nodes to the EE, it sends the results back to the driver and to the UI (front end)

Hive stays in contact with the Hadoop file system and its daemons via the execution engine. The dotted arrow in the diagram shows that communication.

Different modes of Hive

Hive can operate in two modes depending on the size of the data nodes in Hadoop. These modes are:

  • Local mode
  • MapReduce mode

When to use local mode

  • If Hadoop is installed under pseudo-distributed mode with one data node, we use Hive in this mode
  • If the data size is small enough to be limited to a single local machine, we can use this mode
  • Processing will be very fast on smaller data sets present in the local machine

When to use MapReduce mode

  • If Hadoop has multiple data nodes and data is distributed across the different nodes, we use Hive in this mode
  • It performs on large amounts of data and the query executes in parallel
  • Processing of large data sets with better performance can be achieved through this mode

In Hive we can set a property to state which mode Hive should work in. By default it works in MapReduce mode, and for local mode you can use the following setting:

SET mapred.job.tracker=local;

From Hive version 0.7 onward it supports a mode that runs MapReduce jobs in local mode automatically. On Hive 4.x the default engine is Apache Tez, so this property applies to the legacy MapReduce path.

What is Hive Server2 (HS2)?

HiveServer2 (HS2) is a server interface that performs the following functions:

  • Enables remote clients to execute queries against Hive
  • Retrieves the results of those queries

Current versions add advanced features based on Thrift RPC, such as:

  • Multi-client concurrency
  • Authentication

HS2 sits behind Beeline and every JDBC or ODBC session, which is why it replaced the single-user Hive CLI. The HiveQL operators and built-in functions reference is the natural next step.

FAQs

Hive is a batch query layer that scans large tables through distributed jobs. HBase is a NoSQL store built for millisecond random reads and writes on single rows. They solve opposite access patterns and often run side by side.

Hive runs on MapReduce, Apache Tez or Apache Spark. MapReduce is deprecated and Hive 4.x defaults to Tez, which keeps intermediate results in memory instead of writing them to disk between stages.

A managed table gives Hive ownership of metadata and files, so dropping it deletes the data from the warehouse directory. An external table stores only metadata, and dropping it leaves the underlying files untouched.

Learned cost models feed execution statistics back into the planner to predict scan volume, join order and partition pruning more accurately than static estimates. Cloud platforms surface the same signals as compaction and partition-layout recommendations.

Copilot drafts HiveQL joins, window functions and Java UDF skeletons from a comment, which shortens boilerplate work. Column names, partition keys and data types still need checking against the real metastore first.

Yes. Hive 0.14 added UPDATE and DELETE, and later releases delivered full ACID support for transactional tables. Hive 4.x extends this through Apache Iceberg tables with snapshot isolation and schema evolution.

All three expose SQL over the same files. Hive favours long batch jobs and owns the metastore the others read. Spark SQL suits mixed pipelines; Trino targets interactive queries across several sources.

Yes, largely through the Hive Metastore, which remains the catalog standard Spark, Trino, Presto and Flink all read. Hive itself still serves scheduled batch transformations and warehouse loads.

Summarize this post with: