Big Data Testing Tutorial: What is, Strategy, How to test
Prafulla Ranadive
⚡ Smart Summary
Big Data Testing verifies that a big data application processes terabytes of data correctly, quickly and securely, combining data staging validation, MapReduce validation and output validation with architecture and performance checks across a distributed Hadoop cluster.
What is Big Data Testing?
Big Data Testing is a testing process of a big data application in order to ensure that all the functionalities of a big data application work as expected. The goal of Big Data Testing is to make sure that the big data system runs smoothly and error-free while maintaining the performance and security.
Big data is a collection of large datasets that cannot be processed using traditional computing techniques. Testing of these datasets involves various tools, techniques, and frameworks to process. Big data relates to data creation, storage, retrieval and analysis that is remarkable in terms of volume, variety, and velocity. You can learn more about Big Data, Hadoop and MapReduce before you start testing.
What is Big Data Testing Strategy?
Testing a Big Data application is more verification of its data processing rather than testing the individual features of the software product. When it comes to Big Data Testing, performance and functional testing are the keys.
In a Big Data Testing strategy, QA engineers verify the successful processing of terabytes of data using a commodity cluster and other supportive components. It demands a high level of testing skills as the processing is very fast. Processing may be of three types:
- Batch processing: Stored data is processed on a schedule, so tests target job completion and accuracy.
- Real-time processing: Records are processed on arrival, so tests target latency and data loss.
- Interactive processing: Analysts query directly, so tests target ad hoc query response time.
The diagram below summarises the strategy.
Along with this, data quality is also an important factor in Hadoop testing. Before testing the application, it is necessary to check the quality of data, and this should be considered as a part of database testing. It involves checking various characteristics like conformity, accuracy, duplication, consistency, validity, data completeness, etc. Next in this Hadoop Testing tutorial, we will learn how to test Hadoop applications.
How to Test Hadoop Applications
The following figure gives a high-level overview of the phases in testing Big Data applications.
Big Data Testing, or Hadoop testing, can be broadly divided into three steps.
Step 1: Data Staging Validation
The first step in this Big Data Testing tutorial is referred to as the pre-Hadoop stage, and it involves process validation.
- Data from various sources like RDBMS, weblogs, social media, etc. should be validated to make sure that correct data is pulled into the system
- Comparing source data with the data pushed into the Hadoop system to make sure they match
- Verify the right data is extracted and loaded into the correct HDFS location
Tools like Talend and Datameer can be used for data staging validation.
Step 2: “MapReduce” Validation
The second step is a validation of “MapReduce”. In this stage, the Big Data tester verifies the business logic validation on every node and then validates it after running against multiple nodes, ensuring that:
- MapReduce process works correctly
- Data aggregation or segregation rules are implemented on the data
- Key value pairs are generated
- Validating the data after the MapReduce process
Step 3: Output Validation Phase
The final or third stage of Hadoop testing is the output validation process. The output data files are generated and ready to be moved to an EDW (Enterprise Data Warehouse) or any other system based on the requirement.
Activities in the third stage include:
- To check the transformation rules are correctly applied
- To check the data integrity and successful data load into the target system
- To check that there is no data corruption by comparing the target data with the HDFS file system data
Architecture Testing
Attention now moves from the data to the cluster carrying it.
Hadoop processes very large volumes of data and is highly resource intensive. Hence, architectural testing is crucial to ensure the success of your Big Data project. A poorly or improperly designed system may lead to performance degradation, and the system could fail to meet the requirement. At a minimum, performance and failover test services should be run in a Hadoop environment.
Performance testing includes testing of job completion time, memory utilization, data throughput, and similar system metrics. The motive of the failover test service is to verify that data processing occurs seamlessly in case of failure of data nodes.
Performance Testing
Performance Testing for Big Data covers three main areas.
- Data ingestion and throughput: In this stage, the Big Data tester verifies how fast the system can consume data from various data sources. Testing involves identifying the number of messages that the queue can process in a given time frame. It also includes how quickly data can be inserted into the underlying data store, for example the insertion rate into a MongoDB and Cassandra database.
- Data processing: It involves verifying the speed with which the queries or MapReduce jobs are executed. It also includes testing the data processing in isolation when the underlying data store is populated within the data sets. For example, running MapReduce jobs on the underlying HDFS.
- Sub-component performance: These systems are made up of multiple components, and it is essential to test each of these components in isolation. For example, how quickly the message is indexed and consumed, MapReduce jobs, query performance, search, etc.
Performance Testing Approach
Performance testing for a Big Data application involves testing of huge volumes of structured and unstructured data, and it requires a specific testing approach to test such massive data.
The workflow below shows the sequence a performance test follows.
Performance Testing is executed in this order.
- The process begins with the setting up of the Big Data cluster which is to be tested for performance
- Identify and design corresponding workloads
- Prepare individual clients (custom scripts are created)
- Execute the test and analyze the results (if objectives are not met, then tune the component and re-execute)
- Optimum configuration
Parameters for Performance Testing
Various parameters to be verified for performance testing are:
- Data storage: How data is stored in different nodes
- Commit logs: How large the commit log is allowed to grow
- Concurrency: How many threads can perform write and read operations
- Caching: Tune the cache settings “row cache” and “key cache”
- Timeouts: Values for connection timeout, query timeout, etc.
- JVM parameters: Heap size, GC collection algorithms, etc.
- MapReduce performance: Sorts, merge, etc.
- Message queue: Message rate, size, etc.
Test Environment Needs
Test environment needs depend on the type of application you are testing. For Big Data software testing, the test environment should encompass the following.
- It should have enough space for storage and processing of a large amount of data
- It should have a cluster with distributed nodes and data
- It should have minimum CPU and memory utilization to keep performance high to test Big Data performance
Big Data Testing vs. Traditional Database Testing
The table below contrasts the two disciplines property by property.
| Properties | Traditional database testing | Big Data Testing |
|---|---|---|
| Data | Tester works with structured data | Tester works with both structured as well as unstructured data |
| Testing Approach | Testing approach is well defined and time-tested | The testing approach requires focused R&D efforts |
| Testing Strategy | Tester has the option of the “Sampling” strategy done manually or the “Exhaustive Verification” strategy by an automation tool | “Sampling” strategy in Big Data is a challenge |
| Infrastructure | It does not require a special test environment as the file size is limited | It requires a special test environment due to large data size and files (HDFS) |
| Validation Tools | Tester uses either Excel-based macros or UI based automation tools | No defined tools; the range is vast, from programming tools like MapReduce to HiveQL |
| Testing Tools | Testing tools can be used with basic operating knowledge and less training | It requires a specific set of skills and training to operate a testing tool. Also, the tools are in their nascent stage and over time they may come up with new features. |
Tools Used in Big Data Scenarios
The table below groups the common tools by cluster layer.
| Big Data Cluster | Big Data Tools |
|---|---|
| NoSQL: | CouchDB, Databases MongoDB, Cassandra, Redis, ZooKeeper, HBase |
| MapReduce: | Hadoop, Hive, Pig, Cascading, Oozie, Kafka, S4, MapR, Flume |
| Storage: | S3, HDFS (Hadoop Distributed File System) |
| Servers: | Elastic, Heroku, Google App Engine, EC2 |
| Processing: | R, Yahoo! Pipes, Mechanical Turk, BigSheets, Datameer |
Challenges in Big Data Testing
Three practical obstacles recur on almost every Big Data project.
- Automation: Automation testing for Big Data requires someone with technical expertise. Also, automated tools are not equipped to handle unexpected problems that arise during testing.
- Virtualization: It is one of the integral phases of testing. Virtual machine latency creates timing problems in real-time Big Data performance testing. Also, managing images in Big Data is a hassle.
- Large dataset: Three pressures come with volume.
- Need to verify more data and need to do it faster
- Need to automate the testing effort
- Need to be able to test across different platforms
Performance testing challenges
- Diverse set of technologies: Each sub-component belongs to a different technology and requires testing in isolation
- Unavailability of specific tools: No single tool can perform the end-to-end testing. For example, NoSQL might not fit for message queues
- Test scripting: A high degree of scripting is needed to design test scenarios and test cases
- Test environment: It needs a special test environment due to the large data size
- Monitoring solution: Limited solutions exist that can monitor the entire environment
- Diagnostic solution: A custom solution is required to drill down into the performance bottleneck areas
