· 7 min read
Lessons from operating a small Hadoop data cluster
Practical failure modes and fixes across HDFS, YARN, Spark, ZooKeeper, HBase, and ClickHouse.
Loading post…
· 7 min read
Practical failure modes and fixes across HDFS, YARN, Spark, ZooKeeper, HBase, and ClickHouse.
Loading post…
Running a small data cluster is mostly an exercise in managing shared failure domains. HDFS, YARN, Spark, ZooKeeper, HBase, and ClickHouse may be separate systems, but on a small fleet they compete for the same memory, disks, hostnames, clocks, and maintenance windows.
These are the operating lessons I would carry into the next cluster. The examples are generic, but each failure mode came from a real debugging session.
A scheduler sees the resources a NodeManager advertises, not the RAM that a RegionServer or ClickHouse process will need ten minutes later. Advertising nearly all of a mixed-role host to YARN works until a Spark burst overlaps an HBase scan, a merge, or a control-plane failover.
I now start with a per-node budget:
| Role | Reserve first | Add only with a measured budget |
|---|---|---|
| Control plane | NameNode, ResourceManager, ZooKeeper, JournalNode | Small service workloads |
| Compute | NodeManager, Spark local disks | DataNode if disk and memory are reserved |
| HBase | RegionServer heap, off-heap use, OS cache | A bounded NodeManager |
| ClickHouse | Server memory, page cache, background merges | Lightweight ingestion services |
YARN node labels or separate queues can keep ordinary executors off stateful hosts. When co-location is unavoidable, make the rollback lever simple: stopping or shrinking the NodeManager should immediately return headroom to the database.
Keep the memory-to-vcore shape consistent across compute nodes. With dominant-resource scheduling, an oversized CPU or memory advertisement can strand the other dimension.
One of the nastiest failures I saw looked healthy in the ResourceManager UI. A new
NodeManager heartbeated successfully, so YARN marked it RUNNING and kept scheduling
ApplicationMasters to it. Every launch failed because the RPC listener had resolved
the hostname to 127.0.1.1 and bound only to loopback.
The fix was to correct host resolution and bind the service listener explicitly:
<property>
<name>yarn.nodemanager.bind-host</name>
<value>0.0.0.0</value>
</property>
<property>
<name>yarn.nodemanager.webapp.bind-host</name>
<value>0.0.0.0</value>
</property>
The advertised address should still be a stable hostname. After joining a node, test both directions:
getent hosts worker-03
ss -ltnp | grep -E ':45454|:8042'
yarn node -list
yarn jar hadoop-mapreduce-examples.jar pi 4 100
That last canary matters. Registration proves the outbound heartbeat path; a real container proves the inbound launch path, localization, permissions, Java runtime, and local disks.
Time synchronization belongs in the same checklist. Lease-based systems and cross-host logs are much harder to reason about when clocks disagree.
yarn rmadmin -updateNodeResource is useful for reducing a busy node without a
NodeManager restart. I learned not to mistake it for configuration management: after
a ResourceManager failover, a live override reverted and the node quietly advertised
its old capacity again.
For every live change:
The same rule applies to scheduler queues. Edit both ResourceManager hosts before
refreshing queues. Also remember that a queue's practical single-user ceiling can be
lower than maximum-capacity because capacity × user-limit-factor also applies.
Process-state monitoring missed a streaming outage because the relaunched Spark app
was RUNNING, its receivers were not receiving records, and the upstream service kept
consuming input files. Nothing crashed; fresh data simply stopped appearing.
There were two useful causes in separate incidents:
The good fix was not a larger timeout. It was to restore the receiver's minimum core budget, restart the owning ingestion service so it created a clean application, and monitor the timestamp of the newest stored event.
flowchart TD Input[Input files or socket] --> Engine[Ingestion engine] Engine --> Receiver[Spark receiver] Receiver --> Batch[Processing tasks] Batch --> Store[(Data store)] Store --> Freshness[Newest event age]
Each arrow needs an observation. Application state alone only checks the middle of the path. A freshness alert on the destination detects a pipeline that is alive but deaf.
For interactive PySpark, executor Python is another common trap. A packed virtual
environment may contain a symlink to a system interpreter that is absent on workers.
Use a genuinely relocatable environment or ensure every node has the matching Python,
ship it with YARN's archive mechanism, and set PYSPARK_PYTHON before creating the
Spark context:
export PYSPARK_PYTHON=./environment/bin/python
spark-submit --archives pyspark-env.tar.gz#environment job.py
The PySpark packaging guide describes the archive pattern and the limitations of packed virtual environments.
Removing a DataNode safely is a state transition, not a service stop:
stateDiagram-v2 [*] --> InService InService --> Decommissioning: add to exclude file and refresh Decommissioning --> Decommissioned: replicas restored Decommissioning --> InService: remove from exclude file and refresh Decommissioned --> Stopped: fsck healthy, then stop service
My runbook now follows this order:
dfsadmin -report and run hdfs fsck /.hdfs dfsadmin -refreshNodes.Decommissioned, then require a healthy fsck before stopping it.The official HDFS user guide documents that decommissioning completes only after replicas have been restored.
A cluster with hundreds of thousands of tiny blocks made this much slower than its data volume suggested. Decommission throughput was limited by replication scheduling and the source node, not network bandwidth. Carefully raising NameNode replication work limits improved it, but those values depend on NameNode heap and DataNode capacity. Change them gradually and monitor RPC queues, heap, block health, and application I/O.
Adding empty disks or using a capacity-aware placement policy affects new writes; it does not move existing blocks. Run the HDFS balancer separately, with a bandwidth cap and outside peak ingestion windows.
Host-file updates looked harmless until changing several ZooKeeper quorum members in parallel caused a ResourceManager to lose its session and fail over. An odd-sized quorum does not help if maintenance temporarily removes a majority.
For quorum and HA changes:
That “open port is not ready” distinction also appeared with JournalNodes. A restarted JournalNode was listening, but had not begun serving edits. Restarting the next member too quickly left the active NameNode without a writable quorum. The recovery worked, but waiting on the NameNode's view of JournalNode readiness would have avoided it.
An HBase LeaseException initially looked like an expired scanner and led toward ever
larger timeouts. Correlating the scanner ID with the RegionServer log showed a slow
scan, an overlapping request, and a long host pause—but no RegionServer restart or
region movement. The job continued afterward.
The more useful response was to correlate:
JvmPauseMonitor output and GC logs;Large timeouts can hide contention without removing it. If RegionServers share hosts with YARN or ClickHouse, reduce the competing workload first and observe whether the pauses disappear.
When moving a RegionServer, use HBase's graceful stop so regions drain before the process exits. Move one server at a time, require zero dead servers and zero stuck regions, then consider an off-hours major compaction to rebuild locality on the new DataNodes. The HBase reference guide describes the graceful restart path.
I encountered repeated MEMORY_LIMIT_EXCEEDED inserts while process RSS was far below
the server limit. The global tracker was charging allocator-retained mappings, which
spiked on a schedule even though active allocations and individual query peaks were
modest.
The diagnostic mistake would have been to raise every limit blindly. The useful comparison was:
MemoryTracking;system.query_log;After correcting the accounting behavior, I kept a measured per-query
max_memory_usage guard. Disabling one conservative global signal without adding a
replacement can turn a false positive into an out-of-memory kill. ClickHouse's
memory-limit guidance
also recommends sizing per-query limits from the actual query classes.
Background work deserves its own audit. In one case, system log tables were responsible for most merge CPU because their retention was effectively unbounded. Add realistic TTLs, and when changing a ClickHouse TTL through configuration, keep the server config and table metadata expression consistent. Otherwise a restart can treat the table definition as different and rename or recreate internal log tables.
For replicated tables, separate replica identity from host identity. Stable shard and
replica macros plus ZooKeeper paths make hostname changes a configuration update rather
than a data migration. Check system.replicas for lag and read-only replicas after
every restart.
The safest changes all ended up looking similar:
The main lesson is simple: “the service is running” is the beginning of cluster verification, not the end.
Comments
View on GitHub