HDFS fsck command

$ hdfs fsck /orc-hive-staging/<dir> -files -blocks -locations

Status: HEALTHY
 Total size:    5041073 B
 Total dirs:    24
 Total files:   430
 Total symlinks:                0
 Total blocks (validated):      415 (avg. block size 12147 B)
 Minimally replicated blocks:   415 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       0 (0.0 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    2
 Average block replication:     2.0
 Corrupt blocks:                0
 Missing replicas:              0 (0.0 %)
 Number of data-nodes:          3
 Number of racks:               1
FSCK ended at Thu Sep 21 10:19:04 IST 2017 in 12 milliseconds
Other options available:
Usage: hdfs fsck <path> [-list-corruptfileblocks | [-move | -delete | -openforwrite] [-files [-blocks [-locations | -racks]]]] [-includeSnapshots] [-storagepolicies] [-blockId <blk_Id>]
        <path>  start checking from this path
        -move   move corrupted files to /lost+found
        -delete delete corrupted files
        -files  print out files being checked
        -openforwrite   print out files opened for write
        -includeSnapshots       include snapshot data if the given path indicates a snapshottable directory or there are snapshottable directories under it
        -list-corruptfileblocks print out list of missing blocks and files they belong to
        -blocks print out block report
        -locations      print out locations for every block
        -racks  print out network topology for data-node locations
        -storagepolicies        print out storage policy summary for the blocks
        -blockId        print out which file this blockId belongs to, locations (nodes, racks) of this block, and other diagnostics info (under replicated, corrupted or not, etc)
        -replicaDetails print out each replica details
If any under block replication error:
$ hdfs fsck / | grep 'Under replicated'

/user/hdfs/.staging/job_1505283552010_0142/job.jar:  Under replicated BP-1499440246-192.168.3.166-1504355970266:blk_1076326609_2585951. Target Replicas is 10 but found 3 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
.
/user/hdfs/.staging/job_1505283552010_0142/job.split:  Under replicated BP-1499440246-192.168.3.166-1504355970266:blk_1076326610_2585952. Target Replicas is 10 but found 3 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
..
...
/user/hdfs/.staging/job_1505283552010_0142/libjars/hive-hcatalog-core.jar:  Under replicated BP-1499440246-192.168.3.166-1504355970266:blk_1076326608_2585950. Target Replicas is 10 but found 3 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).

For the above files set replication to 2 manually:
$ hadoop fs -setrep 2 /user/hdfs/.staging/job_1505283552010_0142/job.jar
Replication 2 set: /user/hdfs/.staging/job_1505283552010_0142/job.jar

$ hadoop fs -setrep 2 /user/hdfs/.staging/job_1505283552010_0142/job.split
Replication 2 set: /user/hdfs/.staging/job_1505283552010_0142/job.split

$ hadoop fs -setrep 2 /user/hdfs/.staging/job_1505283552010_0142/libjars/hive-hcatalog-core.jar
Replication 2 set: /user/hdfs/.staging/job_1505283552010_0142/libjars/hive-hcatalog-core.jar
Now try the fsck command
$ hdfs fsck / | grep " under replicated" | wc -l
0

0 Comments

There are no comments yet

Leave a comment

Your email address will not be published. Required fields are marked *