Wednesday, February 06, 2013

HBase+HDFS: fsck showing Average Block Replication more than Default Replication Factor

At the first go, it looks very surprising when you get Average Block Replication more than Default Replication Factor, usually what we expect is it to be equal or momentarily less than default Replication Factor.

On a cluster,
  • I saw Average Block Replication as 3 where as Replication Factor was 2.
  • Moreover, to make it more interesting, Over-Replicated blocks were 0%.

After delving deeper we figured out that, replication factor on HDFS was 2, but
HBase didn't have any configuration defined, hence was getting defaulted to 3.

So, nothing was counted as over replicated blocks, as corresponding to those HBase tables the replication factor was 3.

It's not a problem as such, just a state of confusion, to avoid it you can put your hdfs-site.xml in HBase or configure replication factor in Hbase-site.xml

1 comment:

Deepak mundhada said...

Hi Mayank,

If I am developing HBase application in Java to make a client, Then what you recommend to use the raw HBase API or Thrift, REST, Avro, etc.