Please see my other blog for Oracle EBusiness Suite Posts - EBMentors

Search This Blog

Note: All the posts are based on practical approach avoiding lengthy theory. All have been tested on some development servers. Please don’t test any post on production servers until you are sure.

Friday, June 23, 2017

Managing HDFS Quotas


The Hadoop Distributed File System (HDFS) allows the administrator to set quotas for the number of names used and the amount of space used for individual directories. Name quotas and space quotas operate independently, but the administration and implementation of the two types of quotas are closely parallel.

Name Quotas
The name quota is a hard limit on the number of file and directory names in the tree rooted at that directory. File and directory creations fail if the quota would be exceeded.

Space Quotas
The space quota is a hard limit on the number of bytes used by files in the tree rooted at that directory. Block allocations fail if the quota would not allow a full block to be written.
[hdpsysuser@nn01 ~]$ hdfs dfsadmin -setQuota 1 /userdata/inam

[hdpsysuser@nn01 ~]$ hdfs dfs -put /tmp/mydata/test3.txt /userdata/inam
put: The NameSpace quota (directories and files) of directory /userdata/inam is exceeded: quota=1 file count=3


[hdpsysuser@nn01 ~]$ hdfs dfsadmin -clrQuota /userdata/inam
[hdpsysuser@nn01 ~]$ hdfs dfs -put /tmp/mydata/test3.txt /userdata/inam


[hdpsysuser@nn01 ~]$ hdfs dfsadmin -setSpaceQuota 1m /userdata/inam

[hdpsysuser@nn01 ~]$ hdfs dfs -put /tmp/mydata/Tamm-Sample.txt /userdata/inam
17/05/07 15:36:09 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota of /userdata/inam is exceeded: quota = 1048576 B = 1 MB but diskspace consumed = 402653256 B = 384.00 MB

[hdpsysuser@nn01 ~]$ hdfs dfsadmin -clrSpaceQuota /userdata/inam
[hdpsysuser@nn01 ~]$ hdfs dfs -put /tmp/mydata/Tamm-Sample.txt /userdata/inam


[hdpsysuser@nn01 ~]$ hdfs dfs -count -q /userdata/inam
           1              -2            none             inf            1            2            2097176 /userdata/inam


No comments: