Managing HDFS Directories

Now let us have a look at how to create directories and manage ownership.

• By default hdfs is superuser of HDFS
• hadoop fs -mkdir or hdfs dfs -mkdir – to create directories
• hadoop fs -chown or hdfs dfs -chown – to change ownership of files
• chown can also be used to change the group. We can change the group using -chgrp command as well. Make sure to run -help on chgrp and check the details.
• Here are the steps to create user space. Only users in HDFS group can take care of it.
  • Create directory with user id itversity under /user
  • Change ownership to the same name as the directory created earlier (/user/itversity)
  • You can validate permissions by using hadoop fs -ls or hdfs dfs -ls command on /user. Make sure to grep for the user name you are looking for.
• Let’s go ahead and create user space in HDFS for itversity. I have to login as sudoer and run below commands.

sudo -u hdfs hdfs dfs -mkdir /user/itversity
sudo -u hdfs hdfs dfs -chown -R itversity:students /user/itversity
hdfs dfs -ls /user|grep itversity

You should be able to create folders under your home directory.

hdfs dfs -ls /user/`whoami`
hdfs dfs -mkdir /user/`whoami`/retail_db
hdfs dfs -ls /user/`whoami`

• You can create the directory structure using mkdir -p. The existing folders will be ignored and non existing folders will be created.
  • Let us run hdfs dfs -mkdir -p /user/${USER}/retail_db/orders/year=2020.
  • As /user/${USER}/retail_db already exists, it will be ignored.
  • Both /user/${USER}/retail_db/orders as well as /user/${USER}/retail_db/orders/year=2020 will be created.

hdfs dfs -help mkdir

hdfs dfs -mkdir -p /user/`whoami`/retail_db/orders/year=2020

We can delete non empty directory using hdfs dfs -rm -R and empty directory using hdfs dfs -rmdir. We will explore hdfs dfs -rm in detail later.

hdfs dfs -rmdir /user/`whoami`/retail_db/orders/year=2020
hdfs dfs -rm -R /user/`whoami`/retail_db