HDFS Shell Commands

Category : Hadoop | Sub Category : HDFS Commands | By Prasad Bonam Last updated: 2020-10-05 07:00:26 Viewed : 706

---

    HDFS Shell Commands

1.               Get the list of Files and Directories in HDFS

Command: hdfs dfs –ls

Run and Result:

[root@dev user]# hdfs dfs -ls

Found 5 items

drwx------   - root    supergroup          0 2020-08-27 16:00 .Trash

drwxr-xr-x   - root    supergroup          0 2020-10-05 10:39 .sparkStaging

drwx------   - root    supergroup          0 2020-08-30 18:19 .staging

drwxr-xr-x   - root    supergroup          0 2020-04-01 16:14 Bkup_file

drwxrwxrw   - runnerdev supergroup   0 2020-04-03 14:17 runnerdebBatch

 

2.               Create the directory in HDFS.

Command: hdfs dfs –mkdir

Run and Result: [root@dev user]# hdfs dfs -mkdir /testfolder

3.                Create a file in HDFS

Command: hdfs dfs – touchz

Run and Result:

[root@dev user]#  hdfs dfs -touchz  /testfolder/testFile.txt

[root@dev user]# hdfs dfs -ls /testfolder/

Found 1 items

-rw-r--r--   3 root supergroup          0 2020-10-05 10:58 /testfolder/testFile.txt

4.               Check the file size in HDFS

Command: hdfs dfs –du -s

Run and Result:

[root@dev user]# hdfs dfs -du -s /testfolder/testFile.txt

0  0  /testfolder/testFile.txt

5.               Read a file in HDFS

Command: hdfs dfs –cat

Run and Result:

Example 1:

[root@dev user]# hdfs dfs -cat /user/admin/a.txt

{"event":{"msgenvelope":{"a":"a"},"msgBody":{"b":"b"}}}

Example2:

[root@dev user]#  hdfs dfs -cat /user/admin/testFile.txt

The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple ..

6.               Copy the file from  Local file system to HDFS

Command: hdfs dfs -copyFromLocal   <local source>   <hdfs destination>

Run and Result:

[root@dev user]# hdfs dfs -copyFromLocal /home/user/testLocalFolder/test.txt /user/admin/

7.                Copy single source or multiple sources from local file system to the destination file system

Command: hdfs dfs -put <local source>  <destination>

Run and Result:

[root@dev user]#   hdfs dfs –put /home/user/testLocalFolder /user/admin/

8.               Copy the file from  HDFS to  Local file system

Command: hdfs dfs - copyToLocal <hdfs source>   <local destination>

Run and Result:

[root@dev user]# hdfs dfs -copyToLocal /user/root/testfile.txt /home/admin/testLocalFolder/

9.               Copy files from hdfs to the local file system using get

Command: hdfs dfs -get <hdfs source> < local destination >

Run and Result:

[root@dev user]# hdfs dfs –get /user/root  /home/hadoop/

10.            Remove the file from HDFS

Command: hdfs dfs -rm  <path>

Run and Result:

[root@dev user]#    hdfs dfs –rm /user/root/testFile.txt 

11.               Remove  entire directory from HDFS

Command: hdfs dfs -r  <path>

Run and Result : [root@dev user]# hdfs dfs -rm -r  / user/hadoop /

(Or)

Command: hdfs dfs -rmdir <path>

Run and Result: [root@dev user]#  hdfs dfs –rmdir /user/ hadoop/

12.               Copy files from source to destination in HDFS

Command: hdfs dfs -cp <src> <dest>

Run and Result: [root@dev user]#   hdfs dfs -cp /user/hadoop/testFile.txt /user/hadoop/

13.               Move files from source to destination in HDFS

Command: hdfs dfs -cp <src> <dest>

Run and Result: [root@dev user]#   hdfs dfs -mv /user/hadoop/testFile.txt /user/hadoop/

14.               Get HDFS version

Command: hdfs version

Run and Result: [root@dev user]# hdfs version

Hadoop 2.6.0-cdh5.16.2

15.               Check the health of the Hadoop file system

Command: hdfs fsck

Run and Result:

Go!