HDFS

> Prefer using full path of the hadoop like /opt/software/hadoop-2.7.0/bin/hdfs

In Hadoop bin folder, we do see hadoop and hdfs. If you are using hadoop, we see following request

SDS-bash3.2$ hadoop dfs
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
...
...

basic syntax

hdfs fs [generic options]

    [-help [cmd ...]]
    [-ls [-d] [-h] [-R] [<path> ...]]
    [-mkdir [-p] <path> ...]


    [-chgrp [-R] GROUP PATH...]
    [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
    [-chown [-R] [OWNER][:[GROUP]] PATH...]

    [-copyFromLocal [-f] [-p] [-l] ... ]
    [-copyToLocal [-p] [-ignoreCrc] [-crc] ... ]

    [-moveFromLocal ... ]
    [-moveToLocal ]
    [-mv ... ]

    [-df [-h] [<path> ...]]
    [-du [-s] [-h] <path> ...]

    [-rm [-f] [-r|-R] [-skipTrash] ...]
    [-rmdir [--ignore-fail-on-non-empty]

Examples

/opt/software/hadoop-2.7.0/bin/hdfs fs -get /hdfs/source/path /tmp/sam/new_folder
/opt/software/hadoop-2.7.0/bin/hdfs fs -copyToLocal /hdfs/source/path /tmp/sam/new_folder

copy files from local

bin/hadoop fs -put /localfs/destination/path /hdfs/source/path
bin/hadoop fs -copyFromLocal /localfs/destination/path /hdfs/source/path 

copy files to local

bin/hadoop fs -get /hdfs/source/path /localfs/destination/path
bin/hadoop fs -copyToLocal /hdfs/source/path /localfs/destination/path
Advertisements