Eu apenas executei isso em uma VM (Kubuntu 16.04) com a versão hadoop 2.7.3.
Pré-requisitos:
- Ubuntu OS
- Usuário não raiz com privilégios de sudo
- Java instalado
Etapas:
-
Faça o download da versão hadoop de aqui por
- Clicando na opção binária na coluna tarball da versão desejada
-
Clique no link abaixo Sugerimos o seguinte site espelho para o seu download: ou clique direito e salve o link
-
se o link foi salvo, faça o download usando:
wget http://www-us.apache.org/dist/hadoop/common/hadoop-2.7.3/hadoop-2.7.3.tar.gz
-
Verifiqueaintegridadedodownload:
Façaodownloaddoarquivo
.md5dodownload
Execute
shasum-a256hadoop-2.7.3.tar.gzecompareoresultadocomcathadoop-2.7.3.tar.gz.mds,nestalinha...hadoop-2.7.3.tar.gz:SHA256=D489DF3808244B906EB38F4D081BA49E50C4603DB03EFD5E594A1E98B09259C2...
Instaleohadoop:
Descomprimaemova:
tar-xzvfhadoop-2.7.3.tar.gz&&sudomvhadoop-2.7.3/usr/local/hadooplocalizeojavanoseusistemacom:
readlink-f/usr/bin/java|sed"s:bin/java::" # result /usr/lib/jvm/java-8-openjdk-amd64/jre/-
edite o
hadoop-env.shfilesudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh:-
Opção 1 (valor estático):
#export JAVA_HOME=${JAVA_HOME} export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/jre/ -
Opção 2 (alterações no valor dinâmico com alteração no java no sistema):
#export JAVA_HOME=${JAVA_HOME} export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::")
-
-
Executar hadoop:
-
/usr/local/hadoop/bin/hadoop-
Saída:
Usage: hadoop [--config confdir] [COMMAND | CLASSNAME] CLASSNAME run the class named CLASSNAME or where COMMAND is one of: fs run a generic filesystem user client version print the version jar <jar> run a jar file note: please use "yarn jar" to launch YARN applications, not this command. checknative [-a|-h] check native hadoop and compression libraries availability distcp <srcurl> <desturl> copy file or directories recursively archive -archiveName NAME -p <parent path> <src>* <dest> create a hadoop archive classpath prints the class path needed to get the credential interact with credential providers Hadoop jar and the required libraries daemonlog get/set the log level for each daemon
-
-
-
Teste:
mkdir ~/input cp /usr/local/hadoop/etc/hadoop/*.xml ~/input /usr/local/hadoop/bin/hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar grep ~/input ~/grep_example 'principal[.]*';-
Verifique se há erros no resultado:
File System Counters FILE: Number of bytes read=1247674 FILE: Number of bytes written=2324248 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=2 Map output records=2 Map output bytes=37 Map output materialized bytes=47 Input split bytes=114 Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=47 Reduce input records=2 Reduce output records=2 Spilled Records=4 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=61 Total committed heap usage (bytes)=263520256 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=151 File Output Format Counters Bytes Written=37
-
Fonte:
Adicione o caminho para .bashrc:
# hadoop executable
export PATH="/usr/local/hadoop/bin:$PATH"