Eu quero usar o hadoop no ArchLinux, então eu tentei aqui ( link ), mas eu pude fazer isso .
Eu tentei ...
alteração de permissão de uso
$sudo chmod o+r /etc/hadoop && sudo chmod o+rw /tmp/hadoop/log ...
adicione este código em /etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk
altere o nome em /etc/conf.d/hadoop
HADOOP_USERNAME="<my user name>"
(Eu posso ssh para localhost sem uma senha, então eu não fiz nada para o ssh)
tente hadoop namenode -format
Eu tenho alguma mensagem WARM
WARNING: HADOOP_SLAVES has been replaced by HADOOP_WORKERS. Using value of HADOOP_SLAVES.
WARNING: Use of this script to execute namenode is deprecated.
WARNING: Attempting to execute replacement "hdfs namenode" instead.
Então, altero /etc/profile.d/hadoop.sh da seguinte forma
export HADOOP_CONF_DIR=/etc/hadoop
export HADOOP_LOG_DIR=/tmp/hadoop/log
export HADOOP_WORKERS=/etc/hadoop/workers
export HADOOP_PID_DIR=/tmp/hadoop/run
E eu faço hadoop namenode -format novamente (não consigo encontrar hdfs no meu shell)
SHUTDOWN meu hadoop Eu recebi essa mensagem
[<my user name>@<pc name> etc]$ hadoop namenode -format
WARNING: Use of this script to execute namenode is deprecated.
WARNING: Attempting to execute replacement "hdfs namenode" instead.
2017-12-29 05:08:01,679 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = <my user name>/10.0.2.15
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.0.0
STARTUP_MSG: classpath = ...
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r c25427ceca461ee979d30edd7a4b0f50718e6533; compiled by 'andrew' on 2017-12-08T19:16Z
STARTUP_MSG: java = 1.8.0_144
************************************************************/
2017-12-29 05:08:01,685 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2017-12-29 05:08:01,688 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-d2e0965e-76ff-456b-8552-9da5706b8a6b
2017-12-29 05:08:02,355 INFO namenode.FSEditLog: Edit logging is async:true
2017-12-29 05:08:02,389 INFO namenode.FSNamesystem: KeyProvider: null
2017-12-29 05:08:02,391 INFO namenode.FSNamesystem: fsLock is fair: true
2017-12-29 05:08:02,392 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: fsOwner = <my user name> (auth:SIMPLE)
2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: supergroup = supergroup
2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: isPermissionEnabled = true
2017-12-29 05:08:02,397 INFO namenode.FSNamesystem: HA Enabled: false
2017-12-29 05:08:02,452 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2017-12-29 05:08:02,471 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2017-12-29 05:08:02,471 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2017-12-29 05:08:02,486 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2017-12-29 05:08:02,486 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Dec 29 05:08:02
2017-12-29 05:08:02,488 INFO util.GSet: Computing capacity for map BlocksMap
2017-12-29 05:08:02,488 INFO util.GSet: VM type = 64-bit
2017-12-29 05:08:02,490 INFO util.GSet: 2.0% max memory 878.5 MB = 17.6 MB
2017-12-29 05:08:02,494 INFO util.GSet: capacity = 2^21 = 2097152 entries
2017-12-29 05:08:02,546 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2017-12-29 05:08:02,555 INFO Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS
2017-12-29 05:08:02,555 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2017-12-29 05:08:02,555 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2017-12-29 05:08:02,555 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: defaultReplication = 3
2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: maxReplication = 512
2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: minReplication = 1
2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: redundancyRecheckInterval = 3000ms
2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: encryptDataTransfer = false
2017-12-29 05:08:02,555 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
2017-12-29 05:08:02,625 INFO util.GSet: Computing capacity for map INodeMap
2017-12-29 05:08:02,626 INFO util.GSet: VM type = 64-bit
2017-12-29 05:08:02,626 INFO util.GSet: 1.0% max memory 878.5 MB = 8.8 MB
2017-12-29 05:08:02,626 INFO util.GSet: capacity = 2^20 = 1048576 entries
2017-12-29 05:08:02,627 INFO namenode.FSDirectory: ACLs enabled? false
2017-12-29 05:08:02,627 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2017-12-29 05:08:02,627 INFO namenode.FSDirectory: XAttrs enabled? true
2017-12-29 05:08:02,627 INFO namenode.NameNode: Caching file names occurring more than 10 times
2017-12-29 05:08:02,631 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true
2017-12-29 05:08:02,643 INFO util.GSet: Computing capacity for map cachedBlocks
2017-12-29 05:08:02,643 INFO util.GSet: VM type = 64-bit
2017-12-29 05:08:02,643 INFO util.GSet: 0.25% max memory 878.5 MB = 2.2 MB
2017-12-29 05:08:02,643 INFO util.GSet: capacity = 2^18 = 262144 entries
2017-12-29 05:08:02,649 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2017-12-29 05:08:02,654 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2017-12-29 05:08:02,654 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2017-12-29 05:08:02,670 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2017-12-29 05:08:02,670 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2017-12-29 05:08:02,685 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2017-12-29 05:08:02,685 INFO util.GSet: VM type = 64-bit
2017-12-29 05:08:02,685 INFO util.GSet: 0.029999999329447746% max memory 878.5 MB = 269.9 KB
2017-12-29 05:08:02,686 INFO util.GSet: capacity = 2^15 = 32768 entries
Re-format filesystem in Storage Directory /tmp/hadoop-elect/dfs/name ? (Y or N) Y
2017-12-29 05:08:04,846 INFO namenode.FSImage: Allocated new BlockPoolId: BP-142461510-10.0.2.15-1514491684838
2017-12-29 05:08:04,846 INFO common.Storage: Will remove files: [/tmp/hadoop-<my user name>/dfs/name/current/fsimage_0000000000000000000, /tmp/hadoop-<my user name>/dfs/name/current/fsimage_0000000000000000000.md5, /tmp/hadoop-<my user name>/dfs/name/current/seen_txid, /tmp/hadoop-<my user name>/dfs/name/current/VERSION]
2017-12-29 05:08:04,863 INFO common.Storage: Storage directory /tmp/hadoop-<my user name>/dfs/name has been successfully formatted.
2017-12-29 05:08:04,870 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-<my user name>/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
2017-12-29 05:08:04,981 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-<my user name>/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 387 bytes saved in 0 seconds.
2017-12-29 05:08:04,986 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2017-12-29 05:08:04,989 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at <my user name>/10.0.2.15
************************************************************/
iniciar os serviços use sudo systemctl start hadoop-datanode(jobtracker etc...)
verifique se os serviços usam sudo systemctl status hadoop-...
● hadoop-datanode.service - Hadoop Datanode
Loaded: loaded (/usr/lib/systemd/system/hadoop-datanode.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-12-29 05:29:15 JST; 53s ago
Process: 30450 ExecStart=/usr/bin/hadoop datanode (code=exited, status=1/FAILURE)
Main PID: 30450 (code=exited, status=1/FAILURE)
Dec 29 05:29:14 <my user name> hadoop[30450]: at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2728)
Dec 29 05:29:14 <my user name> hadoop[30450]: at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2872)
Dec 29 05:29:14 <my user name> hadoop[30450]: at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2896)
Dec 29 05:29:14 <my user name> hadoop[30450]: 2017-12-29 05:29:14,859 INFO util.ExitUtil: Exiting with status 1: java.io.IOException: Incorrect configuration: namenode address dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not configured.
Dec 29 05:29:14 <my user name> hadoop[30450]: 2017-12-29 05:29:14,865 INFO datanode.DataNode: SHUTDOWN_MSG:
Dec 29 05:29:14 <my user name> hadoop[30450]:
/************************************************************
Dec 29 05:29:14 <my user name> hadoop[30450]: SHUTDOWN_MSG: Shutting down DataNode at elect/10.0.2.15
Dec 29 05:29:14 <my user name> hadoop[30450]:
************************************************************/
Dec 29 05:29:15 elect systemd[1]: hadoop-datanode.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 05:29:15 <my user name> systemd[1]: hadoop-datanode.service: Failed with result 'exit-code'.
● hadoop-namenode.service - Hadoop Namenode
Loaded: loaded (/usr/lib/systemd/system/hadoop-namenode.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-12-29 05:29:13 JST; 10min ago
Process: 30449 ExecStart=/usr/bin/hadoop namenode (code=exited, status=1/FAILURE)
Main PID: 30449 (code=exited, status=1/FAILURE)
Dec 29 05:29:13 <my user name> hadoop[30449]: at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:929)
Dec 29 05:29:13 <my user name> hadoop[30449]: at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653)
Dec 29 05:29:13 <my user name> hadoop[30449]: at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
Dec 29 05:29:13 <my user name> hadoop[30449]: 2017-12-29 05:29:13,366 INFO util.ExitUtil: Exiting with status 1: java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
Dec 29 05:29:13 <my user name> hadoop[30449]: 2017-12-29 05:29:13,368 INFO namenode.NameNode: SHUTDOWN_MSG:
Dec 29 05:29:13 <my user name> hadoop[30449]:
/************************************************************
Dec 29 05:29:13 <my user name> hadoop[30449]: SHUTDOWN_MSG: Shutting down NameNode at <my user name>/10.0.2.15
Dec 29 05:29:13 <my user name> hadoop[30449]:
************************************************************/
Dec 29 05:29:13 <my user name> systemd[1]: hadoop-namenode.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 05:29:13 <my user name> systemd[1]: hadoop-namenode.service: Failed with result 'exit-code'.
● hadoop-jobtracker.service - Hadoop Jobtracker
Loaded: loaded (/usr/lib/systemd/system/hadoop-jobtracker.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-12-29 05:26:04 JST; 17min ago
Process: 29726 ExecStart=/usr/bin/hadoop jobtracker (code=exited, status=1/FAILURE)
Main PID: 29726 (code=exited, status=1/FAILURE)
Dec 29 05:26:04 <my user name> hadoop[29726]: queue get information regarding JobQueues
Dec 29 05:26:04 <my user name> hadoop[29726]: sampler sampler
Dec 29 05:26:04 <my user name> hadoop[29726]: streaming launch a mapreduce streaming job
Dec 29 05:26:04 <my user name> hadoop[29726]: version print the version
Dec 29 05:26:04 <my user name> hadoop[29726]: Daemon Commands:
Dec 29 05:26:04 <my user name> hadoop[29726]: tput: No value for $TERM and no -T specified
Dec 29 05:26:04 <my user name> systemd[1]: hadoop-jobtracker.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 05:26:04 <my user name> hadoop[29726]: historyserver run job history servers as a standalone daemon
Dec 29 05:26:04 <my user name> hadoop[29726]: SUBCOMMAND may print help when invoked w/o parameters or with -h.
Dec 29 05:26:04 <my user name> systemd[1]: hadoop-jobtracker.service: Failed with result 'exit-code'.
● hadoop-secondarynamenode.service - Hadoop Secondary Namenode
Loaded: loaded (/usr/lib/systemd/system/hadoop-secondarynamenode.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-12-29 05:25:59 JST; 20min ago
Process: 29433 ExecStart=/usr/bin/hadoop secondarynamenode (code=exited, status=1/FAILURE)
Main PID: 29433 (code=exited, status=1/FAILURE)
Dec 29 05:25:59 <my user name> hadoop[29433]: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:231)
Dec 29 05:25:59 <my user name> hadoop[29433]: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:194)
Dec 29 05:25:59 <my user name> hadoop[29433]: at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:690)
Dec 29 05:25:59 <my user name> hadoop[29433]: 2017-12-29 05:25:59,387 INFO util.ExitUtil: Exiting with status 1: ExitException
Dec 29 05:25:59 <my user name> hadoop[29433]: 2017-12-29 05:25:59,400 INFO namenode.SecondaryNameNode: SHUTDOWN_MSG:
Dec 29 05:25:59 <my user name> hadoop[29433]:
/************************************************************
Dec 29 05:25:59 <my user name> hadoop[29433]: SHUTDOWN_MSG: Shutting down
SecondaryNameNode at <my user name>/10.0.2.15
Dec 29 05:25:59 <my user name> hadoop[29433]:
************************************************************/
Dec 29 05:25:59 <my user name> systemd[1]: hadoop-secondarynamenode.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 05:25:59 <my user name> systemd[1]: hadoop-secondarynamenode.service: Failed with result 'exit-code'.
● hadoop-tasktracker.service - Hadoop Tasktracker
Loaded: loaded (/usr/lib/systemd/system/hadoop-tasktracker.service; disabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Fri 2017-12-29 05:26:04 JST; 23min ago
Process: 29727 ExecStart=/usr/bin/hadoop tasktracker (code=exited, status=1/FAILURE)
Main PID: 29727 (code=exited, status=1/FAILURE)
Dec 29 05:26:04 <my user name> hadoop[29727]: queue get information regarding JobQueues
Dec 29 05:26:04 <my user name> hadoop[29727]: sampler sampler
Dec 29 05:26:04 <my user name> hadoop[29727]: streaming launch a mapreduce streaming job
Dec 29 05:26:04 <my user name> hadoop[29727]: version print the version
Dec 29 05:26:04 <my user name> hadoop[29727]: Daemon Commands:
Dec 29 05:26:04 <my user name> hadoop[29727]: tput: No value for $TERM and no -T specified
Dec 29 05:26:04 <my user name> hadoop[29727]: historyserver run job history servers as a standalone daemon
Dec 29 05:26:04 <my user name> hadoop[29727]: SUBCOMMAND may print help when invoked w/o parameters or with -h.
Dec 29 05:26:04 <my user name> systemd[1]: hadoop-tasktracker.service: Main process exited, code=exited, status=1/FAILURE
Dec 29 05:26:04 <my user name> systemd[1]: hadoop-tasktracker.service: Failed with result 'exit-code'.
Não consigo encontrar esta solução
Por favor, me diga a solução.
Tags java hadoop arch-linux