O tutorial Pig no modo local retorna OutOfMemoryError

Navegue suas respostas
1

Eu sou novo com porco e tento aprender com porco tutorial no site da Apache. Eu estou trabalhando com hadoop 1.0.1 e PigVersion 0.11.1. No tutorial, propõe-se experimentar o porco em dois exemplos (chamados script1-local.pig e script2-local.pig). Mas quando tento executar o primeiro exemplo com a seguinte CLI: 

pig -x local script1-local.pig

Eu recebo o seguinte erro: (parece lidar com o tamanho de heap Java ...) 

2013-07-16 15:26:59,033 [main] INFO  org.apache.pig.Main - Apache Pig version 0.11.1 (r1459641) compiled Mar 22 2013, 02:13:53
2013-07-16 15:26:59,033 [main] INFO  org.apache.pig.Main - Logging error messages to: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:26:59,246 [main] INFO  org.apache.pig.impl.util.Utils - Default bootup file /root/.pigbootup not found
2013-07-16 15:26:59,332 [main] INFO  org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2013-07-16 15:27:00,287 [main] WARN  org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_CHARARRAY 3 time(s).
2013-07-16 15:27:00,287 [main] WARN  org.apache.pig.PigServer - Encountered Warning USING_OVERLOADED_FUNCTION 3 time(s).
2013-07-16 15:27:00,300 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY,ORDER_BY,DISTINCT,FILTER
2013-07-16 15:27:00,446 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2013-07-16 15:27:00,469 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.CombinerOptimizer - Choosing to move algebraic foreach to combiner
2013-07-16 15:27:00,492 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 5
2013-07-16 15:27:00,493 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 5
2013-07-16 15:27:00,518 [main] INFO  org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2013-07-16 15:27:00,533 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-07-16 15:27:00,536 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2013-07-16 15:27:00,537 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=208348
2013-07-16 15:27:00,537 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2013-07-16 15:27:00,557 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2013-07-16 15:27:00,563 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2013-07-16 15:27:00,563 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2013-07-16 15:27:00,563 [main] INFO  org.apache.pig.data.SchemaTupleFrontend - Distributed cache not supported or needed in local mode. Setting key [pig.schematuple.local.dir] with code temp directory: /tmp/1373988420563-0
2013-07-16 15:27:00,564 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting identity combiner class.
2013-07-16 15:27:00,637 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2013-07-16 15:27:00,648 [JobControl] WARN  org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2013-07-16 15:27:00,652 [JobControl] WARN  org.apache.hadoop.mapred.JobClient - No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
****file:/root/pigtmp/excite-small.log
2013-07-16 15:27:00,681 [JobControl] INFO  org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-07-16 15:27:00,681 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2013-07-16 15:27:00,688 [JobControl] INFO  org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2013-07-16 15:27:00,922 [Thread-3] INFO  org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
2013-07-16 15:27:00,925 [Thread-3] INFO  org.apache.hadoop.mapred.Task -  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1b4b0967
2013-07-16 15:27:00,936 [Thread-3] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/root/pigtmp/excite-small.log:0+208348
2013-07-16 15:27:00,942 [Thread-3] INFO  org.apache.hadoop.mapred.MapTask - io.sort.mb = 100
2013-07-16 15:27:01,059 [Thread-3] WARN  org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:949)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:674)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:756)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
2013-07-16 15:27:01,138 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
2013-07-16 15:27:01,138 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases clean1,clean2,houred,ngramed1,raw
2013-07-16 15:27:01,138 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: raw[28,6],clean1[31,9],clean2[-1,-1],houred[39,9],ngramed1[42,11] C:  R: 
2013-07-16 15:27:01,145 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2013-07-16 15:27:01,148 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2013-07-16 15:27:01,148 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local_0001 has failed! Stop running all dependent jobs
2013-07-16 15:27:01,149 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2013-07-16 15:27:01,149 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2013-07-16 15:27:01,149 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Detected Local mode. Stats reported below may be incomplete
2013-07-16 15:27:01,150 [main] INFO  org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 

HadoopVersion   PigVersion  UserId  StartedAt   FinishedAt  Features
1.0.1   0.11.1  root    2013-07-16 15:27:00 2013-07-16 15:27:01 GROUP_BY,ORDER_BY,DISTINCT,FILTER

Failed!

Failed Jobs:
JobId   Alias   Feature Message Outputs
job_local_0001  clean1,clean2,houred,ngramed1,raw   DISTINCT    Message: Job failed! Error - NA 

Input(s):
Failed to read data from "file:///root/pigtmp/excite-small.log"

Output(s):

Job DAG:
job_local_0001  ->  null,
null    ->  null,
null    ->  null,
null    ->  null,
null


2013-07-16 15:27:01,150 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2013-07-16 15:27:01,152 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,153 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,154 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,154 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log
2013-07-16 15:27:01,155 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job failed, hadoop does not return any error message
Details at logfile: /root/pigtmp/pig_1373988419029.log

Como alguém encontra o mesmo problema?

Obrigado.

    
por MarAja 16.07.2013 / 17:33

1 resposta

1

Como é referido nesta resposta , pode tentar aumentar o espaço máximo da memória alocada: 

export HADOOP_CLIENT_OPTS="-Xmx1024m"
    
por 22.09.2013 / 11:10

Tags

O Samsung Magician não detecta SSD quando está definido para o modo AHCI Uma maneira de acessar um servidor linux remoto via SSH anonimamente e rodando um navegador gráfico?