Processo Weblogic acabou de ser morto repentinamente no redhat linux

Question

Um dos processos no meu EAS foi morto ao amanhecer.

6 processo executado neste servidor. Mas apenas um processo acabou de ser morto.

Eu não sei como consertar isso. Este é o último log do Weblogic que antes do processo ser morto.

/app/orarecp/bea/domains/HRRECP/bin/startWeblogic.sh: line 180: 27066 killed               ${JAVA_HOME}/bin/java ${JAVA_VM} {MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava-security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTION} ${PROXY_SETTINGS} ${SERVER_CLASS}

E este é o script de shell startWeblogic. startWeblogic Shell Script: linha 180

E isso é partes do script de shell do processo de início.

USER_MEM_ARGS="-Xms3072m -Xmx3072m -Xverbose:gc -Xverboselog:${LOG_HOME}/gclogs/gc.${SERVER_NAME}_'date + '%Y%m%d_%H%M%S''.out"

JAVA_OPTION="${JAVA_OPTIONS} -Xbootclasspath/p:/app/orarecp/jennifer/agent/lwst.jdk.jar:/app/orarecp/jennifer/agent/lwst.boot.jar:/app/orarecp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/orarecp/jennifer/agent/R11.conf -javaagent:/app/orarecp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom"

especificação do servidor é ...

• CPU: 8 núcleos, 2,4 GHz // O tempo que a utilização das CPUs ficou abaixo de 15%
• Memória: 32Gb
  • Mem: 32922024k total, 32817492k usado, 104580k livre, 147464k buffers
  • Troca: total 33551744k, 148k usado, 33551596k livre, 15231864k em cache

pilha do servidor é ...

• weblogic 11g
• jrockit 1.6
• redhat linux 5,8 (64 bits)

Eu me pergunto por que o processo é morto. O que devo fazer?

Por favor, responda se você precisar de informações adicionais.

Todas as respostas serão úteis para mim.

Obrigado.

Informação aditiva.

1. antes desse último log Mas esse log ocorre com frequência.

<[ServletContext@228988411[app:Recruit-Front module:WebContent path:/recruit spec-version:2.5]] Servlet failed with Exception

java.lang.NullPointerException at cj.recruit.syscommon.filter.SessionLocaleFilter.doFilter(SessionLocaleFilter.java:41) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at frameone.core.common.web.filter.EnvironmentSetupFilter.doFilter(EnvironmentSetupFilter.java:88) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at cj.recruit.syscommon.filter.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:22) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

<[ServletContext@228988411[app:Recruit-Front module:WebContent path:/recruit spec-version:2.5]] Servlet failed with Exception java.lang.NullPointerException

at cj.recruit.syscommon.filter.SessionLocaleFilter.doFilter(SessionLocaleFilter.java:41) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at frameone.core.common.web.filter.EnvironmentSetupFilter.doFilter(EnvironmentSetupFilter.java:88) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at cj.recruit.syscommon.filter.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:22) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

2. outro processo JAVA (ps -ef | grep java)

ps -ef | grep java

3. outro processo Java de servidor

   ps -ef | grep java

oraehrp 818 786 0 19:34 pts/1 00:00:00 grep java

root 9689 1 1 2015 ? 1-01:05:58 /usr/java/jdk1.6.0_23/bin/java -Djava.awt.headless=true -jar /app/CJ_agent//cjagent.jar /app/CJ_agent//conf/cjagent.conf

oraehrp 10128 10074 2 2015 ? 1-10:06:08 /usr/java/jrockit1.6/bin/java -jrockit -Xms1024m -Xmx1024m -Dweblogic.Name=EHRREP1 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 14795 14740 2 2015 ? 1-10:31:48 /usr/java/jrockit1.6/bin/java -jrockit -DAdminServer -Xms1024m -Xmx1024m -verbosegc -Dweblogic.Name=AdminServer -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 23456 23395 67 18:25 ? 00:46:23 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD11/gclogs/gc.EHRPRD11_20160201_182544.out -Dweblogic.Name=EHRPRD11 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehrp/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E11.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 24233 24179 49 18:27 ? 00:33:03 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD12/gclogs/gc.EHRPRD12_20160201_182747.out -Dweblogic.Name=EHRPRD12 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehrp/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E12.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 25006 24944 53 18:29 ? 00:34:35 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD13/gclogs/gc.EHRPRD13_20160201_182933.out -Dweblogic.Name=EHRPRD13 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehRP/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E13.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

    

Answer 1

Isto é devido à falta de memória. Você tem um pouco menos de 300MB de RAM livre (free + buffers) e cada nova instância do Java Weblogic estará comendo 3GB.

Existe uma funcionalidade do kernel do Linux chamada killer da OOM que mata um processo aleatório sempre que não houver memória suficiente.

Como configurar o assassino de falta de memória do Linux

Dê uma olhada nos argumentos JAVA e nos Scripts Java em fuga. Você está usando muita RAM .

Avalie se você precisa ajustar o Java Xms / Xmx ou adicionar mais RAM ao servidor.

Eu tive a oportunidade de verificar os requisitos do WebLogic agora. Como Personalizando o tamanho de heap da JVM WebLogic afirma:

The default JVM heap size for WebLogic is 3GB. ... USER_MEM_ARGS="-Xms128m -Xmx3072m ...

Preste atenção nos 128m, a configuração recomendada é de 128m ao iniciar o processo e o crescimento necessário para 3GB.

Xms é a memória mínima alocada, Xmx a memória máxima. Se você alterar Xms para 128 m, os encadeamentos só consumirão a memória necessária ao iniciar, e não 3 GB de uma só vez como padrão. Isso provavelmente usará menos memória RAM ao longo do tempo.

De Quais são os parâmetros Xm e Xmx quando iniciando JVMs?

The flag Xmx specifies the maximum memory allocation pool for a Java Virtual Machine (JVM), while Xms specifies the initial memory allocation pool. The memory flag can also be specified in multiple sizes, such as kilobytes, megabytes, and so on.

Eu também avaliaria após essa alteração se você precisa de mais RAM, já que o valor que você tem agora não é suficiente para uma operação adequada (por exemplo, muito baixa).