enviar json via rsyslog para elasticsearch

Question

em nossa empresa, precisamos de estatísticas agregadas de serviços

por algum motivo, decidimos usar o rsyslogd para enviar diretamente as estatísticas dos aplicativos (formato json) para o sistema elasticsearch ( este tutorial ).

mas enfrentamos esse problema:

quando enviar estatísticas json para rsyslogd local pelo comando logger , tudo foi ok.

mas

quandooaplicativo(java)enviaessasestatísticasparaorsyslogd(comaestruturadobacklog) 

errommjsonparse:

9448.407432204:mainQ:Reg/w0:Calledaction,loggingtommjsonparse9448.407443744:mainQ:Reg/w0:Action1transitionedtostate:itx9448.407450424:mainQ:Reg/w0:enteringactionCalldoAction(),state:itx,actionNbr19448.407465385:mainQ:Reg/w0:mmjsonparse:noJSONcookie:'{"subject":                "Report","report_no":              2411,"report_time_from":       1479309445405,"report_time_until":      1479309448406,"report_time_duration":   3001,"upload_lessThan1M_count":      0,"upload_lessThan1M_size":       0,"upload_btw1Mto2M_count":      0,"upload_btw1Mto2M_size":       0,"upload_btw2Mto5M_count":      0,"upload_btw2Mto5M_size":       0,"upload_btw5Mto10M_count":      0,"upload_btw5Mto10M_size":       0,"upload_btw10Mto20M_count":      0,"upload_btw10Mto20M_size":       0,"upload_btw20Mto50M_count":      0,"upload_btw20Mto50M_size":       0,"upload_btw50Mto100M_count":      0,"upload_btw50Mto100M_size":       0,"upload_moreThan100M_count":      0,"upload_moreThan100M_size":       0,"upload_total_size":  0,"upload_total_count": 0,"thumb_count":  0,"thumb_time":   0,"download_lessThan1M_count":      0,"download_lessThan1M_size":       0,"download_btw1Mto2M_count":      0,"download_btw1Mto2M_size":       0,"download_btw2Mto5M_count":      0,"download_btw2Mto5M_size":       0,"download_btw5Mto10M_count":      0,"download_btw5Mto10M_size":       0,"download_btw10Mto20M_count":      0,"download_btw10Mto20M_size":       0,"download_btw20Mto50M_count":      0,"download_btw20Mto50M_size":       0,"download_btw50Mto100M_count":      0,"download_btw50Mto100M_size":       0,"download_moreThan100M_count":      0,"download_moreThan100M_size":       0,"download_total_size":  0,"download_total_count": 0,"cache_served_count": 0,"cache_served_size":  0,"cache_new_count":    0,"cache_new_size":     0}'
9448.407510073:main Q:Reg/w0  : Action 1 transitioned to state: rdy
9448.407517838:main Q:Reg/w0  :     PRIFILT 'local2.*'
9448.407527643:main Q:Reg/w0  :     pmask:  X  X  X  X  X  X  X  X  X  X  X  X  X  X  X  X  X  X FF  X  X  X  X  X  X  X 
9448.407598847:main Q:Reg/w0  : PRIFILT condition result is 1
9448.407604631:main Q:Reg/w0  :     ACTION 2 [omelasticsearch:action(type="omelasticsearch" ...)]
9448.407617909:main Q:Reg/w0  : executing action 2
9448.407622728:main Q:Reg/w0  : Called action, logging to omelasticsearch
9448.407631995:main Q:Reg/w0  : action 3 queue: qqueueAdd: entry added, size now log 1, phys 1 entries

detalhes do software: (rsyslog 8.4.2-1 + deb8u2, rsyslog-elasticsearch)

syslog-config:

#load needed modules
#load needed modules
module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
module(load="mmjsonparse") #for parsing CEE-enhanced syslog messages
module(load="omelasticsearch") #for indexing to Elasticsearch


#try to parse structured logs
local2.* action(type="mmjsonparse")

#define a template to print field "foo"
template(name="justFoo" type="list") {
    property(name="$!all-json")
}

#and now let's write the contents of field "foo" in a file
#action(type="omfile"
#    template="justFoo"
#    file="/tmp/foo")

local2.*    action(type="omelasticsearch"
       server="192.168.218.42"
       serverport="9200"
       template="justFoo"
       searchIndex="stats"
       searchType="stats"
       bulkmode="on"
       queue.type="linkedlist"
       queue.size="5000"
       queue.dequeuebatchsize="300"
       action.resumeretrycount="-1")

    

Answer 1

Eu encontrei a resposta!

local2.* action(type="mmjsonparse")

mudaria para:

action(type="mmjsonparse" cookie="")

Detalhes:

Diretivas de configuração específicas de ação:

cookie [string] é padronizado como “@cee:”

Permite definir o cookie que deve estar presente na frente da parte JSON da mensagem.

Mais importante ainda, isso pode ser definido como a string vazia ("") para não requerer nenhum cookie. Nesse caso, os espaços iniciais são permitidos na frente do JSON. Nenhum caractere que não seja espaço em branco é permitido após o JSON. Se tal for necessário, deve-se usar mmnormalise.

Ref: link

Most importantly, this can be set to the empty string (“”) in order to not require any cookie. In this case, leading spaces are permitted in front of the JSON. No non-whitespace characters are permitted after the JSON. If such is required, mmnormalize must be used.