excluindo linhas entre linhas em um arquivo de texto usando awk ou sed

Question

Eu estou querendo saber se existe um comando sed ou awk para excluir todas as linhas entre os cabeçalhos 'Query_' na coluna 1, se o número de linhas entre cada cabeçalho for menor que 5. O seguinte é um extrato de um arquivo grande ~ 1Gb. Eu tentei vários métodos diferentes, mas todos falharam.

Query_10      26   KMGKWYPTEDAPAKKRKTQSWRQNKSKLRGGIVPGQVLIILAGKHKGKRVVYLTQLSTGE  205
XP_010718494  131  KMPRYYPTEDVPRKSHGKKPFSQHKRRLRASITPGTVLILLTGRHRGKRVVFLKQLGTGL  192
NP_001291831  111  KMPRYYPTEDVPRKSHGKKPFSQHVRKLRASITPGTILIILTGRHRGKRVVFLKQLSSGL  172
Query_10      206  IVVTGPHKFNRCPLKKLAQSFTMPTSTFVDI*GLNFDITEQHFVKEKP**SSEEAQFFAK  385
XP_010718494  193  LLVTGPLVVNRVPLRRAHQKFVIATSTKVDISGVKIHLTDAYFKKKKLRKPKQEGEIFDT  255
NP_001291831  173  LLVTGPLSLNRVPLRRTHQKFVIATSTKIDISSVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_012359817  173  LLVTGPLVLNRVPLRRTHQKFVIATSTKIDISNVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_009246541  173  LLVTGPLVLNRVPLRRTHQKFVIATSTKIDISNVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_003225150  155  LLVTGPLAINRVPLRRAHQKFVIATSTKVDISSVKLHLNDVYFKKKKLRKPKQEGEIFDT  217
Query_13      31    MEEQKEKGLSNPEVV*KYRQCSEIVNQVLSTVVSSCVPGADVASICTNGDFLIEDGLRNI  210
XP_002947167  7     IQGEQEPNLSVPEVVTKYKAAADICNRALQAVIDGCKDGSKIVDLCRTGDNFITKECGNI  66
XP_004993505  1     MELDRQSKVVDADALSKYRAAAAIANDCVQQLVANCIAGADVYTLAVEADTYIEQKLKEL  60
XP_006961234  1     MSETKEYSLNNPDTLTKYKTAAQISEKVLAAVSDLCVPGAKIVDICQQGDKLIEEELAKV  62
XP_008089018  1     MSEETDYTLNNPDTLTKYKTAAQISEKVLAAVAELVVPGEKIVTICEKGDKLIEEELAKV  60
Query_13      211   EPDTNIEKGIAIPVCLNINNICSYYSPLPDASTTLQEGDLVKVDLGAHFDGYIVSAASSI  390
XP_004029906  65    YTKKKVEKGPAFPTCISINEICGHYSPLLSDSSLLKEGDVVKIDLGTHIDGFIALGAHTV  131
XP_004031065  64    FTKKKLQKGPAFPTCISVNEICGHYSPLISDSSLLKEGDVVKIDLGAQIDGFIALAAHTV  130
XP_003223249  65    KKEKDMKKGIAFPTSISVNNCVCHFSPLKDQDYILKEGDLVKIDLGVHVDGFISNVAHSF  125
XP_002947167  67    YKGKQIEKGVAFPTCVSVNSVVGHFSPNADDTSALKAGDVVKFDMGCHIDGFIATQATTV  126
XP_003880798  73    ENGKKMEKGIAFPTCISINEICGHFSPVEENAETLTEGDVVKIDMGCHIDGYISVVAYTV  135
XP_004348044  69    KANKKVKKGIAFPTCVSLNSTVCHQSPLSDAAITLQAGDVAKVDLGVHVDGLIAVVAHTI  129
XP_003284133  69    HSKKKIEKGIAFPTCISVNNCVGHYSPLKATSRSLVDGDIVKIDLGVHINGFIAVGAHTI  128
NP_001241588  65    YKNVKIERGVAFPTCLSINNVVCHFSPLASDEAVLEEGDILKIDMACHIDGFIAVVAHTH  126
XP_009039553  76    YQKKIIDKGVAFPTCVSVNECVCHNSPLESDTTSLSEGDLVKLDVGCYVDGYIAVAAHTM  141

O resultado desejado seria o seguinte:

Query_10      206  IVVTGPHKFNRCPLKKLAQSFTMPTSTFVDI*GLNFDITEQHFVKEKP**SSEEAQFFAK  385
XP_010718494  193  LLVTGPLVVNRVPLRRAHQKFVIATSTKVDISGVKIHLTDAYFKKKKLRKPKQEGEIFDT  255
NP_001291831  173  LLVTGPLSLNRVPLRRTHQKFVIATSTKIDISSVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_012359817  173  LLVTGPLVLNRVPLRRTHQKFVIATSTKIDISNVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_009246541  173  LLVTGPLVLNRVPLRRTHQKFVIATSTKIDISNVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_003225150  155  LLVTGPLAINRVPLRRAHQKFVIATSTKVDISSVKLHLNDVYFKKKKLRKPKQEGEIFDT  217
Query_13      211   EPDTNIEKGIAIPVCLNINNICSYYSPLPDASTTLQEGDLVKVDLGAHFDGYIVSAASSI  390
XP_004029906  65    YTKKKVEKGPAFPTCISINEICGHYSPLLSDSSLLKEGDVVKIDLGTHIDGFIALGAHTV  131
XP_004031065  64    FTKKKLQKGPAFPTCISVNEICGHYSPLISDSSLLKEGDVVKIDLGAQIDGFIALAAHTV  130
XP_003223249  65    KKEKDMKKGIAFPTSISVNNCVCHFSPLKDQDYILKEGDLVKIDLGVHVDGFISNVAHSF  125
XP_002947167  67    YKGKQIEKGVAFPTCVSVNSVVGHFSPNADDTSALKAGDVVKFDMGCHIDGFIATQATTV  126
XP_003880798  73    ENGKKMEKGIAFPTCISINEICGHFSPVEENAETLTEGDVVKIDMGCHIDGYISVVAYTV  135
XP_004348044  69    KANKKVKKGIAFPTCVSLNSTVCHQSPLSDAAITLQAGDVAKVDLGVHVDGLIAVVAHTI  129
XP_003284133  69    HSKKKIEKGIAFPTCISVNNCVGHYSPLKATSRSLVDGDIVKIDLGVHINGFIAVGAHTI  128
NP_001241588  65    YKNVKIERGVAFPTCLSINNVVCHFSPLASDEAVLEEGDILKIDMACHIDGFIAVVAHTH  126
XP_009039553  76    YQKKIIDKGVAFPTCVSVNECVCHNSPLESDTTSLSEGDLVKLDVGCYVDGYIAVAAHTM  141

Script Python que experimentei:

lines = [line.rstrip() for line in open('infile.txt')]
for line in lines: 
    data = line.split()
    sequence = data[2]
    if data[0].startswith("Query_"):
        hits = [i for i,c in enumerate(sequence) if c == <50]
        continue
    else:
        print(list(sequence[plus50] for plus50 in hits))

    

Answer 1

com sed :

sed '
    /^Query_/{                #starts loop when meet patten
        :a
        $!{
            N
            /\nQuery_/!ba     #untill meet next pattern
        }
        /\(\n.*\)\{6,\}/{     #checks how many lines in block
            $b                #for end of file
            s/\nQuery_/\n&/   #marks lines to print
        }
    }
    /\n\n/P                   #prints marked lines
    D                         #remove 1st line in block, go to start
    '

Outro formato de script para awk :

awk '
    /^Query/{c=0;lines=$0;next}
    ++c<5{lines=lines "\n" $0;next}
    c==5{print lines}
    1                         #short for {print}
    '

    

Answer 2

com awk :

awk '{if($1~/^Query_/){c=0;delete a;a[0]=$0}else{c++}
    if(c<5){a[c]=$0}
    if(c==5){for(i in a){print a[i]}}
    if(c>5){print}}' file

• Na primeira linha, o primeiro campo $1 é verificado se começa com Query_ . Nesse caso, a variável do contador c está definida como 0 . A matriz a é removida e o primeiro elemento da matriz é definido com o valor dessa linha. Senão a variável do contador é incrementada.
• Na segunda linha, a matriz é preenchida linha por linha, até que mais 5 linhas estejam nela.
• Terceira linha: Se tivermos mais 5 linhas, percorra a matriz e imprima seus elementos linha a linha.
• Quarta linha: todas as linhas de agora em diante podem ser impressas.

Saída com seus dados de exemplo:

Query_10      206  IVVTGPHKFNRCPLKKLAQSFTMPTSTFVDI*GLNFDITEQHFVKEKP**SSEEAQFFAK  385
XP_010718494  193  LLVTGPLVVNRVPLRRAHQKFVIATSTKVDISGVKIHLTDAYFKKKKLRKPKQEGEIFDT  255
NP_001291831  173  LLVTGPLSLNRVPLRRTHQKFVIATSTKIDISSVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_012359817  173  LLVTGPLVLNRVPLRRTHQKFVIATSTKIDISNVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_009246541  173  LLVTGPLVLNRVPLRRTHQKFVIATSTKIDISNVKIHLTDAYFKKKKP--RHQEGEIFDT  235
Query_13      211   EPDTNIEKGIAIPVCLNINNICSYYSPLPDASTTLQEGDLVKVDLGAHFDGYIVSAASSI  390
XP_004029906  65    YTKKKVEKGPAFPTCISINEICGHYSPLLSDSSLLKEGDVVKIDLGTHIDGFIALGAHTV  131
XP_004031065  64    FTKKKLQKGPAFPTCISVNEICGHYSPLISDSSLLKEGDVVKIDLGAQIDGFIALAAHTV  130
XP_003223249  65    KKEKDMKKGIAFPTSISVNNCVCHFSPLKDQDYILKEGDLVKIDLGVHVDGFISNVAHSF  125
XP_002947167  67    YKGKQIEKGVAFPTCVSVNSVVGHFSPNADDTSALKAGDVVKFDMGCHIDGFIATQATTV  126
XP_004348044  69    KANKKVKKGIAFPTCVSLNSTVCHQSPLSDAAITLQAGDVAKVDLGVHVDGLIAVVAHTI  129
XP_003284133  69    HSKKKIEKGIAFPTCISVNNCVGHYSPLKATSRSLVDGDIVKIDLGVHINGFIAVGAHTI  128
NP_001241588  65    YKNVKIERGVAFPTCLSINNVVCHFSPLASDEAVLEEGDILKIDMACHIDGFIAVVAHTH  126
XP_009039553  76    YQKKIIDKGVAFPTCVSVNECVCHNSPLESDTTSLSEGDLVKLDVGCYVDGYIAVAAHTM  141

    

Answer 3

com GNU awk

$ awk -F'\n' -v RS='Query_' -v ORS= 'NF>6{print RS $0}' ip.txt
Query_10      206  IVVTGPHKFNRCPLKKLAQSFTMPTSTFVDI*GLNFDITEQHFVKEKP**SSEEAQFFAK  385
XP_010718494  193  LLVTGPLVVNRVPLRRAHQKFVIATSTKVDISGVKIHLTDAYFKKKKLRKPKQEGEIFDT  255
NP_001291831  173  LLVTGPLSLNRVPLRRTHQKFVIATSTKIDISSVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_012359817  173  LLVTGPLVLNRVPLRRTHQKFVIATSTKIDISNVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_009246541  173  LLVTGPLVLNRVPLRRTHQKFVIATSTKIDISNVKIHLTDAYFKKKKP--RHQEGEIFDT  235
XP_003225150  155  LLVTGPLAINRVPLRRAHQKFVIATSTKVDISSVKLHLNDVYFKKKKLRKPKQEGEIFDT  217
Query_13      211   EPDTNIEKGIAIPVCLNINNICSYYSPLPDASTTLQEGDLVKVDLGAHFDGYIVSAASSI  390
XP_004029906  65    YTKKKVEKGPAFPTCISINEICGHYSPLLSDSSLLKEGDVVKIDLGTHIDGFIALGAHTV  131
XP_004031065  64    FTKKKLQKGPAFPTCISVNEICGHYSPLISDSSLLKEGDVVKIDLGAQIDGFIALAAHTV  130
XP_003223249  65    KKEKDMKKGIAFPTSISVNNCVCHFSPLKDQDYILKEGDLVKIDLGVHVDGFISNVAHSF  125
XP_002947167  67    YKGKQIEKGVAFPTCVSVNSVVGHFSPNADDTSALKAGDVVKFDMGCHIDGFIATQATTV  126
XP_003880798  73    ENGKKMEKGIAFPTCISINEICGHFSPVEENAETLTEGDVVKIDMGCHIDGYISVVAYTV  135
XP_004348044  69    KANKKVKKGIAFPTCVSLNSTVCHQSPLSDAAITLQAGDVAKVDLGVHVDGLIAVVAHTI  129
XP_003284133  69    HSKKKIEKGIAFPTCISVNNCVGHYSPLKATSRSLVDGDIVKIDLGVHINGFIAVGAHTI  128
NP_001241588  65    YKNVKIERGVAFPTCLSINNVVCHFSPLASDEAVLEEGDILKIDMACHIDGFIAVVAHTH  126
XP_009039553  76    YQKKIIDKGVAFPTCVSVNECVCHNSPLESDTTSLSEGDLVKLDVGCYVDGYIAVAAHTM  141

• -v RS='Query_' set Query_ como separador de registro de entrada
• -v ORS= set string vazia como separador de registro de saída
• -F'\n' definir nova linha como separador de campos de entrada
• NF>6 questão é preservar blocos com 5 entradas. Incluindo o cabeçalho há 6 linhas, o que significa 6 novas linhas. Dividir essa string mínima obrigatória fornecerá 7 campos - daí a condição NF>6
• print RS $0 print RS, bem como registro de entrada quando a condição é satisfeita