corresponde às mesmas linhas [fechadas]

0

Eu tenho um arquivo para o qual preciso saber se ele tem entradas iguais

O arquivo contém entradas como esta:

dn: cn=ccb2fa1a-6efb-4f29-b18b-72e226d76935,ou=Named,ou=Identities,ou=Active,o
rdcPosition: cn=936480,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>70
dn: cn=715f55d1-e940-42f9-8ae5-25ff1eff6f55,ou=Named,ou=Identities,ou=Active,o
rdcPosition: cn=7292,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>4024
rdcPosition: cn=8910,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>5209
rdcPosition: cn=7263,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>6725
rdcPosition: cn=936480,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>11
dn: cn=f61e2769-a9c8-486a-914b-92333055b5e5,ou=Named,ou=Identities,ou=Active,o
rdcPosition: cn=938936,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>74
rdcPosition: cn=942380,ou=Entities,ou=Active,ou=Vault,o=rdc#5#<position><cn>51
dn: cn=7548d048-1288-4b66-97f4-efe15c68fc50,ou=Named,ou=Identities,ou=Active,o
rdcPosition: cn=311432,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>43
dn: cn=e51f3d78-b9d8-4bcf-b8c5-321519f19515,ou=Named,ou=Identities,ou=Active,o
rdcPosition: cn=938936,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>35
dn: cn=cf6ddfb2-4261-4169-9e6e-0d6963262b49,ou=Named,ou=Identities,ou=Active,o
rdcPosition: cn=938936,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>82

Preciso saber se há entradas duplicadas para as linhas "rdcPosition" sob "dn:", por exemplo, esta:

dn: cn=65fb5990-4d2f-492e-83fb-c2cbd72d8988,ou=Named,ou=Identities,ou=Active,o
rdcPosition: cn=7688,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>2323
rdcPosition: cn=7688,ou=Entities,ou=Active,ou=Vault,o=rdc#3#<position><cn>2323

Você tem alguma idéia de qual comando do Unix eu devo usar?

    
por user120119 19.06.2015 / 13:52

3 respostas

1

O tipo de scripts quick'n'dirty que escrevo todos os dias:

#!/usr/bin/perl
#

use strict;
use warnings;

#data structures we're gonna need
my %positions; #how many times have we seen a given position
my %registered_lines; #the concatenated lines for the given position 
my $dn; # the current dn section we're in

while (<>)
{
    if (/^dn:/) #beginning of a new dn section (and end of the previous one)
    {
        my $printed = 0; #we want to print the dn line only once
        foreach my $key (keys %positions) #we look at all positions seen in last section
        {
            if ($positions{$key} gt 1) # has the current position been seen more than once
            {
                print $dn unless $printed;
                $printed = 1;
                #print "position $key is repeated $positions{$key} times\n";
                print $registered_lines{$key}; #print all the lines with the position
            }
        }

        #reset variables for the next section
        $dn = $_;
        %positions = ();
        %registered_lines = ();
    }

    if (/^rdcPosition/) #new line
    {
        /(\d+)$/; #have a look at the digits at the end of the line
        my $pos = $1;
        if (exists $positions{$pos}) #have we already seen this position
        {
            $positions{$pos} += 1; #increment the counter
            $registered_lines{$pos} .= $_; #record the line
        }
        else
        {
            $positions{$pos} = 1;
            $registered_lines{$pos} = $_;
        }
    }
}

Execute como:

perl script.pl < input_data_file
    
por 19.06.2015 / 14:21
0

Se você estiver interessado em saber "Há duplicatas?" então eu recomendo comparar os resultados de cat <file> | sort | wc -l e cat <file> | sort | uniq | wc -l . Se houver duplicatas, o uniq as removerá e o número diminuirá. Se você quiser ver essas diferenças, veja o script perl @Igeorget postado.

    
por 19.06.2015 / 14:51
0
awk '/^dn:/ {d=1} {if (d) {print buf | "sort|uniq -d"; d=0; buf=""} else {buf=buf$0"\n"}} END {print buf | "sort|uniq -d"}'|grep -v '^$'

Muito menos digitação do que na versão perl =). Pode ser ainda mais simples, mas não tenho como executar a regra awk 'em qualquer padrão OU no END', então ele contém uma pequena duplicação de código shell.

    
por 19.06.2015 / 15:29