Extraindo strings entre caracteres especiais

4

Eu tenho um grande arquivo de texto criado a partir da combinação de muitos arquivos html.

cat *.html > all_html_files.txt

Dentro do arquivo de texto estão strings específicas que eu quero extrair para outro arquivo de texto. Por exemplo:

book title>The Edge of the Round World< font 32 - extra

Eu quero extrair todo o texto que ocorre entre os símbolos > e < .

Eu quero extrair The Edge of the Round World e todas as outras strings no documento que aparecem entre os mesmos símbolos.

Eu tentei encontrar uma solução, mas não consigo adaptar os comandos que encontrei porque não consigo descobrir exatamente o que substituir - não consigo entender a lógica.

Estou familiarizado recentemente com o uso do sed e do awk graças a este fórum.

    
por speld_rwong 26.12.2015 / 06:53

4 respostas

4

sed -ne's/<\([^>"]*\("[^"]*"\)*\)*\)*>//g;/./p' <infile >outfile

... com GNU ou BSD sed s:

sed -Ene's/<([^>"]*("[^"]*")*)*>//g;/./p' <infile >outfile

Aqui está algo um pouco mais complicado como prova de conceito:

url='http://stackoverflow.com/questions/1732348/regex-match-open-tags-except-xhtml-self-contained-tags'
curl -s  "$url"   |
sed -Ene:n -etD   \
        -e's/ ?[^ "]*"[^"]*"//g;/"/'bN  \
        -e's/[[:space:]]*($|<)/\n/'   \
        -e'/^Moderator.s Note/q'        \
        -e'/.\n/P;/\n</!t'        -e:D  \
        -e'/\n/D;/^<script>/!s/>/&\n/'  \
        -e'/\n/!s/<\/script>/\n/' -e:N  \
        -e'/\n/!{N;s///;}' -e//tD -etn

a parte mais difícil é filtrar todo o javascript

html - RegEx match open tags except XHTML self-contained tags - Stack Overflow
current community
chat
        Stack Overflow
        Meta Stack Overflow
                        Stack Overflow Careers
your communities
Sign up
 or
log in
 to customize your list.
more stack exchange communities
company blog
Stack Exchange
Inbox
Reputation and Badges
sign up
log in
tour
        help
                            Tour
                                Start here for a quick overview of the site
                        Help Center
                            Detailed answers to any questions you might have
                            Meta
                                Discuss the workings and policies of this site
                    Stack Overflow
Questions
Jobs
beta
Tags
Users
Badges
Ask Question
Sign up
&times;
            Stack Overflow is a community of 4.7 million programmers, just like you,
 helping each other. Join them; it only takes a minute:
RegEx match open tags except XHTML self-contained tags
up vote
1326
down vote
favorite
4475
I need to match all of these opening tags:
&lt;p&gt;
&lt;a&gt;
But not these:
&lt;br /&gt;
&lt;hr /&gt;
I came up with this and wanted to make sure I've got it right. I am only capturing t
he
a-z
.
&lt;([a-z]+) *[^/]*?&gt;
I believe it says:
Find a less-than, then
Find (and capture) a-z one or more times, then
Find zero or more spaces, then
Find any character zero or more times, greedy, except
/
, then
Find a greater-than
Do I have that right? And more importantly, what do you think?
html
regex
xhtml
share
edited
May 26 '12 at 20:37
            community wiki
        11 revs, 7 users 58%
Jeff
locked
 by
Robert Harvey
&#9830;
Jun 7 '12 at 19:41
This post has been locked due to the high amount of off-topic comments generated. Fo
r extended discussions, please use
chat
.
comments disabled on deleted / locked posts / reviews
&nbsp;|&nbsp;
                                35 Answers
35
            active
            oldest
            votes
1
2
 next
up vote
4427
down vote
accepted
You can't parse [X]HTML with regex. Because HTML can't be parsed by regex. Regex is
not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-
regex questions here so many times before, the use of regex will not allow you to co
nsume HTML. Regular expressions are a tool that is insufficiently sophisticated to u
nderstand the constructs employed by HTML. HTML is not a regular language and hence
cannot be parsed by regular expressions. Regex queries are not equipped to break dow
n HTML into its meaningful parts. so many times but it is not getting to me. Even en
hanced irregular regular expressions as used by Perl are not up to the task of parsi
ng HTML. You will never make me crack. HTML is a language of sufficient complexity t
hat it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML usi
ng regular expressions. Every time you attempt to parse HTML with regular expression
s, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp.
 Parsing HTML with regex summons tainted souls into the realm of the living. HTML an
d regex go together like love, marriage, and ritual infanticide. The &lt;center> can
not hold it is too late. The force of regex and HTML together in the same conceptual
 space will destroy your mind like so much watery putty. If you parse HTML with rege
x you are giving in to Them and their blasphemous ways which doom us all to inhuman
toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he
comes. HTML-plus-regexp will liquify the n​erves of the sentient whilst you observe,
 your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are t
he cancer that is killing StackOverflow
it is too late it is too late we cannot be saved
 the trangession of a chi͡ld ensures regex will consume all living tissue (except fo
r HTML which it cannot, as previously prophesied)
dear lord help us how can anyone survive this scourge
 using regex to parse HTML has doomed humanity to an eternity of dread torture and s
ecurity holes
using rege
x as a tool to process HTML establishes a brea
ch between this world
 and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but
more corrupt) a mere glimp
se of the world of reg​
ex parsers for HTML will ins
​tantly transport a p
rogrammer's consciousness i
nto a w
orl
d of ceaseless screaming, he comes
, the pestilent sl
ithy regex-infection wil​
l devour your HT
​ML parser, application and existence for all time like Visual Basic only worse
he comes he com
es
do not fi
​ght h
e com̡e̶s, ̕h̵i
​s un̨ho͞ly radiańcé de
stro҉ying all enli̍̈́̂̈́ghtenment, HTML tags
lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur eye͢s̸ ̛l̕ik͏e liq
​uid p
ain, the song of re̸gular exp​re
ssion parsing
will exti
​nguish the voices of mor​
tal man from the sp
​here I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​
he f
inal snuf
fing o
f the lie​
s of Man ALL IS LOŚ͖̩͇̗̪̏̈́T A
LL I​S L
OST th
e pon̷y he come
s he c̶̮om
es he co
me
s t
he
 ich​
or permeat
es al
l MY FAC
E MY FACE ᵒh god n
o NO NOO̼
O​O N
Θ stop t
he an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨ
e̠̅s
 ͎a̧͈͖r̽̾̈́͒͑e
 n
​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ T
O͇̹̺ͅƝ̴ȳ̳ TH̘
Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝
S̨̥̫͎̭ͯ̿̔̀ͅ
Have you tried using an XML parser instead?
    
por 26.12.2015 / 07:03
0

Sou fã do grep e do regex Perl para fazer este tipo de trabalho. Você pode gostar de tentar isso

grep -oP '(?<=book title>).*(?=<)' all_html_files.txt
    
por 26.12.2015 / 12:50
0

Usar regexes para extrair informações do HTML raramente é uma boa ideia, especialmente se os elementos sintáticos puderem abranger linhas no seu arquivo.

Se você quiser fazer isso apenas uma vez, eu abro o arquivo no seu editor de texto favorito e uso as macros de pesquisa e substituição para reduzir as coisas. Eu fiz isso apenas hoje, na verdade :) mas demorei um tempo relativamente longo.

Se você quiser fazer isso regularmente, use algo projetado para o trabalho. Veja htmlparsing.com e Wikipedia Comparação de analisadores HTML .

    
por 26.12.2015 / 10:36
-1

Eu resolvi um cenário simples. O texto da amostra está abaixo:

text.txt

book title>The Linux Command Line< font 32 - extra
book title>How Linux Works< font 32 - extra
book title>UNIX and Linux System Administration Handbook< font 32 - extra
book title>Raspberry Pi Cookbook< font 32 - extra
book title>Linux Bible< font 32 - extra
book title>The Linux Programming Interface< font 32 - extra

comando

$ cat text.txt | awk 'BEGIN {FS=">"} {print $2} | awk 'BEGIN {FS="<"} {print $1}'

saída

The Linux Command Line
How Linux Works
UNIX and Linux System Administration Handbook
Raspberry Pi Cookbook
Linux Bible
The Linux Programming Interface
    
por 28.12.2015 / 06:01