Acabei com o seguinte script awk
.
awk '{gsub(/ {1,5}/,"\t|")}1' infile.txt | \
awk -F'|' 'NR>1{for(i=1;i<=NF;i++) if ($i=="\t") c++;
printf("%*s",(c*8),"\t-");
for (fill=1;fill<=length($0)+c;fill++)printf "-";
printf("\n%s\n",$0);c=0;next} {print $0}'
A saída é:
col1 |col2 |col3 |subcol1 |subcol2 |subcol3
-----------------------------------
| | |subcol4 |subcol5 |subcol6
--------------------------------------
| |col4 |subcol7 |subcol8 |subcol9
-------------------------------------
| | |subcol10 |subcol11 |subcol2