executando códigos simultaneamente durante a saída do resultado um por um

Question

Eu tenho um script parecido com:

for simplify in 0.1 ;do
  for lmbda in 0.9 1.1 1.3;do
    for mu in 2.1 3.4 4.2;do
      rm eci.out;
      csce.py --mu $mu --lmbda $lmbda --simplify $simplify  \
        --favor-low-energy 0.01 --bias-stable \
        --save-energies ce-energies.dat --save-weights ce-weights.dat \
        --casm-eci-file eci.in eci.out --save-hull ce-hull.dat \
        --preserve-ground-state 100 
     done
   done
 done

Durante a execução sequencial, a saída será

 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.0002 --lmbda 0.005 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.0002,  lambda = 0.005,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.03952735        0.04380240        0.02230235       -0.00185235
    230 clusters      0.03734292        0.04294355        0.02049885       -0.00162721
   Leave-one-out CV score wrt. input   : 13.8588
   Leave-one-out CV score wrt. full fit: 0.2201
   Writing CV energies to file 'cv-energies.dat'.
 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.0002 --lmbda 0.5 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.0002,  lambda = 0.5,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.04017143        0.04451601        0.02267050       -0.00188774
    219 clusters      0.03826959        0.04392544        0.02088397       -0.00166322
   Leave-one-out CV score wrt. input   : 7.9875
   Leave-one-out CV score wrt. full fit: 0.1674
   Writing CV energies to file 'cv-energies.dat'.
 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.002 --lmbda 0.5 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.002,  lambda = 0.5,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.14367173        0.15893736        0.07206284       -0.00912716
    237 clusters      0.12351319        0.17167930        0.05376677       -0.00158861
   Leave-one-out CV score wrt. input   : 5.4561
   Leave-one-out CV score wrt. full fit: 0.1119
   Writing CV energies to file 'cv-energies.dat'.
 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.002 --lmbda 0.005 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.002,  lambda = 0.005,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.14299857        0.15820042        0.07228279       -0.00898964
    237 clusters      0.12292319        0.17103283        0.05359447       -0.00138365
   Leave-one-out CV score wrt. input   : 5.3224
   Leave-one-out CV score wrt. full fit: 0.1104
   Writing CV energies to file 'cv-energies.dat'.
 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.02 --lmbda 0.5 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.02,  lambda = 0.5,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.19027749        0.20991052        0.09677830       -0.01602039
    237 clusters      0.17531289        0.27618581        0.06910020        0.00220309
   Leave-one-out CV score wrt. input   : 2.4223
   Leave-one-out CV score wrt. full fit: 0.0638
   Writing CV energies to file 'cv-energies.dat'.
 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.2 --lmbda 0.5 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.2,  lambda = 0.5,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.26328087        0.28804954        0.15654138       -0.03325137
    237 clusters      0.20897747        0.39607952        0.07887236        0.00877210
   Leave-one-out CV score wrt. input   : 1.9517
   Leave-one-out CV score wrt. full fit: 0.0477
   Writing CV energies to file 'cv-energies.dat'.
 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.02 --lmbda 0.005 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.02,  lambda = 0.005,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.18831434        0.20789816        0.09380533       -0.01523797
    237 clusters      0.17809687        0.28067519        0.06919726        0.00308981
   Leave-one-out CV score wrt. input   : 2.4420
   Leave-one-out CV score wrt. full fit: 0.0642
   Writing CV energies to file 'cv-energies.dat'.
 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.0002 --lmbda 0.05 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.0002,  lambda = 0.05,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.03972713        0.04402288        0.02232628       -0.00187813
    220 clusters      0.03735457        0.04295330        0.02008527       -0.00161760
   Leave-one-out CV score wrt. input   : 9.9223
   Leave-one-out CV score wrt. full fit: 0.1863
   Writing CV energies to file 'cv-energies.dat'.
 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.2 --lmbda 0.005 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.2,  lambda = 0.005,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.23518508        0.25836295        0.13669345       -0.02483126
    237 clusters      0.21460451        0.36901059        0.08558721        0.00618068
   Leave-one-out CV score wrt. input   : 1.7372
   Leave-one-out CV score wrt. full fit: 0.0490
   Writing CV energies to file 'cv-energies.dat'.
 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.002 --lmbda 0.05 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.002,  lambda = 0.05,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.14375418        0.15902969        0.07218794       -0.00912434
    237 clusters      0.12364959        0.17204399        0.05379933       -0.00144881
   Leave-one-out CV score wrt. input   : 5.5220
   Leave-one-out CV score wrt. full fit: 0.1124
   Writing CV energies to file 'cv-energies.dat'.
 /home1/03631/key01027/local_python_lib/bin/csce.py --mu 0.02 --lmbda 0.05 --simplify 0.1 --favor-low-energy 0.01 --bias-stable --save-energies ce-energies.dat --save-weights ce-weights.dat --casm-eci-file eci.in eci.out --save-hull ce-hull.dat --preserve-ground-state 100
 Split-Bregman Input Parameters:  mu = 0.02,  lambda = 0.05,  eci_cutoff = 1e-08
 Attempting to simplify the CE using a target RMSE of 0.1
 Warning: Unable to achieve target RMSE of 0.1
                      RMSE              RMSE (no wght)    MAE               ME
    237 clusters      0.19066043        0.21033595        0.09672442       -0.01606650
    237 clusters      0.17565376        0.27643274        0.06917755        0.00238796
   Leave-one-out CV score wrt. input   : 2.4257
   Leave-one-out CV score wrt. full fit: 0.0639
   Writing CV energies to file 'cv-energies.dat'.

No entanto, para tornar a computação mais eficiente, sei que poderíamos executá-la simultaneamente com "&":

for simplify in 0.1 ;do
  for lmbda in 0.9 1.1 1.3;do
    for mu in 2.1 3.4 4.2;do
      rm eci.out;
      csce.py --mu $mu --lmbda $lmbda --simplify $simplify  \
        --favor-low-energy 0.01 --bias-stable \
        --save-energies ce-energies.dat --save-weights ce-weights.dat \
        --casm-eci-file eci.in eci.out --save-hull ce-hull.dat \
        --preserve-ground-state 100 &
     done
   done
 done

Mas a saída seria caótica ... O que eu poderia fazer para que a saída estivesse em sequência enquanto as executasse simultaneamente? Obrigado.

    

Answer 1

Se você forçar todos esses comandos, a saída para STDOUT não estará em ordem, já que os garfos não serão executados em ordem.

Você pode redirecionar todas as chamadas csce.py para arquivos individuais e, em seguida, codificar esses arquivos. Use as variáveis de loop para os nomes dos arquivos para entender qual arquivo representa qual processo.

Mas, por questão de escalabilidade e controle, talvez valha a pena usar o paralelo GNU .

A propósito, os nomes dos arquivos *.dat não dependem dos parâmetros de sua chamada, portanto, dependendo de como o script python funciona, eles podem ser corrompidos (sobrescritos) ou crescer com todas as chamadas para o seu script. / p>     

Answer 2

yes gnu parallel funcionará bem, mas se você quiser fazer isso através de & Para o operador, você precisa saber que o processo bifurcado só será executado quando ele se apossar do ressago, uma vez que os recursos estão compartilhando o mesmo stdout, parece que a saída é caótica.