Existe uma diferença entre aspas simples e duplas, e não é uma questão de estilo. Citações simples irão literalmente ecoar o que você escreve, então você pode ter certeza de que o que você escreve é o que será impresso. Por outro lado, as aspas duplas interpolarão o valor das variáveis.
De man bash
:
Enclosing characters in single quotes preserves the literal value of each character within the quotes. A single quote may not occur between single quotes, even when preceded by a backslash.
Enclosing characters in double quotes preserves the literal value of all characters within the quotes, with the exception of $, backtick, \, and, when history expansion is enabled, !. The characters $ and backtick retain their special meaning within double quotes. The backslash retains its special meaning only when followed by one of the following characters: $, ', ", \, or newline. A double quote may be quoted within double quotes by preceding it with a backslash. If enabled, history expansion will be performed unless an ! appearing in double quotes is escaped using a backslash. The backslash preceding the ! is not removed.
Então, no seu exemplo:
#!/bin/bash
PLACE="World"
echo "Hello $PLACE !"
echo 'Hello $PLACE !'
resultará nessa saída:
Hello World !
Hello $PLACE !