Existem duas coisas diferentes acontecendo aqui, ambas documentadas no manual bash
$ '
A citação simples de cifrão é uma forma especial de citar:
Words of the form $'string' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.
$ "
Aspas duplas em dólar são para localização:
A double-quoted string preceded by a dollar sign (‘$’) will cause the string to be translated according to the current locale. If the current locale is C or POSIX, the dollar sign is ignored. If the string is translated and replaced, the replacement is double-quoted.