Acabei de descobrir por acaso que o Prezto faz isso por padrão (!):
~ ❯❯❯ ll
drwx------@ 3 mflander staff 102 17 Mar 14:21 [1] Applications
drwx------+ 3 mflander staff 102 17 Mar 12:10 [2] Desktop
drwx------+ 3 mflander staff 102 17 Mar 12:10 [3] Documents
drwx------+ 10 mflander staff 340 20 Mar 11:12 [4] Downloads
drwx------@ 55 mflander staff 1870 17 Mar 15:23 [5] Library
drwx------+ 3 mflander staff 102 17 Mar 12:10 [6] Movies
drwx------+ 3 mflander staff 102 17 Mar 12:10 [7] Music
drwx------+ 3 mflander staff 102 17 Mar 12:10 [8] Pictures
drwxr-xr-x+ 5 mflander staff 170 17 Mar 12:10 [9] Public
drwxr-xr-x 23 mflander staff 782 17 Mar 15:00 [10] anaconda3
drwxr-xr-x 5 mflander staff 170 20 Mar 09:36 [11] projects
~ ❯❯❯ cd 4
~/Downloads ❯❯❯