The simplest way to uninstall device drivers before removing the hardware from your PC is to open Device Manager, right-click on the device in question, and then select the Uninstall option. You then click on the OK button in the subsequent 'Confirm Device Removal' dialogue box and, once the deed is done, power down your PC and remove the hardware. However, this method doesn't delete the driver files from your hard drive. A more thorough approach may be available via the Control Panel's 'Add or Remove Programs' applet. Many recent audio interfaces provide a dedicated uninstall option as part of their setup application. Annoyingly, though, even running such an uninstall routine may leave behind various files (and the setup application itself will remain).
The very best option is to run a dedicated stand-alone uninstall utility that will completely erase all references to a particular piece of hardware once and for all, before or after you physically remove it. A classic example is M Audio's Delta Uninstaller, available for download from M Audio's web site Perhaps unsurprisingly, many manufacturers don't publicise such utilities, but you may find one tucked away somewhere on the orginal CD-ROM bundled with the product, or available as an option if you re-run the original Setup.exe file from the CD-ROM. You can even run the latest driver installer utility if it first uninstalls the old drivers and then provides an option to install the new ones. If all else fails, contact the manufacturer or post a request on an associated support forum to find the best approach for a thorough uninstall.