Is there a cryptographic algorithm that encrypts files with a password and produces garbage when the password is incorrect?
Standard cryptographic algorithms will do this. If you use the wrong key in a standard symmetric cipher (like AES) it will decrypt the data into a random series of bytes that will look like nonsensical garbage with overwhelming probability.
A symmetric encryption/decryption function is best thought of as a pseudorandom permutation. For any given key K and any fixed block-sized input (typically block size like 128-bits or 256 bits), your Encrypt function will map each n-bit block to another n-bit block and the Decrypt function will do the reverse mapping. For a slightly different key, you’ll get a completely different mapping.
That said, this inherent property is usually undesired.
So often times there’s an extra check to ensure the key was correct (and if not indicate that decryption failed) — that is they’ll either make sure the padding is valid (which should rarely happen for an incorrect key — though this will occassionally happen) or they’ll check against a Message Authentication Code that verifies the key was valid.