Direct answer
On Linux, a filename is just bytes. It does not have to be valid UTF-8. Until this week, fd --format '{}' took a name like test_\xFEinvalid.txt and, when the output went to a pipe, swapped the FE byte for EF BF BD (the Unicode replacement character, �). The next tool in the pipeline, such as xargs -0 rm or xargs -0 cp, then looked for a file that did not exist.
My fix, sharkdp/fd #2197, writes the original bytes on Unix whenever stdout is not a terminal. It is merged upstream and will ship in the next fd release.
The computer that knew better 🤖
Imagine you hand your computer a file with an unusual name, and it says:
“Hmm, I don’t like that. Let me fix it for you!”
It doesn’t rename the file. The file on disk stays exactly as it was. What changes is how the name is passed along to the next tool. Ask for that name and you get a “cleaned-up” version instead, which matches nothing on disk.
Nothing crashes and nothing logs an error. You find out later, when xargs says No such file or directory and you start doubting yourself.
Software shouldn’t be that judgmental.
Reproduce it in 10 seconds
The steps are from the original report, sharkdp/fd #2189:
mkdir t && touch "t/test_$(printf '\xfe')invalid.txt"
# plain fd: the raw byte survives
fd invalid t | xxd | head -1
# --format before the fix: FE became EF BF BD
fd --format '{}' invalid t | xxd | head -1Plain fd already did the right thing. Only the --format path went through to_string_lossy(), which is Rust’s polite way of saying “I’ll replace anything I don’t understand”. There was even a comment marking the spot:
// TODO: support writing raw bytes on unix?That TODO is the whole story. Someone had already seen the problem and left a note for later.
The fix: don’t change what you weren’t asked to change
The change in src/output.rs is small. If we are on Unix and stdout is not an interactive terminal, write the bytes of the OsStr exactly as they are. Otherwise keep the old behavior:
#[cfg(unix)]
{
use std::os::unix::ffi::OsStrExt;
if config.interactive_terminal {
print_entry_format_base(stdout, &output, config)
} else {
// Preserve the original path bytes when stdout is not an interactive terminal.
stdout.write_all(output.as_os_str().as_bytes())
}
}Why keep the lossy version on a terminal? Because a human is reading it. A � on screen is honest and readable, while a raw 0xFE byte can garble the terminal. A pipe has a program on the other end, and the program needs the real bytes.
This mirrors what fd’s normal output path already did, so --format now behaves the same way as plain fd.
The tests carry the weight
The 22 lines of code mattered less than the 54 lines of tests. test_format_invalid_utf8 creates a real file with a \xFE byte in its name and checks every placeholder byte for byte:
| Placeholder | Expected output |
|---|---|
{} |
test1/test_\xFEinvalid.txt |
{/} |
test_\xFEinvalid.txt |
{//} |
test1 |
{.} |
test1/test_\xFEinvalid |
{/.} |
test_\xFEinvalid |
{} with --print0 |
test1/test_\xFEinvalid.txt\0 |
{} with --path-separator = |
test1=test_\xFEinvalid.txt |
The test is skipped on macOS, because APFS coerces names to UTF-8, so you cannot create the broken file there to begin with. It is also skipped on Windows, which stores paths as UTF-16, and that is a different problem.
What I took away
- Read the TODOs. Some of the best first contributions are already written down in the code as a question.
- Reproduce it with
xxd, not with your eyes. A terminal will “help” you and hide the bug. A hex dump won’t. - Keep the scope small. Terminal output, Windows and
--path-separatorbehave as before. Only the broken case changed. - Be polite about overlap. The issue’s author said they had a fix ready, but I couldn’t find a linked PR. So I opened mine and said in the PR that I’d close it in favor of theirs if their approach differed.
- Credit the reviewer. Thayne McCombs (@tmccombs) reviewed the PR, pushed the final commit and merged it. A maintainer’s time is the scarcest resource in open source.
By the numbers
- Project: sharkdp/fd, the fast
findalternative written in Rust - PR: #2197: fix: write raw bytes for piped –format output on unix
- Fixes: #2189
- Diff: 3 files, +78 / −2 (
src/output.rs,tests/tests.rs,CHANGELOG.md) - CI: 20 checks, 19 passed and 1 skipped, none failed
- Status: merged into
masteron 7 November 2026
No unwanted makeover. No disappearing files. Just let the filenames be themselves. 💖
FAQ
Why does xargs say “No such file or directory” for a file that exists?
Often because a tool earlier in the pipeline changed the name’s bytes. If a filename is not valid UTF-8, a tool that converts it to text may swap the bad byte for �, and xargs then asks the kernel for a name that does not exist. Pipe the list through xxd and compare it with the real name.
Can a Linux filename contain invalid UTF-8?
Yes. To the Linux kernel a filename is any sequence of bytes except / and the NUL byte. UTF-8 is a convention, not a rule. macOS (APFS) is different: it requires valid UTF-8 names.
What is the � character (U+FFFD)?
The Unicode replacement character. Software inserts it where it found bytes it could not decode. In UTF-8 it is the three bytes EF BF BD, so one bad byte becomes three different ones.
Which fd version includes the fix?
It is in the changelog for the release after fd 10.5.1. Until then you can build fd from master.
How do I see the real bytes of a filename?
Use a hex dump instead of your eyes: ls | xxd or fd pattern | xxd. For the basics of reading those bytes, see ASCII table and hex to ASCII in the terminal.