---
title: "Your Filename Can Be Weird. Your Tools Shouldn't Make It Weirder."
description: "fd --format quietly rewrote non-UTF-8 filenames in pipes, so xargs got paths that did not exist. My fix (sharkdp/fd PR #2197) is merged upstream."
author: "Mohammad Pouya Ghourchiani"
canonical: "https://ghourchiani.ir/blog/fd-weird-filenames-raw-bytes"
image: "https://ghourchiani.ir/og/mark-square.png"
date: "2026-11-07"
lang: "en"
generated: "2026-11-19"
---
# Your Filename Can Be Weird. Your Tools Shouldn't Make It Weirder.

> **TL;DR:** When fd's --format output went to a pipe, filenames that were not valid UTF-8 had their bad bytes swapped for U+FFFD, so tools like xargs -0 received paths that did not exist on disk. PR #2197 writes the original bytes on Unix whenever stdout is not a terminal. It was merged into sharkdp/fd on 7 November 2026.

## Direct answer

On Linux, a filename is just bytes. It does not have to be valid UTF-8. Until this week, `fd --format '{}'` took a name like `test_\xFEinvalid.txt` and, when the output went to a pipe, swapped the `FE` byte for `EF BF BD` (the Unicode replacement character, `�`). The next tool in the pipeline, such as `xargs -0 rm` or `xargs -0 cp`, then looked for a file that **did not exist**.

My fix, [sharkdp/fd #2197](https://github.com/sharkdp/fd/pull/2197), writes the original bytes on Unix whenever stdout is not a terminal. It is merged upstream and will ship in the next fd release.

## The computer that knew better 🤖

Imagine you hand your computer a file with an unusual name, and it says:

> "Hmm, I don't like that. Let me fix it for you!"

It doesn't rename the file. The file on disk stays exactly as it was. What changes is **how the name is passed along** to the next tool. Ask for that name and you get a "cleaned-up" version instead, which matches nothing on disk.

Nothing crashes and nothing logs an error. You find out later, when `xargs` says `No such file or directory` and you start doubting yourself.

Software shouldn't be that judgmental.

## Reproduce it in 10 seconds

The steps are from the original report, [sharkdp/fd #2189](https://github.com/sharkdp/fd/issues/2189):

```sh
mkdir t && touch "t/test_$(printf '\xfe')invalid.txt"

# plain fd: the raw byte survives
fd invalid t | xxd | head -1

# --format before the fix: FE became EF BF BD
fd --format '{}' invalid t | xxd | head -1
```

Plain `fd` already did the right thing. Only the `--format` path went through `to_string_lossy()`, which is Rust's polite way of saying "I'll replace anything I don't understand". There was even a comment marking the spot:

```rust
// TODO: support writing raw bytes on unix?
```

That TODO is the whole story. Someone had already seen the problem and left a note for later.

## The fix: don't change what you weren't asked to change

The change in `src/output.rs` is small. If we are on Unix and stdout is **not** an interactive terminal, write the bytes of the `OsStr` exactly as they are. Otherwise keep the old behavior:

```rust
#[cfg(unix)]
{
    use std::os::unix::ffi::OsStrExt;

    if config.interactive_terminal {
        print_entry_format_base(stdout, &output, config)
    } else {
        // Preserve the original path bytes when stdout is not an interactive terminal.
        stdout.write_all(output.as_os_str().as_bytes())
    }
}
```

Why keep the lossy version on a terminal? Because a human is reading it. A `�` on screen is honest and readable, while a raw `0xFE` byte can garble the terminal. A pipe has a program on the other end, and the program needs the real bytes.

This mirrors what fd's normal output path already did, so `--format` now behaves the same way as plain `fd`.

## The tests carry the weight

The 22 lines of code mattered less than the 54 lines of tests. `test_format_invalid_utf8` creates a real file with a `\xFE` byte in its name and checks every placeholder byte for byte:

| Placeholder | Expected output |
|---|---|
| `{}` | `test1/test_\xFEinvalid.txt` |
| `{/}` | `test_\xFEinvalid.txt` |
| `{//}` | `test1` |
| `{.}` | `test1/test_\xFEinvalid` |
| `{/.}` | `test_\xFEinvalid` |
| `{}` with `--print0` | `test1/test_\xFEinvalid.txt\0` |
| `{}` with `--path-separator =` | `test1=test_\xFEinvalid.txt` |

The test is skipped on macOS, because APFS coerces names to UTF-8, so you cannot create the broken file there to begin with. It is also skipped on Windows, which stores paths as UTF-16, and that is a different problem.

## What I took away

- **Read the TODOs.** Some of the best first contributions are already written down in the code as a question.
- **Reproduce it with `xxd`, not with your eyes.** A terminal will "help" you and hide the bug. A hex dump won't.
- **Keep the scope small.** Terminal output, Windows and `--path-separator` behave as before. Only the broken case changed.
- **Be polite about overlap.** The issue's author said they had a fix ready, but I couldn't find a linked PR. So I opened mine and said in the PR that I'd close it in favor of theirs if their approach differed.
- **Credit the reviewer.** [Thayne McCombs (@tmccombs)](https://github.com/tmccombs) reviewed the PR, pushed the final commit and merged it. A maintainer's time is the scarcest resource in open source.

## By the numbers

- **Project:** [sharkdp/fd](https://github.com/sharkdp/fd), the fast `find` alternative written in Rust
- **PR:** [#2197: fix: write raw bytes for piped --format output on unix](https://github.com/sharkdp/fd/pull/2197)
- **Fixes:** [#2189](https://github.com/sharkdp/fd/issues/2189)
- **Diff:** 3 files, +78 / −2 (`src/output.rs`, `tests/tests.rs`, `CHANGELOG.md`)
- **CI:** 20 checks, 19 passed and 1 skipped, none failed
- **Status:** merged into `master` on 7 November 2026

No unwanted makeover. No disappearing files. Just let the filenames be themselves. 💖

## FAQ

### Why does xargs say "No such file or directory" for a file that exists?

Often because a tool earlier in the pipeline changed the name's bytes. If a filename is not valid UTF-8, a tool that converts it to text may swap the bad byte for `�`, and `xargs` then asks the kernel for a name that does not exist. Pipe the list through `xxd` and compare it with the real name.

### Can a Linux filename contain invalid UTF-8?

Yes. To the Linux kernel a filename is any sequence of bytes except `/` and the NUL byte. UTF-8 is a convention, not a rule. macOS (APFS) is different: it requires valid UTF-8 names.

### What is the � character (U+FFFD)?

The Unicode replacement character. Software inserts it where it found bytes it could not decode. In UTF-8 it is the three bytes `EF BF BD`, so one bad byte becomes three different ones.

### Which fd version includes the fix?

It is in the changelog for the release after fd 10.5.1. Until then you can build fd from `master`.

### How do I see the real bytes of a filename?

Use a hex dump instead of your eyes: `ls | xxd` or `fd pattern | xxd`. For the basics of reading those bytes, see [ASCII table and hex to ASCII in the terminal](/blog/ascii-table-hex-to-ascii-terminal).

## Links

- [ASCII table, hex to ASCII and back, in the terminal](/blog/ascii-table-hex-to-ascii-terminal)
- [All upstream contributions: /open-source](/open-source)
- [Upstream contribution diary: Biome, Nitro, Watchexec, ripgrep](/blog/upstream-contribution-diary)
- [Fixing a Biome false positive for forwardRef](/blog/biome-forwardref-false-positive)
- [About the author](/about)
