TopolearnReal file systems contain names that are not valid UTF-8; tools that crash on them are fragile.
Operating systems store file names differently. Linux file names are arbitrary bytes (anything except / and NUL); Windows uses UTF-16. Python presents them as str, so it needs a translation.
On POSIX, Python decodes names with the filesystem encoding (sys.getfilesystemencoding(), UTF-8 on modern systems) and the surrogateescape error handler: each byte that isn't valid UTF-8 becomes a lone surrogate character such as '\udcff'. That way any name round-trips: os.fsencode turns the str back into exactly the original bytes, and os.fsdecode goes the other way.
The catch: such a string cannot be encoded to UTF-8 for printing or JSON; it raises UnicodeEncodeError. Encode with errors="surrogateescape" or "replace" when displaying.
Most os functions also accept bytes paths and then return bytes: os.listdir(b".") gives raw names with no decoding at all.
os.PathLike is the protocol that lets open, os and shutil accept path objects: any class with __fspath__() returning str or bytes qualifies. os.fspath(p) returns a str/bytes unchanged and calls __fspath__ otherwise; library code should call it on every path argument it receives. pathlib.Path implements this protocol.
Example
import os
raw = b"report-\xff.txt" # not valid UTF-8
open(raw, "w").close()
name = os.listdir(".")[0]
print(repr(name)) # 'report-\udcff.txt'
print(os.fsencode(name) == raw) # True: exact round trip
print(os.listdir(b".")[0] == raw) # True: bytes in, bytes out
print(name.encode("utf-8", "replace")) # b'report-?.txt'
class Asset:
def __init__(self, key):
self.key = key
def __fspath__(self):
return f"asset-{self.key}.bin"
with open(Asset(7), "wb") as f: # open accepts any PathLike
f.write(b"\x00")
print(os.fspath(Asset(7))) # asset-7.bin
print(os.path.exists(Asset(7))) # TrueRemember
- POSIX names are bytes; Python decodes them with surrogateescape so they always round-trip
os.fsencode/os.fsdecodeconvert names; bytes arguments return bytes- Implement
__fspath__(os.PathLike) and callos.fspathin your own path-taking APIs
Watch out for
- Printing or JSON-encoding file names without an error handler; undecodable names raise UnicodeEncodeError.