Parsing binary formats with Python's struct
Decoding a header by hand is the best way to learn a format. Doing it a second time is a waste: write a parser. Python's struct module, in the standard library, turns bytes into numbers with a short format string, and it's enough for most formats you'll meet.
The format string
struct.unpack(fmt, data) reads bytes according to fmt and returns a tuple. The first character sets the byte order:
<little-endian (ELF on x86, PE, most in-memory data)>big-endian (PNG, JPEG, network protocols)
Then one letter per field:
B unsigned char 1 b signed char 1
H unsigned short 2 h short 2
I unsigned int 4 i int 4
Q unsigned 8-byte 8 q signed 8-byte 8
s bytes, e.g. 4s = 4 raw bytes x one padding byteWith < or >, no automatic padding is added: the format describes the bytes exactly.
Example 1: the ELF header
In "The ELF header, byte by byte" we decoded /bin/ls by hand. The same thing in Python:
import struct
FIELDS = ("type machine version entry phoff shoff flags "
"ehsize phentsize phnum shentsize shnum shstrndx").split()
def elf_header(path):
with open(path, "rb") as f:
data = f.read(64)
if data[:4] != b"\x7fELF":
raise ValueError("not an ELF file")
if data[4] != 2 or data[5] != 1:
raise ValueError("only 64-bit little-endian here")
values = struct.unpack_from("<HHIQQQIHHHHHH", data, 16)
return dict(zip(FIELDS, values))
for k, v in elf_header("/bin/ls").items():
print(f"{k:10} {v:#x}")unpack_from reads at an offset (16, right after e_ident) without slicing. The format <HHIQQQIHHHHHH is just the field sizes in order: 2, 2, 4, 8, 8, 8, 4, then six 2-byte fields. On Ubuntu 24.04 it prints type 0x3, machine 0x3e, entry 0x6d30: the same values readelf -h shows.
Example 2: walking PNG chunks
From "Reading a PNG by hand": a chunk is a big-endian length, a 4-byte type, the data, and a CRC.
import struct, zlib
def png_chunks(path):
data = open(path, "rb").read()
if data[:8] != b"\x89PNG\r\n\x1a\n":
raise ValueError("not a PNG")
pos = 8
while pos < len(data):
length, ctype = struct.unpack_from(">I4s", data, pos)
body = data[pos + 8 : pos + 8 + length]
(crc,) = struct.unpack_from(">I", data, pos + 8 + length)
ok = crc == zlib.crc32(ctype + body)
yield pos, ctype.decode("ascii", "replace"), length, ok
pos += 12 + length
for off, name, size, ok in png_chunks("tiny.png"):
print(f"{off:#06x} {name} {size:5} crc {'ok' if ok else 'BAD'}")0x0008 IHDR 13 crc ok
0x0021 IDAT 22 crc ok
0x0043 IEND 0 crc okA few lines, and you have a validator that also tells you about extra data after IEND or chunks with bad CRCs: exactly the things you look for when a file seems "off".
Habits that keep parsers honest
- Check magic numbers first and fail with a clear message.
- Bounds-check every length before you use it:
if pos + 12 + length > len(data): raise .... Real files are truncated or deliberately malformed, and a parser for untrusted input must not trust the lengths inside it. - Print offsets in hex next to every value. You'll compare against a hexdump constantly.
- For bigger formats,
struct.Struct("<HHI")compiles the format once and gives you.sizefor free, andctypes.Structureor a library likeconstructlet you describe the layout declaratively.
When to stop and use a library
For ELF and PE, pyelftools and pefile already handle every edge case; for anything you're learning, writing the first version yourself is how the format sticks.