Hex, binary and two's complement
Reverse engineering means reading numbers all day. Before anything else it's worth getting fast at three things: converting between hex and binary, knowing the sizes of the common integer types, and reading negative numbers in two's complement.
Hex is shorthand for binary
One hex digit is exactly four bits, so conversion is a lookup, not arithmetic:
0 0000 4 0100 8 1000 C 1100
1 0001 5 0101 9 1001 D 1101
2 0010 6 0110 A 1010 E 1110
3 0011 7 0111 B 1011 F 11110xB6 is 1011 0110. 0x80 is 1000 0000: only the top bit of a byte. That's why hex is everywhere in tools: two digits per byte, and the bit pattern is visible at a glance.
Worth memorising: powers of two up to 0x10000 (65536), and that 0x100 = 256, 0x1000 = 4096 (a memory page on x86), 0x10000 = 64 KiB.
Sizes
On x86-64 Linux:
char: 1 byte (8 bits)short: 2 bytesint: 4 byteslong, pointers: 8 bytes (on 64-bit Windowslongis still 4)
Assembly names them by size: byte, word (2), dword (4), qword (8). That's where DWORD PTR [rbp-0x4] comes from.
Unsigned is easy
An 8-bit unsigned value covers 0x00 to 0xFF, 0 to 255. A 32-bit one covers 0 to 0xFFFFFFFF = 4,294,967,295.
Signed: two's complement
CPUs store negative numbers in two's complement. The rule for an n-bit number: the top bit counts as minus 2^(n−1) instead of plus.
For a byte, the top bit is worth −128:
0x7F=0111 1111= 127, the largest positive byte0x80=1000 0000= −1280xFF=1111 1111= −128 + 127 = −1
To negate a value by hand: flip every bit and add 1. 5 is 0000 0101; flipped it's 1111 1010; plus 1 is 1111 1011 = 0xFB = −5.
The same bits mean different things depending on how the code uses them. 0xFFFFFFFF is 4,294,967,295 as an unsigned int and −1 as a signed int. The bytes don't say which; the instructions do. That's why you'll learn to look at the comparison: jl/jg are signed, jb/ja are unsigned.
Sign extension
Widening a signed value copies its top bit into the new bits, so the value stays the same: the byte 0xFB (−5) becomes the dword 0xFFFFFFFB, still −5. Widening an unsigned value fills with zeros instead.
You'll see both in disassembly:
movsx eax, BYTE PTR [rdi] ; signed char -> int (sign-extend)
movzx eax, BYTE PTR [rdi] ; unsigned char -> int (zero-extend)
cdqe ; eax -> rax, sign-extendedSpotting movsx vs movzx is one of the cheapest ways to recover whether a variable in the original source was signed.
Practice
Convert without a calculator, then check with python3 -c 'print(hex(...))':
0xC8as an unsigned byte and as a signed byte- −2 as a 16-bit value in hex
0xFFFFFF9Cas a signed 32-bit int
Answers: 200 and −56; 0xFFFE; −100.