TopolearnKnowing the bytes explains compatibility rules, size limits and proxy behaviour.
Protobuf encoding is a sequence of key-value records. Each key is a varint (field_number << 3) | wire_type. Wire types: 0 = VARINT (ints, bools, enums), 1 = I64 (fixed64, double), 2 = LEN (strings, bytes, nested messages, packed repeated), 5 = I32.
- Varints use 7 bits per byte with a continuation bit, so 1 takes one byte and 300 takes two (
AC 02). Negativeint32/int64always take 10 bytes;sint32/sint64use ZigZag encoding to keep small negatives small. - Field numbers up to 15 fit in a one-byte key with the wire type.
- Unknown fields can be skipped because the wire type says how long they are. That is the basis of forward compatibility.
- Repeated scalars are packed into one LEN record in proto3. A later value for a non-repeated field overwrites an earlier one, so concatenating two messages merges them.
gRPC on HTTP/2: a call is one HTTP/2 stream. The request starts with headers (:method POST, :path /pkg.Service/Method, content-type: application/grpc, te: trailers, optional grpc-timeout). Each message is a length-prefixed frame: 1 byte compression flag, 4 bytes big-endian length, then the protobuf bytes. The response ends with trailers carrying grpc-status and grpc-message, which is why a proxy that drops trailers breaks gRPC.
Default maximum receive message size is 4 MB in most implementations; large payloads belong in streams or object storage.
Example
// Hand-encode HelloRequest { name: "Ann" } and frame it like gRPC does
const name = Buffer.from("Ann", "utf8");
const key = (1 << 3) | 2; // field 1, wire type 2 (LEN)
const msg = Buffer.concat([Buffer.from([key, name.length]), name]);
console.log(msg.toString("hex")); // 0a03416e6e
const frame = Buffer.alloc(5 + msg.length);
frame.writeUInt8(0, 0); // not compressed
frame.writeUInt32BE(msg.length, 1); // 4-byte length
msg.copy(frame, 5);
console.log(frame.toString("hex")); // 00000000050a03416e6e
// varint of 300: 7 bits per byte, low group first
const varint = (n) => { const out = []; do { let b = n & 0x7f; n >>>= 7; if (n) b |= 0x80; out.push(b); } while (n); return Buffer.from(out); };
console.log(varint(300).toString("hex")); // ac02Remember
- Key = (field_number << 3) | wire_type, encoded as a varint
- Use sint types for negative numbers (ZigZag)
- gRPC frames: 1-byte flag + 4-byte length + message
- Status travels in HTTP/2 trailers
Watch out for
- Proxies or CDNs that strip HTTP trailers, making every call look like an error.