Mini Kafka
A distributed message broker in Java, written to understand the parts of Kafka that actually carry the design: how a log is stored so records can be found without scanning it, how many connections one process can really hold, and how a consumer knows a record arrived intact.
The problem
Reading about message brokers taught me the vocabulary and none of the constraints. I could describe a segmented, indexed log without being able to say what goes wrong without one. The only way to learn why the design is the way it is, is to build the version that is not, and measure where it stops holding up.
What I built
A distributed Java message broker over TCP: a segmented commit log, a sparse index over it, per-record integrity checking, and a JUnit suite over the whole thing. Maven for the build, Python on the load-generating side.
The interesting decision
Three of them, and each came from a measurement rather than a preference.
Concurrency. One OS thread per connection topped out at 2,072 connections. A virtual-thread model scaled to 10,000, because a thread blocked on a socket read now costs a continuation on the heap instead of a kernel thread — the resource that was scarce stopped being the resource in play.
Lookup. Finding a record meant scanning a full 64 MB segment. I added a sparse index holding one 8-byte entry per 4 KB of log, which turns that scan into a binary search plus a bounded 4 KB read. Sparse is the point: an entry per record would search marginally faster and would not fit in memory, while the 4 KB ceiling makes the difference between them invisible.
Integrity. A corrupted record must never reach a consumer, so every record carries a CRC32 checksum verified on read. I trusted it only after testing it the honest way — a unit test that flips each of the 30 body bytes in turn and asserts that all 30 are caught.
What I would change
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.