computer science
metacognition
software engineering
- You must try, and then you must ask
- Runbooks
- Scale later
- The power and cost of abstraction
- Judgment over technique
- Documentation
- Design with migration in mind
- Read the code
- RPC codegen
- Perfect is the enemy of good
- Everything is a trade-off
storytelling
tools
code
Blogs that I've enjoyed reading over the years.
- https://0xadada.pub
- http://dtrace.org/blogs/ahl
- https://blog.acolyer.org
- http://www.brendangregg.com/blog
- http://dtrace.org/blogs/bmc
- https://danluu.com
- http://code.dblock.org
- https://blog.codinghorror.com
- http://gordonbrander.com
- https://blog.jessfraz.com
- https://jvns.ca
- https://graydon2.dreamwidth.org
- https://www.dampfkraft.com
- https://scattered-thoughts.net
- https://bcantrill.dtrace.org
- https://caseyhandmer.wordpress.com
- https://xeiaso.net
- https://twos.dev
- https://brooker.co.za/blog
- https://matt-rickard.com
- https://fasterthanli.me
- https://eatonphil.com
- https://john-millikin.com
- https://patrickcollison.com
- https://sahillavingia.com
- http://the-witness.net/news
- https://www.aspiringpeasant.com
- https://grahamc.com
- https://paul-snively.github.io
- https://yacinemtb.substack.com
- https://fchollet.com
- https://fly.io/blog
- https://blog.cloudflare.com
- https://lamport.azurewebsites.net/pubs/pubs.html
- https://martin.kleppmann.com/archive.html
- https://aws.amazon.com/builders-library/
- https://aphyr.com/posts
- http://dancres.github.io/Pages
- https://www.igvita.com/2012/07/19/latency-the-new-web-performance-bottleneck
- https://research.google/pubs/the-tail-at-scale
- http://christophermeiklejohn.com/distributed/systems/2013/07/12/readings-in-distributed-systems.html
- https://shipilev.net
- https://huyenchip.com/blog
- https://copyconstruct.medium.com
- https://mcfunley.com/writing
- https://eli.thegreenplace.net/archives/all
- https://ericsink.com/index.html
- https://fgiesen.wordpress.com
- https://fabiensanglard.net
- https://bellard.org
- https://www.akalin.com
- https://rakyll.org
- https://blog.regehr.org
- https://preshing.com
- https://www.snellman.net/blog
- https://karla.io
- https://kate.io
- https://lindzey.github.io/index.html
- https://tratt.net/laurie/blog
- https://idea.popcount.org
- https://tonsky.me
- http://psy-lob-saw.blogspot.com
- https://www.windytan.com
- http://pvk.ca
- http://rachelbythebay.com/w
- https://blog.pizzabox.computer
- http://yosefk.com/blog
- https://flownet.com/ron
- https://blog.rongarret.info
- http://blog.nullspace.io
- https://huonw.github.io
- https://manybutfinite.com
- https://neverworkintheory.org
- https://kavyajoshi.me
- https://blog.lambdaclass.com/austral
- https://borretti.me
- https://rust-lang.github.io/rfcs
- https://dr-knz.net
- https://rsdoiel.github.io
- https://matklad.github.io
- https://drewdevault.com
- https://parallelthoughts.xyz/
I play a lot of pool/billiards, and under a lot of different rule sets. One of the formats I play is APA 9-ball rules. (Technically, now that I live in Japan, it's JPA, but the rules are the same as its parent American association.)
For a brief background of the rules, in 9-ball, you shoot the balls 1 through 9 sequentially in order, and typically the winner of a game is the player that pockets the 9 ball. APA rules changes the scoring of the game, where instead of winning 1 game point for each 9 ball pocketed, you win 1 point for each ball pocketed, and an extra point for pocketing the 9.
While playing a 9-ball match recently, I was the victim of a truly incredible streak of "rolls" by my opponent. (For non pool players, a "roll" is when you get lucky.) My opponent missed 11 times total in the match. 8 of those times I was totally snookered (no view of the object ball), and 1 time they fluked a ball in and continued their inning for a break and run. What this basically means is, I only had 2 capitalizable opportunities at the table following my opponent missing during the whole match.
I don't begrudge my opponent at all; they are a good player and a good person. It was an opportunity for me to try to exercise some stoicism in the face of misfortune. 🙃
⅔ of the way through the match, after watching another roll go by, I caught myself thinking "if this were a set match rather than APA rules, it would seem more winnable despite these rolls."
And that got me to thinking... It would be fun to create a simulation of two players playing 9-ball, add luck into the parameters, and see what effect the rules had on the results...
The simulation of luck in 9-ball
In my simulation (code here), there are two players: Alice and John. Alice is a better player than John, but they are both skill level 7s on the APA skill rating system. Alice is just a little bit more likely to put together strings of 5+ ball runs than John is.
In this twisted simulation world, Alice can never get lucky. Only John can get lucky. I define "getting lucky" as "that player gets another typical turn at the table." I think this is an accurate proxy implementation for "getting a roll". For example, safeing your opponent on a miss or fluking a ball in.
Each turn a player takes, they basically roll a weighted 10-sided die (marked with numbers 0 through 9). The number they roll indicates the amount of balls they run. The only difference is, after John misses, he gets to roll another die, and if he rolls high, he gets another turn.
Note that nothing in this simulation accounts for the differences in strategy between the games. Arguably, accounting for strategy differences would even further attenuate the effect of luck in set match rules, since the goal is not to take as many points as possible, the goal is to make the 9 ball, and a more skilled player would be better at crafting an executable plan to make the 9 ball. The point of this simulation is just to check what effect the rules difference has on the effects of luck.
The experiment
Chance to run # balls in a single inning | John | Alice |
---|---|---|
0 | 7% | 5% |
1 | 13% | 10% |
2 | 21% | 20% |
3 | 21% | 20% |
4 | 18% | 18% |
5 | 11% | 12% |
6 | 4% | 5% |
7 | 2% | 4% |
8 | 2% | 3% |
9 | 1% | 2% |
As you can see, Alice is just a little bit better than John. Alice has a 15% chance to make 1 or fewer balls, while John has a 20% chance of doing so. Alice has a 14% chance at running 6 or more balls, but John has a 9% chance. Small margins, but they add up over lots of matches.
Running the simulation
OK, let's have Alice and John play a million matches against each other and
we'll record the results. (To be exact, I simulated 131,072
matches across 8 different configurations, for a total of 1,048,576
simulated
matches of 9 ball.)
I had John and Alice alternate who starts the break (i.e. who wins the lag) for each match.
Zero luck (setting up ground truth)
Rules format | Winning % for Alice |
---|---|
APA | 68.9% |
Set match race to 5 | 57% |
With no luck involved, Alice wins 69% of the time (nice!) in APA, and 57% of the time in the set match (race to five).
Adding luck
Rules | Alice Win % (20% luck) | (40% luck) | (50% luck) |
---|---|---|---|
APA | 34% | 9% | 3% |
Set match | 43% | 27% | 19% |
As you can see, with only 20% luck (every five shots, John gets a roll) Alice's
69% APA winning chance is already cut in half to 34% and she's a big underdog!
At 50% luck (every other miss, John gets a roll), Alice only wins 3% of
131,072
matches!
In contrast to the APA rules, even with a 50% luck rate for John, Alice still manages to take about ⅕ of the race to five set matches. In a set match, Alice's winning rate reduces linearly with the amount of luck John gets, but in APA her winning rate reduces geometrically. Basically, if your opponent is getting a bunch of rolls, you'd rather be playing a set match.
Getting rolls
Now, you might say "OK, but it's not that realistic for your opponent to get lucky 20-50% of the time", but APA skill level 7s, 8s, and 9s only really miss a handful of balls a match. So the variance is high — they only need a couple of rolls on those few misses to edge out their opponent, even if they're technically outclassed.
Another point worth mentioning is, when you get a roll, you also get another chance at getting a roll. Because getting a roll means getting back to the table in a favorable position, you essentially get a free roll at another roll. So you're not only gaining points but you gained another chance at rolling the die on getting lucky again.
Intuition for set matches being less affected by luck
In a set match, if you get a roll, that roll only matters it if contributes to you pocketing the 9 ball. On the other hand, every roll you get under APA 9 ball rules contributes to your winning chances (in an amortized sense) because you can model an expected number of points per inning at the table.
Wrap up
When your opponent is getting lucky every other shot in APA, and it feels like you can't win... don't despair! Under those conditions, statistically speaking, you should lose 97% of the time!
And all of this is to say nothing of the human element: that we tend to play worse when nothing is going our way. But we need to work hard to be stoic under pressure when we're at the mercy of bad luck, because the odds are ever-so stacked against us.
RPC code generation
I imagine most of you are familiar with RPC client and server codegen patterns.
- Write a interface description, for example OpenAPI or gRPC and Protocol Buffers.
- From those declarative descriptions of the types and functions, generate the stub code for the server.
- From the same declarative descriptions, generate clients that can call the server.
What makes this great is that you can generate a client for whatever languages you like with minimal marginal cost. Those clients stay in sync via the interface description and modulo deployment timing differences, everything is always in sync. The generated code has native types and an SDK in the client language of choice and everything gets serialized the same on the wire, so the server is happy.
dropshot taught me something new
Dropshot is a general-purpose crate for exposing HTTP APIs from a Rust program released by Oxide Computer Company. Notably, Dropshot is OpenAPI compatible, which is where it comes into this story.
Dropshot does not do Step 1 or Step 2 in the above pattern. Instead, Dropshot generates an OpenAPI spec from the server code itself. This is brilliant, because it dodges a few concrete problems and it gets at a deeper essential truth about this pattern: The server code is the authoritative source of the interface description, not the other way around.
Why is this concretely better?
- You no longer have to write an interface description. This is saved work.
- When the interface changes, merging the generated server code with the existing server code is messy. Because there no longer is any generated server code, you get to dodge this error-prone step.
- You benefit from the language's static analysis and compilation checks insteading of having a two-step process of write interface description -> merge generated code into server code.
- You no longer have the extra complexity of code-generation in your dev-workflow for the server.
Why is the server code actually the root of authority?
Unless you have or plan to have multiple implementations of the server using different stacks, there's only ever one implementation of the server.
As long as the server is serving requests, it is axiomatic that the behavior and interface of the server is exactly specified by the server's code. On the other hand, it is possible that the server simply diverges from the interface description. For example, when merging the generated code, human error might cause the divergence. Or it may happen naturally during some refactoring or bugfix even when you weren't trying to change the interface.
However, if you generate the interface description from the server code itself, it is not possible for it to diverge. (A bug in the generation code is possible, but this case is also true for generating the server code.)
Things I learned about Japanese culture living in Japan. Especially how it differs from U.S. culture.
- The lack of a car culture is transformative and its effects cannot be understated.
- Trains craft their own surrounding culture and urban fabric.
- Train stations become nexuses of community, culture, and place.
- The "last train" of the night is its own framing device of timelines, culture, and community.
- You walk a lot in Japan.
- Walking allows you to perceive places differently, deeper.
- Walking is fundamentally more human(ist).
- When you walk, you interact with passers-by and your environment. They interact with you. It's a bidirectional relationship.
- People almost always appear exceptionally polite.
- There is what people say and do in public and what people think. You are expected to read the room. If you can't, you're going to be missing a high-bandwidth channel of information. (本音・建前)
- Saying goodbye or see you later in Japan is expected to take much longer than in the U.S. If you don't drag it out a bit, it feels callous.
- A group going their separate ways at the end of a night is a complicated dance.
- Separation is delayed and rejected until physics intervenes.
- In the U.S., we'd say "see you next time", walk away and not look back. It's 3 seconds.
- In Japan, it's drawn out. Many goodbyes are said, many bows are exchanged. You're meant to look back on the escalator and keep waving until they're out of sight.
- The Japanese life is a very scheduled one; adhoc meetings are rarely a thing.
- In the U.S., "let's grab coffee sometime" without scheduling anything is genuine in 90% of cases.
- In Japan, a commitment without a date is a polite way of saying "no" in 80% of cases.
- The most typical "hard no" you'll hear in Japan is "that's a little difficult."
- Most experiences in Japan are highly polarized, inherently extremist.
- You'll often get the best experience in a category, or the worst.
- Most of the time, it's the best experience.
- Excellence is the norm; deviation is hammered like the nail that sticks out.
- Tokyo's hustle and bustle is somehow self-fulfilling; even if you're not in a hurry, you begin to feel as though you ought to be.
- I didn't expect to like Tokyo. Megacities were never my thing; I figured an Osaka or Kyoto would be more my speed. Tokyo is different, it's less of a megacity than it is a large collection of individual neighborhoods, each with their own unique culture and atmoshphere.
- When leaving leftover food, I've seen Japanese people bring their hands together and apologize for the neglected remainder. It wasn't about being seen or being ashamed, it was a genuine prayer of remorse.
- When confused or don't know what to, a common strategy is 前の人 (follow the person in front of you). This comes in handy at train stations. If you're caught in a swarm of moving people and don't know what way to go, going with the flow of people is usually a good bet.
- Conformity in Japan is kind of a trope and oversimplified. It's also deeply related to (10). Conformity in public is quite high, but the honne-tatemae divide allows people to be non-conformist in their inner worlds. In some ways, the U.S. is more conformist than Japan.
- No particularly sensitive way to put this. Many Japanese men have a bit of pedophilia. It wouldn't be shocking to hear a mid-thirties salaryman say 18 year olds are his type.
Anecdotes from my time living in Japan.
The woman with the fan at the coffee bar
Summer 2022, 8AM, Tsukiji Peppers Cafe.
100% humidity, 35 degrees Celsius. I sit down on a wooden crate stool at a coffee bar in a narrow, intimate alley of the famous Tsukiji fish market. I'm sweating bullets, just like everyone else. I order an iced coffee, hoping to kill two birds with one stone: beat the heat and jettison my jet lag.
A fashionably-dressed woman in her mid-thirties wearing a flowery dress and a decidedly dignified air sits down to the left of me. The owner gives her a warm welcome. She must be a regular. She pulls out a fan and starts fanning herself without reserve.
Her gaze wanders to the foreigner sweating bullets on her right and she opens a conversation in an all-too-familiar way: 暑いですね (it's hot, isn't it?). You hear this phrase all the time on hot days. Usually it has the same semantics as "how's it goin'" or "howdy" — it's a polite way to acknowledge the existence of another human in your space, but doesn't signal a desire to open up a conversation. But something about the way she said it was different — she wanted to chitchat.
I gave her my agreement that it was indeed too hot today. Before I could say anything else she snapped her fan closed and held it out to me. "Fan yourself, you're too hot." I was taken aback by the directness, but I wasn't about to disagree; I accepted the fan and the task. As I fanned myself for 5 minutes or so, we chatted while we waited for our coffee. She asked all the usual questions: where are you from, how long have you been here, what brought you here, and so on. We had a fun chat. She introduced me to the owner and the other regular patrons.
In a lull of the conversation, I offered her my thanks for the fan and held it out for return. She studied my face with a suspicious furrow in her brow. Rejecting my outstretched arm returning the fan, she said "I still see sweat. You're still too hot. Keep fanning."
As it would happen, it was the owner's birthday that day. Just as I was finishing my coffee and egg sandwich, the whole family rained down on the place, a good number of them, as far as I could tell, still drunk from last night. The old man of the family, smartly dressed in a pin-stripe suit and Tag Heuer tells his son, the owner, "a round of champagne for everyone, on me!"
"Everyone?" he says from behind the counter, eyebrows raised.
After a small pause, "Yeah, everyone."
The old man taps me on shoulder with a wry smile, losing his balance a bit and leaning on me. "You drink alcohol?" I glance at the clock. 8:14AM. "Yeah, here and there." The old man starts gleefully rubbing his hands together and makes hurried hand gestures at his son to send the next glass my way.
That's the story of my first morning in Japan: delicious coffee and breakfast, a flat rejection of an attempt to return a borrowed fan, and a glass of champagne, all before 8:30AM.
My first time using a ramen ticket machine
Summer 2022, 9PM, Higashi Ginza Ramen-ya (Ramen Takahashi, IIRC)
It was one of my first days in Japan. Maybe the second or third day.
I'm fiddling with the ramen ticket vending for only a ten or so seconds when a salaryman queues behind me. "Just great," I think, feeling the pressure of holding someone up, — "my first time using one of these and I don't even get ten seconds to get my bearings."
I know what I want and I thought I understood the process and how to use the machine, but when I press the button, nothing happens. I was expecting it to prompt me with an amount on the screen.
After spending another 10 seconds, I look back, apologize and wave him forward to go before me, 「どうぞ、お先に」(please go ahead of me).
「いいのかな?」(but is that really alright?).
Then, he stepped up and graciously helped me get it to work.
It turns out you put the money in first, then push the button. Duh!
The wallet and passport at the Ginza hotel
Fresh off a 10 hour red-eye from the U.S., I was checking into a business hotel in Ginza, Tokyo. Sleep deprived and just through the coronavirus precautionary checks in NRT, I was a bit exhausted. After checking in, I was in a bit of a hurry to collapse into the bed, and booked it to the elevator. Ninth floor.
As I'm fumbling with the key card to open the door, I hear the sound of the phone ringing in my room. Well, that's a tad bit strange. I figure out the door and find my way to the phone, picking it up.
"Hello," I answer, as I hear a knock at my door.
"Hello, I'm calling to let you know that you left your wallet and passport at the front desk. We're running it up to you now."
"Oh, thank you so much."
I go to answer the door. The other front desk staff member had ran the wallet up to me. As I profusely thank them, they hand me my wallet and passport with both hands, apologizing for the interruption.
The Yukata in the department store
I was on the way to Meiji Jingu Hanabi Taikai (a big fireworks festival), donned in Yukata (a summer kimono) along with everyone else. We make a stop at a department store on the way for the restroom. I'm waiting outside and an old man walks by me and, noticing my Yukata, says 「かっこい!」(basically the equivalent of "badass" or "cool").
Contrary to western intuition, Japanese are very happy to see foreigners in traditional Japanese clothing. It is seen as an appreciation of the culture rather than appropriation.
The confused woman in the elevator
Bic Camera (dept. store, kind of like Target I guess). I'm alone in the elevator. (The store has 8 floors.) I'm on my way up to the 8th floor and the lift stops at the fifth floor.
A middle-aged woman gets on, and presses the button for the fifth floor. Confused that nothing happens, she presses it again. Then again. I tell her, in Japanese, "the fifth floor is the current floor". As if the victim of a terrible jump scare she jolts her body, looking at me and says, 「日本語上手!」 ("Your Japanese is good!"). With a nervous laugh, she selects another floor.
The girl in front of me in the ramen line
I queue to select my order from a ramen vending machine. It's quite late, maybe 10pm. The young high-school girl (unmistakably, because school uniforms) in front of me hesitates for perhaps 3 seconds before making her order. She collects her change and food ticket, then turns to me and says 「済みません」(sumimasen), as if to apologize for inconveniencing me by taking 8 seconds instead of 5 seconds while in front of me in line.
The school girl that crosses the street in front of a taxi
A young school girl waits to cross an intersection. Unexpectedly, a taxi stops and waits for her to cross. She bows deeply to the taxi, sprints across the street, pivots on a dime to face back towards the taxi with a ballerina-esque pirouette and snaps her body into another deep bow to the taxi as it drives away.
All of this occurred in just a handful of seconds, on a rainy evening in Tokyo. I watched through my window, seated in another taxi waiting at a red light on the orthogonal street.
The elderly customers leave a boutique in Ginza
An elderly couple walks out of a boutique store with a shopping bag of purchased goods. A man and woman, staff of the store, see them to the exit. In extremely polite language, they thank the elderly customers, and deeply bow, holding the bow. The elderly customers accept the thanks and take their leave.
The staff of the store continue to hold the bow together, not looking up, not moving, not speaking, standing in the doorway of their store. The elderly couple keep walking away, getting smaller and smaller in the alleyway. The staff hadn't moved an inch.
I began to feel uncomfortable with how long I had been watching, even though I was confident my observation was undetected. I walk away. I steal a last glance before I turn a corner. The customers were long gone.
The two staff hadn't yet moved. I never did see them rise from their bow.
The office workers at the entrance to the train station late at night
After a night of heavy drinking with teammates, a gaggle of office workers clogs up the ticketing gate flow late at night at Higashi Ginza station.
In the politeness hierarchy, it seems there are some trump cards. To prematurely depart a group outing without the protracted "goodbyes dance" is, it seems, the worse offense than inconveniencing and impeding the ticketing gates of a train station.
Everyone succssively continues to bow at each other and say their goodbyes in various forms. Once the cycle begins, it catalyzes. Planning the next meeting, more goodbyes, more bows. They take a few steps away, then turn back, and start saying their goodbyes again, bow some more for good measure. This process can repeat quite a number of times. It seems almost to be a game of chicken: Who will blink first and start walking away without looking back?
The only customer at a brunch restaurant
Lunch. Kagurazaka, Tokyo. Saturday afternoon. I'm the only customer at a brunch cafe, eating my food. The waitress stands near the cash register, eyes fixed on me, standing at attention with hands folded neatly behind her back. She's waiting to see if there's anything I need, so she can pounce upon any request I have with sub-second response time. I try to pretend I cannot feel her gaze.
Service in Japan is impeccable, and you come to love aspects of it. But sometimes it has a palpable intensity.
The married couple I befriended takes me to dinner
I made some friends with a married couple perhaps 10-15 years my senior. We sometimes go out for dinner. Despite great efforts, I'm not allowed to contribute to paying for any part of the meals. Every time we go, it's their treat.
I confided to a mutual (even older) friend of ours that I felt guilty to always be treated. I told him I wish they would let me pay. He said "What do you mean? That's just the way it is: The older ones pay for the younger ones. Just say 「ご馳走様でした」(thank you for the meal) and move on!"
Hiring is hard. Deciding whom to let hire you is at least as hard.
I think the complexity class of deciding whom to let hire you is a superset of the complexity class of deciding whom to hire. It has the same hardness properties of hiring: deciding what kind of people you want to work with, and then selecting a set of people that are those kind of people. Then it adds more complexity: deciding what kind of leadership chain you want to work with, what kind of domain you're interested in working in, what level of compensation you're willing to accept, what kind of workplace culture you find appealing, and so on and so forth.
Just knowing what you want is hard enough
But we also have the (difficult!) task of evaluating if a given opportunity matches our criteria. Both of these tasks are superbly difficult.
A simple framework for evaluating jobs
Maybe a lightweight framework for evaluating jobs using simple multiple linear regression will help us develop a pattern of thinking for better evaluating job offers.
Factors
Factors are attributes or qualities that a job can have. Try to rank your factors and assign a weight to each of them.
Lots of people go job seeking without even knowing what their factors are, let
alone their weights. Most people end up defaulting to a revealed preference of
factors={compensation=1.0}
. I think this is a false revealed preference, not a
true one, born from a lack of deliberateness. At the least, it is certainly true
for me that in the past, I over-indexed on compensation as a factor due to
under-deliberating my job-seeking process.
It is important that this step comes early in the process, because your factors and their weights are an input to:
- who you accept interviews with
- how you conduct your reverse interviews, and
- what data points you're looking for
Some examples of factors
people
: Do you like your co-workers? Are they kind? Do they make you better?culture
: Is the culture inspiring? Do its values align with yours?mission
: Is the mission doing good? Is it interesting? Will it change the world?technology
: Is the technology robust, scalable, "boring", efficient, simple?compensation
: Does the compensation allow you the lifestyle you want? Can you buy a house?location
: Do you like the office location? Do you have to relocate?scope
: Does the role have the breadth you want? The impact you want?learning
: Will you be learning in the role? Understanding new domains or systems?growth
: How much will this role grow your career? Is there upward mobility?remote
: Does the role allow for remote working? 0→zero tolerance, 0.5→hybrid, 1.0→full remote. This measure is interesting because full remote is not necessarily better than hybrid for me. I prefer a hybrid arrangement, with flexibility.
Interview and evaluate
I wish I had a robust framework on how to evaluate your factors. This is one of the hardest parts. From your factors and their weights, try to derive a strategy for reverse interviewing during your loop. Try to find good questions and ask them to get data points. Try to find other means of gathering data on your factors, out-of-band of the interviews. Contact employees that recently left the company, read forums, ask friends. Read their launch posts and blogs.
After calculating your weighted scores (sum of each factor's score times its weight), each role should have a score, modulo some epsilon error bounds.
Don't get too caught up in false rigor. This process isn't meant to be the decision process. It's just a framework to help you gather better data and ask yourself the right questions. If you end up disagreeing with the scores, you can try to feed this back into your factors and weights.
What are my weights?
My weights are always a work in progress, and I learn new things about my preferences every year. Right now, this is what seems right to me.
people = 1.00
culture = 1.00
location = 1.00
mission = 1.00
technology = 1.00
learning = 1.00
growth = 1.00
scope = 0.80
remote = 0.10
compensation = 0.10
Deriving from these numbers I just scribbled down, for me to consider a job with a 50% score on people, all other things being equal, it should pay 5x the job with a 100% people score to be in the running. Put differently, I'm willing to take at least a 1/5th pay cut to work with excellent people as opposed to people that I can merely tolerate. This feels ~order-of-magnitude correct.
Feedback loop
When you're in a job that you like, try to figure out if there were unidentified factors that make it great. Add them to your list. When you're quitting a job that didn't work out for you, try to identify the testable factors that capture why you didn't like the job. Weight them, and add them to your list. If something is worth quitting a job over, it's probably worth a highly-weighted spot in your factors ranking.
Questions to ask prospective teams
How are engineers asked to track work? #culture
- JIRA→bad
- weekly status updates→bad
- burndown charts→bad
- kanban→good
- we don't→good
How do you deploy your software? #tech
- not deployment pipelines→bad (modulo company size and what they deploy)
How do you build your software? #tech
- standard build system→good
- random→bad
Does your software also build and test locally on dev machines? #tech
- no→bad
Can I run native linux on my laptop? #culture
- no→makes me sad but I understand IT constraints
How many source repos do you have? #tech
- monorepo→best
- less→better
- microrepo koolaid→not great
Do you have sprints? #culture
- yes→kinda bad
- qualified yes→better
- no→good
- why would we do that→best
- kanban→best
What is the process to expense something? How long does it take? Do you do it often? #culture
My experience here is that overly frugal (frupid) organizations are often more wasteful. Especially when the mechanism to "keep frugal" is to add tons of friction for expensing things. This costs me patience and my time (which is also your time).
Anecdotally, companies that have a low friction means for expensing things and do it more regularly are less wasteful (Stripe, Google). If you can't trust your employees within basic expensing bounds, you have a hiring problem, not an expensing system problem.
Do engineers change code on repos owned by other teams? #culture
- sometimes→good
- never→mild concern
- cannot→dealbreaker
Do teams share common infra/practices for build systems, deployment, and monitoring? #tech
- no→bad
Do teams share common infra for tickets, design documents, chat/email, etc? #culture
- no→bad
What's are the most important qualities in a leader? #people #culture
Good
- works in service of autonomy mastery, and purpose
- sets vision
- humanist, empathetic
- thinks big
- servant leadership
- builds culture (over managing work)
- removes wasteful process
- deeply understands that ICs do the real work
- says no a lot
- asks questions, especially "dumb" questions
- opinionated but approachable
- individuals matter
🚩
- manages work instead of creating culture
- sees ICs as fungible assets
- anything that's basically a dogwhistle for "good at politics"
- sees reports as something to extract value from
- over-indexes on OKRs
- "pay for performance"
- process > people
- takes agency away from reports
- "alignment"
- indexes on "performance management" (that doesn't work)
The best managers I've had shared some things in common:
- Asked lots of questions, even "dumb questions".
- Unbounded willingness to look stupid.
- Understands the importance of data and metrics, but knows when a decision should not be data-driven.
- Focused on the bigger picture, almost always.
- Work backwards from customer
- How does this tie into our strategy
- Found out what intrinsically motivated me and funneled those opportunities towards me
What did you find surprising when you first joined, that wasn't obvious externally? #culture #tech #people
Here, I'm interested in surprising-good and surprising-bad things. These surprising little facts sometimes become the most important thing.
Companies with a good culture will generally have employees that openly mention bad things. The interviewer having nothing negative to say isn't evidence of bad signal, it's just absence of good signal.
What kind of window dressing does your company do? #culture #people
This is an explicit "tell me a bad thing please", but honed in on the concept of window dressing. I want an answer for both "to customers" and "to prospective employees / the industry at large".
What are the best three reasons to join this company?
Pretty self-explanatory. It can be really useful to hear from an engineer what they think are the unique reasons to join their company.
What are the best three reasons to decline to join this company?
This question catches folks off-guard, but it's worded in a clever way to trick an engineer into playing a game that engineers like to play. Assuming that you were to decline to join, which set of reasons make that decision most rational and true?
How many external nouns are directly referenced in your systems? #tech
I have a hypothesis that systems with more edges to external nodes are worse. One way to interpret this question is "are you glue or are you paper?", but there are other interpretations.
Questions for startups
- What's the most distilled, essentialist mission statement that captures the ethos of what you want to build?
- What does your cap table look like?
- What's your capitalization and runway?
- Are you raising? How much? Why?
- Do you have a concrete plan to become cash-flow positive or profitable?
- Do you offer options or RSUs? (Greatly prefer RSUs here.)
- Is there a timeline or expectation I can hold about future liquidity?
- Is your cap table structured such that investors and founders are made whole before employees get liquidity, in the event of a liquidity event?
- (In the event of hypergrowth) Can I ask for a guarantee that I won't get layered for at least $X years? (I like to continue working for someone I've vetted.)
Best reasons to not hire me
I believe in the importance of toolmaking.
Tools have unseen, unknowable payoffs. This is hardest when the deadlines are tight and opportunity costs seem high, but it's not a distraction. It's an investment in future velocity. (Sharpening the Axe: The Primacy of Toolmaking)
I have a low tolerance for bull.
I don't like scurrying about the organization and playing back-room politics, playing visibility games, or being asked or expected to spew bull. I just want to build the right stuff the right way and have fun doing so.
I said "low tolerance" on purpose. My tolerance is not zero; I know the world is not perfect. I've been in those rooms and navigated those sensitive situations where finesse and restraint and choosing one's words were important. I just don't want it to be my every day job. I know how to play that game, but I've chosen to play a different game called "building stuff that matters".
I'm skeptical of many common management practices.
I think a lot of common engineering management practices are bull.
Some things I'm skeptical of:
- Having "alignment meetings" that happen too regularly
- Weekly status reports
- Self-preservation over honesty and communication
- Pay for performance, and more generally performance management
- Daily standups (can be useful, but often cargo culted)
- Making everything transactional and incentive based
Further reading
- I recently read the book Alfie Kohn's Punished by Rewards and found it to be a convincing critique of how society tries to motivate people.
Yojijukugo are four-character compound Japanese words, often expressive idioms. The direct translation is four (四) character (字) idiom (熟語). A beautiful part of the Japanese language (and culture!), they carry much meaning in little space. Brevity is the soul of wit, someone said. According to jisho.org, Japanese has over 3000 yojijukugo.
Another aspect of yojijukugo is that they often evoke powerful metaphors and analogs to convey their meaning. For example, the equivalent of "survival of the fittest" in Japanese is 弱肉強食 (jakunikukyoushoku). The literal translation is weak (弱) meat (肉) strong (強) food (食). Isn't that a fun way of putting it?
Yojijukugo I treasure
質実剛健 (shitsujitsugouken)
Translation: Unaffected and sincere, with fortitude and vigor.
質 (shitsu) on its own can mean quality, value, or nature. 実 (jitsu) on its own can mean truth, sincerity, or honesty. 剛 (gou) on its own means robust or strong. 健 (ken) on its own might mean healthy or fit.
But, if you instead parse it as two words:
質実 (shitsujitsu) is a word meaning simplicity. 剛健 (gouken) is a word meaning vigor.
I admire that regardless of at which level of granularity you resolve the components of this word, they always seem to add meaning to the yojijukugo.
一日一歩 (ichinichiippo)
Direct translation: one day, one step.
I like the imagery: every day, taking a single step. Slow but steady.
一期一会 (ichigoichie)
Direct translation: one time, one meeting.
This one is more difficult to translate naturally. I think of it as a mix of "once-in-a-lifetime" and "nothing should go left uncherished". By "one time, one meeting", it means to say that you'll only get to experience every thing once. By underscoring the finiteness of our experiences, it asks of us to cherish each of them.
臨機応変 (rinkiouhen)
Natural translation: playing it by ear.
臨機 (rinki) means tailor-made or suited to the occasion. 応変 (ouhen) means to respond to change.
“Shame is a soul-eating emotion."
—Carl Jung
Summary
I was born and raised in Seattle, and I've tried to develop a good sense of taste, so I think I'm a good source of recommendations in Seattle.
Because I am a relatively price-inelastic consumer, I will employ the following dollar score:
$ => This is notably inexpensive and good value.
$$ => Not extravagant but not notable good value.
$$$ => Pay-to-win tier. Your utility function discards cost.
Food
Thai Tom, Thai
$
University District
NOTE: Cash-only.
Thai Tom is a mainstay of the University District, with notably good prices, fast service, and typically long lines. Being owned and operated wholly by Thai helps keep the authenticity.
What you get here in food, you pay for in comfort. This place is cramped, decidedly hole-in-the-wall, and uncompromising. Go here if you want the best Thai food for the best price, and care exactly zero about being comfortable while you're scarfing it down.
Luc, French
$$
Madison Park
Excellent value during happy hour, but the nice dinners can cross into $$$
territory. A terrific kitchen with a deep bench of talent, and high-class
service to go with it. Excellent ambience. The whole package.
Try the beef bourguignon.
Altura, American Fine Dining
$$$
North Capitol Hill
You absolutely hate your money and you love high-end, multi-course culinary experiences. That's Altura. Get the wine pairing.
When you get the bill, just pay without looking at the numbers. And don't look at the credit card bill that month, either. And only go here once every year or two. That's my formula.
Probably the best pure restaurant dining experience in Seattle.
Tuta Bella, Pizza
$$
Multiple locations, but I know Wallingford best.
Excellent pizzeria. Italian style. Hard to go wrong here. Good place for families and dates alike.
Mean Sandwich, Sandwiches
$$
Ballard
The best chicken sandwich I've ever had. The potato skin sides are unique, clever, and amazing. I'm sure the rest of the menu is just as amazing, but I still haven't managed to order anything else.
Sweet Alchemy, Ice Creamery
$$
Ballard, University District, Capitol Hill
The best ice creamery I've ever been to. Korean-American owned, with some fun culturally-influenced flavors. Try the makgeolli. It's my favorite. Very simple. Persian Rose is also incredible, more complex. I buy my pints here, but they're perfect for stopping by and grabbing a scoop/waffle cone too.
Gordito's Healthy Mexican Restaurant, Mexican
$
Greenwood
This isn't a fancy hipster culinary place. This is a family owned and operated, Seattle mainstay for simple, good burritos, with fresh ingredients. Open kitchen. Killer value for what you get. Order the Grande Burrito. You'll get three epic meals out of it. The soup is dang good too. Don't forget to hit the salsa bar.
Ramen Danbo, Japanese Ramen
$$
Capitol Hill
Authentic Fukuoka-style ramen. Very flavorful. My personal favoirte ramen in Seattle, although between Betsutenjin and Danbo, it comes down to preference of style.
Betsutenjin, Japanese Ramen
$$
Capitol Hill
A pure ramen shop. Wonderfully pure broth and noodles. Nothing too highly opinionated; a pursuit of simple perfection. Fairly authentic atmosphere. My second choice in seattle, mostly due to a matter of preference.
Yoroshiku, Japanese Izakaya
$$
Wallingford
Excellent izakaya with rare offerings, opinionated takes, and a wonderfully diverse selection. The unique karaage is worth a try.
Redmill, Burgers
$$
Phinney Ridge, Ballard, Interbay locations.
A local favorite. Massive juicy burgers, killer shakes, and good sides. Pretty darn good value. If you try only one burger place in Seattle, I think Redmill gets the nod.
Parks
Sunset Hill Park
Free
Ballard / Sunset Hill
Locals call this "the Bluff". Come here at sunset on a clear day. Witness the splendor of the Olympic Mountains. Don't leave at sunset. Stay for the next hour. That's when it gets the prettiest. Bring some wine or tea or cheese and a friend.
Washington Park Arboretum
Free
North Capitol Hill
Great walking and biking trails here. See all of the pretty trees and gardens.
Kerry Park
Free
Queen Anne
The canonical view of the Seattle skyline. Usually busy, but worth a stop during off-hours for the view on a clear sunny day. Gorgeous on the right day.
~Daytrips
// TODO: Add Port Townsend
// TODO: Add Leavenworth
// TODO: Add Mount Rainier
// TODO: Add Lake Quinault
Niche
// TODO: Add favorite pool hall
// TODO: Add favorite theaters
// TODO: Add favorite farmer's markets
This is a pattern I call validate or calibrate. This pattern comes into play you're not sure about a feeling you have, especially about something sensitive, for example toxic workplace politics, a particularly damaging manager, or anything generally outside of your control that negatively impacts you. This is when you talk to a colleague whom you trust, to either validate your perspective, or to have it callibrated by a different perspective you trust.
The goal of the conversation is simple. One of two outcomes:
- your uncertain feeling is validated by the shared experience of someone else (helps you cope, can sometimes be the catalyst to finding a solution)
- someone you trust helps you calibrate your understanding of the situation and your feelings (re-calibrating is important, and helps you stay within likely-correct bounds).
I've been on both ends of these conversations. I recommend only validating when you truly agree and have a shared take on the situation. It is okay to console if you are not able to validate. Consolation is different from validation. Validation produces new knowledge. It is another data point that indicates "maybe it isn't just me". Consolation is merely a treatment of the symptoms.
The dual to this is: If someone comes to you with an issue and your viewpoint is different, it's probably worth having that callibrating discussion. They should listen to you. After all, they came to you.
Philosophy on tools
Given eight hours to chop down a tree, spend six sharpening the axe.
A poor craftsman blames his tools. But not because good tools are unneeded. It is precisely because a good craftsman knows the value of a good tool that it is poor form to blame them. A good craftsman knows how to select good tools. And if those tools don't exist, a good craftsman creates their own good tools.
Every good tool is born from a craftsman having had only the wrong tools.
If something is worth doing even once, it's worth building a tool to do it.
I have built an entire world around my tools.
Software tools
Operating system: Gentoo Linux
Why do I use Gentoo Linux?
Most operating systems and distributions use pre-compiled
binaries as the primitive unit of
installable package. For example, if you want to install firefox
, your
operating system would have you download a bunch of pre-compiled object code
,
wrapped in a script that would move the executables and files to the right
locations. This is the typical flow to "install software" on your computer.
Gentoo, unlike most operating systems, downloads the source code rather than its compilation artifacts. Then, the installation process actually compiles that source code into the build artifacts which are placed in the right locations.
This is the most important reason as for why I've ultimately landed on Gentoo as my operating system of choice: Because it is a source-based distribution. Gentoo is often referred to as a metadistribution. It is a distribution that enables you to build the exact distribution you want.
Why do I care if it is source-based?
It makes it extremely easy for me to inspect what my software is doing, and to
make patch it and recompile, on the fly. In short, it is the ideal setup
for a hacker and open-source enthusiast, because the ability to modify software
that is running on my computer is inherently baked into the very DNA of the
operating system itself! It is not a workaround or weird hack for me to make a
change to Firefox and re-compile it. It is the standard installation flow.
This makes the edit-compile-test loop for open-source software highly optimized
for me, which makes it more likely that I will fix and upstream changes.
NixOS
I've been considering experimenting with NixOS as a new daily driver. It has a
lot of overlap with what I like from Gentoo, but has a better package manager
and robust pattern for managing the state and configuration of the system.
Windowing system: i3
I use i3 as my windowing system. The short answer as to why is: I am a proponent of a mouse-less existence insofar as most applications are concerned.
A mouse is an interface best suited for continuous-shaped requirements, e.g. aiming a pointer in games, drawing, and other inherently 2-dimensional arbitrary precision input modalities.
Textual interfaces (e.g. code), reading, navigating filesystems, etc. is inherently discrete (folders have files, files have words, words have letters). For discrete applications such as these, I (and many others) prefer mouse-less modes of interaction. A mouse can point at many different precise points for a given character in a word, but it's still the same logical place.
Almost all of my modalities of interacting with my computer are inherently discrete, therefore, I optimize all of my tools to be a keyboard-only interaction modality.
i3wm (and any tiling window manager in general) is the natural conclusion to the question, "How do we build a window manager optimized for the keyboard?"
- Window navigation happens at the speed of thought.
- I have a consistent layout of windows that I build muscle memory with.
- i3wm action latency is staggeringly lower than other window managers.
- No superfluous animations.
- Re-arranging windows side-by-side is one keystroke.
Editor: Emacs
I use emacs, but with vi keybindings. Don't ask, it's a long story. I'm at home with either, but I prefer emacs for its extensibility and vi for its modal UX. I solved for this by extending emacs to have the same modal editing UX as vi. I don't like elisp and emacs has had some performance problems in the past. That being said, I don't know of anything better.
- org-mode for notes/docs
- magit for git
- emacs macros
Language: Rust
These days, I try to write most of my personal things in rust, because it aligns with my values, and I believe that it will grow to be an important language. Its performance, reliability, safety, and language design landed it a special place in my heart and mind.
Everything else (software)
I do everything else in my web browser or in my terminal of choice. (These days, it's alacritty, but my terminal choices seem to be much less sticky than OS/editor, and often times I'm using one of the shells within emacs directly.)
I typically only have three applications open on my computer, each in their own i3 workspace:
- emacs
- terminal emulator
- browser
Hardware tools
Keyboard: Happy Hacking Keyboard
There is no better keyboard for a bible-thumping UNIX software engineer that I am aware of.
Control
key whereCaps Lock
normally isEsc
on the same row as digits, closer to home row- Tilde and grave to right of
|\
Super
key where it belongs, on the left thumb to left of space- Hackable keymapping
- Compact, ideal for staying on homerow
- The hardware and feel of the topre switches are top-notch
Headphones: Sennheiser HD800s
DAC/Amp: Audeze Deckard
Webcam: Fujifilm XT-2
Microphone: RE20
Desktop: Dual AMD Milan 128-core + 1024GB ECC RAM Supermicro
Kathy McQueen was my mom. She was everything a son could ever ask for: nurturing, strong, smart, didn't put up with any crap, gave everything for her two boys. She was a business owner, a body builder, a climber, a mountaineer, a world traveler, a sun-loving, book-reading go-getter.
She was diagnosed with early-onset Alzheimer's when I was around 12 years old. We (my dad, brother, and I) cared for her at home for about the next decade.
Somewhere along the way, she forgot how to speak, how to eat, and who we were.
This story might as well be my own.
These days, other than close friends or family, when someone asks me how my mom is, I say, "she passed away when I was a teenager." They usually say, "Oh, I'm so sorry to hear that." Then we move on.
The real story doesn't fit into a small-talk conversation. It's too long and nuanced. Too brutal. Too unresolved.
There was a phase where she would slip out of the house and go for walks on her own. I remember following her at a distance for a bit, letting her enjoy the agency and solitude of a walk around the neighborhood, before catching up to her and holding her hand so she didn't feel lost and alone.
One time, she gave us the slip real good. This was when I was maybe 16 years old. I searched the whole neighborhood. I eventually found her about a mile away, talking to strangers and laughing at the nearest grocery store. It was a strange phase of the disease — she could still walk and talk mostly, but would often forget major things.
Unfortunately on this day, she didn't know who I was or what "home" was. When I said "let's go home, mom," she started yelling at me in the grocery store, trying to fight me off and run away.
"Get away from me," "I don't know who you are." "Stop!" Everyone watched. I don't think the watchers understood what was happening. I couldn't help but think I was somehow perversely benefiting from being on the wrong side of the bystander effect.
She fought me the whole way home. The next day, it was like nothing happened.
Every son or daughter that has cared for a parent with dementia has hundreds of such stories. I know I do. It is the ultimate form of hothousing for suffering and loss.
And yet, despite these stories, this couldn't be more true:
So far, with the help of her family, the day program and the sheer force of her own will and devotion to Jo, Robin has managed to provide all his care-giving at home. Families of people with dementia land in different places on this issue, but to her, it's beyond question that Jo is not here anymore; most everything that made him Jo has been stolen. "But it’s almost like an echo that’s left," she says. "It’s like I want to take care of this Jo because it’s honouring the real Jo."
When she's forced to move Jo into a nursing home and Robin loses her ability to care every day for the echo of the man she married, that will mean he's really gone. "Sometimes people very kindly say, 'Oh, it will be better when Jo is living somewhere else.' No, it won’t. That’s the part that people don’t understand. It won’t be better. It will be the worst part," she says through sobs. "Because I don’t consider what I'm doing a negative thing."
As sad as these stories are, eventually you end up longing for those brutal days. The days mom would sometimes remember you. The days mom would sometimes say something or laugh.
In this world, much unintended harm is inflicted.
We create a false dichotomy. You are asked to choose a tribe: the insensitive, always-offenders; or the over-sensitive, never-offenders. But there is a middle way.
What does Jon Postel's law tell us? Be conservative in what you do, be liberal in what you accept from others. This principle about API and protocol design can help inform us on how to interact with each other. Applied to the topic of giving and taking offense, the principle tells us that our aim ought to be to not offend and to not take offense.
Though it seems oceans away, interlinked with this principle is another, the principle of charity.
The principle of charity or charitable interpretation requires interpreting a speaker's statements in the most rational way possible and, in the case of any argument, considering its best, strongest possible interpretation.
Put differently:
We make maximum sense of the words and thoughts of others when we interpret in a way that optimizes agreement.
When Jon Postel was designing TCP, why did he recommend "to be liberal in what you accept from others"?
Because making the maximum sense of the system is furthered by interpreting what you receive from others in a way that optimizes agreement. Although no amount of disagreement will hurt the feelings of a congestion control algorithm, it can certainly increase packet retransmissions, packet loss, and other Bad Things. This conclusion equally applies to human communication.
Principally, this is most interesting as applied to resolving apparent deltas of thought. This wording, apparent deltas of thought, seems a bit clumsy — and perhaps it is — but I use it to avoid words with subtly different meanings, such as disagreement or misunderstanding. I avoid these words because they have a connotation of finality. That we aren't actively working towards convergence. (Or at the least, working towards convergence in the sense that we understand each other's apparent delta in thought.)
The principle of charity has a beautiful symmetry. It states how to receive communication, but its implicit normative dual is about transmitting. Not only should you receive communication in the most charitable way, but you should also communicate in a way that optimizes the likelihood of a charitable interpretation.
One of the most important tricks I've ever learned is how to convert apparent doubt, disagreement, and misunderstandings into a question about the crux of why I think I might disagree. Usually there is an assumption, a fact, a goal, or something else upon which the decision in question is wholly determined. If I can identify that particular thing, I can assume my understanding of it is flawed, then ask the question. By asking a question, I maximize a charitable outcome: Maybe I'm missing context. Maybe they're missing context, and don't know it yet. Either way, we'll find out.
There is never a reason to not apply the principle of charity. It always optimizes better outcomes.
These words ring so true. See here for a good performance.
there's a bluebird in my heart that
wants to get out
but I'm too tough for him,
I say, stay in there, I'm not going
to let anybody see
you.
there's a bluebird in my heart that
wants to get out
but I pour whiskey on him and inhale
cigarette smoke
and the whores and the bartenders
and the grocery clerks
never know that
he's
in there.
there's a bluebird in my heart that
wants to get out
but I'm too tough for him,
I say,
stay down, do you want to mess
me up?
you want to screw up the
works?
you want to blow my book sales in
Europe?
there's a bluebird in my heart that
wants to get out
but I'm too clever, I only let him out
at night sometimes
when everybody's asleep.
I say, I know that you're there,
so don't be
sad.
then I put him back,
but he's singing a little
in there, I haven't quite let him
die
and we sleep together like
that
with our
secret pact
and it's nice enough to
make a man
weep, but I don't
weep, do
you?
To not bury the lede: Everything is a trade-off, so making a decision is about understanding which parameters you want to optimize, and their relative weights. This decision process can be easy, but sometimes it is exceptionally hard. Sometimes, a disagreement on the correct solution is actually a subtle disagreement on the relative weights allocated to each parameter. Framing these decisions as optimization problems can be a useful method for driving alignment or achieving consensus. At the very least, it can drive a mutual understanding of why the parties disagree.
For fun, let's model making a decision as some psuedo-code:
/// An outcome is something you decide to do, and it has Consequences.
struct Outcome {
name: String,
consequences: Vec<Consequence>
}
/// For our simple model, a Consequence can only be one of three things.
enum Consequence {
/// How long until we get the outcome -- i.e. Wall clock time.
TimeToComplete(magnitude: int64),
/// How much "effort" will it be, in engineer-hours (or person-hours)?
Effort(magnitude: int64),
/// An Outcome will have certain Effects, for example if you decide to mow
/// your lawn, an Effect might be Effect("House looks better", 30). For
/// simplicity, we're doing this in a loosey-goosey Stringly-typed way, but
/// it gets the idea across. The int64 is the magnitude of the Effect. Each
/// Effect should abide by a contract about how to interpret its magnitude.
Effect(name: String, magnitude: int64)
}
/// A function that decides between Outcomes, given some weights
fn decide(Vec<Outcome> outcomes, HashMap<Consequence, f64> weights) {
// The decision's Outcome, and how well it scored. Higher is better.
let mut decision: Option<(Outcome, f64)> = None;
// Find the best outcome, by summing the magnitude of each outcome's
// consequence's magnitude, weighted by the weights parameter.
for outcome in outcomes {
let score = outcome
.consequences
.iter()
.map(|c| c.magnitude * weights[c])
.sum();
// If the outcome we're currently evaluating looks better than any of
// the previous ones, then it becomes our defacto candidate decision.
if decision.is_none() || decision.1 < score {
decision = Some(outcome, score);
}
}
return decision
}
Let me tell a story of an engineer's journey, with three chapters: Junior, Mid-level, and Senior.
Junior
A junior may have some preferences or ideas, but until you've seen how something fails, you don't have well-founded justified beliefs on how to best do things. Even more problematically, you might not be good at enumerating the possible outcomes.
There is a difference between keeping a beginner's mind and being inexperienced. A beginner's mind can bring salient fresh perspectives on a problem. Inexperienced juniors may have a beginner's mind, but not expertise. If you couple expertise with a beginner's mind, that's where the real gains come into play.
Mid-level
A mid-level engineer has started to see how things can fail, and begins to develop strong opinions on how things ought to be. These engineers should be reasonably decent at enumerating the possible outcomes, but they may inevitably miss some important possibilities. A mid-level engineer develops heuristics like Gang of Four patterns are best practices, or everything should be immutable, or good code means no duplicated logic.
The thinking patterns often held by mid-level engineers work in 90+% of cases. They're good enough, for most things. But they break down. Best practice is an oxymoron. A practice is only the best if it's always the best, and nothing is best for every case. Absolutes like "everything should be X" are useful as an approximate default case, but they don't hold true in all cases. Nuance is everywhere, and second- or third-order consequences matter.
A good example of second- and third-order consequential thinking presents itself
in the question of code deduplication. Many of us grew up learning how terrible
duplicated code is. But did anyone ever teach us how bad deduplication can be?
Deduplication is normally achieved by introducing abstraction. This is where
the question of Consequence.magnitude
comes into play. Ceteris
paribus, that code duplication is to be avoided. But all things are
not equal. The cost of a wrong abstraction is in fact orders of magnitude worse
than duplicated code. Then, we can model this as follows: do we allow some
duplication, or do we risk the wrong abstraction? Generally, you should allow
duplication until that duplication becomes an actual problem. By that time, you
should have sufficient experience in the problem space that the likelihood of
choosing a bad abstraction is sufficiently low. Some present this idea as the
rule of three. I propose to go even further, the rule of pain:
duplicate until it hurts, then refactor. The reason is simple: duplication
pain grows approximately linearly, but pain from the wrong abstraction is
unbounded and grows in super-linear ways.
Senior
A senior understands that everything is a trade-off. Even things that seem obviously correct are in fact a trade-off — it's just that your weights are so skewed that it doesn't seem like a trade-off to you.
Things a senior might think:
- "Code duplication isn't ideal, but avoiding it could be even worse."
- "I may know a better solution, but the juice wouldn't be worth the squeeze."
- "Many things can be modeled in terms of risk, and that viewpoint can be useful."
- "Immutability is a good default, but mutability has a place when performance is important."
- "Design patterns are often cargo-culting, and should be critically evaluated."
- "'Premature optimization is the root of all evil' is the root of all evil."
- "Performance always matters, the only question is: To what magnitude?" (c.f. Would you browse Facebook if it took a year to load the homepage?)
Things a senior might think about:
- Tying engineering back to the business need. e.g., "Our quality of engineering is of the utmost importance, our $50B business depends on our infrastructure platform running smoothly", vs. "We're a startup, we simply need to deliver results quickly, before running out of runway."
- There are better options we could pursue, but the differences are too minute to be worth arguing.
- How to communicate and drive a discussion from a trade-off perspective, and specify the upsides and downsides of each outcome and the return on investment.
Imagine you've written the following hack to see which OS you're running.
OperatingSystem determine_os(uname: &str) {
if (uname.contains("Darwin")) {
return OperatingSystem.MacOS;
} else if (uname.contains("Linux")) {
return OperatingSystem.Linux;
} else {
return OperatingSystem.Unknown;
}
}
You send it for code review, and get the following comment.
Nit: you can remove the final
else
.
Unfortunately, a common code review comment. But does it hold up to scrutiny?
What does it look like after applying the suggestion?
OperatingSystem determine_os(uname: &str) {
if (uname.contains("Darwin")) {
return OperatingSystem.MacOS;
} else if (uname.contains("Linux")) {
return OperatingSystem.Linux;
}
return OperatingSystem.Unknown;
}
By omitting the final else
, we've broken the structural and visual alignment
of logic that belongs to the same category.
Indentation is a visual manifestation of logical structure. Things that are alike logically and structurally should reflect that visually.
This function is effectively a tri-branch conditional return. And the code
should show that as explicitly as possible. The else
logically groups the code
and encodes that meaning into it via structure. There's a cargo
cult against using else
anywhere. Don't fall
for that trap.
If you're still not convinced, consider this. Would you write a switch statement like this?
switch(color) {
case "red":
return Color.Red;
case "blue":
return Color.Blue;
}
return Color.Unknown;
Or would you write it like this?
switch(color) {
case "red":
return Color.Red;
case "blue":
return Color.Blue;
default:
return Color.Unknown;
}
It's more than dozens of times I have heard or participated in a conversation where people would ask each other, "How many countries have you been to?" At some point it would sound like a competition. 20, 30, 40, “and this year I plan to do A and B and C”.
Yes, you have been to 30 countries in 5 years, but have you really “been” there? How many places do you really know and understand? How many of those amazing, life-changing moments do you actually retain, internalize and use to help yourself become a better human, a better citizen of this world?
Travel should be challenging. It should change the way you think and feel. It isn't about checking the boxes or seeing the sights. It disappoints me to have to say: It's also not about being seen at those sights. The obtuse misappropriation of the art of photography, to travel somewhere — with its own culture, nature, and context — and make yourself the primary object of your photos: Fremdscham is the only word I can muster.
Travel slowly, explore deeply
Go somewhere you don't understand. Take the time, absorb what it offers. Learn something about the place. Learn something about yourself.
Perfect is the enemy of good. Shipping something imperfect is usually better than shipping nothing at all. (There are exceptions, but they aren't common.)
Most sufficiently advanced systems have to evolve to become good. Evolution requires exposure and experience in the real world, battle-tested behavior, and importantly, failures. The only path to this kind of evolution is to actually put something out there, even if it isn't perfect. Especially if it isn't perfect.
The elbow in the curve of time spent improving design and analysis comes earlier than you think. You should be spending less time designing, and more time learning from your failures, missteps, and the complex realities of your system in the wild. Your designs are wrong anyway. Natural selection is going to be a far more efficient process for showing you why and how your design is wrong than anything else.
An old essay by Richard P. Gabriel.
Note that this doesn't mean to ship unrobust systems. This means that real-world feedback is valuable and systems should be built with evolution in mind.
When you become stuck on a problem, try for another 15 minutes, and then you must ask.
Document what you try. It becomes a guide that may lead you to the answer. If not, it certainly leads to a better question.
The temptation to immediately ask an expert is tantalizing — but resist! The documentation of your effort serves as a guide for mentoring and teaching.
More importantly, you earn the trust of those experts that you go to them on matters which you've already put effort into solving yourself.
Some of the time in that ~15 minute buffer, you will find the answer yourself. Knowledge gained by trial can often be stronger than knowledge from tutelage. Sometimes, you will even learn something that no one else knows. This is one of the greatest feelings, and often bears incredible fruit.
After all, experts are just people that have already tried and failed a lot.
Any sufficiently advanced form of expertise should look similar to trying a lot — and failing a lot. The best experts become intimate with this feeling. They may even revel in it.
(This writing was inspired by a director of software engineering at Google.)
Reading the code shouldn't be your last resort.
Reading the code isn't always comfortable. It can feel like an arduous chore, especially if it isn't a core habit of yours (yet). This feeling is self-inhibiting. Recognize that it is born from discomfort, not logic.
The code is truth. The only complete truth.
If physicists had all the source code for the universe, how many experiments do you think they would run? My guess is zero.
Read the code and all else will follow.
Always read the code. Even when they tell you it doesn't or shouldn't work that way. Especially when they tell you that.
I long searched for the perfect playlist to listen to as I worked. I never found it. So I tried to create it.
Flow is my playlist for concentration, coding, problem solving, etc. As of Dec 2019, it has over 77 hours of music, 771 tracks, and a healthy few hundred software engineer followers.
Its aim is to optimize for the mental state of Flow. Using nearly exclusively non-vocal tracks, the genres are primarily spacemusic and ambient.
Thought is useful when it motivates for action and a hindrance when it substitutes for action.
—Bill Raeder
The great benefit of computer sequencers is that they remove the issue of skill, and replace it with the issue of judgement.
— Brian Eno
Collapse the distance between imagination and implementation by building tools.
— Gordon Brander
Less rote practice; more imaginative delivery.
Writing a runbook is a declaration of temporary failure.
The act of writing a runbook is the incantation to a dark, forbidden contract with the better angels of our nature. It is a promise to do the right thing in the future, but not now. Right now, we're going to do the evil thing: write a runbook.
How often do we go back and do the right thing?
Rarely.
How many of these promises do we break?
A lot of them.
An error doesn't become a mistake until you refuse to correct it.
If you have runbooks, consider them as backlog tasks. Try not to add to their numbers. Failure begets runbooks; runbooks beget runbooks; and runbooks beget failure.
A good, simple writeup on the fallacies of distributed computing.
A good index resource on consistency models, their guarentees, proofs, origins, etc.
Everything has been said before, but since nobody listens we have to keep going back and beginning all over again.
André Gide
"It is an attitide that can be encapsulated in a simple but demanding rule: always think big picture and fine detail."
Will Gompertz, Think Like an Artist
Everyone belongs to a tribe. Everyone underestimates the influence those tribes have on their thinking.
Acknowledge your tribes. Try to separate your thinking from your identity.
(Inspiration from pg, if I recall correctly.)
I'm convinced that the best kind of learning is interdisciplinary. Fields of study have much to learn from each other. The most interesting learnings are those that transcend fields of study. The rest is just details.
Keep Your Identity Small (2009) is an essay by Paul Graham which describes something I've intuitively practiced since my early twenties. But I'd never formalized the thought.
My set of justified true beliefs come not as an expression or extension of my identity, but from reasoned logic combined with a set of morals, ideal outcomes, and other justified true beliefs. I've internalized the idea that I shouldn't let these beliefs constitute my identity.
A related idea is that of “Strong Opinons, Weakly Held", a framework for thinking espoused by Palo Alto's Institute for the Future. The idea is that opinions should be "weakly held" — be unattached to the idea, so that you do not irrationaly reject its refutation. The other side of this coin is that you must not let this lack of attachment dampen your rigor, passion, or strength in forming, understanding, and justifying your beliefs.
This talk by Brendan Gregg is a fantastic introduction to the tools available in linux for performance instrumentation, monitoring, and debugging.
perf-tools
: https://github.com/brendangregg/perf-toolsebpf
: https://ebpf.io/strace
: https://man7.org/linux/man-pages/man1/strace.1.htmlvalgrind
: https://valgrind.org/ftrace
: https://lwn.net/Articles/608497/uprobes
: https://www.brendangregg.com/blog/2015-06-28/linux-ftrace-uprobe.htmluptime
dmesg | tail
iostat xz 1
sar -n DEV 1
sar -n TCP,ETCP 1
We are what they grow beyond. That is the true burden of all masters.
—Master Yoda
This essay corroborates a lot of my own views about how I would choose to raise my children. Whenever interacting with children, I prefer to treat them exactly as I do adults, respecting their ability to reason, learn, and comprehend.
As a kid, I resented being treated like a child. I didn't like it when adults would dumb down a concept or hand-wave things because it was assumed I wouldn't be able to understand.
I think if we treated kids more like adults, the adults of tomorrow would be better.
What is a deadlock?
A deadlock is when you're at a restaurant waiting to get the check, and your waiter is wating for you to ask for it.
And you're both wondering why it is taking so long.
You're both waiters now.
First you learn the value of abstraction, then you learn the cost of abstraction, then you're ready to engineer.
Always design your programs as a member of a whole family of programs, including those that are likely to succeed it.
—Edsger W. Dijkstra
Computer science is no more about computers than astronomy is about telescopes.
—Edsger Dijkstra
Functionality is an asset; code is a liability.
—Kirk Pepperdine, @kcpeppe
The lurking suspicion that something could be simplified is the world's richest source of rewarding challenges.
— Edsger W. Dijkstra
Perfecting oneself is as much unlearning as it is learning.
— Edsger W. Dijkstra
A delicate matter, requiring taste and judgement. I tend to err on the side of eliminating comments, for several reasons. First, if the code is clear, and uses good type names and variable names, it should explain itself. Second, comments aren't checked by the compiler, so there is no guarantee they're right, especially after the code is modified. A misleading comment can be very confusing. Third, the issue of typography: comments clutter code.
— Rob Pike, Notes on Programming in C
I used to be staunchly against comments, but I came to appreciate them. Like many things, it isn't black and white.
It is true in most cases, code with good names, types, and abstractions should explain itself. However, explaining the what of the code is only one of the useful modes a comment has. And it's easily the least common/useful mode.
Let's explore the questions that comments can help us answer.
Why does this code exist?
It is not self-evident why all code exists. One of the best use cases for a comment is to answer "why?", not "what?".
/*
* Distribute memory according to CPU & memory use on each node,
* with 3/4 hysteresis to avoid unnecessary memory migrations:
*
* faults_cpu(dst) 3 faults_cpu(src)
* --------------- * - > ---------------
* faults_mem(dst) 4 faults_mem(src)
*/
return group_faults_cpu(ng, dst_nid) * group_faults(p, src_nid) * 3 >
group_faults_cpu(ng, src_nid) * group_faults(p, dst_nid) * 4;
This is an example of a good comment from the Linux kernel, which concisely captures both the "what?" and the "why?" of a non-trivial expression.
What are we doing?
Distributing memory, according to the formula shown above.
Why do it like this?
We're adding a 3/4 hysteresis to avoid the churn and burn of twitchy, sensitive-to-change migration formulas.
Cargo cult programming is a style of computer programming characterized by the ritual inclusion of code or program structures that serve no real purpose.
— Wikipedia
Most engineers go through a progression of understanding the balance of simplicity versus other values. Oftentimes, simplicity should win.
Examples
Why write a switch statement here instead of employing the strategy pattern?
More generally:
How about using XYZ Gang of Four pattern here instead?
I'm not against these patterns because I'm unfamiliar with them. Quite the opposite. Healthy experience with these patterns should tend to reduce the urge to reach for them.
Refactoring (noun): a change made to the internal structure of software to make it easier to understand and cheaper to modify without changing its observable behavior.
Refactoring is often misunderstood. It shouldn't change the behavior. That's the point.
Gods and Dogs, Jiří Kylián
The i3 window manager is the best thing to happen to my desktop in a long time.
Much like modal editing, it is a paradigm shift in how to interface with your computer. No floating windows. Everything is snapped. If you're a vim/emacs user, you'll like tiling window managers. Try i3.
A few days ago, our team launched a new component. A week after launch, while dealing with some ops tickets, we discovered that the detailed, extensive wiki document for that component was already out-of-date.
This is a familiar story to any dev. Documentation is hard to keep accurate. The knee-jerk reaction is to assess that the miss here was that we allowed the documentation to become inaccurate. The knee-jerk reaction is wrong. It is not feasible to maintain documentation if that documentation is too specific.
The solution is to change the philosophical approach to documentation. For software systems of any reasonable complexity, the code is the only source of truth for its behavior and its descriptive reality. Documentation ought to answer the following questions, in basic non-technical english prose:
- Why did we create this component? (prescriptive statement)
- At a high-level, how does it solve its problem? (This answer should absolutely not map anywhere near 1:1 to its implementation. It should be at an extremely high-level.)
- Going no deeper than a file/class level view, describe in a paragraph the function of the most critical classes/files/modules in the code. This is as specific as it should get, and this serves as a guide for an engineer to dive into the code.
Again, documentation should never map 1:1 to anything more specific than a class. Anything more specific than that, and it effectively has to change whenever the code changes. If you're changing the function of tons of classes and files, it is reasonable to expect that the documentation would necessarily have to change in kind.
Writing documentation at a high-level helps to minimize the potential that your documentation will become out-of-date and inaccurate. It is easier to understand, too. Which is kind of the point. If someone wants to know the objective truth exactly as it exists, the documentation is absolutely the wrong place for that. That's what the code is for.
A home is something personal, a space that can shape your habits, emotions, and thoughts. I do not see it as overly materialistic to dream of an ideal place to call home.
My wishlist
- Eastern light (I wake best by natural light)
- Western light (sunsets are romantic)
- In Seattle, preferably Capitol Hill, Queen Anne, Montlake
- Ceiling-to-floor windows, and plenty of them
- Grand views of mountains and water
- High ceilings
- Open floor plan, without needless walls or partitioning elements
- A modest and easily maintainable yard
- A spacious, open kitchen that allows for entertaining while cooking
- Relatively medium-to-small house — 3br, less than 2500ft²
- Stone exterior
- Sitting high above the road
- Nearby or adjacent to green space
- Minimalism over complicated or noisy designs
- Some styles I like: Tudor, Mid-Century Modern, Gothic, Craftsman, Pacfic Lodge
- Understated or modest from the exterior — I don't want it to be showy
Examples of homes I like
Starting a business is antithetical to scaling efforts.
Early in my career, I worked at an early stage startup. I fought tooth and nail in design meetings to apply solid engineering principles to everything we did: scalability, extensibility, modularity, generalization, etc.
I was wrong to do so.
I was attacking the problem from a different perspective, a perspective that would fare well in a large company with massive distributed systems — a company inherently optimizing for the long run. Startups, however, are not optimizing for the long run. They can afford to worry about that later. They need to solve for tomorrow.
This doesn't mean you should throw your engineering principles away. They're not inherently wrong; they are simply the wrong tool for the job, if you're at a startup. Everything is a matter of compromise, and at a startup, you will be compromising more often than not.
Murali Krishnan, my manager at the time, got this right. I was often on the other side of the table, arguing for engineering excellence. I wasn't yet wise enough to understand. It was only later I understood: startups are optimizing a much different set of parameters.
As a young engineer that wants to hone these areas of excellence, it can be a very hard pill to swallow, intentionally setting aside engineering best practices. It's okay. You're not optimizing for the long run. Not yet.
Every time you ask an engineer to postpone what he/she was working on and immediately work on some new task, you've done the equivalent of grabbing at least $500 from your company's piggy bank and flushing it down the toilet. You'd better make sure that new task is worth it.
Any progress the engineer has made on the previous task, you can now kiss goodbye. You should consider that previous task as completely unbegun.
Context switching is incredibly expensive for engineers. It can take upwards of an hour for an engineer to build state on a problem. Building this state is a fixed cost of entry for the engineer to enter a state of flow that enables productivity. Breaking that flow and wasting all that state is expensive.
If you like imaginative/challenging math/logic/CS problems, take a gander at FiveThirtyEight's The Riddler.
FiveThirtyEight is Nate Silver's
website that does precise, academic, and statistical commentary on a variety
of issues. Their The Riddler
column features hard and interesting problems
on a weekly cadence.
In the past, I have solved several, with at least one of my solutions being featured with a shout-out. My solution with code is on GitHub. This particular problem was a fun little geometry problem which I opted to solve via simulation and numerical geometry, though others found an elegant analytical solution.
If you think good architecture is expensive, try bad architecture.
A clever person solves a problem.
A wise person avoids it.—Albert Einstein
omne trium perfectum
— Latin for "everything that comes in threes is perfect."
Kudos to a previous manager of mine, Murali Krishnan, whom I met while working at Transform for demonstrating true mastery over the rule of three and opening my eyes to its power.
The rule of three
(of which there is so much written that it
needs a disambiguation page) is an incredibly potent
tool to communicate ideas, persuade others, and to tell stories.
- A threefold model lends explanatory power to difficult-to-explain topics, simplifying the conceptual model and providing and elegant framework to teach.
- Problems often cleanly subdivide into three sub-problems.
- The cadence and presentation of threes tend to be inherently agreeable to humans, augmenting credibility and persuasiveness.
“An error doesn't become a mistake until you refuse to correct it.”
—Orlando Aloysius Battista
Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.
—Antoine de Saint-Exupery
You will often hear X is almost Y. Almost in this context should
be read as not. Almost
is a face-saving mechanism when you don't want to
say no.