Debates and Arguments
A colleague of mine recently suggested building a new service as a Lambda function rather than the traditional stateful service the company would normally build. As is usually the case when a new tool is put forward to the group, there’ll be a few strong opinions to the pro, a few strong opinions to the con, with the rest of the team with no strong opinion either way waiting to swayed to one side or the other.
This is not uncommon for most of the teams I’ve worked with, its rare for an engineering team to be always totally of the same mind about new tools and a bit of debate is healthy for a project. Whats also unfortunately common though is when two strong opinions lock horns in a seemingly intractable way - unstoppable force meets immovable object, with a mix of bored and slightly uncomfortable developers waiting for the dust to settle. There’s a subtle art to nudging calls like this back to being useful discussion and away from battles of tech ideology.
Find the problem
The first step is to break down the argument into problems and solutions. Ask “What problems does this tool solve for us?”. “Why should we use a Lambda function when we usually spin up an EC2 instance to run a java service?”.
This feels obvious of course, but the act of listing the problems a tool purports to solve and the problems a team believes it has highlights disagreement in a team about what the problems actually are. Sometimes I’ll try to introduce a tool to solve a theoretical problem that doesn’t exist yet, other times I’ll be ignorant of a major problem the team has and dismiss a tool as over-engineering. Am I introducing a Lambda function because I’m optimizing for bursts of simple requests at peak times with no load the rest of the day? Does my colleague believe I’m trying to build a service to handle a constant load with stateful data?
In these disagreements is often hidden the reason behind strong like or dislike of a tool by developers who are otherwise generally in agreement. Each are trying to choose a tool to solve a problem as best they can, but they’re solving fundamentally different problems without realizing it - the tool solves one of these problems well and another poorly. Once everyone’s on the same page about the problem you’re trying to solve, you can have a productive discussion about tools to solve that problem.
“I just need something quick to grab data from Dynamo and serve back to the dashboard a couple of times a day, It feels wasteful to leave a service idling 90% of the time”.
Problems you can’t measure
Developers are almost always predisposed to think in terms of technical first-order effects of tools - size, speed, amount of boilerplate code. This is natural, they’re easy to measure and progress feels more tangible when it can be quantified, even if more meaningful changes aren’t as easily charted. Its common for example to track a metric like “unit test line coverage” to gauge how good your automated testing is rather than asking “do these tests make you confident - would you be stressed if we released this with no manual testing?” - the latter is usually a much better indicator.
The benefits of many tools are in the less-technical or even the non-technical second-order effects. Does choosing this tool make hiring easier? Do other engineers in the org already use this tool? Is it better to build something perfect in-house and risk it being orphaned when engineers leave the organization, or go off-the-shelf and risk the tool a product is built around becoming abandonware?
“A Java service does make sense, but we lost most of our Java devs in the layoff, if we do small Lambda functions we could have the Javascript and Ruby devs help with backend services”
Not all experience is equal
Once you’ve gathered your list of the problems the team face and the problems the tool might solve, move on to asking “How well does the tool solve our problems?”. Tools often shine on a handful of these problems and leave much to be desired on others. This isn’t necessarily a reason to dismiss the tool out of hand, but neither is it a reason to adopt it.
This is the second area where objections tend to get shaken out. A developer may have had a really poor experience with a tool and believe the drawbacks they faced weren’t worth the benefits on another project. How a tool is used is just as important as the choice of tool itself - seeing a tool used badly on a project makes it all to easy to discount using it again on another totally different project. Most people loathe Jira, but typically its because its being used badly, not because Jira itself is objectively bad.
“I had to do Lambda at my last company, yeah its cheap and it scales well but it was so slow, we were getting 10 second response times to requests!”
Tabs or spaces
If you’re paying close attention, there’s also chance to discover differences of opinion on what developers find painful here - one developer may prize terseness of code, where another cares more about readability, so a tool introducing a lot of boilerplate code will be the bane of one engineer and only a mild inconvenience to another.
List these opinions regardless of if people agree terse code is better or worse. Its easy to be drawn off-track here, the objective is simply to record these opinions not agree on them, nows not the time to decide once and for all if tabs are better than spaces.
“James likes that we could have many tiny single-responsibility functions and we could write them in multiple languages, Dan thinks its better to have one bigger service to enforce Java as our primary language even if we’re short on Java devs right now.”
The alternatives
Finally, once the preceding questions are answered, move onto finding alternatives with extreme prejudice. This may feel needlessly hostile to the proponent of the tool, but they should also participate in this exercise too. Often I’ll suggest a tool because I think it’d solve a problem perfectly, but after peeling away many layers of post-rationalization I realize its because I just really enjoyed using it, and that there’s a better choice thats just less fun. Once you get into the habit of this, there’s almost a perverse sense of pride in trying to rip your own idea to shreds to see if theres a better choice lurking.
“We’ve got another Java service that handles data from that Dynamo DB, couldn’t we just extend that, rather than creating a new service or using Lambda functions?”
ADRs are your friend
The important thing with all of these steps is to do it in the open and in good faith, embrace loud opinions but don’t let them soak up the whole meeting - make them feel heard, record them, then bring the discussion back to the questions above. Oh and - document it, it doesn’t need to be war and peace lost forever to the Confluence black hole, but recording a quick ADR in your repo with the reasoning and the alternatives is great for team members who weren’t lucky enough to get the live performance.
Its been my experience that even if they don’t really like a tool, a developer who at least understands the reasoning behind why it was chosen will embrace it much moreso than one who doesn’t. Instead of complaints at every retro and gentle acts sabotage, they redirect their energy productively into finding a better alternative to the benefit of the entire team. Give them space to do this - usually the best way to convince someone that their pet theory won’t work is letting them try it and convince themselves instead. And if they were right, even better, you’ve just found a better solution to the problem - win-win.