In my experience, the teams that run the best are full of generalizing specialists. So you have three people that are quite handy with the DB: One is the best of the lot, and is mostly doing DB work. The others are good at it, but they bring unique strengths in other areas. In the same way, you have a guy that is pretty strong at Unix administration, a language lawyer, some people that are big into UIs and such.
So while people will probably dedicate 80% of a week to a given kind of task, they can, and do, play multiple roles, depending on how much their specific expertise is needed that week.
For instance, I am the number one choice on UIs, but I also play support managing Postgres, because I did a whole lot of that in a past life, and we don't need a full time DBA. I am also leveling up in our scheduling system, all Akka actors.
At my previous job, the team had all the Database experience it needed, but domain knowledge was weak, so I spent much of my work working with users trying to figure out what they needed, and why they were unhappy with the product. They hired me for my UI strength, but reality said that I would be more useful doing customer facing activities half the time, so I did.
The developers should know how to do ops. And vice versa.
We are moving into weird heterogeneous bugs territory. A lot of stuff that has wasted my time last year was caused by interaction of technology stacks and not by pure logical errors. Or error localized in even one package.
I had problem with premature socket close on the response with just the wrong setup for all of nginx, haproxy, slim and php 5.4 ... lets pure ops or pure dev figure that out.
Whatever part of dev/ops that applies to me personally is very much tied up in 'full-stack' debugging of intermittent performance problems and heisenbugs.
I rely on the ops guys to keep vanilla installs steady and to keep everything patched, but there's very little way that they can get insight into application problems (especially in the framework du jour) that express themselves at some other location in the stack. I just have to know the whole thing.
They are not developers - they have no rights to write commit and promote code but they are good enough at reading it to not be helpless when something breaks down and they need to take a peek what has changed recently in the codebase and stuff like that.
I on the other hand have no admin rights (only read access) but I know enough of out network and ops structure to be able to make an educated guess when something is behaving strange.
So if it hits the fan they could probably patch an emergency bug and I could fire up some VM instance. It is not about wearing a lot of hats, but being able quickly to put on a new one if needed.
I argue that the shared responsibility mentality is distracting and diminishes the power of an individual. It's hard enough being a good programmer let alone one that also knows how to configure networks, firewalls, and operating systems; one who knows how to manage security policies, LDAP organization charts and ACLs, etc. I still believe that the jack-of-all-trades is a useful individual but I don't believe an entire organization should be formed around their ideals.
As in all things in life -- the truth is somewhere in the middle. If you build significantly sophisticated systems you will need specialists at some point. I don't recommend wasting their talents on getting reamed out for forgetting to configure the firewall properly.
I don't think that's what the author is arguing for. More like, "You build it, you run it" (quote via article). Instead of handing your code off to someone else to be 100% responsible for testing, for release, for production support, the author argues that it is better to have developers share responsibility in those roles. It's very similar to what I'm doing now.
We don't hand off code to be installed, we give a release plan, an installation plan, and the code. We are on call for first runs, and there's always a subject matter expert developer on call to support overnight issues if the support team needs help. If you have this responsibility, you write more responsible code. If I may support something overnight, I'm not going to unthoughtfully make it hard to support. We share testing and documentation responsibility as well (although documentation is an issue), so again testers and support are doing their job in a vacuum because they don't have the developer's insight to the program. In fact, devs do most of the testing. It's not always easy to write test cases for a system you didn't write or design. I've kind of always thought that it's common sense to support your own product in test/production as needed (though not the first point of contact). You wrote it, you're the expert.
Reading this article surprised me because I thought it was the same everywhere. Having no responsibility for testing or prod seems like asking for bad code. If you're just writing code, you're not going to focus on what it's going to be like to support it, install it, maintain it. You'll have other, more concrete (to you) things to focus on. Coding standards may become more important. But are standards really as effective if they're not informed by / don't care about responsibility for a production product?
Perhaps it's that funny definition of "devops" blurring the lines.
I think it's perfectly reasonable for a developer to be aware of excessive memory fragmentation, leaking FD's, and anything else related to the process their program directs. It's well within my domain to debug such issues.
However the "devops" role is also pushing responsibility for configuring firewalls, LDAP ACLs, securing the OS, managing init daemons, and the entirety of the infrastructure on developers. Since it's all code now there can be more people involved and the silos can come down. "You build it, you run it," as you say.
What I'm suggesting is that for significantly sophisticated applications it won't be tenable to spread the responsibility on everyone. It's not reasonable to suggest that everyone be familiar with the implementation of a simulated-annealing algorithm in the continuous-domain constraint search solver. I think a support person would be well within reason to defer questions, bugs, and requests with that particular part to the developer responsible for it. And yet we're arguing that it's perfectly acceptable for that developer to be familiar with the intricacies of managing a data center of compute clusters? I don't know about you but I don't have an infinite amount of time on this Earth to take on such responsibilities.
I do think that there's a middle-ground to be exploited. If our expert system is ballooning in memory usage and crashing our servers it's reasonable for the developer to profile the memory usage under simulated loads and data sets and look for memory fragmentation or leaks, etc. I think it's a little too much to ask one person to also be intimately familiar with managing compute clusters, RAID failures, security policies, and package management. They, like many of us, may have a good, general idea of how it works but there exist specialists for a reason: it's hard to go beyond a certain degree of complexity without focusing. It's the reason why we work in teams!
>It's not reasonable to suggest that everyone be familiar with the implementation of a simulated-annealing algorithm in the continuous-domain constraint search solver. [...] we're arguing that it's perfectly acceptable for that developer to be familiar with the intricacies of managing a data center of compute clusters? I don't know about you but I don't have an infinite amount of time on this Earth to take on such responsibilities.
No one is arguing for that, as I said. Where in the article do you see something like that? No mention of firewalls, data centers, developers doing their own server administration. This seems to be a more important takeaway from the article:
> Ultimately, DevOps is about empathy, with everyone on your team and with your customers.
No one is suggesting "everyone does everything" at all... you're sort of creating an argument here.
Another way to look at this is that this yet another bog-standard business magazine article telling you should be working harder, longer, and appearing more dedicated than your peers to better differentiate yourself in this "fast-changing" world of ours.
I agree. Devops is just management speak for doing two jobs on one salary. Plus they find it hard to convince ops guys to wake up every night responding to alarms so calling everyone devops they can spread that pain among more people.
This reminds me of going back towards a 90s job title trend - the webmaster. While that applied very much for small Geocities sites, there seems to be an awful lot of overlap and even business alignment toward minimizing operations staff / budget such that today's so-called devops engineers would really have been the webmasters then and in SaaS shops we're just a team of webmasters that have deeper specialization / interest in specific areas.
Regardless, this pretty much only applies for SaaS shops I'd say. There's still software delivered the old way that's viable.
I have never been in a shop where as a developer I was isolated from quality assurance or production. Fixing production issues was part of the job and insuring the fixes and original code got through QA was there too.
However I have also seen this branded as "soft skills". Something that seems to be yet another way to excuse many people for their lack of skills and reward them for doing other stuff. I have seen it used to keep really bad developers, if not reward them. It also tends to be a favorite term of those who cannot manage their team properly.
There are companies out there where there is that isolation. Especially in the financial sector. I worked as a vendor to several big banks and credit card companies. Devs were expected to write code for functionality, that's about it. If there was an issue with the production environment and it wasn't related to a deployment, it was operations' problem. For example if my app has major memory issues every 3 hours or so it was operations' responsibility to restart that app when need be and ensure it stays up and running not the devs' responsibility to fix the memory issues. This isn't always the case and often what was mentioned above were temporary solutions intended to give devs more time but there was definitely that disconnect where operations is responsible maintaining the running state of the environment and the devs were primarily concerned with writing code that works and provides the functionality it is supposed to but not concerned with making the code run well.
These are great examples and response to the parent. Just this morning I was on a call with a financial sector customer who made the statement "we don't want developers to have that information / access to that information" when referring to an operations dashboard. The silos created in corporate IT are huge, with corporate IT organizationally structured in a kind of polar opposite to the typical software venture.
Programming these days seems to be more of an exercise in managing unexpected complexity between various components or pieces of code. Here's a few tips:
1) Unit test. 95% of your test suite should be unit tests. Objects under test should not require the entire codebase to be loaded in order to perform... ideally they depend on nothing, or just 1 or 2 things (which in turn hopefully depend on nothing, or just 1 or 2 things, ad nauseum).
2) Don't mutate passed-in arguments. In fact, mutate as little as possible.
3) Don't use objects with potentially unexpected, surprising or unknown behavior that makes it difficult to reason around the code (QUICK- what happens when you merge a ruby HashWithIndifferentAccess with a regular hash containing both similarly-named symbol and string keys?)
4) Use function objects, that have zero side effects, wherever it makes sense. (Or just use an entirely functional language.)
6) Microbenchmark things. Individually, a series of tasks might all "look" fast, but when run 10000 times in a row, might expose unreasonable resource utilization.
I think I'm a bit skeptical of how different the responsibilities of a 'DevOps' are different from a traditional 'Developer' in terms of actual complexity. The whole discussion seems to reduce technological change to just a little tiny part of this 'transition' to DevOps.
How can you even develop a web SaaS application without having a basic knowledge of sysadmin/networking/IT/etc?
In my opinion technology changed and development methodologies followed.
I wish the author proposed an alternative to status quo [that would resemble the past a bit more]. Can anyone?
Of course adopting PaaS and especially BaaS solutions can change it, but again, that would be a technological change rather than just a methodology thing.
> Putting developers in charge of not just building an app, but also running it in production, benefits everyone in the company, and it benefits the developer too.
How do companies which do this keep the support workload of developers low enough so that they have time for development? It's a great idea, because as the linked interview with Wener Vogels proposes, it creates a developer that has more frequent contact with customers. I suppose eventually you have to shift resources around to start other projects, but then who supports what they're leaving? Even in large teams, eventually expertise will dwindle until you're adding new people to support an existing system.
I don't really agree with the original totem-pole idea that devs can do anything QAs and ops can do. While it may be true that a good dev is capable of muddling their way through any challenge because they know how to get around the numerous blocks thrown up by technology, there are definite skills to QA and Ops that developers may be lacking and which will lead to the dev being a poor substitute.
That said, I think devs have skills which are applicable anywhere in an organization. This goes back to the virtue of a programmer being laziness. Being able to see how to build systems and automate things is something that is potentially useful everywhere. In the cases of ops and product support, there is an especially large amount of value from having devs poking their head in from time to time to pick off easy targets for improved automation, reporting, logging, or what-have-you.
Yes. My company has tried for a good many years to keep product development partitioned from operations. We build stuff, through it over the wall, and let the ops team deal with it. Certainly, we give support where necessary, but it tends to be grudgingly given, and we never actually track time spend doing that support. I've been pushing, along with several other developers, that we at least need some sort of rotating product support team so we can work with ops to resolve problems in the field without disrupting our own timelines. We've moving in the right direction, but there's a lot of inertia to overcome.
> How do companies which do this keep the support workload
> of developers low enough so that they have time for
> development?
Development and operations are not distinct things. A novel algorithm, say, needs to be run to be useful, and the developer is best-equipped to shepherd his or her code through a production environment.
Of course, that's distinct from tier-1 customer support. "Devops" doesn't normally encompass client interaction.
The concept of "everyone on support" is practiced in a lot of orgs, with varying implmentations (e.g. "one whole day a month", "one whole week per quarter", "one week of half days per quarter"). What works for your company depends a lot on it's current workflows and workload, how long support cases tend to last for. Most mentions of it refer to his post = http://signalvnoise.com/posts/3676-everyone-on-support
I read both articles and felt they both made good points.
It is in a company's (and the industry's) best interest to pair down well-paid developers and open up more entry-level positions, however, it's in the consumer's best interest for developers to maintain high accountability and "closeness" to the product.
I don't see these approaches as mutually exclusive I see how full stack development and silos can both exist in the same environment like if the Dentist in the counterpoint article's analogy takes over the hygienist's role when the hygienist is out on vacation.
It's always good to have a wide base of knowledge, and it's always good to avoid ivory-tower silos. But at the same time, it's economically inefficient to use a specialized developer to do tasks that could be done a lot cheaper by others. It's like using jet airplane to haul a tractor-trailer. Maybe you can do it, but eventually someone will exploit that inefficiency and out-compete you.
Maybe this can relate to the other story in HN about building boring systems that just work... If you are in-charge of keeping the app running, maybe you wont want to mess around with cutting-edge technologies.
To me, the rise of DevOps coincided with the commodization of server services, where provisioning of services could be performed by the software itself.
It's another abstraction layer, and requires new skills.
Elephants in the room: unfunded mandates, disparate equity, control issues, and deadlines.
Ideally, people should support their own work. If I build something and have complete ownership of it, I'll make sure it runs. If I'm the one who's getting the money and career progress, and I'm picking the deadlines, I'll be the "3:00 am guy". It seems fair that it should be that way.
On the other hand, if I'm asked to build someone else's idea, to their deadlines, making technical compromises that I wouldn't choose, then I'm not going to be the "3:00 am guy" for shots someone else called. Fuck that shit, life's too short. If you want that from me, either pay a consulting fee ($250+ per hour) or give me serious equity (like, founder levels). Otherwise, That's Not My Job.
Salary jobs are for the stability of pay and work (i.e. not being the 3:00am Guy) and for career advancement (resume, respected titles, networking). That's what I'm paying for when I accept a salary 1/10 of what I'm worth to the business. If I'm expected to take on extra duties that don't advance my career, and not getting that stability, then I'm not getting what I paid for.
I'll gladly take the negatives of ownership if I get to partake in the positives (autonomy, self-direction, participation in the reward) but expecting me to take only the first is just unreasonable.
So while people will probably dedicate 80% of a week to a given kind of task, they can, and do, play multiple roles, depending on how much their specific expertise is needed that week.
For instance, I am the number one choice on UIs, but I also play support managing Postgres, because I did a whole lot of that in a past life, and we don't need a full time DBA. I am also leveling up in our scheduling system, all Akka actors.
At my previous job, the team had all the Database experience it needed, but domain knowledge was weak, so I spent much of my work working with users trying to figure out what they needed, and why they were unhappy with the product. They hired me for my UI strength, but reality said that I would be more useful doing customer facing activities half the time, so I did.