I have nothing against Go (in fact I like it) but they seem to have built GOV.UK with every damn bit of technology there is available. I've seen Varnish, Go, Ruby, Python, Scala, Java, Mongo, MySQL, nginx, rails, sinatra, Django and PILES of Not Invented Here.
Also I'm not sure it's a great use of public money to build stuff like this when they should have nginx/apache up front and some configuration.
In fact this whole thing stinks of crappy information architecture resulting in a massive front end router hack that they did in Scala, binned and moved to Varnish, then binned and moved to Go.
This is not one "cohesive" web front end. It's a rat's nest and an increasingly expensive one.
We did have some configuration - it was stored in Varnish and Nginx. The time cost of shipping new redirects when content is moved or new routing configuration for new tools coming online is non-negligible, particularly as we're in the middle of moving 300+ government agencies onto the GOV.UK platform. Also having all those redirects, friendly URLs, and application mappings in static configuration makes self-service publishing tools very difficult to build.
This is all outlined in the blog post - perhaps not as clearly as I would have liked, though!
They're incredibly easy to build. I notice you use puppet. That's enough tooling to push configuration out and restart services for nginx or apache+mod_proxy. You can break configuration out into separate files per agency if you need to.
Varnish is different as that requires restarting.
We do the same with commercial kit (Riverbed) with over 140 HTTP application endpoints and that is higher friction than the equivalent setup yet we manage it with a mere 1 person...
Deployment is a solved problem. No offense but you're not Google!
"OMG we're the government and everything we do is huge and unprecedented so it is impossible for us to use off-the-shelf anything and we can learn nothing from best practises" is step 0 of every failed big government project.
There's significant thawing of that attitude (and gov.uk, for their imperfections, is definitely a very positive part of the trend) but the idea is alive and well.
Yes because a real world organization with IT systems going back 50+ years is exactly like some me to macbook and MBA start up with two men and a pantomime horse in old street.
Are you a price comparison site? Or Experian or Equifax perhaps?
Unless you clearly say you're not just some enterprise business that aren't exactly known for their technical competence this comment is totally ambiguous.
Do you just think you know what you're talking about, or do you actually?
So you are handling a small set of related services to a large number of people. That's hardly comparable to the task of migrating 300+ (distinct) government agencies/services to the new system, along with whatever old systems they currently use.
I tend to agree. The post makes a point of saying "we had a router built in scala, but no one on the team knew how to maintain it".
It seems they're allowing team members to play with "language of the week" (lotw), build something that's then used in production without the rest of the team knowing the lotw, that member moving on and then rebuilding it in a new lotw 6 months later. no doubt this GO implementation will go the same way at a later date.
I'm not opposed to using a lotw in production with other languages and such, but it needs to be done in a thought out manner with the whole "what if the project lead gets hit by a big red double decker" mentality.
I wonder if the technology churn is more of an issue with the structure and organisation of GOV.UK rather than a technical one. In the post he states that the previous router was unmaintainable as there was no-one with a deep knowledge of Scala in the team and that this was the main reason for the re-write. Why was this allowed to happen? Is there a rapid/high turn over of staff or are teams moved around without regard for skill balance?
It does not look that bad actually. Your list lacks Oracle, Microsoft technologies to really complain about inefficient usage of public money. I would love if my country used open source more.
New legislation means local councils have to justify their requirement for Oracle/MS licencing costs over open-source/free software. They can still buy the licences if they can show they really need it, but loads of councils are now properly evaluating solutions like Canonical's OS/Cloud stack.
Sadly it's easy to justify because they just play the "this is a critical system and Oracle / MS provide round the clock support and quick resolution milestones" card.
I've lost count of the amount of times I've seen Oracle products bought based off the back of Oracle's keen sales people overstating the value of their post sales support.
Plus since there's so many redundancies happening in local government, they don't like train and retain skilled in house staff who would have been able to offer the same level of support. So 3rd party support is seen as more reliable than in house.
Lastly, many project managers push for such contracts because it means they can deny responsibility (ie if Oracle fucks up, they can blame Oracle. But if the system goes down and they're responsible for their own support, then the buck stops at the project managers). So you often get project managers push for such contracts just out of laziness.
There might be a case that this legislation might see more contracts signed with open source vendors who provide support, but local governments will always still have the argument that MS / Oracle are big trusted names where as a less known (outside of the world of IT) open source vendor may not be able to provide the scale of service that the government wants (complete BS of course, but all to often these decisions are made by project managers who know jack shit about IT)
Having worked along side a number of government projects, I can confidently say that all the projects based on "off the shelf" products end up becoming vastly more expensive because you usually end up with hugely expensive support contracts with BT / IBM / MS / Oracle / etc and often the solutions need some bespoke tuning to work the way needed, so you add on expensive consultancy costs to have the software adapted to fit.
Where as developing stuff in house means that you're employing significantly cheaper labour (namely, the rubbish salaries that most get in the public sector vs the private sector) and as a bonus the cost is more likely going to stay within the UK (ie you're not paying multinational companies nor their overseas consultants (Oracle was particularly bad for this as they dumped our stuff in a US data centre and our support contacts were all living somewhere in eastern Europe. So very little - if anything - provided was UK based).
Sadly our government doesn't seem to understand about doing anything in between - or at least not that I observed when I used to work there. It was either entirely bespoke or entirely bought, developed and set up by some overpriced conglomerate (or worse yet, outsourced completely). So going by the trends that I've witnessed, I'd rather the technology playgrounds just so long as they remain developed in house.
Maybe. I don't know how huge that project is. Maybe technologies that have been chosen are the best for the tasks at hand. Maybe some of those technologies are only minor part, not in critical area, maybe it is some technology research project. Maybe it is kind of managerial pride "let's write how many lines our source code contains". As well, I guess, you don't assume that software developers should learn new technologies on their own time (while I personally love to do that).
That's a silly argument. I'd rather not be pissing an estimated £6400+maintenance up the wall on a project which could have been realised with off the shelf technology.
Just because the organisation uses open-source software doesn't excuse them from public and professional scrutiny.
An MP spending £1645 on a duck house under expenses got a lot more attention than this little bit of waste. Just redressing the balance.
Anything can be realised with off-the-shelf technology plus time. Your posited solution further down the page would also have involved work in setting up the pool configuration and switching, so let's not pretend that it comes at zero cost.
Presumably by having a list of different technologies they are taking the approach of right tool for the job. The obvious side affect also being that engineers who work on this need to be polyglot as opposed to any specific tech. I certainly categorise this as a good thing, a developer and as a stakeholder (UK resident and user of the site).
Actually they're using the wrong tools for the job because they're having to rewrite bits regularly in different tech. There is obviously no evaluation taking place.
Establishing technical standards and homogenity is important on projects of this scale.
I think they established the reasons for it fairly well in the blog post, and even better in the comments elsewhere. I'd say they are the right tools for the job.
Maybe. Maybe not. The proof is in the pudding. It's immeasautably better than before and is a lot cheaper to run. Until otherwise I am of the opinion that they are making the right decisions.
In their defense having worked on sorting out problems on old complex large scale publishers sites. If your trying to make a collection of services hang together in a seamless way you do need some sort of front end to tie it all together and that needs to be usable by non gurus.
Though they do seem to be using all the new toys maybe a bit more thought about which technology to use rather than jumping from technology to technology.
We publish several huge legacy applications, several huge new applications, several integrations, public web site, documentation, online support. A mere 80-100 million HTTP transactions a day.
People make changes to their infrastructure. The govuk team are at least brave enough to blog about it, and offer the code up for inspection. Can you post links to similar public contributions you have made?
I'm sure if you joined the team you'd wave your magic wand and everyone would fall in line, and things would be beautiful. Right.
Oh, and I'm sure the golang community are really, really pleased that "in fact, [you] like it" - very charitable of you!
The UK government is a 19th Century 'filing cabinet bandwidth' information architecture. The IT systems as always reflect the organisation structure that produces them. The data redundancy and unnecessary domain separation is frustrating and archaic.
Uk.gov is just the cherry on the top of an old and creaky enterprise design that hasn't kept pace with changes to its mission or the world around it.
Actually the stuff GDS is doing starts from a clean slate and does away with much of what you are complaining about, in fact it is intended to fix a lot of it!
If GDS was the IT department for the whole of government and the departments became a team of policy analysts and ministerial support then you'd be right.
Overwhelmingly the core information management and processing is on the local IT infrastructure of each department. GDS is unifying the web presence and some of the citizen identity systems, its not fundamentally taking over the core information systems of state.
EDIT: To be clear, I think something like GDS should take a long hard look at the core information systems of state and start again with a clean slate breaking the technology and the data out of the policy silos.
I'm not sure why you have such a negative attitude towards gov.uk, the site seems refreshingly good at it's job. It's simple, clear and easy to use. As long as they can execute fast what does the tech stack matter?
I think it's fair to say it's going to take a little while to migrate all of the systems over. Give credit where it's due, they're doing a good job with the new site and I'm sure that when it arrives the replacement for the tax disc application will be of a similar standard.
I think the site is good. I have no problem with the technology ultimately but to put a fine point on it, I'm not paying the government lots of money to invent their own infrastructure components when perfectly good ones exist already.
A lot of the comments here seem to be missing something quite important: Scala and Go are not similar languages.
Scala is a large and complex language, encompassing multiple programming idioms. Teams programming Scala (as with C++ and many other "large" languages) often have difficulties nailing down what portion of the language they are going to use. "Java-without-semicolons" all the way through to idiomatic, functional Scala.
Go is a relatively small, simple language. You could easily learn Go's basic syntax and semantics in an afternoon or two. Simplicity and suitability for programmers is an explicit design goal.
Switching away from Scala due to lack of experience while simultaneously picking up Go is not as illogical as everyone seems to be implying.
This, whilst likely true, is not a self-evident fact.
I think the argument in the blog post is weak because it does not quantify what makes Go more suitable for learning than say, Erlang or Clojure, which are dismissed outright, before moving on to discuss some of the advantages of Go in depth.
You're right, but i think the point you're missing (and i don't mean this in a rude way) is that they had a system built in scala, somehow, but now no one on the team knows scala.
They lack an architectural oversight, that they should really be using one(ish) language rather than a big mix of languages so that this doesn't occur again when no one on the team knows GO because the project leads taken a new job.
Retention. Also, I'd suggest that your average dev will learn Go quick enough to work on this router quicker than they'll learn Scala; I prefer Scala myself, though! It's a balance between proper tech choice, correct management choices moving forward, and dealing with management mistakes (like letting someone build the router in Scala when the rest of the team don't know it) from the past.
A crappy PHP dev will learn Go a hell of a lot quicker than Scala, IMO. A safer choice, for their team and position, IMO.
If Scala and Go were the only two technologies mentioned you would have a point. But this is a monstrous fad bloated lineup of technologies they have there. Integrating this mess is way way more complex than any individual language.
There are good arguments for trying new technologies, but this particular instance seems to be a very good example of how such a strategy can go terribly wrong.
[Edit]
On second thought, I might be jumping to conslusions as I don't know anything about how successful they are in actually achieving their goals, doing it on budget or how fun it is to work there. Their actual approach may be less confused than the blog post makes it sound.
I like Erlang, but in shops I've worked with that have your "average" dev in it, it's syntax and semantics would be something they struggle with (if it ain't PHP, they'll struggle with it...). Unfortunately I don't blame them. That said, I think if they truly needed to build a somewhat stateful concurrent router, Erlang is a better choice (solely because it's battle tested in those highly concurrent workloads). Picking tech for big things like this when you have to balance a teams knowledge with the right tool is a hard problem in my experience.
I agree, but if you want to learn something today then GO would fit better for the task at hand and easier to learn. That and the cost involved is really good value given the training involved with learning Scala over GO.
That and I suspect there was some wishlists on what people wanted to do in the mix, like COBOL, still used but people just do not want to learn that as it is deemed so last century in comparsision to say java, though they are altogether different fish. Bit like the difference between a proxy and a router, what sounds better on a CV.
I suspect that from a maintenance aspect that the GO code will be much easier to support and grow as and when needed as well as scaleability.
Still it is most encouraging to see such a young language like GO get so much respect and use so quickly in a enviroment that has to many in the past and present been slow to adapts new technology. But there again many Govermental departments still have XP, albeit used as a fancy terminal to some old mainframe application that just ticks along.
On the github page it says the router sits between nginx and varnish and does:
* Reverse proxy, forwarding requests to and serving responses from multiple backend servers on a single domain.
* Redirector, serving HTTP 301 and 302 redirects to new URLs.
* Gone responder, serving HTTP 410 responses for resources that used to but no longer exist.
Does anyone know why they have this as a separate piece of software instead of getting nginx to do it?
No but updating static routes via puppet/ansible and reloading is a piece of piss.
If your site is volatile enough that the routing needs to change that often then there is something wrong either with your information architecture or your development process.
Or maybe you're trying to deal with a huge legacy content migration of several hundred different government agencies and want to deploy several times a day without waiting for 10 minutes for nginx to load 1MB of config each time?
The back-end to GOV.UK is designed to be very configurable by non-technical users, and many of their routes are dynamic. While Puppet could be used, the process of automating those edits, then rolling them out to all the servers, but in batches so that no downtime is experienced with restarts, configuring load balancers to be aware of this, etc, etc, the whole process has lots of points that could fail.
It really sounds like a dynamic front-end that is aware of the routes would be a much better idea, far less that can go wrong.
Oh, then it's OK. It's always reasonable and a safe bet to rewrite something instead of understanding how something works, because after all, your predecessors were a dumb bunch and you (using Go! how cool is that!?) obviously know much more.
"Computer performance is characterized by the amount of useful work accomplished by a computer system compared to the time and resources used." --wikipedia
Is there a different definition I'm missing? I think, a car can be "fast" and take too much fuel, but (in my book) a programming language cannot be "great for performance" while being bad at resource usage.
I asked on twitter, and got a reasonable response. Investing in learning Scala is a much longer term thing than learning Go (which is, in their experience, a much quicker thing to do).
I'm a big fan of Go and we use it internally, but I too wonder why nginx + Lua wouldn't be the chosen path.
If it's good enough for Cloudflare it probably is good enough here.
Besides, Varnish also would have excelled (yes I read the first post in the series) if they had just stored the configuration in a saner format and then output the required VCL when the config changed and reloaded on the fly.
Both of the technologies I would've picked (nginx and Varnish) were in their stack.
I love working in Go, but reading this feels like a reinvention of the wheel. Go would be my choice only if these other things didn't exist already and you had to start from scratch.
The biggest pain identified was the writing, maintaining and loading of the router config. Openresty could've solved that.
Openresty and Lua are great, but Go is a more generally applicable language for building web services. So when taken in that context Go makes more sense if we look at it as a learning exercise as well as a new part of the stack.
"Erlang, Go, Scala, and Clojure are all good fits and represent the current state of the art." - why is the choice so limited? It seems to be quite JVM-focused as well, I would consider F#/Nemerle as well. Erlang was invented in 1986 by the way - so a safe bet rather than "state of the art".
F# and Nemerle are both based on the CLR, and unfortunately the performance of Mono is pretty bad, and not really suitable yet for high performance use-cases like this.
Also I'd Erlang is state of the art, even though it is so old (Haskell too), but I wouldn't call it an entirely safe bet because it requires quite a different style of programming and experience, so might not work well for certain developers.
Isn't Erlang just a (semi-)functional language with Actor Model baked in? I might be wrong but it seems that you just need any dev familiar with message queues and functional style.
I have nothing against Go (in fact I like it) but they seem to have built GOV.UK with every damn bit of technology there is available. I've seen Varnish, Go, Ruby, Python, Scala, Java, Mongo, MySQL, nginx, rails, sinatra, Django and PILES of Not Invented Here.
Also I'm not sure it's a great use of public money to build stuff like this when they should have nginx/apache up front and some configuration.
In fact this whole thing stinks of crappy information architecture resulting in a massive front end router hack that they did in Scala, binned and moved to Varnish, then binned and moved to Go.
This is not one "cohesive" web front end. It's a rat's nest and an increasingly expensive one.