Ask HN: A Microsoft engineer is hoping to hear your devops pains

[brendanp]

Hey Hacker News,

I'm an engineer at Microsoft in the Cloud and Enterprise division - one of the happy outcomes of the last few years is that we now have the freedom to talk directly with people about the problems they're facing.

An area in which we think we've potentially underserved customers is in the devops arena - many of our toolchains are tuned for The Old Ways, and sometimes don't offer the flexibility/composability of alternate solutions.

It's typically pretty easy for us to talk with people who already use our stuff; we have conferences, hosted forums, etc, that give us insight into how to tune things we've already built for existing customers. What I'm hoping for are some thoughts from people who don't use our tools, even if it's for philosophical rather than practical reasons. You certainly don't need to be a Windows user - we're not wed to any particular technology; we just want to build things that people find useful.

So, what prevents you from practicing continuous delivery in the fashion that you'd prefer? Are there areas that require constant investment to keep them functional? Do security or compliance concerns slow down your pipeline? Are there specific investments that you'd want us to make to help ease your pain?

If you're up for talking to us directly, we have a surveymonkey survey [https://www.surveymonkey.com/s/ND3ZRY9] that asks a few questions - we'll get back to you and setup a phone call.

Thanks!

[spdustin]

First, my bona fides: I own a long-standing SharePoint training and consulting company, was one of the first SharePoint MVPs (there were only two of us in the first award cycle for STS), and I continue to devote my professional life to teaching about and extending SharePoint.

What I like: PowerShell. Nearly everything about it. Composability and grammar, pipelining, introspection ... It's a wonderful thing. Installing SharePoint via PowerShell is a system automator's dream, IMHO.

When you're using the Windows machine it's on. Or using objects with remoting support baked in. Or have configured PowerShell remoting, which is a bit of a security black box for me to understand, and also, still requires Windows.

Now, the devops dream, one I think is shared by more people than are willing to speak up about it. I know there are third party apps that enable this dream, but they're so unknown and themselves have security issues, so devops-focused folks haven't embraced it. But with this dream, envisioned with the below sample fictional exchange from my terminal, a new world is open. One with Ansible/Chef/Salt/Vagrant/etc singing in the choir. One with GitHub-hosted repos proclaiming how a new Capistrano plugin will deploy their new ASP.Next app from their Mac development environment to their Windows IIS host. One where I and others can say, "hallelujah, hot damn, now we're talkin'!"

    $ ssh [e-mail]
    Connecting to windowsserver.example.com...Connected
    User:PS>

[Terretta]

> shared by more people than are willing to speak up about it

I'll speak up about it. WinRM is cake, but the cake is a lie.

DSC has potential. Nano and DSC together with spdustin's request is nirvana.

@brendanp, you should have shared contact info in your profile. Since you didn't, check my profile, check my name in Microsoft press releases[1] for bona fides, then hit me up because we are working on this aggressively and have just recently been talking about reaching out to you on this topic.

And to anyone else who thinks this is a cool space, email me, I'm hiring.

1. Read between the lines of my second quote here: http://news.microsoft.com/2000/06/12/microsofts-new-digital-...

[brendanp]

Sure - I'm (perhaps predictably) brendanp at microsoft. I'd be happy to talk more.

thanks!

[stephengillie]

I'm hardly your target audience, but I just left a .NET shop. Devs had trouble using TFS and all of the new tools (like New Relic) because the majority of their web app was written in VB. Due to bad design decisions, like circular DLLs, the application won't even compile properly in TFS. They're in the middle of replacing it with a ground-up .NET 4 rewrite.

And architecture choices, like creating a home-grown module for URL rewrites - and then later, when the URL Rewrite module was released with IIS8, choosing to continue using the home-grown module instead of the MS official module.

The same shop had a data import application, to ingest and ETL data from hundreds of sources. Likewise, they were heavily hamstrung in that most of the ETL was written in the depreciated DTS format, not something any modern SSIS can work with.

So...I guess my only answer is one you probably aren't really interested in - what's held back devs in my space has been old, depreciated code that nobody makes tools for anymore.

[brendanp]

I'm interested in everything!

In this case, what delayed updating the web app to a more modern architecture? Company culture? Lack of resources?

It does seem like pipelines need not just to be maintained, but also continuously improved, in order to avoid being left behind by the rest of the tool ecosystem.

[stephengillie]

Company culture was a huge part of it - lack of owner interest, previous architect apathy, intense micromanagement, ticket/maintenance focus encouraged small targeted fixes instead of necessary rewrites, circular DLLs heavily complicated development.

And brain drain - not many developers (or engineers or anyone else) willingly stay in environments like this. So the people who have spent years learning the application and how to maintain it find other work, and are replaced by people who have to analyze the system anew.

---

A little more targeted:

- TFS 2013 has AD integration, but it works in the dumbest way possible - you can't just add someone to an AD security group and they get TFS permissions, nope you have to go into TFS and find their AD account and add them.

- It's difficult to debug websites on IE8 because it doesn't have modern debugging tools. Would it be possible at all to have a browser release with the IE8 engine and the IE11 debugging tools? (I'm currently working at a job with an IE8 dependency on a web app, so I understand why it's still around.)

- Sharepoint Online is a decent document repository, and TFS is a decent document repository - both have their advantages and disadvantages. But they in no way integrate. This caused major issues when I was providing Ops documentation from a Sharepoint Online site, trying to work with Devs putting their documentation in our local TFS. (Working with either in Jira is beyond painful)

---

Unrelated, I wish WDS and WSUS were more integrated - when I push an image from WDS, I want it to have all security patches slipstreamed in from WSUS. Instead, I deploy an image with one and use the other to patch it.

[toomuchtodo]

Can you edit your post? You may want to make specific mention that you're talking about the MS toolchain. The open source toolchain for CI and integration with Linux/FreeBSD hosting/cloud providers is sufficiently mature to not run into the pain points you describe (build systems, containerization through lxc and docker, aws and every other cloud provider's api, and so forth).

Disclaimer: Infrastructure engineer who does DevOps as well.

[brendanp]

I'm definitely not trying to limit things to the Microsoft toolchain. I recognize that composability is super important, and anything we build should participate in the greater ecosystem.

[Nelkins]

One feature I'd really like to see would be the option of deploying a single binary (even if it's not a native binary; that would be really great, and I know the LILC effort is working toward something like that[1]). I know you can sort of accomplish this[2][3], but it would make me really happy if there were first class support for it.

[1]: https://github.com/dotnet/llilc/ [2]: http://research.microsoft.com/en-us/people/mbarnett/ILMerge.... [3]: http://blogs.msdn.com/b/microsoft_press/archive/2010/02/03/j...

[leap_ahead]

Drop the prices for Azure VMs to the reasonable level similar to that of the other hosting companies. Right now the first tolerable configuration A1 costs $57 and this can be had for $15 elsewhere.

Please also make your payment processor accept virtual credit cards so I can actually pay you money with the means I have (not in a possession of a real credit card). PayPal will also do nicely.

[brendanp]

I don't have much insight into Azure pricing, unfortunately. You need the bigger disk with the standard tier A1 for your app/service?

[leap_ahead]

No, the disk is fine. I just wish that VM cost $20, perhaps $25, but not $57 which it costs now (I've corrected my original post, it was not $67 but $57 as I remembered wrongly).

I wish to publish a relatively simple app for a reasonable price and I'm sadly skipping Azure from my consideration. I'm seeing VMs similar to A1 cost about $12-15 with many hosting companies. And for $20-25 I can get your level A2 which costs $115.

I realize you may not be in a position to influence pricing. I just would like to point out that the pricing of Azure VMs is not simply uncompetitive, but plainly prohibitive. I hope Microsoft does something about it.

Saying it as a veteran Microsoft developer since the 90s.

[janpieterz]

I think the lowest entry point for this is an A0, which offers an admittedly very small (half an A1) VM for 9.98 euro a month. Good to start off with, and if you build your application slightly differently you can actually run it on two machines, total power is the same, with only the communication overhead between them which luckily isn't that high if you put them in the same region. I find that I can run a very decent amount of software on Azure for a very very low price, but sometimes I needed to make certain architectural changes (for example using Cloud Services + Table Storage + Service Bus instead of VM's and SQL Server makes the whole system very cheap).

A very nice thing that NServiceBus does on Azure is giving the opportunity to have multiple endpoints hosted in the same Cloud Service [1]. Taking this mindset, for small applications, you can build it so when needed (and presumably when money is also less of a thing since you need more), you can scale out easily, but when just starting up you can do it super cheap. This depends a lot on your application though, but for example the Topshelf framework [2] can help a lot.

I'm not affiliated with either Microsoft or NServiceBus, but I am an avid user of both and really love the ecosystem, including the pricing ;)

[1] http://docs.particular.net/nservicebus/azure/shared-hosting-...

[2] https://github.com/Topshelf/Topshelf

[Eridrus]

The thing keeping us from CD is a lack of faith in our automated testing, mostly because we don't have enough automated testing, but partly because our problem domain makes it very hard to automate the testing that we are really concerned about doing. Part of this is definitely lack of testing culture though.

[brendanp]

Yeah - I'm sympathetic. It's often difficult to sell people on the notion that integration testing might be just as expensive to implement as the service/feature that you're building in the first place.

It sounds like this is partly an education issue - are there specific areas within your problem domain that more tooling would actually help with?

[Eridrus]

Our fundamental problem is that we have a Typescript (<3) analytics payload that has to deal with lots of weird JS execution environments that we fundamentally can't recreate in a lab, which means we have no idea if collection code is working as intended until we run it in prod and have a manual look at the data.

But I started thinking about this in a bit more detail, and the things that make our problem domain hard don't necessarily have to block doing fast releases, since the failure scenarios that we want to block should largely be automatically detectable.

The parts that we haven't been able to automate though are around verifying that our payload, does not have any user visible impact on sites we run in, including DOM elements we want hidden definitely still being hidden, verification that we don't trip any SSL or other warnings, and we don't generate console warnings from the browser, etc. So if Selenium was not a complete hack job and had more knowledge of the browser chrome/UI, we could probably fully automate it.

[penguinlinux]

Hello and Thanks for reaching out to the Dev and Operations community. I am a devops engineer who is very familiar with Amazon AWS, Backspace Cloud and Google Compute Engine. Is there a way that we can get free trials of Amazon azure. AWS provides you with free credits for a year to run a few services and try their APIs and also try their services. This has been useful because it allowed me to learn about their products without having to pay for them. Then I have been able to use my knowledge and promote AWS service at companies I have worked and pay for AWS.

Is there such a resource with Microsoft Azure?

Thanks

[janpieterz]

Hi,

I'm not affiliated with Microsoft, but you can sign up for a free trial [1]. They'll give you 150 euro in credits for a month to play around with. Besides that, a lot of their services are available very cheaply to play around with, for example table storage and blob storage, Azure websites, would hardly cost you anything or you can even use it for free if you hardly have any data/usage to get to know the system. There are of course things that will cost you, but if you're just willing to play around, for 50 euro a month investment yourself you can get quite far.

What Microsoft also offers is Bizspark [2], which if it is applicable, will help you along the way nicely.

I fully agree that the Amazon credits is setup a little bit nicer to just play with, but with that first month (if you dedicate some time for it) you should get a good feel of it, plus the opportunity for Bizspark helps a lot, and if you don't fit in the Bizspark requirements I think it's more than reasonable to assume you can pay 50 euros a month to test it for a longer period than a month!

[1] http://www.microsoft.com/bizspark/default.aspx

[2] http://azure.microsoft.com/en-us/pricing/free-trial/

[Someone1234]

You should seriously re-post this on Reddit's /r/sysadmin there are tons of DevOps people in there.

[brendanp]

That's a great idea, though I'm a little worried about being perceived as an astroturfer.

[Someone1234]

You could try contacting the mods first, state your reasoning and ask for their blessing?

[antod]

I work in a small heterogeneous Linux/Windows SaaS environment where most coding is done in Java or Python. We don't use an MS toolchain because historically MS tools want you to use other MS tools almost exclusively.

But in terms of DevOps things MS as a whole could do that would make our life easier:

* Contribute to Vagrant, Packer, Saltstack and Ansible etc to help make Windows clients better first class citizens with those tools.

* Keep shrinking the disk and deployment (both time and space) footprints of Windows Server and SQL Server. And making sure they stay close to that original size after extensive patching without ballooning out.

* Somehow make licensing and activation etc less painful.

* Native SSH and rsync servers and clients shipped with Windows Servers for better interoperability. No need for a posix command shell, we'd be happy with executing Powershell over SSH - eg if remote Powershell had an SSH transport option as well as WinRM. And being able to forward ports over SSH too.

* Make high availability, robustness and security features common across SQL Server editions. We don't care about business intelligence, reporting etc etc but do want things like mirroring and compressed backups etc without paying through the nose for it.

[neduma]

My 2 cents. (By the i really love the recent MS efforts - When was the last time heard M$? Can't think of it..)

I heard a lot of good things about Powershell. Why don't you start from there? I see it is a 'carrot' hook. Build an friendly/developer centric ecosystem based on that and build it from there..

If you really look at vagrant/docker/ansible ecosystem, It's all about cli/api/scriptable and disposable environments. Can't we come up with some kind of framework (called Ultron, for example) which nicely wraps around vms/containers/cmt tools based on powershell semantics with full integration of Azhure. I would love to check that out.

Any effort to bring Azure into developer desktop/laptop to play around would be compelling for devs.

[aprdm]

I am another one that also enjoys the recent microsoft effort to open source and be more open with the community. I think devops envolves a lot of scripting, having a POSIX compatible shell out of the box in Windows would help a lot. Same for a decent ssh client.

[brendanp]

Thanks for the suggestions! We've definitely heard this feedback consistently - especially around SSH.

[jurymatic]

There are some things we've encountered that should have been relatively basic but turned out to be a nightmare. One is configuring a VM to have a public, static IP. I currently have a Windows VM running that somehow has no fewer than 3 different static IP's, depending on where you look. Yes, we followed the directions and had our dev ops guy give it a try.

Second, we deal with some third party vendors who require us to white list our IP's. We ended up having to config a VPN through Digital Ocean because even our dev ops guy and our IT guy together couldn't figure out how to route our traffic.

[lawnchair_larry]

Test 1:

Install 3 systems with a base OS install, and nothing else - Windows, Mac, Linux.

Put a programmer in front of each. Race to create and run "Hello World" in Python, Ruby, C/C++, and Java.

Test 2:

Set up a Windows Server and a Linux server running a web erver. Using a Mac or a Chromebook, remotely edit the web server configuration file over a dialup-speed connection.

Test 3a:

Get a fresh installation of Windows, Linux, Mac. Race to get up and running on a local instance of Django and Rails using a sqlite db.

Test 3b:

From the previous test, swap out sqlite for postgres. Then try swapping it out for SQL server.

Test 4:

Try to do even 1 single thing in powershell against a remote Windows Server when using a Mac or Linux.

Test 5:

Without syncing, search for a function by name across your repos hosted on Github/Gitlab/Gitweb. Try the same thing on Visual Studio TFS or Visual Studio Online.

Test 6:

Delete HTTP.SYS on a Windows Server and try to do anything at all. You can't even use remote powershell! There is no way to get up and running with a simple userland webserver using anything in your entire ecosystem. All routes lead to WCF and HTTP.SYS. Can you imagine if you needed a linux kernel module loaded in order to use SSH?

Test 7:

Install Windows, Linux, Mac on the first Wednesday of the month. Assuming updates are released for your major services on all three platforms that month - survive until the following Wednesday without a reboot.

Test 8:

Survive the Hello World test in #1, using C#, without a reboot.

Test 9:

Write a simple script to query a TFS server from Mac/Linux. Write a simple script to query a redmine server from Mac/Linux.

Test 10:

From a fresh install, capture a single packet over loopback on all platforms.

Test 11:

Deploy a repo server for all of your internally developed packages, and have any other developer fetch and install that using a native package manager and a single command.

Test 12:

<I have nothing here, but it should be something to demonstrate to how cumbersome it is to deal with manipulating XML for everything, compared to any other text-based format>

Test 13:

Try to integrate Gitlab with whatever Oauth2 thing Azure provides. Compare this with trying to use oauth2 via Google or anyone else.

--

By doing these things, you'll find that they are nearly frictionless on the other operating systems in comparison. You'll find that some are actually impossible to do on Windows.

Actually, you'll have them all finished for Mac and Linux before Visual Studio is finished installing in preparation for test #1.

[ramon]

Time to think about cloud, virtualization and containers when talking about CI. The scenarios I see currently are: 1) AWS - S3 / EC2 / Cloudfront 2) Containers (Docker) 3) Virtualization (Citrix, VMWare)

[brendanp]

Just to make sure I understand, you're suggesting that we need to ensure that our tools fully integrate with non-Microsoft cloud providers, container providers and virtualization providers (rather than being primarily targeted at Azure/Windows containers/Hyper-V)?

[munishm]

I don't think that would be feasible for something like MSFT as they can loose their market control but it would be a dream come true for devOPs - switching between the Clouds or using S3/SQS with Azure.