I've been using CoreOS & Docker for about 3 months now in production (stable channel on AWS). At the moment I have a "cluster" of 2 machines on AWS and 1 simple CI server on DigitalOcean, also on CoreOS & Docker. It wasn't easy for me to get used to "the docker way" of doing things, but I think I'm quite fluent in using Docker & building containers now. Setting up everything is very easy & productive, they have a great documentation (example: https://coreos.com/docs/running-coreos/cloud-providers/ec2/). If you're interested, you can setup a test VMs using Vagrant, this takes like 5 minutes: https://coreos.com/docs/running-coreos/platforms/vagrant/
However, I don't really feel comfortable with Docker security and I will probably switch to rkt - more focus on security and better approach to containers imo. CoreOS is incredibly good product, these people see the future. Full disclosure: I'm very happy user of CoreOS products.
I'm in a similar place with regards to Docker versus Rocket, but I'm not a big fan of CoreOS right now. I think it's got some real neat ideas, but in my experience with it "everything in a container" is as yet unmanageable at scale. And I'm very uncomfortable with signing over the entirety of my infrastructure to a small company whose--venture-backed, which makes me extremely uncomfortable when it comes to running my platform and infrastructure--goals are not clear to me. (I have less of a beef with, say, Chef, with much more of a history and more generally understood goals and a business model, but I'm still not super comfortable with them, either.)
I also don't much care for etcd, because IMO Zookeeper's rep for complexity is hugely overblown and most folks end up re-implementing Zookeeper poorly in etcd, but that's a side thing.
All that said, I agree with you that Rocket is a much, much better idea and design, and that despite my misgivings about their corporate goals CoreOS is a way more serious project from a security standpoint than Docker. I'm excited to see this, if only because I think Rocket will pick up some dV from this.
I believe that the current state of containers is just the beginning of the trend and containers development. IMO, containerization simplifies security, fast deployment cycles, dependency problems and improves some other things. There are problems, of course and things that are not solved in a containers world. The obvious thing is a persistent storage - we don't have a good solution to this that uses containers. There are "patterns" (like, for example: running specialized database container and sharing "volume" with the host), but it's just a hacky trick and not a solution. I really hope it'll be solved. Using AWS RDS (Amazon Database in the Cloud) and S3 is kind of a "workaround" (pretty good workaround, to be honest), but not everybody wants to use AWS.
I really like Ansible and I believed that Ansible could help with Docker deployments (http://www.ansible.com/docker), but at the moment I think that using Dockerfiles is faster and simpler. I'm still using Ansible for Vagrant provisioning, but don't need it for setting up production/ci anymore. Sure, Ansible/Chef/Puppet will be probably on the market and used by some big players, but I'd be very surprised if we won't see big shift into containers for most of the technology companies. Being faster, more productive and cost effectiveness (related: https://www.youtube.com/watch?v=xK0njkATf84) are good enough arguments imo.
Just anecdotally, I've had the shittiest experience in recent memory with CoreOS. Their choice of BTRFS bites me pretty much daily whenever logs start writing stack traces for errors. A similar problem happened when Deis chose CephFS for their registry container.
While I'm a huge fan of Docker I will maintain that CoreOS is not ready for production in most capacities and recommend against using it in its current form.
Can you elaborate on "whenever logs start writing stack traces for errors."? What caused the problem? And what exactly was the problem with Deis & CephFS? I was considering using those some time ago.
Sure. Whenever the log messages start to get pretty decent in size, like, when the log messages start including tracebacks due to errors, and you have lots of errors happening back to back, like lets say high traffic is causing some deadlock exception at the DB layer or something(this was our actual problem), this high write causes BTRFS to lock up. Usually this happens because of some kind of kernel level error, our ops guy is familiar with the exact details of the exception BTRFS throws. Suffice it to say, when that happens, it's not immediately obvious the instance is unavailable. If you have plenty of errors happening in a short time span, this behavior will start to roll across your cluster and depending if all of the hosts are running the offending container, your entire fleet of hosts will lock up. This sucks bad. The only real solution is to bounce the host machines -- though they seem to come back just fine.
CephFS had a similar-ish problem, where if you happened to ever own less than 3 nodes (almost always because of the above), it would have a real problem self-healing and start to get confused and re-elect bad nodes to quorum leadership, possibly clobbering all your registry data. We contributed a patch back to Deis to use S3 as a registry persistence layer because, well, having a volatile registry sucked. We were about to develop a control layer for quorum services to live on separate of the application services we were developing, and using a proxy in the quorum layer to communicate stuff like etcd. I would highly recommend this approach if you end up with any services that require a quorum.
I would prefer to avoid many disk operations in the first place. Not sure, if it's applies to your problem, but have you thought about using something like Sentry (https://getsentry.com/ || https://github.com/getsentry/sentry)? Maybe there is some other tool that could help you here, I'm often impressed by the Open Source community and the vast number of different packages for (almost ;)) every technological problem.
Do you mean the differences to the way I was setting up servers before CoreOS? Well, this is subjective, of course, but I feel that it's easier & faster to isolate everything now. I still have to add iptables rules (I also use AWS ports restrictions) and add SSH keys, but that's all I have to do. Auto updates with scheduled reboots and safe rollback work out of the box, I have only few vector attacks in terms of services exposed to the public network. When I have everything in 1 data center, "internal" tools (etcd, fleet, confd, locksmithd) are open only in private network and just ports 22, 80 & 443 have to be exposed. Containers are also "linked" in private network, I can easily securely connect services/applications in containers running on different machines without much overhead.
Often the easiest attack vector is/are security vulnerabilities in applications exposed to the public network. Containers are great here, because when application is compromised, just 1 particular container is dead (and probably other containers running that app). I can just run docker kill app && docker rm app and the rest of my containers are (probably) ok. The problem is that attacker can gain access to data in etcd, since it's not encrypted by default and has no per-user permission (but you can use HTTPS in etcd cluster, which is good), however as of now you can use something like crypt: https://github.com/xordataexchange/crypt to use gpg in etcd (with natural API).
Docker gives you root for everything in the container, which may not be the best option. Also, only lately Docker have the option to verify downloaded images. rkt already has this feature, and it's just 0.5.5. (see also: https://github.com/coreos/etcd/blob/master/Documentation/sec...). Docker is great, don't get me wrong, I think it's a very good software and I'm glad I can use it. CoreOS team has just a better vision and priorities, IMO.
I'm considering a blog post about CoreOS/Docker/rkt and similar tools, not sure if there's interest?
The title is slightly misleading but not entirely. Docker was, at first, a container technology. However, after it received its funding it has tried to convince the community and investors that it can be more and started to provide container orchestration tools, i.e Swarm, Machine and so on. Docker has received most of its momentum and attention because of the work they did around the container format. Docker build, run, stop, encapsulates most users experience of Docker.
Docker wants to be known as a platform providing container management tools and it needs to do this quickly because Mesosphere and Kubernetes provide this type of functionality at a more mature level, albeit using somewhat different philosophies. The container format part of Docker is ultimately replaceable.
Google (Ventures) backing Tectonic is significant because Tectonic will provide a commercial enterprise-ready distribution of Kubernetes supporting Rocket. Rocket doesn't need to reach feature-parity with Docker to be a notable replacement because Docker already does too much. All Rocket needs to do is provide the much needed enterprise features that Docker is lacking and integrate well with Mesosphere or Kubernetes. If this happens before Swarm and Machine mature, we could be wondering 6 months down the road what the hype around Docker was all about.
If Tectonic succeeds in the enterprise marketplace then Google will have stealthily marginalised Docker using community efforts through rkt and kubernetes and not had to fight them directly.
hmmm, my mistake. I should have read this more carefully.
"Google also offers cloud computing services a la Amazon, and it was the first big-name cloud company to embrace Docker. Since then, Amazon, Microsoft, and others have followed, responding to Docker’s enormous popularity among Silicon Valley developers. But now, Google is backing Rocket as well, rolling the technology into its Kubernetes cloud computing software. Google calls this “an important milestone for the Kubernetes project.”
Yes, rkt App Container runtime is based on cgroups and namespaces. Same as runtimes of Docker, lmctfy, systemd-nspawn, lxc or (almost) any other container technology for Linux.
The big difference is on specification of images. rkt introduced the App Container specification for its images: https://github.com/appc/spec/blob/master/SPEC.md#image-archi...
I believe there's even at least one production deployment of Docker on a different stack of underlying container technology, the one Joyent runs on top of SmartOS (which uses Solaris/Illumos zones as the container tech).
Disclosure: I work at Google and am a co-founder of the Kubernetes project.
Actually no. Google Ventures runs quite autonomously from the product teams; I speak to them from time-to-time but they make their own decisions and don't influence ours.
We supported the rkt/appc PR because (1) we try hard to be an open community and to not play favorites, and (2) because we think the project has good promise as an open standard and as a lightweight modular runtime.
Please note that we are trying hard not to play favorites. Docker support will continue indefinitely and we will continue to make investments in the Docker community.
I think the title should be "Kubernetes is going to support X" where X is either rkt or appc. CoreOS is a company, developing an open container spec (appc). rkt is CoreOS's implementation of that spec, but other implementations are possible. Currently X=CoreOS, which is most likely to imply CoreOS's operating system, and that isn't what the article is about.
Headline: Google backs alternative to AngularJS, the web's Last Next Big Thing. Google reveals the entire AngularJS project is a ruse designed to ruin countless weekends of aspiring web dev's! No backward compatibility, no human-readable documentation and no implementation of their own technology in their own core product line! In fact, Google states, AngularJS is their first experiment with using their "considerable weight" to trick engineers into learning a technology which is completely useless and unsupported in the long run.
Before I get down-voted to oblivion, apologies for snark but I think I'm making a valid point. A lot of engineer's decided to throw in with AngularJS because of Google's "considerable weight" but as we're seeing now, we're all getting burned. It's a fine technology but the pain points have been severe. My point being just because Google backs a technology doesn't mean it's going to be useful, and should instead be based on its merits.
I think your point is fair: all technology should be reviewed strictly on its merits. We certainly don't have a magic formula that gets it right 100% of the time. I can however say we are quite committed to this area and this project in particular.
Here are the things I think about most days:
In the case of Kubernetes, we are doing is a little different to anything we have done before. We have a commercial offering (Google Container Engine) based on this project that we ultimately hope the world to consider as a great way to run containers for money (yes, we hope to create a business around this). The nice thing is we are a service provider so we just need to convince someone to use our compute services for us to benefit from an open source project we sponsor. We aren't really hiding our intentions here; we do containers well inside Google and have some whizz-bang container hosting infrastructure and services. The problem is that people are not likely to use it unless there is a 100% open, fully compatible alternative available for off-Google use. By way of background I was the product lead for Google Compute Engine (our infrastructure-as-a-service product) for several years. The product is going really well now, but I learned the hard way that big customers like choices and no-one likes to be locked in. This isn't just us being Googley (though I have a personal predilection to openness); this is a simple fact of the marketplace today. We could not convince someone to bet on a disruptive technology if an open and credible alternative does not exist.
The second thing is that we see tremendous power in the open source community that we want to tap into, both for our cloud product line, and for our own internal use (it is no secret we use a lot of OSS internally). Kubernetes has achieved an almost ridiculous velocity (check out our PR acceptance rates) because the community is supporting it. It is a better product because we benefit not only from having a bunch of Google engineers working on it, but because we also get engineers from Red Hat, CoreOS and others who bring different perspective and capabilities. We aren't enterprise guys for example, but the Red Hat folks are -- the combined team is way better than the sum of the parts. Our service (Google Container Engine) is stronger for it, and in the limit we expect to use Kubernetes to run a lot of our internal services too.
Thanks, right on. It's a huge organization - I think small potatoes like my dev shoppe need to be wary of just doubling down 'because Google.' Angular was hopefully an outlier! Looking forward to seeing how Kubernetes evolves.
Google supports Docker, too; Google supporting multiple approaches to the same problem isn't new (heck, its not even that uncommon to have multiple solutions to the same problem originating at Google.)
However, I don't really feel comfortable with Docker security and I will probably switch to rkt - more focus on security and better approach to containers imo. CoreOS is incredibly good product, these people see the future. Full disclosure: I'm very happy user of CoreOS products.