Wow that is a long and awesome list! If you don't mind me asking, how much did it cost you to setup / maintain it? Also what is the main reason driving you to self host so much stuff?
It was cheap to set up, all the cost is in time to learn to implement in a secure and performant fashion
Hardware actually cost money. Here's a breakdown:
> 4 pcengines alix boxes for openbsd router/firewall appliances
These were around $120 each with 4GB flash storage at the time. They're half that now. Low power, no cooling required, x86, 3 ethernet. You could buy APU2 now for more power
> 3 supermicro Opteron servers with KVM/corosync/Sheepdog/csync2 for hosting all VMs
I used cheap cases, eBay MB/CPU/RAM, tiered storage (green/black/ssd) to keep costs down and infiniband for 10gbit interconnects ($15/card on ebay!). I made sure to get quality components (esp MB/power supplies). One of the servers is also my desktop. I'm guessing they were about $800 each. Having the 3 node cluster is nice. When we had a forest fire threaten our town and we were ordered to evacuate I just grabbed one box and all my data was already replicated to it. When we returned home I plugged it back in and it re-synched back up.
Maintenance has been a non-issue. I haven't had any components die except the occasional HD. Power costs are the main thing. Estimate about $500/yr at $0.10/KWh
> PF + CARP + pfsync + OpenBGPD for routing
I use a local indy ISP that gives me a bunch of static IPs and lets me route a /29 with BGP. I know them fairly well, and get a sweet deal. Doesn't cost more than a regular consumer connect, though it is slower.
I've used OpenBSD since the early 2.x days, and find it very easy to administer. The release and documentation quality are second to none, and I've found the community to be very helpful as long as you've tried to help yourself first.
> Unbound + NSD + Bind for DNS
I keep my Bind server vlanned off and serve everything out thru unbound/nsd. Both of those programs are very easy to set up, the real beast being bind. I know there are better alternatives out there, but I know Bind well and have lots of custom config I don't want to throw away.
> SSH/OpenBSD ipsec/apache Guacamole for roaming and permanent site-to-site VPN (pcengines ALIX hosted at my inlaws in Japan)
If I were to pick one outstanding program on this list it'd probably be Guacamole. Pure HTML5 rdp/vnc/ssh/telnet/etc client that is seriously amazing. I've set it up at a half dozen places now, and it's never so much as hiccuped.
OpenBSD IPSEC is VERY easy to set up, if you've had nightmare experiences with other packages!
> Apache + Lets Encrypt + awstats + relayd for serving web pages and analysis
Apache is the old standard, and awstats is cool for keeping tabs on what is going on in the logs (geoip as well).
Lets encrypt was amazingly easy. I'm using certbot and set it up in under an hour. I'm forcing SSL on all my web services now.
Relayd is another "so simple and it just works" package from OpenBSD. I use it as a front-end load balancer.
> ZoneMinder for video monitoring. Tied into legacy security system for automation
On Debian (my Linux distro of choice), this was simple to set up. Perl scripts to integrate it into my DSC security system. Auto arm/disarm camera recording and relay light control required the IT serial integration board and programmers manual for the system
> Postgres for database work. Some mysql/redis
I've been using postgres forever, so setup and use are second nature. An absolutely incredible piece of software engineering.
> NetDisco + Nagios + NagVis + NFSen + MRTG + Smokeping + PNP4Nagios + NUT + Splunk + Racktables for monitoring. All configs are dynamically generated from netdisco db
This is another stack I've set up at many locations (including businesses). They are a real timesink to integrate together. I have MANY custom scripts to make the config generation from netdisco work properly, but once setup you have total insight into every aspect of your network (and I forgot to list RANCID!). Netdisco/NFSen on their own are still a killer combination, and work as well or better than packages that cost tens of thousands of dollars. I'm happy to help any one trying to set these up if you PM me.
> OpenSMTPD + Citadel (webcit) for email delivery and webmail
Citadel is maybe the weakest thing I have in my stack. I'm looking at the other webmail solutions in this thread carefully
> Minetest server for kids. We use this tons as a family, and the kids spend lots of time modding. TW2002 server. TShock server.
When your kids are asking to learn LUA, you know something is working!
> OpenELEC for diskless netboot KODI machines around the house
Amazing and easy to set up if you already have your own DHCP server you can modify. Just need tftp and nfs after that. Using OLD desktop PCs for this works great. I'm using cast off dell gx290s
> Samba4 Domain controller + NFS for sharing files in different applications
I've been using Samba4 since pre-alpha (TP series) when you had to run your own LDAP server. Things are so easy now its hard to overstate. Using Bind makes it a bit trickier since I need to add some magic entries, but if you use the builtin its a single python script between you and a full SSO AD domain.
> SVN for source control and Config diffs for all servers/tools/network devices
I found SVN config to be a bit of a head-scratcher. I think this is another one where other tools are probably better nowadays. I'm looking at some of the other things people are suggesting.
> Asterisk via FreePBX / NCID for all phone/CallerID services, including remote handsets at VPN locations.
Another timesink. PBXs are hard to configure, and I'd move to another system if there was something less esoteric.
As to why I do it? I find it satisfying to learn how things work, like the idea that I'm master of my own destiny and know how my data is being used
How do you secure your publicly accessible IP at home from the web. I see you use static IP's I guess it's no different than renting a sever? You probably have separate networks (local home vs. serving)
Looking at Netdisco I have 12 VLANs. ISP1, ISP2 (from Dad's house across the valley), OSPF backbone, DMZ, LAN, Guest, Cameras, Servers, Japan, Kids, Management and Test. Everything passes thru PF and only exposes services as required. The Guest VLAN, for example, has no way to get to the rest of the VLANs. Camera VLAN can only talk to the DVR software, etc.
The important thing with securing my public IPs (as with securing anything) is to understand the surface area and minimize risks. With the number of services I'm exposing, I have to be careful. First thing is to keep all OSs up to date. OpenBSD every 6 months and Debian on an ongoing basis.
Next, whenever possible use single-purpose proxies that have been well audited. In my case OpenSMTPD, Unbound and NSD protect the always popular mail and dns servers from attack.
Keeping complicated things off the internet is important. Big CMS packages are constantly under attack. I only run either static pages on my webserver, or very carefully audited custom PHP pages (written with an attacker's eye to exploit injection)
When I expose something more complicated like Guacamole or Citadel that might end up with a known hole in the login screen, I put it behind both SSL and an HTTP simple auth login prompt. It's ancient, and so well tested that it is unlikely to end up with an exploit against it. Or at least less likely than the app its protecting. I've actually been toying around with doing 2 factor auth on these services with a dynamically generated simple auth sent via SMS bridge or IFTTT...
On my internal network, I have my family running linux wherever possible, and keep Java and Flash off the windows PCs. Everyone runs with a nonprivledged account and the OS/software packages/PDF readers are all kept up to date. I think the wife and kid's PCs are probably the most likely things to be compromised on the network, so I keep a close eye on them.
I also take the reactive approach of log alerting, change monitoring and general diligence to the state of my network. If you know what has changed in the last 24 hours it tends to be a small enough dataset to eyeball, and will tend to tip you off when things are wonky. I also have enough logging and data sources (syslog, configs diffs, apache logs, nfsen data, mrtg data) that it would be exceptionally difficult for an attacker to wipe out all trace of their presence.
I do not know of a successful attack against my network, tho I'm not naive enough to think that it hasn't happened. I just haven't realized it if it has.
Wow thanks a lot for the details this is not my field haha regarding the tech you mentioned in the beginning.
Curious what your family thinks having to run Linux. It's my primary os myself with i3-wm. People usually complain "install apps command line" ui, etc .. whatever Ubuntu being as supported as it is that UI is so slow/bloated IMO.
I have Windows as well but use Virtual box and Linux to access the web if it's not just YouTube/Gmail/regular pages with adblock/uorigin running.
I don't even know how many ports there are on computers, I mean in my experience I've seen up to port 10000, I think you have to enable them? I usually use 21/22/80/443 though I've seen 3000 (websocket), 4200 (angular), 246 or 286 (windows RDP) I don't know...
Windows rdp is 3389, but really, anything can go on any port up to 65535. The common ports like 80 and 443 are just there as "it's common to use these for services, so you don't have to put in http://google.com:80". And there's no real "enabling" them, it's just a question of if your firewall allows it, and if you have something actually listening.
Also, websocket is just a protocol upgrade slapped on http/https, so it normally goes over 80/443. If someone's running it on another port, it's probably cause the server they have on their normal http/https doesn't work well with websockets.
Yeah the 3000 I just recalled that with a tutorial on socket.io I did a couple years back.
Yeah my bad in the RDP, you know I messed up on that. Was trying to get around firewall rules, I edited the registry and changed the port for it... Could not get back in. This was one of Amazon's windows servers, and I mounted it as a volume to another one and could not access the registry to change it back so yeah... Locked myself out haha.
I have a long history of having my family use Linux. I'm a Debian guy myself, so I've always either used Ubuntu or basic Debian. I run dwm on any machines that I put X on (very few, honestly).
I started my Dad on Ubuntu around 2008, and he hasn't used anything else since. Shortly thereafter my Mom, then my Sisters. Most recently my Step-mom, and Grandparents have gone to Ubuntu.
When I got some castoff laptops from work, I turned them into Sugar notebooks for my kids, and once that became too limiting for them I helped them install mainline Debian. Some of them run Gnome and my oldest runs fvwm.
My wife still runs Windows because of inertia, more than anything TBH.
In my experience, I've only had 1 peripheral that someone has bought that was totally unusable due to drivers (scanner), and the only programs my relatives have asked for that weren't available were my kids wanting to play Roblox, which I didn't want them playing anyways. Thanks to Minetest and buying most Humble Indie bundles I actually have a pretty good library of Linux games for them to play, so there hasn't been much bellyaching from them. Well, that and the Windows gaming/home theatre PC.
I've gone to a 2 strikes and you're out policy on Windows installs. If I have to re-install it for you more than one, and you're either getting Linux or finding someone else to fix your computer. My Grandparents got caught on this policy, but my stepmom actually asked for "That system that Dad has that doesn't get viruses". Happy ever since.
Most people that aren't highly technical tend be served by Firefox/Thunderbird/LibreOffice for 99.99% of their needs. It's mostly Facebook these days TBH.
It makes support dead simple. no-ip.com and SSH let me fix almost anything remotely, and no one has every gotten pwned that I know of.
Other people have answered the port question, but I'll try to go at a slightly lower level. Each open port will have a program running on the host that has opened a listening socket on that port. Netstat can help you find out what is listening, and on what interface/port. As a rule, only root can open ports under 1024, and any well written server will drop all non-required privs. You can check with the ps command. This is somewhat enforced on some OSs, eg OpenBSD with pledge.
From a security standpoint you should verify that you know and understand every server listening on a socket, which interfaces they are bound to (netstat asterisk means all interfaces), and whether they are exposed to the internet via direct interface/proxy/port forward/etc.
One trick to secure services that need remote access: if the service is only for technical users, you can give them each an account with no interactive shell, and then they can ssh port-forward to the port they want to access. Eg. you can make 3389 (RDP) only listen on the local LAN or the loopback device, ssh to your router with port forwarding local port 3399 (or whatever), to remote IP:3389 and point your local rdp client to localhost:3399. Great for ad-hoc limited VPN type connections. That way you only have to be aware of SSH remote holes, and not the more-likely RDP server.
Hopefully if any of this is incorrect, someone will correct me
Man that's cool and you save yourself buying windows licenses.
I'll have to check that out no-ip, so you install openssh server to SSH int their computers? Yeah I like that feature myself too lazy to get up and use the dev desktop.
That's my hope regarding Linux that it's safe, using virtual box and Linux.
Haven't heard of Sugar notebooks.
Yeah I used to run Linux Mint, then Debian, then Ubuntu mostly because of their good driver support, a few times I've had laptops with Debian installed and they couldn't connect to WiFi right off the bat. Then i3-wm because my computers are generally garbage. At least now both my desktop/laptop have 8gb ram. Could go higher maxing out chrome tabs.
Anyway thanks and for the info on the ports, I have used netstat before it's intimidating haha, so many at least when I checked on my windows laptop if I recall right.
Hardware actually cost money. Here's a breakdown:
> 4 pcengines alix boxes for openbsd router/firewall appliances
These were around $120 each with 4GB flash storage at the time. They're half that now. Low power, no cooling required, x86, 3 ethernet. You could buy APU2 now for more power
> 3 supermicro Opteron servers with KVM/corosync/Sheepdog/csync2 for hosting all VMs
I used cheap cases, eBay MB/CPU/RAM, tiered storage (green/black/ssd) to keep costs down and infiniband for 10gbit interconnects ($15/card on ebay!). I made sure to get quality components (esp MB/power supplies). One of the servers is also my desktop. I'm guessing they were about $800 each. Having the 3 node cluster is nice. When we had a forest fire threaten our town and we were ordered to evacuate I just grabbed one box and all my data was already replicated to it. When we returned home I plugged it back in and it re-synched back up.
Maintenance has been a non-issue. I haven't had any components die except the occasional HD. Power costs are the main thing. Estimate about $500/yr at $0.10/KWh
> PF + CARP + pfsync + OpenBGPD for routing
I use a local indy ISP that gives me a bunch of static IPs and lets me route a /29 with BGP. I know them fairly well, and get a sweet deal. Doesn't cost more than a regular consumer connect, though it is slower. I've used OpenBSD since the early 2.x days, and find it very easy to administer. The release and documentation quality are second to none, and I've found the community to be very helpful as long as you've tried to help yourself first.
> Unbound + NSD + Bind for DNS I keep my Bind server vlanned off and serve everything out thru unbound/nsd. Both of those programs are very easy to set up, the real beast being bind. I know there are better alternatives out there, but I know Bind well and have lots of custom config I don't want to throw away.
> SSH/OpenBSD ipsec/apache Guacamole for roaming and permanent site-to-site VPN (pcengines ALIX hosted at my inlaws in Japan)
If I were to pick one outstanding program on this list it'd probably be Guacamole. Pure HTML5 rdp/vnc/ssh/telnet/etc client that is seriously amazing. I've set it up at a half dozen places now, and it's never so much as hiccuped. OpenBSD IPSEC is VERY easy to set up, if you've had nightmare experiences with other packages!
> Apache + Lets Encrypt + awstats + relayd for serving web pages and analysis
Apache is the old standard, and awstats is cool for keeping tabs on what is going on in the logs (geoip as well). Lets encrypt was amazingly easy. I'm using certbot and set it up in under an hour. I'm forcing SSL on all my web services now. Relayd is another "so simple and it just works" package from OpenBSD. I use it as a front-end load balancer.
> ZoneMinder for video monitoring. Tied into legacy security system for automation
On Debian (my Linux distro of choice), this was simple to set up. Perl scripts to integrate it into my DSC security system. Auto arm/disarm camera recording and relay light control required the IT serial integration board and programmers manual for the system
> Postgres for database work. Some mysql/redis
I've been using postgres forever, so setup and use are second nature. An absolutely incredible piece of software engineering.
> NetDisco + Nagios + NagVis + NFSen + MRTG + Smokeping + PNP4Nagios + NUT + Splunk + Racktables for monitoring. All configs are dynamically generated from netdisco db
This is another stack I've set up at many locations (including businesses). They are a real timesink to integrate together. I have MANY custom scripts to make the config generation from netdisco work properly, but once setup you have total insight into every aspect of your network (and I forgot to list RANCID!). Netdisco/NFSen on their own are still a killer combination, and work as well or better than packages that cost tens of thousands of dollars. I'm happy to help any one trying to set these up if you PM me.
> OpenSMTPD + Citadel (webcit) for email delivery and webmail
Citadel is maybe the weakest thing I have in my stack. I'm looking at the other webmail solutions in this thread carefully
> Minetest server for kids. We use this tons as a family, and the kids spend lots of time modding. TW2002 server. TShock server.
When your kids are asking to learn LUA, you know something is working!
> OpenELEC for diskless netboot KODI machines around the house
Amazing and easy to set up if you already have your own DHCP server you can modify. Just need tftp and nfs after that. Using OLD desktop PCs for this works great. I'm using cast off dell gx290s
> Samba4 Domain controller + NFS for sharing files in different applications
I've been using Samba4 since pre-alpha (TP series) when you had to run your own LDAP server. Things are so easy now its hard to overstate. Using Bind makes it a bit trickier since I need to add some magic entries, but if you use the builtin its a single python script between you and a full SSO AD domain.
> SVN for source control and Config diffs for all servers/tools/network devices
I found SVN config to be a bit of a head-scratcher. I think this is another one where other tools are probably better nowadays. I'm looking at some of the other things people are suggesting.
> Asterisk via FreePBX / NCID for all phone/CallerID services, including remote handsets at VPN locations.
Another timesink. PBXs are hard to configure, and I'd move to another system if there was something less esoteric.
As to why I do it? I find it satisfying to learn how things work, like the idea that I'm master of my own destiny and know how my data is being used
Any specific questions, let me know!