[DrPhish]

I self host everything I can at home, so this list might be a bit too exhaustive, but there wasn't any scope on this ask HN, so...

Overall hardware platform:

4 pcengines alix boxes for openbsd router/firewall appliances

3 supermicro Opteron servers with KVM/corosync/Sheepdog/csync2 for hosting all VMs

Software:

PF + CARP + pfsync + OpenBGPD for routing

Unbound + NSD + Bind for DNS

SSH/OpenBSD ipsec/apache Guacamole for roaming and permanent site-to-site VPN (pcengines ALIX hosted at my inlaws in Japan)

Apache + Lets Encrypt + awstats + relayd for serving web pages and analysis

ZoneMinder for video monitoring. Tied into legacy security system for automation

Postgres for database work. Some mysql/redis

NetDisco + Nagios + NagVis + NFSen + MRTG + Smokeping + PNP4Nagios + NUT + Splunk + Racktables for monitoring. All configs are dynamically generated from netdisco db

OpenSMTPD + Citadel (webcit) for email delivery and webmail

Minetest server for kids. We use this tons as a family, and the kids spend lots of time modding. TW2002 server. TShock server.

OpenELEC for diskless netboot KODI machines around the house

Samba4 Domain controller + NFS for sharing files in different applications

SVN for source control and Config diffs for all servers/tools/network devices

Asterisk via FreePBX / NCID for all phone/CallerID services, including remote handsets at VPN locations

And that's just the ones that I really enjoy using off the top of my head. I hope to find lots more things to try in this thread. Metabase already looks like an awesome candidate!

[zweicoder]

Wow that is a long and awesome list! If you don't mind me asking, how much did it cost you to setup / maintain it? Also what is the main reason driving you to self host so much stuff?

[DrPhish]

It was cheap to set up, all the cost is in time to learn to implement in a secure and performant fashion

Hardware actually cost money. Here's a breakdown:

> 4 pcengines alix boxes for openbsd router/firewall appliances

These were around $120 each with 4GB flash storage at the time. They're half that now. Low power, no cooling required, x86, 3 ethernet. You could buy APU2 now for more power

> 3 supermicro Opteron servers with KVM/corosync/Sheepdog/csync2 for hosting all VMs

I used cheap cases, eBay MB/CPU/RAM, tiered storage (green/black/ssd) to keep costs down and infiniband for 10gbit interconnects ($15/card on ebay!). I made sure to get quality components (esp MB/power supplies). One of the servers is also my desktop. I'm guessing they were about $800 each. Having the 3 node cluster is nice. When we had a forest fire threaten our town and we were ordered to evacuate I just grabbed one box and all my data was already replicated to it. When we returned home I plugged it back in and it re-synched back up.

Maintenance has been a non-issue. I haven't had any components die except the occasional HD. Power costs are the main thing. Estimate about $500/yr at $0.10/KWh

> PF + CARP + pfsync + OpenBGPD for routing

I use a local indy ISP that gives me a bunch of static IPs and lets me route a /29 with BGP. I know them fairly well, and get a sweet deal. Doesn't cost more than a regular consumer connect, though it is slower. I've used OpenBSD since the early 2.x days, and find it very easy to administer. The release and documentation quality are second to none, and I've found the community to be very helpful as long as you've tried to help yourself first.

> Unbound + NSD + Bind for DNS I keep my Bind server vlanned off and serve everything out thru unbound/nsd. Both of those programs are very easy to set up, the real beast being bind. I know there are better alternatives out there, but I know Bind well and have lots of custom config I don't want to throw away.

> SSH/OpenBSD ipsec/apache Guacamole for roaming and permanent site-to-site VPN (pcengines ALIX hosted at my inlaws in Japan)

If I were to pick one outstanding program on this list it'd probably be Guacamole. Pure HTML5 rdp/vnc/ssh/telnet/etc client that is seriously amazing. I've set it up at a half dozen places now, and it's never so much as hiccuped. OpenBSD IPSEC is VERY easy to set up, if you've had nightmare experiences with other packages!

> Apache + Lets Encrypt + awstats + relayd for serving web pages and analysis

Apache is the old standard, and awstats is cool for keeping tabs on what is going on in the logs (geoip as well). Lets encrypt was amazingly easy. I'm using certbot and set it up in under an hour. I'm forcing SSL on all my web services now. Relayd is another "so simple and it just works" package from OpenBSD. I use it as a front-end load balancer.

> ZoneMinder for video monitoring. Tied into legacy security system for automation

On Debian (my Linux distro of choice), this was simple to set up. Perl scripts to integrate it into my DSC security system. Auto arm/disarm camera recording and relay light control required the IT serial integration board and programmers manual for the system

> Postgres for database work. Some mysql/redis

I've been using postgres forever, so setup and use are second nature. An absolutely incredible piece of software engineering.

> NetDisco + Nagios + NagVis + NFSen + MRTG + Smokeping + PNP4Nagios + NUT + Splunk + Racktables for monitoring. All configs are dynamically generated from netdisco db

This is another stack I've set up at many locations (including businesses). They are a real timesink to integrate together. I have MANY custom scripts to make the config generation from netdisco work properly, but once setup you have total insight into every aspect of your network (and I forgot to list RANCID!). Netdisco/NFSen on their own are still a killer combination, and work as well or better than packages that cost tens of thousands of dollars. I'm happy to help any one trying to set these up if you PM me.

> OpenSMTPD + Citadel (webcit) for email delivery and webmail

Citadel is maybe the weakest thing I have in my stack. I'm looking at the other webmail solutions in this thread carefully

> Minetest server for kids. We use this tons as a family, and the kids spend lots of time modding. TW2002 server. TShock server.

When your kids are asking to learn LUA, you know something is working!

> OpenELEC for diskless netboot KODI machines around the house

Amazing and easy to set up if you already have your own DHCP server you can modify. Just need tftp and nfs after that. Using OLD desktop PCs for this works great. I'm using cast off dell gx290s

> Samba4 Domain controller + NFS for sharing files in different applications

I've been using Samba4 since pre-alpha (TP series) when you had to run your own LDAP server. Things are so easy now its hard to overstate. Using Bind makes it a bit trickier since I need to add some magic entries, but if you use the builtin its a single python script between you and a full SSO AD domain.

> SVN for source control and Config diffs for all servers/tools/network devices

I found SVN config to be a bit of a head-scratcher. I think this is another one where other tools are probably better nowadays. I'm looking at some of the other things people are suggesting.

> Asterisk via FreePBX / NCID for all phone/CallerID services, including remote handsets at VPN locations.

Another timesink. PBXs are hard to configure, and I'd move to another system if there was something less esoteric.

As to why I do it? I find it satisfying to learn how things work, like the idea that I'm master of my own destiny and know how my data is being used

Any specific questions, let me know!

[ge96]

Not original asker

How do you secure your publicly accessible IP at home from the web. I see you use static IP's I guess it's no different than renting a sever? You probably have separate networks (local home vs. serving)

[DrPhish]

Looking at Netdisco I have 12 VLANs. ISP1, ISP2 (from Dad's house across the valley), OSPF backbone, DMZ, LAN, Guest, Cameras, Servers, Japan, Kids, Management and Test. Everything passes thru PF and only exposes services as required. The Guest VLAN, for example, has no way to get to the rest of the VLANs. Camera VLAN can only talk to the DVR software, etc.

The important thing with securing my public IPs (as with securing anything) is to understand the surface area and minimize risks. With the number of services I'm exposing, I have to be careful. First thing is to keep all OSs up to date. OpenBSD every 6 months and Debian on an ongoing basis.

Next, whenever possible use single-purpose proxies that have been well audited. In my case OpenSMTPD, Unbound and NSD protect the always popular mail and dns servers from attack.

Keeping complicated things off the internet is important. Big CMS packages are constantly under attack. I only run either static pages on my webserver, or very carefully audited custom PHP pages (written with an attacker's eye to exploit injection)

When I expose something more complicated like Guacamole or Citadel that might end up with a known hole in the login screen, I put it behind both SSL and an HTTP simple auth login prompt. It's ancient, and so well tested that it is unlikely to end up with an exploit against it. Or at least less likely than the app its protecting. I've actually been toying around with doing 2 factor auth on these services with a dynamically generated simple auth sent via SMS bridge or IFTTT...

On my internal network, I have my family running linux wherever possible, and keep Java and Flash off the windows PCs. Everyone runs with a nonprivledged account and the OS/software packages/PDF readers are all kept up to date. I think the wife and kid's PCs are probably the most likely things to be compromised on the network, so I keep a close eye on them.

I also take the reactive approach of log alerting, change monitoring and general diligence to the state of my network. If you know what has changed in the last 24 hours it tends to be a small enough dataset to eyeball, and will tend to tip you off when things are wonky. I also have enough logging and data sources (syslog, configs diffs, apache logs, nfsen data, mrtg data) that it would be exceptionally difficult for an attacker to wipe out all trace of their presence.

I do not know of a successful attack against my network, tho I'm not naive enough to think that it hasn't happened. I just haven't realized it if it has.

Its a great hobby if you like this stuff!

[ge96]

Wow thanks a lot for the details this is not my field haha regarding the tech you mentioned in the beginning.

Curious what your family thinks having to run Linux. It's my primary os myself with i3-wm. People usually complain "install apps command line" ui, etc .. whatever Ubuntu being as supported as it is that UI is so slow/bloated IMO.

I have Windows as well but use Virtual box and Linux to access the web if it's not just YouTube/Gmail/regular pages with adblock/uorigin running.

I don't even know how many ports there are on computers, I mean in my experience I've seen up to port 10000, I think you have to enable them? I usually use 21/22/80/443 though I've seen 3000 (websocket), 4200 (angular), 246 or 286 (windows RDP) I don't know...

It's a lot to keep track of web security. OSWAP

Anyway thanks for your time rambled on my end.

[jlgaddis]

Take a look at /etc/services on your Linux box for some hints or Google something like "IANA well-known ports" for a list of "registered" ones.

[Vendan]

Windows rdp is 3389, but really, anything can go on any port up to 65535. The common ports like 80 and 443 are just there as "it's common to use these for services, so you don't have to put in http://google.com:80". And there's no real "enabling" them, it's just a question of if your firewall allows it, and if you have something actually listening.

Also, websocket is just a protocol upgrade slapped on http/https, so it normally goes over 80/443. If someone's running it on another port, it's probably cause the server they have on their normal http/https doesn't work well with websockets.

[DrPhish]

I have a long history of having my family use Linux. I'm a Debian guy myself, so I've always either used Ubuntu or basic Debian. I run dwm on any machines that I put X on (very few, honestly).

I started my Dad on Ubuntu around 2008, and he hasn't used anything else since. Shortly thereafter my Mom, then my Sisters. Most recently my Step-mom, and Grandparents have gone to Ubuntu.

When I got some castoff laptops from work, I turned them into Sugar notebooks for my kids, and once that became too limiting for them I helped them install mainline Debian. Some of them run Gnome and my oldest runs fvwm.

My wife still runs Windows because of inertia, more than anything TBH.

In my experience, I've only had 1 peripheral that someone has bought that was totally unusable due to drivers (scanner), and the only programs my relatives have asked for that weren't available were my kids wanting to play Roblox, which I didn't want them playing anyways. Thanks to Minetest and buying most Humble Indie bundles I actually have a pretty good library of Linux games for them to play, so there hasn't been much bellyaching from them. Well, that and the Windows gaming/home theatre PC.

I've gone to a 2 strikes and you're out policy on Windows installs. If I have to re-install it for you more than one, and you're either getting Linux or finding someone else to fix your computer. My Grandparents got caught on this policy, but my stepmom actually asked for "That system that Dad has that doesn't get viruses". Happy ever since.

Most people that aren't highly technical tend be served by Firefox/Thunderbird/LibreOffice for 99.99% of their needs. It's mostly Facebook these days TBH.

It makes support dead simple. no-ip.com and SSH let me fix almost anything remotely, and no one has every gotten pwned that I know of.

Other people have answered the port question, but I'll try to go at a slightly lower level. Each open port will have a program running on the host that has opened a listening socket on that port. Netstat can help you find out what is listening, and on what interface/port. As a rule, only root can open ports under 1024, and any well written server will drop all non-required privs. You can check with the ps command. This is somewhat enforced on some OSs, eg OpenBSD with pledge.

From a security standpoint you should verify that you know and understand every server listening on a socket, which interfaces they are bound to (netstat asterisk means all interfaces), and whether they are exposed to the internet via direct interface/proxy/port forward/etc.

One trick to secure services that need remote access: if the service is only for technical users, you can give them each an account with no interactive shell, and then they can ssh port-forward to the port they want to access. Eg. you can make 3389 (RDP) only listen on the local LAN or the loopback device, ssh to your router with port forwarding local port 3399 (or whatever), to remote IP:3389 and point your local rdp client to localhost:3399. Great for ad-hoc limited VPN type connections. That way you only have to be aware of SSH remote holes, and not the more-likely RDP server.

Hopefully if any of this is incorrect, someone will correct me

[atmosx]

Sounds like a full time job to me. I mean, I'm getting paid, to monitor less than this...

[purplezooey]

I bet you're not in California. The $.27/KWh we pay here would be a showstopper :)

[fragmede]

Not OP, but canceling Youtube Red, HBO Go/Now, and Hulu easily covers the cost of electricity on beefy server. Just depends on your priorities. I consider $10/month (what I pay for my desktop class i7 machine, not GP's setup) to have a place to call home on the Internet (that isn't in the cloud/on hard drive island/somebody else's computer) to be well worth it, and if you're using smaller hardware (eg Pizerow) it costs less in electricity than a cup of coffee in a month.

[DrPhish]

Haha, your're right about that!

Even in BC with $0.10/KWh it still adds up, but I write it down to education, privacy and entertainment : D