Hacker News new | ask | show | jobs
by corodra 2501 days ago
Thanks for the extra info. Okay so, $20b in equipment for 300mil cameras... eh yea. I'll say that's maybe about right. After I wrote that, I was estimating $35b-$40b to deploy (fuzzy head math), including labor and infrastructure. But they have a lot of in-country manufacturers that can supply a lot of what they need. Which, I may give them shit on quality, but the indoor stuff, mid-range, is really good. Outdoor, no. We started with replacing a lot of dead outdoor chinese camera deploys. They really suck for freezing temps and high heat. But indoor, it's really hard to defend a non-chinese camera's price compared to, let's say Dahua (I'm more familiar with them). But they're whitelabeled by a firm I could never nail down in Shenzhen. But, whatever. I left the industry almost 2 years ago.

But you mention facial rec. So, again, I left 2 years ago. Most of the systems I knew tied to VMSs ended up always being separate boxes that handled the actual facial rec. I mean, some did inline. There's a thing called edge analytics (industry keyword) to cameras. Meaning the processor on the camera can do motion detection, facial, object track, etc, then pump that info out to a VMS that can handle that specific type of data (nightmare compatibility issues in this industry, ONVIF works until it doesn't). And, basic stuff works great. Motion and object. Yea. Facial... it "works". By works... eh, 70% success rate was the best I ever saw. Which means lots of false positives and lots of never caught. Enough to where staff tells me "This shit is useless". That's if the VMS can handle said data, because it wasn't part of ONVIF 2 when I was around. Oh and, it lags the hell out of the camera and recording if on the same server. Enough to where you have missing video. Which is bad. Real bad.

The way around, separate servers and software with real world ~90% success. Good enough to where filtering through the false positives by someone is pretty easy. This... SUPER EXPENSIVE. Not little expensive. Really bloody fucking expensive to deploy in hardware alone. Petabytes of data being crunched every minute. I'd probably get bjs from both Nvidia and Intel for selling that job. Plus, the bigger the database of bad faces, the longer the crunching takes. I'm trying to remember what my one customer needed... I think it was a dual E5-2695 v4 (maybe it was v3) along with whatever the high end Quadro at the time wise. I think it was the P4000. I'm looking at parts at that time frame, but I'm not 100%. Either way, it was a $10k+ box to just crunch faces (and was above spec since I built it) on about 20 high quality floor cameras (10mp+ wall mounted cameras that had clear face shots). That box was running at around 50%-60% capacity on average days and 80%-90% at peak hours. I think the memory was around 64gb and would max at 50 or so at peak times. Hardware alone would be killer for this china deploy, even if they have inhouse facial rec for "free". And the power draw... God have mercy on that powergrid.

But there are sooooooo, soooooo, soooo many problems with such a deployment. To act like it's a "one system" 300mil camera deploy. No. It's a pain in the ass to do a 1,000 camera deploy for a casino. A real pain in the ass. Different VMSs do these types of deployments differently. But generally what happens is this, you have some master servers that control slave servers. The master servers are what staff log into. "Even if a slave server goes down, the staff can see all other servers just fine." Notice the quotes. You see those quotes. HA! That's brochure talk. I've dealt with most of the big VMS players minus Milestone but ONSSI 5 was built on Milestone so... whatever. God... that was such a cluster fuck. 5.0 was such a mess... Anyways, if a server goes down, it can hiccup the entire federated system. 9 times out of 10, not a big deal. But you can notice it. But the more machines, means that 1 out of 10, the master server can seize up. Which then can crash the slave servers. That's not good. Because they all have to reboot AFTER the master reboots. Then if you have separate archiving servers, these need to sync back up depending on how the redundancy is setup. Lots of downtime.

Look, I can share a lot of stories. But to save time and boring stories of me being on the line with tech support where the lvl 2 and 3 guys go "Yea, I know, it's been an on going issue the past few years. Here's a workaround to make the customer happy. Just feed them xyz bullshit.", here's how to best explain a large scale camera deployment.

Think of a clock. Lots and lots of gears, moving parts. But instead of metal, the gears are made of glass. Some of the gears are thicker and some are really thin. Some are tempered, others you wonder how the hell they didn't shatter from looking at it too hard. At first, it can work great. Maybe a gear or two breaks and you fix them and it works on the initial deploy. But over time, the gears break or wear down in weird ways. Ways where the clock hands still move. It may not be the correct time, but the hands are still moving. Which is what counts at the end of the paycheck, cough, I mean day. So the system "still works". Kind of. Until the gears are so fucked from neglect, the entire thing should just be trashed and you buy another whole clock (not pieces) with glass gears. This is the security industry in a nutshell. Unless you get the expensive stuff from Axis, MOOG or anything from Israel. I'll throw Bosch in there too...eh. There are some companies in Israel that shoot their cameras to prove they're tough. In front of you.

So, yea. Such a wide deployment, I still call bullshit. Especially with maintenance and infrastructure costs. Keeping that system up and running, both electricity, camera replacement, cleaning the housing lens (seriously, smog fucks up those housing lenses hardcore), and we're not even talking about training staff to monitor the stuff yet. It's so expensive with extremely little value. Yes, they're surveillance crazy. But spending so much money on capturing how many "dangerous" people? Really? The guys at Antwerp that get robbed periodically spend less hunting down people who steal $100+ mil in diamonds per person.

And hell, maybe they did deploy this in the way they're talking about. An honest communist...bah. Probably why they're in so much debt. China has been debt heavy for many years now. A project like this can really cripple whoever implements it. It's that long term cost that no one thinks about and seriously hurts a lot of companies. Plus, cameras seriously don't last longer than 10 years. It's a 24/7 "cheap" device. So, they have to replace those cameras every 10 years, EASY. Max, 10 years. If it's a good temperate area. Any sustained lows below 40F or above 95F causes a lot of damage to these things if they're not designed for it (meaning more $$$). You might spend $20b once for parts, but you'll be spending $0.5b to $1b to maintain it, every year on average.

Oh plus, last story. Seattle deployed a city wide mesh network with cameras a number of years ago. It was shut down, I think a week (maybe 2 weeks) into it being turned on. Why? One of the city councilmen (or some other city official of some important status) was caught with a hooker on that camera system and the news spread through the police department, fast. Because. You know. Video. Now, we have separate bodies of power in the country that allow this to happen. The cops spreading the video I mean. Well, hookers too, but that's not the point I'm making. China doesn't really look favorable on the idea of separate bodies of power. But, a true 100% coverage... it's bad for chinese politicians too. Especially ones with wandering peckers and loyalties.

1 comments

Good stuff, that was entertaining and educational. I see, hearing about the many potential problems with such a big surveillance network puts things in perspective. It helped to see through the marketing speak in the mass media, to get a sense of the real-life challenges and current state of technology.

Lots of relevant points, like the (un)reliability of the cameras; difficulty of integrating into a single system and keeping the servers running; processing and storing the data; on-going energy and maintenance costs; dealing with false positives; and (un)reliability of the humans running the whole thing.

So I suppose "Skynet" has a few more decades to be even close to covering the whole country of China. And machine learning is still far from achieving "intelligence" in any meaningful sense of the term.

I can't help but think, though, that given a long enough timeline (say, in the next century) these technologies do seem to have a possibility of merging into a semisentient technopocalypse.