[caust1c]

> I have systems that keep all sorts of data, including every keystroke I type, every step I take and what my computer screen looks like every minute

Yikes? He's smart, so I'm sure he's protected it adequately, but auditing the surface area of this much software seems insane.

[kodah]

I run my own personal infrastructure. Most of what it takes is to research secure setups from the beginning. You don't have other users so upgrades aren't painful. Frankly what I find most difficult is dealing with aging hardware, but this dude probably had the money to buy everything new.

[justinlloyd]

Not really. I keep even more than that. And at a finer grained resolution. And have done so for almost two decades. It's all put on to a write-only-by-the-capturing-device/read-only-by-other-device secured storage system.

[chrisweekly]

Have you written anything public about your setup (and/or rationale)?

[justinlloyd]

You are the third person this week to ask me that. I answer reluctantly.

The rationale is, because I want to.

The system has evolved over the years, current configuration is: Several 1080p SONY cameras with hacked firmware that stream video to a capture device. An older view of the camera rig that has since evolved again, is here: https://youtu.be/dGRDB1vVxyY

Some 4K webcams connected over USB that I don't stream. I capture one full frame every X milliseconds.

Two Kinects set to be out of phase capturing the entirety of the office as a depth map.

Two Rode shotgun microphones capturing audio and feeding it in to a Focusrite box.

Custom built USB "keyboard" with a few arcade buttons that permit "pause/unpause", "forget a little bit" and "forget five minutes."

Two LED lights to indicate recording status for both myself and anybody walking in the office.

Timesnapper on Windows, and a little custom C++ capture program for macOS and Linux that takes a snapshot of my desktop every X seconds.

All that data gets stuffed on to a secured drive on a file server. The data goes back more than a decade. Nobody has access to that data but me.

I use an NVidia Jetson to analyze everything: the desktop images, build up a map of applications, analyze people in the room, identify who they are, what clothes they are wearing, identification of activity, OCR of images, transcription of spoken word to text, identification of websites, identification of music playing, "oh hey, he's listening to the following artist, let me pull that artist's social feed and put it on the ambient screen in the hallway", which is kinda creepy when the software identifies my own music https://soundcloud.com/justinrlloyd and then stalks me and puts up my own social feeds on the household ambient screen. I also have the Jetson watching the front door via the Ubiquiti doorbell camera and can switch on the TV in my office if someone comes to the front door so I can see who it is, and also will notify me that a package is on the doorstep ready to be brought inside via the second high viewpoint door camera performing a "what changed in this scene, is that a package? That looks like a package. Package! ZOMG! Package! Package!!!" That algorithm has one job and it does it really well. Like a hunting dog staring at squirrels.

Lots of this stuff is readily available as ML models, for the most part I just strung them together with simple scripts to move data around.

I have a "virtual assistant" that I wrote, using NLP and key phrases with a speech recognition model that understands specific commands and some free form speech, an early prototype of my virtual assistant is here: https://youtu.be/uhl8wN7Uvv8 and I state for the record that it has gotten far better in the intervening years. And then a text to speech model when absolutely necessary to give me voice prompts.

This virtual assistant can control cameras, e.g. tally lights, zoom and focus, recognize the fact I am holding a receipt from a grocery store, or a book, and take a high resolution picture and tag it with meta data.

I keep a near real-time backup of my computers, and that data goes back probably three decades, any time I retire a machine I take a full drive dump and store that.

Out of office, I take a snapshot of my desktop on the laptop (Microsoft Surface or Macbook Pro), which is then automatically copied to the server when I return to the office. I built my own Sensecam-like device using a J2ME device almost two decades ago, but have since moved to using an Autographer for life logging.

[techdragon]

... WOW

This is just amazing. You've basically built your own JARVIS, and I really want to know how much you would charge to replicate this ? ... its almost impossible to turn something so bespoke into a purchasable product, so I'll never be able to walk into a store and walk out with this entire thing, but its like art, you can pay the artist to make you one thats pretty similar since they know how to do it... I suppose that does open a more interesting question for the wider audience.

How much effort does it take to maintain this?

[justinlloyd]

I have never considered product-izing it. I have no interest in doing so even as a one-off "art piece." And I don't know your net worth, but I doubt there is anybody on this green Earth that I would be willing to work with that could offer up a dollar amount that I'd be interested in. Besides, I'm currently building a task list, https://github.com/justinlloyd/retro-chores/ so have no time for an adventure - nasty, uncomfortable things that make you late for dinner.

With regard to maintenance, I don't know, I haven't touched it in months. Maybe every once in a while I get some weird, catastrophic failure of "I can no longer talk to the ambient screen, but I am still sending commands" and then I have to spend a half-day debugging what the issue is only to realize that the CNC in the workshop stole the IP address because I forgot to lock that IP to the device's MAC when I switched out the old screen for a new screen.

[timdavila]

Curious why you do this and what you feel like it adds to your life?

[justinlloyd]

Why does it have to add anything to my life? Why be the third person to climb a mountain?

I hear tell of a 19th century English gentleman who manually recorded the level of rain water and the barometric pressure in his garden, a dozen times a day, including the wee hours of the morning, waking up several times a night as though afflicted with the worst UTI possible, through all weathers, all seasons, all ills, all trials and travails, any and all privations that life could send his way, and did so for 40+ years. Obssessively and compulsively some would say. Or we may say diligently if being generous. I am sure that at some point someone asked "why do you do this? what do you feel it adds to your life?"

Because I can. And because I want to.

[harperlee]

Feels weird that you have to defend this take on hobbies on a forum called “Hacker News”.

My father is building a small sailboat model (the Santísima Trinidad), and if someone asked “What does that add to his life?” I personally would pity that person, for its lack of empathy and views on a successful life.

[justinlloyd]

People say "I am curious to know..." which is immediately followed by a request that I justify my reason for existing or doing what I do. The question says more about them than it does about me.

In questions of code and of life, we should ask "why about code" and "what about people."

           __
     ^ ^\ /  >
   < ' ' )  )
    (__( ) /

  There is a breed of fox out there, with big bushy tails.
  In which the wind pushes them backward, like big billowing sails.
  They can seen where they've been, but not see where they'll be.
  There are those that sail-walk North
  There are those that sail-walk South
  And they think they control the way that they're going for
  But really they lack all form of direction, no less and no more
  When one day came a fox, who was lacking a big bushy tail
  It was all scrawny and thin, and made the lousiest sail
  "Where do you come from? From where do you hail?"
  Cried a crowd of bushy foxes, fighting the winds to stand still
  As our new little fox, with a scrawny thin tail
  Ambled slowly past them not knowing the question
  He didn't stop for a minute, not even to rest
  Just kept on walking, simply walked into the West
  "What why did you do that?" asked a North-North quick walker. "Don't you know that walking Northwards is best?"
  "Best?" shouted a Southward slow stalker "Everyone knows that is as wrong as can be."
  "But if you walk North, or if you walk South, we know that West is not the best place to see."
  "Why do you walk West? Why do you do that?" puzzled both North-South walkers.
  "I don't know, I just like walking West. I came out of the East, just over yonder, and I'm heading out West, to take a quick gander."
  "But what's over yonder? From where you came. And what's over yander? Is it the same?" quizzed the South-North slow and quick walkers.
  "I don't know. I'll go take a look. I come back when I'm done, I'll walk back to my East."
  "But what if you keep walking? What if you run out of ground?"
  "Then I'll just keep walking, on into the West, my feet out in front of me, until I come right around."

 
  *   /^ ^
  |  ( ' ' >
   \_( )__)

P.S. You and I will probably be the only two people in the world that will read this.