Maintain access 20 year old fitness data? What for?
I'm fairly far along on the data hoarding spectrum. This extends, for example, to over 200K photos from my life, categorized and indexed. It's trivial to look at "what happened on this day 20 years ago" or at least on the nearest date to that day that had pictures taken.
And here's the thing: As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era.
And that's photos. Fitness data? Do I really want to see how I declined over the last 20 years? I see that already from the average speed display on my bike computer. Fitness data is the ultimate "looking forward" thing. But maybe that's my age; it may look different from a younger perspective.
Agree with others that just use a platform that lets you export open-ish files. .gpx doesn't do heart rate, but .fit does, and gpsbabel can translate that to another format as needed.
I think this is a resolution argument. Data should be curated carefully and reduced in resolution as it gets older. If you’re never going to look at it, it’s probably not worth keeping.
As per photos you don’t need 11 pictures of those muffins you baked in 2003. One will probably suffice, if you even care about them 18 years later that is.
The same goes for health data. But overall trends over 20 years are really useful as you get older. Expected decline is inevitable but unexpected decline may be indicated earlier and result in some preventative action to improve your quality of life. There are a lot of long term metrics appearing in Apple health like walking stability on that front.
On photos, I’m not sure what I’d do with 200,000 photos. I have 7,623 which span 4 generations and 125 years at the moment. I spend a lot of time curating these photos, adding metadata, editing and pruning garbage. They are all managed with Apple’s native Photos app which is good enough for the job (non destructive, sqlite DB underneath it, easy to back up)
Out of interest, what is the argument for reducing resolution other than space concerns? Fitness data is not really space-intensive. I recently ran a marathon and the associated GPX file is less than a megabyte; the FIT file is about 180KB. I don't really think even 30 years of regular running data would come to much. I appreciate cycling data will take more space but not that much more.
My point is, it seems like once you have put in place the habits and technological solutions necessary to store fitness data, it actually takes more effort to reduce the resolution of older data. I'd rather have data and not want it than want it and not have it, but then I am an almost pathological hoarder so maybe it's just me.
It’s mostly the ability to be able to locate it afterwards i.e. to improve the signal to noise ratio. I track hikes I do with a GPS and keep the GPX files afterwards but generally don’t refer to them unless uploading curated copies to OS Maps here in the UK for other people to use. So after a few years I nuke them. As for health data, it’s quite large. My apple health data is around 100 Meg compressed now and a lot larger, in the order of 1Gb, decompressed.
I am not a hoarder as a counterpoint. I am always looking at ways to reduce what I have and remain focused. I lived with a hoarder in a house full of trash and don’t want to be that person to someone else. He also had 4TB of photos and videos that needed to be dealt with.
I think at the end of the day my life can probably be compressed to a few images and a few paragraphs…
>On photos, I’m not sure what I’d do with 200,000 photos. I have 7,623 which span 4 generations and 125 years at the moment.
Some of them might be historically interesting enough to preserve on Wikimedia Commons, for example if they depict places that look very different today, or stuff that you don't see in everyday life anymore.
That’s quite possible. I have around 150 historical photos i.e. over 100 years old which depict family life in Switzerland in the 1800s. I will look into that. Thanks for the suggestion.
4TB is a huge amount of data. And it’s costly to keep data spinning. 4TB storage an in apple device is going to set you back a lot of money for example. Not everyone has a NAS or the ability to run one and even more importantly to back it up effectively. It becomes a large monkey on your back and a cost and not an advantage.
Looking at my device storage, I have 91Gb online for 5 people. That’s everything we have ever done. Because we care enough to look after it and curate it.
Fair enough. My older is just under 10 years old, so that kind of sets the "current era" event horizon. 20 years ago is relatives who aren't alive any more, parties full of people I can't even put a name to any more, trips of which the memories have faded, stuff like that.
I have approx. 220k photos, of which 210k are in camera raw, 10k are ‘processed’, which typically means exposure corrected, cropped, etc. They are stored in two parallel directories (raw and processed), each of which is structured into years which are further subdivided into ‘shoot’ folders – one for each distinct upload from the camera. I use Adobe Bridge to do the uploads and to change the raw files names to support my indexing approach. The year folder names are simple (e.g. ‘2021’) and the shoot folders have names like ‘2021-010 garden birds young’ (where 010 is the sequence # of the shoot in the year). The folders thus automatically list themselves in chronological order when sorted alphabetically. A processed image might have a name like ‘20210523_010_my-IRL-name-here_0019.psd’ (so that I can immediately find the raw or processed shoot folder given a file name). I insert my name so that if I a send an image to someone I am identified as a creator (image metadata contains additional contact / copyright info). I’ll often extend the automatically generated file name to include the photo subject as I process specific images.
My indexing system relies on my memory and the fact that the relatively small number of processed pictures are indicative of the contents of the wider raw set – so I can browse the relatively small number of processed shots (using Bridge) if I am looking for something thematically. I usually manage to cull obvious fails from the raw set when I do an upload before doing any processing. Storage costs being what they are I have not yet had to further cull raw or processed images to make room on my hard-drives – currently 1.7Tb of images.
I have left it too late to add meaningful tags to this collection, and because I am the only librarian, I don’t have a need to help other users find particular images.
I'd recommend using one of several mobile phone apps to sync your phone's camera/DCIM directory with a computer/NAS at home, so you have your originals.
I wouldn't recommend syncing from, say, your Google Photos library: even in "original quality," your files may be mucked with (it depends on your platform).
Once you've got them local, there are several apps to share photos via local webserver. I wrote PhotoStructure to deduplicate and manage my 500k+ photos and videos from the past 20+ years.
> Maintain access 20 year old fitness data? What for?
Why not? I have about 12 years worth because that's when I started tracking, no reason to ever delete it. Saving the 0.001% (actual value, I just checked) on disk space isn't a motivation.
It's fun to review old data. Occasionally useful, even. Mostly for fun.
> And here's the thing: As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era.
Wait another 30 years and those now-50 year old photos will be treasure.
I don't look much at my photos from 20 years ago, that's pretty recent so I remember it well. But photos from 50 years ago are quite the treasure. Even older ones (like when grandparents were toddlers) are invaluable. Wish there were tons more of those.
But yes that's photos. Fitness data is certainly less interesting, but still fun and costs approximately zero space to keep.
Hmm. While I don't really need my heart rate or speed for every activity I've done the last years, some of it is actually interesting. Like looking at accumulated time or distance of various activities over the years. I also keep a list of all big events I've been to and my stats from those, which I find cool. But of course, that doesn't have to be exportable.
Just a few days ago I could find out the answer to a Q about when in the year I normally do my first cross country ski session, by looking at the graphs per year. So on average I will be going skiing next weekend 8)
> What app do you use to manage your photo library?
It doesn't seem like a good idea to tie a photo library to any app. Applications come and go, the photo library should last a lifetime (or more if you pass it on). You don't ever want to get caught with your photo data in some proprietary structure when the app that created it is no longer available for whatever reason. My photo library certainly has outlived many applications I've used to process the photos.
After processing I store them in just a directory hierarchy. Top level dirs for each year, event/topic subdirectories under each year. That's it.
That's fine for file storage, but doesn't provide a way to find all photos with dog in them, or photos from 2007 - 2012 with bird and dog, but not cat, in them.
Most apps will let you store your photos how you want them to be stored on the filesystem.
For medical diagnosis when you get older, having that data will actually be useful in the next 20-40 years to diagnosing things that will happen to you then.
Homemade system that is not in any way publishable. It's a folder hierarchy but also viewable as a chronological timeline. The main key is to organize them as you go, and not fall too far behind.
> Homemade system that is not in any way publishable.
I can still access all the photos I took since around 2008, when I first started to nurture an interest in photography as a hobby. I never delete b-rolls unless they're absolute crap (i.e., I triggered with my lens cap on, too dark with no flash---blurry, out-of-focus shots do not count as absolute crap).
I also manage it with a homemade system! This system is closer to my darkest shame than proudest achievement. It kinda reminds me of Destin's pre-Linus-organized system[1] except way smaller and a few scripts more sophisticated (my system has redundancy, ha!). Despite that, I have maintained access despite moving across continents.
In fact, last year, slightly spurred on by the pandemic situation, a bunch of friends asked me if I still have copies of photographs I took years prior. The oldest query was only for 4-5 years ago but I am proud to say that I was able to oblige despite my shameful system.
Kids, don't do this in prod, but homemade systems FTW! :)
I was using GadgetBridge from F-Droid to keep my data and send nothing to the company's server (Xiaomi in this case). But when the battery died, I took a hard look at if I was even using that data, if it was worth it to keep Bluetooth on on my phone, if it was worth trying to go around Xiaomi to get firmware updates and hacks to get bitmap fonts covering wider languages. I decided that the easiest thing to do was toss it and bought a cheap Casio where the battery should last years.
I used gadgetbridge initially but ended up switching to "Notify for Me", which is a lot more hseful. It's written by an individual in Italy but is still closed source and phones home, so I use NetGuard and LineageOS to block all network traffic to the app. It still works fine.
I don't use most functionality, but steps, sleep time, heart rate, and notifications/alarms have all been useful to me.
Thanks for the tip, I just installed it (revoked network permissions + blocked in TrackerControl) and I will check how it works. Lots more options than Gadgetbridge it seems, which is nice.
This is why DICOM is a thing: because hospitals cannot be tied to a vendor for radiology studies. They have to be able to cancel contracts, novate, etc.
Also, IMHO, I think healthcare is where protobuf could really win over JSON: a protobuf comes with the schema and any datatype can ride the message. Everything you need is there.
Is protobuf a heavier lift to get started? Yeah. So? Healthcare is too heavily regulated (and for good reasons) for this to be a significant issue. The barrier to entry is so much higher than the effort to handle protobuf, that it's lost in the wash.
Apple is already offering to export your health data and they use the HL7 FHIR standard.
It's just sad that FHIR has lots of issues as a standard. On one hand it suffers from trying to model the whole world, similar to the problems of the semantic web in the 90s. Even with the cop-out of extensions for things it doesn't support all the objects are very bloated.
A lot of things feel like ad-hoc solutions and there are tons of pitfalls. A primitive value can be there by the primitive value being missing, but an extension being there. FHIR XML cannot be translated to FHIR JSON without access to the resources' definitions, because elements that have a higher maximum cardinality than 1 are serialized as lists in JSON, but the same way in XML. Yet when you read a FHIR XML or JSON you do not know which resource definition applies because there is no way to know the FHIR version. Variant elements that can be different types do not have the element name als the XML element name, but the element name with the type name added to the end after capitalizing the first letter. Profiling the resources is absolutely necessary to ensure interoperability but it's extremely hard to get right.
In the end FHIR brings a whole lot of complexity, while it doesn't bring much of a benefit.
None of those are really valid objections. HL7 FHIR does not even attempt to model the whole world. It only attempts to model the most common healthcare use cases. Most resource elements are optional and you can just ignore those that are irrelevant to your needs. Resource metadata can specify profile conformance, which implies the standard version. There are a variety of tools available now to assist with profiling. The resource definitions are free open source and available to everyone.
If FHIR doesn't bring much benefit then why is it so widely used? If you need to exchange healthcare data with other organizations then what's the alternative? The older HL7 standards weren't any better.
If you think my arguments are not valid, then why don't you reply to them?
FHIR tries to model the whole world in the sense, that addresses are abstracted so much that all kinds of addresses could be represented with it.
Optional elements are one of the biggest problem of FHIR. You cannot just ignore them. If you show a physician just part of the data they might misjudge the situation. If you are writing profiles it is best to set cardinality of not needed elements to 0..0, so nobody gets the bad idea of writing that data or relying on it.
FHIR does not even commit to its own way it works. The system and value elements of Identifier and Coding are also optional. That is just madness, and exactly what I mean, when I say that it's trying to be abstract enough for anything.
To read resource metadata you need to already know which FHIR version the resource is. Even if you can, going through the profile information seems like a huge detour and complexity for such a simple thing. I don't know why there's not simply a different namespace in XML for the different versions. Also widely used JSON formats like OpenAPI include the version.
Profiling is an extreme mess. If you write profiles it's extremely difficult to get everything right. If you need to work with profiled resources it is hard as well, as working on FHIR resources directly is too complicated with extensions and everything. So you need to create Models that represent the profiled resources and need to write converters. You need to do all of this by hand, trying to understand the profiles from a technical perspective and trying to understand what the other human that wrote the profile actually meant.
Why is FHIR used? In my experience there's a push to use international standards and FHIR best fits that. It's not a technology selected by the people that actually have to work with it.
The alternative is to have a schema of your data (for instance with OpenAPI) that exactly matches your data model and is optimized for generating code from it. No need to understand the extreme complexities of FHIR and far less work.
Custom schemas for your data won't work for large scale interoperability across organizations. I've done FHIR profiling before and it's not a serious problem.
Healthcare companies following a schema? Ha. Maybe the closest thing to there being any consistency is companies making sure they can be listed in Apple's Health app.
The closest thing is probably companies assuring they can interact with the transaction formats mandated for electronic transactions under the HIPAA transactions and code sets rule, or EHR standards also mandated/incentivized by the federal government.
Most EHRs and other related healthcare applications now have reasonably correct support of various HL7 interoperability formats. Not perfect, but good enough. Much of this has been driven by federal government regulatory mandates.
Don't file formats like .FIT or .GPX handle this case? I had this problem when migrating nearly 10 years of running data from Nike Running --> Strava. It wasn't perfect but eventually I got "good enough" data after weeks of wrestling with unknown aggregators who either scraped sites or used an unofficial API. Apple Health is the closest thing I've seen where there's somewhat of an open standard despite the walled garden in Apple's ecosystem.
Without some overarching body, like a government regulator, I'm not optimistic there will be widespread adoption of interoperability.
The FIT file format isn't truly open because it's controlled by Garmin. But anyone can download the SDK for free, and most fitness trackers will export data in that format. With Garmin devices you can connect to a USB port and copy the FIT files directly to do whatever you like.
Yes, the three file formats I have seen used are FIT, TCX and GPX. FIT and TCX are controlled by Garmin but the documentation (and, in the case of TCX, the schema) is publicly available. I'm not sure what licence it is available under.
GPX is described on its website as being an open standard and is very widely used, both for fitness data and other applications. The purpose of GPX is to describe GPS data; by itself it is not as suitable for fitness data as FIT or TCX as things like heart rate and cadence data are not part of the standard. However, GPX can be extended.
From an open web perspective, GPX with standard extensions for other fitness-related data would probably be best, but FIT and TCX are getting ever more popular. For now, the situation is fine because it's fairly easy to convert between them. The risk is that Garmin tries to close off the TCX and FIT standards in the future.
Of course this will only work in some jurisdictions, and even then it needs stronger enforcement to ensure that data controllers will comply (or even understand their obligations).
> I'm trying to imagine a future, 30-50 years from now...
I can only hope it's a future in which we collectively realized that smartphones, smartwatches, smartglasses, smartypantstoilets, smartgloves, smartdiapers, etc, were a bloody well idiotic idea and that we've come to our collective senses and realized that they were never what they promised to be, but had, within a few years of launch, turned into just more ways of monopolizing attention for a quick venture capital and IPO payout. Perhaps not the hardware manufacturers, but at least the way the hardware was used.
More realistically, those are likely to be dim memories of the past, back before capitalism ran the planet into the ground, and took most of our technology stacks with it. Modern tech stacks are complexity, piled on complexity, built on a few stacks of complexity, built on broken hardware that's overly complex in an attempt to gain the performance required to run the teetering stacks on top of it (all the uarch vulns, DDR4 Rowhammer is a nice touch on top, etc).
You'll note that neither of those paths involve us running around with the FitBit 2050 gathering... whatever it is it thinks it ought to gather. The way things are going, probably nothing more useful than Covid tracking stuff. "You were in proximity with 1.7 potential individuals who may have been infected with Covid, please quarantine for the next two weeks. confetti"
It would be nice if we had some of our current technology, that open data standards had taken root and we could decide how we wanted to use the stuff, pay appropriately and be paid appropriately for our devices, attention, etc. But that's just not the way things are going, and I fear that's not the way things are going to go, under any reasonable future. There is far, far too much money and "Well, but, see, if you destroy the market you can monopolize it!" money floating around for the near to moderate future to see that future happen. I mean, a bunch of our current companies are literally based on "We will use the venture capital money to undercut everyone else, drive them out of business, and then be the monopoly!" Uber, Lyft, [insert the food delivery service of the week here], the various scooter services, and the list goes on. They're running into the brick wall of, "Wait... what do you mean, we're not going to accomplish that? You mean we have to turn a profit?" I've talked to people who have started taking taxis again, because at least they'll show up, unlike the Sharing Economy Rideshare App of The Future services that weren't remotely pandemic-proof.
We'll see. But I would predict that fitness trackers, in 20 years, are a big pile of ewaste, "... ugh, yeah, you know, it's not worth the hassle to get that data out...," and broken promises. And hopefully not being actively worn by then.
Spot on points here. I love it how everyone i know with a fitness tracker always talks about there 10k steps for a few months and it fades off into oblivion. Like taking steps matters when your holding your phone in envey of whatever the alrogithm feeds you.
Sport will not perpetuate human existence. It was originally created to keep humans in shape for hard the graft of living off the land. Just look at any aboriginal culture, they knew they had done enough steps today when they had enough food.
The fitness tracker was created to make you feel better about being taken for a ride (in life). For those who needs affirmation from their peers that they arnt wasting there life away working for a whatever click bait porn floats onto there device next.
20 yrs from now? we already have big piles of ewaste everywhere.
I once interviewed for a startup making bluetooth diapers for old folks ~2013.
My recommended fitness tracker is to turn off all your devices and go outside into nature at every opportunity. Find a company who will let you be a remote developer and start living for today.*
Fitness trackers are hardly idiotic for those of us who are at least semi serious about endurance sports. Of course you can train and race without one but you'll be less efficient. If you go to any major race just about everyone who podiums will be wearing a fitness tracker. That's because they actually work.
I'm not even competitive, but my tracker devices are hugely motivational for me. And having useful metrics like heart rate as a tachometer for my body is immensely valuable for all sorts of activities.
I'm healthier and more active now because of them, and they've gotten me more in tune with my body, not less.
The brands and models they are wearing are different than the rest of us.
Most of the industry subsides on the aspirational user who doesn't make any lifestyle change and stops wearing the device after a year or less. The vague idea that it'll find some magic ingredient for improving your sleep or exercise.
Even Garmin is getting in on it trying to sell activity trackers for parents to put on their kids. When it's parents who are in full control of their kids' time and activity.
Of course it has a use for people training but that's a small minority of customers.
Here's the copy on a recent device:
"Meet our most advanced fitness & health tracker with tools like an on-wrist ECG app for heart health,* EDA Scan app for stress management and more. Get a 6-month membership of Fitbit Premium™** and optimize your routine with Daily Readiness Score.◆‡ Add a 6-month Premium membership for advanced insights & tools to improve your health"
Guaranteed most customers would benefit from finding exercise they enjoy, drinking less alcohol and worrying less about their sleep rather than being presented with Scores and Goals whose novelty and thus salience will quickly wear off. But that isn't something that can be sold so...
I've seen the race winners wearing the same Garmin, Suunto, and Coros device models as the rest of us. Hardly anyone who actually competes uses Fitbit, it's kind of a toy.
Sounds like we're vigorously agreeing. If you Google "fitness trackers", you might get a Garmin and after a few minutes I found a Coros. But it's mostly Apple, Fitbit, Xiaomi, Withings, Garmin's more fashionable/smartwatch line. If that's what the tech mags are reviewing (for they come up top on search, not fitness websites) that's where the money is.
So, uh, how much less efficient, exactly? 5%? 10%? 90%? How is this efficiency even measured? You improve twice as slow, or the end results are twice as bad, or?..
I'm afraid I can't quantify it for you, but subjectively I'm completely confident that using a high end fitness tracker for training and racing makes me faster in triathlons. Other than that I'm not interested in trying to measure efficiency with versus without the device as that would be a total waste of my time.
Technology has also improved in many other areas: exercise physiology knowledge, nutrition, aerodynamics, lab testing, PEDs, bouncy shoes, etc. It's tough to isolate how much of the total improvement was caused by fitness trackers.
All pro endurance sports could end this instant, forever, and humanity would be fine. Humanity will not be fine if some of the points raised in that comment keep getting ignored.
You are not making the distinction between objective value of fitness trackers (they can measure speed, pace, heart rate, SpO2...), which allows for actionable recourse (I am getting worse/stagnating/improving in this particular area), and sociological/psychological value they are providing.
I think it'd be hard to say measuring something in order to improve on it is useless: and most of the smart devices allow one to extract exactly that value.
Now, I wouldn't be the one to discredit social and psychological value for any smart devices, because they can be very real too, but it's definitely obvious that we should look for alternative solutions to those problems at the same time.
I'm fairly far along on the data hoarding spectrum. This extends, for example, to over 200K photos from my life, categorized and indexed. It's trivial to look at "what happened on this day 20 years ago" or at least on the nearest date to that day that had pictures taken.
And here's the thing: As the memories fade, so does interest in the photos. 10 years ago you still remember clearly. 20 years ago is another era.
And that's photos. Fitness data? Do I really want to see how I declined over the last 20 years? I see that already from the average speed display on my bike computer. Fitness data is the ultimate "looking forward" thing. But maybe that's my age; it may look different from a younger perspective.
Agree with others that just use a platform that lets you export open-ish files. .gpx doesn't do heart rate, but .fit does, and gpsbabel can translate that to another format as needed.