Some background: We launched this archive with the State of North Carolina yesterday. As far as we know, North Carolina is the first government in the world to make web 2.0 records available to the public in this type of extremely searchable, highly interactive fashion. I say "interactive" in reference to the fact that the records look and behave like the original social network. For example, you can expand comments on Facebook posts, view full-sized photos, expand shortened Twitter links, etc.
I'd love to get your feedback on the presentation of data and the usability of the search interface (I recommend clicking on some of the "Example Searches" to see how it works). I'm happy to speak to the underlying technology as well.
Really cool Anil! I can see this dramatically reducing the time it takes and the costs incurred for a government's response to a Freedom of Information Act request.
I appreciate it, and you are exactly right. Most governments take anywhere between two to 10 days to even acknowledge a FOIA request. If they are doing anything about social media, then they have to scour through folders of screenshots to find the "record".
Great work Anil. I did a quick search on beer brewing (near and dear to my heart) and found great government information from legal quantity to some local resources. This site will be valuable if I ever develop a recipe that I will want to sell. Do yo have any plans to do cross search to set up a regional type archive. For example, if I wanted to sell my beer in adjoining states such as VA and SC.
Hey Anil,
I read somewhere that the way ArchiveSocial does the archiving it somehow preserves that info so that it looks just like it did when originally posted on facebook, twitter or wherever. That's kinda hard to imagine. How does it maintain the look of the particular social media environment the content was posted in?
Ron, to clarify, we do present the information back in a way that mimics the original social network, but we don't actually preserve the content pixel-by-pixel. We actually preserve the raw metadata and recreate the look, feel, and behavior. This is in contrast to many other tools that simply capture the HTML presentation from Facebook.com and Twitter.com, but not the underlying data. Our view is that the content should feel very familiar to the user, but that the underlying record and metadata are actually what matters.
Great stuff Anil! I think this nails the archival piece of social data. How can this be extended to use the archived data for insights i.e. run data mining algorithms (think SAS on social data). There are few players in that field but if both of these utilities can be coupled, this will be a powerful tool.
Agreed. We have some really exciting ideas on how to derive value out of this data, now that we have it in this form. I am curious what kind of analytics you think might be useful to government agencies or companies keeping this data for legal reasons?
I appreciate the comment. You are right that, on the surface, this is something fairly easy to say you are doing. There are lots of tools like Backupify, etc that store social media content in some way. The problem is that its virtually impossible to make sense of the data later on because most of the context is lost (i.e. comments are not associated with their posts, you cant view the full-sized photos, etc). It's also a challenge to preserve these ever-changing, non-standardized data formats over the course of several years, while maintaining searchability and accurate presentation of the data.
It's easy to see the commercial value here. Saves time, saves tons of hassle, creates more transparency. Raises the question... why are so many people talking about "Trout Fishing" in North Carolina :)
Ha, thanks for the comment. Funny seeing searches for "trout fishing" increasing on our realtime web analytics :) Speaking of which, we are pretty interested in things like trending topics so that you can see what citizens in a particular geography are discussing. This is a bit different -- and in many ways, more interesting -- than what Twitter does today in regards to trending topics based on location, because discussion on government social networking sites is actually about the region itself.
This makes a lot of sense. It will be interesting to see if this has even more value during emergencies and severe weather when normal citizens have to pull their resources.
Definitely. Financial Services is the flagship market for a solution like this due to SEC and FINRA compliance requirements around record keeping. Imagine not being able to use Twitter, Facebook, or LinkedIn unless you are able to keep records of every interaction for 5 years in a tamper-proof, non-erasable format :)
Lots of innovation over here in NC :) I suppose most folks are working on the "sexy" side of social media (i.e. marketing/engagement). There are some compliance-oriented startups in the valley, but nothing quite like this that we know of. I guess you could draw some parallels to services like Cue on the consumer side. Maybe some other folks here on HN know better?
And yes, we'd love to be the archive for CA (or SF for that matter).
I'd love to get your feedback on the presentation of data and the usability of the search interface (I recommend clicking on some of the "Example Searches" to see how it works). I'm happy to speak to the underlying technology as well.